<div dir="ltr"><br><div>I have a script written to analyze the log message of gluster process.</div><div><br></div><div>It actually scans the log file and identifies the log messages with ERROR and WARNING levels.</div><div>It lists the functions (with either ERROR or WARNING logs) and their percentage of occcurance.</div><div><br></div><div>It also lists the MSGIDs for ERROR and WARNING logs and their percentage of occurance.</div><div><br></div><div>A sample o/p the script:</div><div><br></div><div><div>[root@hal9000 ~]# ./log_analyzer.sh /var/log/glusterfs/mnt-glusterfs.log</div><div>Number<span class="" style="white-space:pre"> </span>Percentage<span class="" style="white-space:pre"> </span>Function</div><div>7<span class="" style="white-space:pre"> </span>0.49<span class="" style="white-space:pre"> </span> __socket_rwv</div><div>4<span class="" style="white-space:pre"> </span>0.28<span class="" style="white-space:pre"> </span> mgmt_getspec_cbk</div><div>4<span class="" style="white-space:pre"> </span>0.28<span class="" style="white-space:pre"> </span> gf_timer_call_after</div><div>3<span class="" style="white-space:pre"> </span>0.21<span class="" style="white-space:pre"> </span> rpc_clnt_reconfig</div><div>2<span class="" style="white-space:pre"> </span>0.14<span class="" style="white-space:pre"> </span> fuse_thread_proc</div><div>2<span class="" style="white-space:pre"> </span>0.14<span class="" style="white-space:pre"> </span> fini</div><div>2<span class="" style="white-space:pre"> </span>0.14<span class="" style="white-space:pre"> </span> cleanup_and_exit</div><div>1<span class="" style="white-space:pre"> </span>0.07<span class="" style="white-space:pre"> </span> _ios_dump_thread</div><div>1<span class="" style="white-space:pre"> </span>0.07<span class="" style="white-space:pre"> </span> fuse_init</div><div>1<span class="" style="white-space:pre"> </span>0.07<span class="" style="white-space:pre"> </span> fuse_graph_setup</div><div><br></div><div>========= Error Functions ========</div><div><br></div><div>7<span class="" style="white-space:pre"> </span>0.49<span class="" style="white-space:pre"> </span>__socket_rwv</div><div>2<span class="" style="white-space:pre"> </span>0.14<span class="" style="white-space:pre"> </span>cleanup_and_exit</div><div><br></div><div>Number<span class="" style="white-space:pre"> </span>Percentage<span class="" style="white-space:pre"> </span>MSGID</div><div>958<span class="" style="white-space:pre"> </span>67.99<span class="" style="white-space:pre"> </span> 109066</div><div>424<span class="" style="white-space:pre"> </span>30.09<span class="" style="white-space:pre"> </span> 109036</div><div>3<span class="" style="white-space:pre"> </span>0.21<span class="" style="white-space:pre"> </span> 114057</div><div>3<span class="" style="white-space:pre"> </span>0.21<span class="" style="white-space:pre"> </span> 114047</div><div>3<span class="" style="white-space:pre"> </span>0.21<span class="" style="white-space:pre"> </span> 114046</div><div>3<span class="" style="white-space:pre"> </span>0.21<span class="" style="white-space:pre"> </span> 114035</div><div>3<span class="" style="white-space:pre"> </span>0.21<span class="" style="white-space:pre"> </span> 114020</div><div>3<span class="" style="white-space:pre"> </span>0.21<span class="" style="white-space:pre"> </span> 114018</div><div>3<span class="" style="white-space:pre"> </span>0.21<span class="" style="white-space:pre"> </span> 108031</div><div>2<span class="" style="white-space:pre"> </span>0.14<span class="" style="white-space:pre"> </span> 101190</div><div>1<span class="" style="white-space:pre"> </span>0.07<span class="" style="white-space:pre"> </span> 7962</div><div>1<span class="" style="white-space:pre"> </span>0.07<span class="" style="white-space:pre"> </span> 108006</div><div>1<span class="" style="white-space:pre"> </span>0.07<span class="" style="white-space:pre"> </span> 108005</div><div>1<span class="" style="white-space:pre"> </span>0.07<span class="" style="white-space:pre"> </span> 108001</div><div>1<span class="" style="white-space:pre"> </span>0.07<span class="" style="white-space:pre"> </span> 100030</div><div><br></div><div>========= Error MSGIDs ========</div><div><br></div><div>1<span class="" style="white-space:pre"> </span>0.07<span class="" style="white-space:pre"> </span>108006</div><div>1<span class="" style="white-space:pre"> </span>0.07<span class="" style="white-space:pre"> </span>108001</div></div><div><br></div><div>It can be found here.</div><div><br></div><div><a href="https://github.com/raghavendrabhat/threaded-io/blob/master/log_analyzer.sh">https://github.com/raghavendrabhat/threaded-io/blob/master/log_analyzer.sh</a>.<br></div><div><br></div><div>Do you think it can be added to the repo?</div><div><br></div><div>Regards,</div><div>Raghavendra</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jan 27, 2016 at 3:44 AM, Aravinda <span dir="ltr"><<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
I am happy to share the `glustertool` project, which is a<br>
infrastructure for adding more tools for Gluster.<br>
<br>
<a href="https://github.com/aravindavk/glustertool" rel="noreferrer" target="_blank">https://github.com/aravindavk/glustertool</a><br>
<br>
Following tools available with the initial release.(`glustertool <TOOLNAME> [ARGS..]`)<br>
<br>
1. gfid - To get GFID of given path(Mount or Backend)<br>
2. changelogparser - To parse the Gluster Changelog<br>
3. xtime - To get Xtime from brick backend<br>
4. stime - To get Stime from brick backend<br>
5. volmark - To get Volmark details from Gluster mount<br>
<br>
rpm/deb packages are not yet available, install this using `sudo<br>
python setup.py install`<br>
<br>
Once installed, run `glustertool list` to see list of tools available.<br>
`glustertool doc TOOLNAME` shows documentation about the tool and<br>
`glustertool <TOOLNAME> --help` shows the usage of the tool.<br>
<br>
More tools can be added to this collection easily using `newtool`<br>
utility available in this repo.<br>
<br>
# ./newtool <TOOLNAME><br>
<br>
Read more about adding tools here<br>
<a href="https://github.com/aravindavk/glustertool/blob/master/CONTRIBUTING.md" rel="noreferrer" target="_blank">https://github.com/aravindavk/glustertool/blob/master/CONTRIBUTING.md</a><br>
<br>
You can create an issue in github requesting more tools for Gluster<br>
<a href="https://github.com/aravindavk/glustertool/issues" rel="noreferrer" target="_blank">https://github.com/aravindavk/glustertool/issues</a><br>
<br>
Comments & Suggestions Welcome<br>
<br>
regards<span class="HOEnZb"><font color="#888888"><br>
Aravinda<br></font></span><span class="im HOEnZb">
<br>
On 10/23/2015 11:42 PM, Vijay Bellur wrote:<br>
</span><div class="HOEnZb"><div class="h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Friday 23 October 2015 04:16 PM, Aravinda wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Gluster developers,<br>
<br>
In this mail I am proposing troubleshooting documentation and<br>
Gluster Tools infrastructure.<br>
<br>
Tool to search in documentation<br>
===============================<br>
We recently added message Ids to each error messages in Gluster. Some<br>
of the error messages are self explanatory. But some error messages<br>
requires manual intervention to fix the issue. How about identifying<br>
the error messages which requires more explanation and creating<br>
documentation for the same. Even though the information about some<br>
errors available in documentation, it is very difficult to search and<br>
relate to the error message. It will be very useful if we create a<br>
tool which looks for documentation and tells us exactly what to do.<br>
<br>
For example,(Illustrativepurpose only)<br>
glusterdoc --explain GEOREP0003<br>
<br>
SSH configuration issue. This error is seen when Pem keys from all<br>
master nodes are not distributed properly to Slave<br>
nodes. Use Geo-replication create command with force option to<br>
redistribute the keys. If issue stillpersists, look for any errors<br>
while running hook scripts inGlusterd log file.<br>
<br>
<br>
Note: Inspired from rustc --explain command<br>
<a href="https://twitter.com/jaredforsyth/status/626960244707606528" rel="noreferrer" target="_blank">https://twitter.com/jaredforsyth/status/626960244707606528</a><br>
<br>
If we don't know the message id, we can still search from the<br>
available documentation like,<br>
<br>
glusterdoc --search <SEARCH_KEY_WORD><br>
<br>
These commands can be programmatically consumed, for example<br>
`--json` will return the output in JSON format. This enables UI<br>
developers to automatically show help messages when they display<br>
errors.<br>
<br>
Gluster Tools infrastructure<br>
============================<br>
Are our Gluster log files sufficient for root causing the issues? Is<br>
that error caused due to miss configuration? Geo-replication status is<br>
showing faulty. Where to find the reason for Faulty?<br>
<br>
Sac(surs AT <a href="http://redhat.com" rel="noreferrer" target="_blank">redhat.com</a>) mentioned that heis working on gdeploy and many<br>
developers<br>
are using their owntools. How about providing common infrastructure(say<br>
gtool/glustertool) to host all these tools.<br>
<br>
</blockquote>
<br>
Would this be a repository with individual tools being git submodules or something similar? Is there also a plan to bundle the set of tools into a binary package?<br>
<br>
Looks like a good idea to me.<br>
<br>
-Vijay<br>
<br>
</blockquote>
<br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
</div></div></blockquote></div><br></div>