<p dir="ltr">A few more: </p>
<p dir="ltr">1.A tool to determine the back end brick location given a file name.<br>
2. A rebalance meter which allows me to determine if it's high time to perform rebalance.</p>
<div class="gmail_quote">On Oct 23, 2015 4:16 PM, "Aravinda" <<a href="mailto:avishwan@redhat.com">avishwan@redhat.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Gluster developers,<br>
<br>
In this mail I am proposing troubleshooting documentation and<br>
Gluster Tools infrastructure.<br>
<br>
Tool to search in documentation<br>
===============================<br>
We recently added message Ids to each error messages in Gluster. Some<br>
of the error messages are self explanatory. But some error messages<br>
requires manual intervention to fix the issue. How about identifying<br>
the error messages which requires more explanation and creating<br>
documentation for the same. Even though the information about some<br>
errors available in documentation, it is very difficult to search and<br>
relate to the error message. It will be very useful if we create a<br>
tool which looks for documentation and tells us exactly what to do.<br>
<br>
For example,(Illustrativepurpose only)<br>
glusterdoc --explain GEOREP0003<br>
<br>
SSH configuration issue. This error is seen when Pem keys from all<br>
master nodes are not distributed properly to Slave<br>
nodes. Use Geo-replication create command with force option to<br>
redistribute the keys. If issue stillpersists, look for any errors<br>
while running hook scripts inGlusterd log file.<br>
<br>
<br>
Note: Inspired from rustc --explain command<br>
<a href="https://twitter.com/jaredforsyth/status/626960244707606528" rel="noreferrer" target="_blank">https://twitter.com/jaredforsyth/status/626960244707606528</a><br>
<br>
If we don't know the message id, we can still search from the<br>
available documentation like,<br>
<br>
glusterdoc --search <SEARCH_KEY_WORD><br>
<br>
These commands can be programmatically consumed, for example<br>
`--json` will return the output in JSON format. This enables UI<br>
developers to automatically show help messages when they display<br>
errors.<br>
<br>
Gluster Tools infrastructure<br>
============================<br>
Are our Gluster log files sufficient for root causing the issues? Is<br>
that error caused due to miss configuration? Geo-replication status is<br>
showing faulty. Where to find the reason for Faulty?<br>
<br>
Sac(surs AT <a href="http://redhat.com" rel="noreferrer" target="_blank">redhat.com</a>) mentioned that heis working on gdeploy and many developers<br>
are using their owntools. How about providing common infrastructure(say<br>
gtool/glustertool) to host all these tools.<br>
<br>
>From my toolkit, following tools are available, planning to create<br>
more such tools for Geo-replication and Gluster.<br>
<br>
volinfo [<VOLNAME>] - Enhanced version of Gluster Volume info<br>
command (<a href="http://aravindavk.in/blog/glusterfs-tools/" rel="noreferrer" target="_blank">http://aravindavk.in/blog/glusterfs-tools/</a> )<br>
<br>
df - df for Gluster Volumes<br>
(<a href="http://aravindavk.in/blog/glusterdf-df-for-gluster-volumes/" rel="noreferrer" target="_blank">http://aravindavk.in/blog/glusterdf-df-for-gluster-volumes/</a> )<br>
<br>
georepsetup - A tool to Create Geo-replication session<br>
easily(<a href="http://aravindavk.in/blog/introducing-georepsetup/" rel="noreferrer" target="_blank">http://aravindavk.in/blog/introducing-georepsetup/</a> )<br>
<br>
gdash - A simple Dashboard for<br>
Gluster(<a href="http://aravindavk.in/blog/introducing-gdash/" rel="noreferrer" target="_blank">http://aravindavk.in/blog/introducing-gdash/</a> )<br>
<br>
gfid <PATH> - To get GFID of a file, works both in Mount and<br>
Backend(<a href="https://github.com/aravindavk/gluster_georep_scripts" rel="noreferrer" target="_blank">https://github.com/aravindavk/gluster_georep_scripts</a> )<br>
<br>
clparser <PATH> - Parse the backend Changelog and print in human<br>
readable format(<a href="https://github.com/aravindavk/gluster_georep_scripts" rel="noreferrer" target="_blank">https://github.com/aravindavk/gluster_georep_scripts</a> )<br>
<br>
xtime <PATH> - To get XTIME xattr from given<br>
path(<a href="https://github.com/aravindavk/gluster_georep_scripts" rel="noreferrer" target="_blank">https://github.com/aravindavk/gluster_georep_scripts</a> )<br>
<br>
stime <PATH> - To get STIME xattr from given path(Used by<br>
Geo-replication <a href="https://github.com/aravindavk/gluster_georep_scripts" rel="noreferrer" target="_blank">https://github.com/aravindavk/gluster_georep_scripts</a> )<br>
<br>
volmark <VOLNAME> - To get Volume Mark of given Volume(Used by<br>
Geo-replication <a href="https://github.com/aravindavk/gluster_georep_scripts" rel="noreferrer" target="_blank">https://github.com/aravindavk/gluster_georep_scripts</a> )<br>
<br>
<br>
Geo-replication developers are already using some tools like Changelog<br>
parser, `arequal-checksum` etc.<br>
<br>
Initial idea for Tools Framework:<br>
---------------------------------<br>
A Shell/Python script which looks for the tool in plugins sub directory, if<br>
exists pass all the arguments and call that script.<br>
<br>
`glustertool help` triggers a python Script plugins/help.py which reads<br>
plugins.yml file to get the list of tools and help messages associated<br>
with it.<br>
<br>
No restrictions on the choice of programming language to create<br>
tool. It can be bash, Python, Go, Rust, awk, sed etc.<br>
<br>
Challenges:<br>
- Each plugin may have different dependencies, installing all tools<br>
may install all the dependencies.<br>
- Multiple programming languages, may be difficult to maintain/build.<br>
- Maintenance of Third party tools.<br>
- Creating Plugins registry to discover tools created by other developers.<br>
<br>
Tool Ideas:<br>
-----------<br>
If you are interested in working on tools for Gluster, I am listing a<br>
few ideas to start with, feel free to add your ideas to the list.<br>
<br>
- A tool to analyze the log file and identify issues. For example,<br>
glustertool georep_log_analize <LOG FILE PATH> --after-date <TIMESTAMP><br>
<br>
Example output: (Illustrative purpose only)<br>
<br>
Number of workers in this node: 2<br>
Number of restarts: 5<br>
Errors: 10<br>
Python Tracebacks: 5<br>
Last state: Active<br>
Files Skipped: 0<br>
Setup issue: No<br>
<br>
- Extract skipped GFIDs from Geo-replication logs and re-trigger sync.<br>
<br>
For example,<br>
glustertool georep_extract_skipped <LOG_FILE> --after-date<TIMESTAMP><br>
<br>
This command will<br>
1. extract Skipped GFIDs list,<br>
2. Mounts Master Volume<br>
3. converts GFID to Path<br>
4. Set Virtual xattr to re-trigger the sync<br>
<br>
- A tool to detect Split brain<br>
<br>
- A tool to convert GFID to Path<br>
<br>
Created a etherpad to record the available tools and ideas <a href="https://public.pad.fsfe.org/p/gluster-tools" rel="noreferrer" target="_blank">https://public.pad.fsfe.org/p/gluster-tools</a><br>
Will update once the I make some progress in creating infrastructure.<br>
<br>
Comments and Suggestions welcome.<br>
<br>
-- <br>
regards<br>
Aravinda<br>
<br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
</blockquote></div>