<div dir="ltr"><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">HI Krutika,</div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif"><br></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">Thanks for the reply. However I am afraid that its not too late for us. I already replaced GlusterFS server and copied my data on the new bricks. Now Again its working flawlessly like it was working before. However, I still have old server and snapshots I'd try to implement your solution and I'll let you know about it. </div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif"><br></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif"><br></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr">Regards<div>Ankur Pandey</div><div>+91 9702 831 855</div></div></div></div>
<br><div class="gmail_quote">On Tue, Sep 22, 2015 at 4:54 PM, Krutika Dhananjay <span dir="ltr"><<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:garamond,new york,times,serif;font-size:12pt;color:#000000"><div>Hi Ankur,<br></div><div><br></div><div>It looks like some of the files/directories are in gfid split-brain.<br></div><div>From the logs that you attached, here is the list of gfids of directories in gfid split-brain, based on the message id for gfid split-brain log message (108008):<br></div><div><br></div><div>[kdhananjay@dhcp35-215 logs]$ grep -iarT '108008' * | awk '{print $13}' | cut -f1 -d'/' | sort | uniq<br><16d8005d-3ae2-4c72-9097-2aedd458b5e0<br><3539c175-d694-409d-949f-f9a3e18df17b<br><3fd13508-b29e-4d52-8c9c-14ccd2f24b9f<br><6b1e5a5a-bb65-46c1-a7c3-0526847beece<br><971b5249-92fb-4166-b1a0-33b7efcc39a8<br><b582f326-c8ee-4b04-aba0-d37cb0a6f89a<br><cc9d0e49-c9ab-4dab-bca4-1c06c8a7a4e3<br><br></div><div>There are 7 such directories.<br></div><div><br></div><div>Also, there are 457 entries in gfid split-brain:<br></div><div>[kdhananjay@dhcp35-215 logs]$ grep -iarT '108008' glustershd.log | awk '{print $13}' | sort | uniq | wc -l<br>457<br><br></div><div>You will need to do the following to get things back to the normal state:<br></div><div><br></div><div>1) For each gfid in the list of the 7 directories in split-brain, get the list of files in split-brain.<br></div><div>For example, for <16d8005d-3ae2-4c72-9097-2aedd458b5e0 , the command would be `grep -iarT '108008' * | grep 16d8005d-3ae2-4c72-9097-2aedd458b5e0`<br></div><div>You will need to omit the repeating messages of course.<br></div><div>You would get messages of the following kind:<br></div><div>glustershd.log :[2015-09-10 01:44:05.512589] E [MSGID: 108008] [afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch] 0-repl-vol-replicate-0: Gfid mismatch detected for <16d8005d-3ae2-4c72-9097-2aedd458b5e0/100000075944.jpg>, d9f15b28-9c9c-4f31-ba3c-543a5331cb9d on repl-vol-client-1 and 583295f0-1ec4-4783-9b35-1e18b8b4f92c on repl-vol-client-0. Skipping conservative merge on the file.</div><div><br></div><div>2) Examine the two copies (one per replica) of each such file, choose one copy and delete the copy from the other replica.<br></div><div>In the above example, the parent is 16d8005d-3ae2-4c72-9097-2aedd458b5e0 and the entry is '100000075944.jpg'.<br></div><div>So you can examine the two different copies at <brick-path>/.glusterfs/16/d8/16d8005d-3ae2-4c72-9097-2aedd458b5e0/100000075944.jpg to decide which one you want to keep.<br></div><div>Once you have decided on the copy you choose to keep, you need to delete the bad copy and its hard link. This is assuming all of the entries in gfid split-brain are regular files. At least that is what I gathered from the logs since they were all .jpg files.</div><div>You can get the absolute path of the entry by noting down inode number of the gfid link on the bad brick and then grepping for the corresponding number under the same brick.<br></div><div>In this example, the gfid link would be <bad-brick-path>/.glusterfs/16/d8/16d8005d-3ae2-4c72-9097-2aedd458b5e0/100000075944.jpg.<br></div><div>So you would need to get its inode number (by doing stat on it) and do a 'find <bad-brick-path> -inum <inodenumber of gfid link> to get its absolute path.<br></div><div>Once you have both, unlink them both. If hard links exist, delete them as well on the bad brick.<br></div><div><br></div><div>There are about 457 files where you need to repeat this exercise.<br></div><div><br></div><div>Once you are done, you could execute 'gluster volume heal <VOL>'. This would take care of healing the good copies to the bricks where the file was deleted from.<br></div><div>After the heal is complete, heal info split-brain should not be showing any entries.<br></div><div><br></div><div>As for the performance problem, it is possible that it was due to self-heal daemon periodically trying to heal the files in gfid-split-brain in vain, and should most likely go away once the split-brain is resolved.<br></div><div><br></div><div>As an aside, it is not clear why so many files ran into gfid split-brain. You might want to check if the network link between the clients and the servers was fine.<br></div><div><br></div><div>Hope that helps. Let me know if you need more clarification.<br></div><div>-Krutika<br></div><hr><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><span class=""><b>From: </b>"Ankur Pandey" <<a href="mailto:ankur.p@ambab.com" target="_blank">ankur.p@ambab.com</a>><br><b>To: </b><a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>, <a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a><br><b>Cc: </b>"Dhaval Kamani" <<a href="mailto:dhaval@ambab.com" target="_blank">dhaval@ambab.com</a>><br><b>Sent: </b>Saturday, September 12, 2015 12:51:31 PM<br><b>Subject: </b>[Gluster-users] GlusterFS Split Brain issue<br><div><br></div></span><div><div class="h5"><div dir="ltr"><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif">HI Team GlusterFS,</div><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif"><br></div><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif">With reference to Question on server fault.</div><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif"><br></div><div class="gmail_default"><span style="font-family:trebuchet ms,sans-serif" face="trebuchet ms, sans-serif"><a href="http://serverfault.com/questions/721067/glusterfs-split-brain-issue" target="_blank">http://serverfault.com/questions/721067/glusterfs-split-brain-issue</a></span><br></div><div class="gmail_default"><span style="font-family:trebuchet ms,sans-serif" face="trebuchet ms, sans-serif"><br></span></div><div class="gmail_default"><span style="font-family:trebuchet ms,sans-serif" face="trebuchet ms, sans-serif">On request of Pranith I am sending you logs. Please tell me if you need anything else. </span></div><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif"><br></div><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif">Attaching logs for 2 master servers. </div><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif"><br></div><div><div><div dir="ltr">Regards<div>Ankur Pandey</div><div>+91 9702 831 855</div></div></div></div><br><div class="gmail_quote"><div dir="ltr"><div><div><div><div dir="ltr"><div><div dir="ltr"><br></div></div></div></div></div></div></div></div><br></div><br></div></div><span class="">_______________________________________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br><a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a></span></blockquote><div><br></div></div></div></blockquote></div><br></div>