<div dir="ltr"><div><div><div><div>OK here is one thing you could do if you are able to recreate this hang easily on your setup.<br></div>The next time you find heal-info hanging, could you do the following:<br><br></div># gluster volume statedump <VOL> <br><br></div>and share the statedump logs? You will find them at /var/run/gluster/<br><br></div>-Krutika<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Apr 14, 2016 at 3:32 PM, Krutika Dhananjay <span dir="ltr"><<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Would you happen to know what those 6 entries that need heal correspond to? Assuming heal-info reported the status at least once without hanging.<br></div>Also, could you share the contents of glfsheal-datastore.log, specifically from the machine where you ran the `heal info` command?<span class="HOEnZb"><font color="#888888"><br><br></font></span></div><span class="HOEnZb"><font color="#888888">-Krutika<br></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Apr 13, 2016 at 8:13 PM, Lindsay Mathieson <span dir="ltr"><<a href="mailto:lindsay.mathieson@gmail.com" target="_blank">lindsay.mathieson@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On 14/04/2016 12:19 AM, Krutika Dhananjay wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hmm what version of gluster was the hang seen on?<br>
</blockquote>
<br>
<br></span>
Ah yes, sorry - 3.7.9<br>
<br>
The heal was triggered by a "killall glusterfsd" on node vna as a test, I then restarted the glusterd service to bring it back up.<br>
<br>
Eventually I shutdown the VM's and stopped all gluster servcies, then brought them back up. Eventually the last 4 shards healed, but it took around an hour.<br>
<br>
Prior to that I had done rolling updates and restarts across all three node, heals between reboots completed reasonably quickly with no i/o freezes.<br>
<br>
I off to bed now I'm afraid. Rather late here :)<span><font color="#888888"><br>
<br>
-- <br>
Lindsay Mathieson<br>
<br>
</font></span></blockquote></div><br></div>
</div></div></blockquote></div><br></div>