<html><head></head><body>Be sure that the "gluster --print-statedumpdir" directory exists first.<br><br><div class="gmail_quote">On May 5, 2016 10:04:41 PM PDT, Ravishankar N <ravishankar@redhat.com> wrote:<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<pre class="k9mail">Thanks for the response. The healinfo outputs 'Possibly undergoing <br />heal' only when the selfheal daemon is performing heal and not when <br />there is IO from the mount. Could you provide the state dump of the 2 <br />bricks (and the mount too if you know from which mount this vm image is <br />being accessed)?<br /><br />The command is `kill -USR1 <pid>` where pid is the process id of the <br />brick or fuse mount. The statedump will be saved in `gluster <br />--print-statedumpdir`<br />Wanted to check if there are any stale locks being held on the bricks.<br /><br />Thanks,<br />Ravi<br /><br />On 05/06/2016 01:22 AM, Richard Klein (RSI) wrote:<br /><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #729fcf; padding-left: 1ex;"> I agree there is activity but it's very low I/O based, like updating log files. It shouldn't be high enough IO to keep it permanently in the "Possibly undergoing healing" state for
days. But just to make sure, I powered off the VM and there is no activity now at all and the "trusted.afr.dirty" is still changing. I will leave the VM in a powered off state until tomorrow. I agree with you that is shouldn't but that is my dilemma.<br /><br /> Thanks for the insight,<br /><br /> Richard Klein<br /> RSI<br /><br /><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #ad7fa8; padding-left: 1ex;"> -----Original Message-----<br /> From: gluster-users-bounces@gluster.org [mailto:gluster-users-<br /> bounces@gluster.org] On Behalf Of Joe Julian<br /> Sent: Thursday, May 05, 2016 1:44 PM<br /> To: gluster-users@gluster.org<br /> Subject: Re: [Gluster-users] Question about "Possibly undergoing heal" on a file<br /> being reported.<br /><br /> FYI, that's not "no activity". The file is clearly changing. The dirty state flipping<br /> back and forth between 1 and 0 is a byproduct of writes occurring. The clients<br /> set the flag, do
the write, then clear the flag.<br /> My guess is that's why it's only "possibly" undergoing self-heal. The write may<br /> have still been pending at the moment of the check.<br /><br /> On 05/05/2016 10:22 AM, Richard Klein (RSI) wrote:<br /><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #8ae234; padding-left: 1ex;"> There are 2 hosts involved and we have a replica value of 2. The hosts are<br /></blockquote> called n1c1cl1 and n1c2cl1. Below is the info you requested. The file name in<br /> gluster is "/97f52c71-80bd-4c2b-8e47-3c8c77712687".<br /><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #8ae234; padding-left: 1ex;"> -- >From the n1c1cl1 brick --<br /><br /> [root@n1c1cl1 ~]# ll -h<br /> /data/brick0/gv0cl1/97f52c71-80bd-4c2b-8e47-3c8c77712687<br /> -rwxr--r--. 2 root root 3.7G May 5 12:10<br /> /data/brick0/gv0cl1/97f52c71-80bd-4c2b-8e47-3c8c77712687<br /><br /> [root@n1c1cl1 ~]# getfattr
-d -m . -e hex<br /> /data/brick0/gv0cl1/97f52c71-80bd-4c2b-8e47-3c8c77712687<br /> getfattr: Removing leading '/' from absolute path names # file:<br /> data/brick0/gv0cl1/97f52c71-80bd-4c2b-8e47-3c8c77712687</blockquote><br /> security.selinux=0x73797374656d5f753a6f626a6563745f723a64656661756c74<br /> 5<br /><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #8ae234; padding-left: 1ex;"> f743a733000<br /> trusted.afr.dirty=0xe68000000000000000000000<br /> trusted.bit-rot.version=0x020000000000000057196a8d000e1606<br /> trusted.gfid=0xb1a49bd1ea01479f9a8277992461e85f<br /><br /> -- From the n1c2cl1 brick --<br /><br /> [root@n1c2cl1 ~]# ll -h<br /> /data/brick0/gv0cl1/97f52c71-80bd-4c2b-8e47-3c8c77712687<br /> -rwxr--r--. 2 root root 3.7G May 5 12:16<br /> /data/brick0/gv0cl1/97f52c71-80bd-4c2b-8e47-3c8c77712687<br /><br /> [root@n1c2cl1 ~]# getfattr -d -m . -e hex<br /> /data/brick0/gv0cl1/97f52c71-80bd-4c2b-8e47-3c8c77712687<br /> getfattr:
Removing leading '/' from absolute path names # file:<br /> data/brick0/gv0cl1/97f52c71-80bd-4c2b-8e47-3c8c77712687</blockquote><br /> security.selinux=0x73797374656d5f753a6f626a6563745f723a64656661756c74<br /> 5<br /><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #8ae234; padding-left: 1ex;"> f743a733000<br /> trusted.afr.dirty=0xd38000000000000000000000<br /> trusted.bit-rot.version=0x020000000000000057196a8d000e20ae<br /> trusted.gfid=0xb1a49bd1ea01479f9a8277992461e85f<br /><br /> --<br /><br /> The "trusted.afr.dirty" is changing about 2 or 3 times a minute on both files.<br /></blockquote> Let me know if you need further info and thanks.<br /><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #8ae234; padding-left: 1ex;"> Richard Klein<br /> RSI<br /><br /><br /><br /> From: Ravishankar N [mailto:ravishankar@redhat.com]<br /> Sent: Wednesday, May 04, 2016 8:52 PM<br /> To: Richard Klein (RSI);
gluster-users@gluster.org<br /> Subject: Re: [Gluster-users] Question about "Possibly undergoing heal" on a<br /></blockquote> file being reported.<br /><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #8ae234; padding-left: 1ex;"><br /><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #fcaf3e; padding-left: 1ex;"> On 05/05/2016 01:50 AM, Richard Klein (RSI) wrote:<br /> First time e-mailer to the group, greetings all. We are using Gluster 3.7.6 in<br /></blockquote></blockquote> Cloudstack on CentOS7 with KVM. Gluster is our primary storage. All is going<br /> well >but we have a test VM QCOW2 volume that gets stuck in the "Possibly<br /> undergoing healing". By stuck I mean it stays in that state for over 24 hrs. This<br /> is a test VM >with no activity on it and we have removed the swap file on the<br /> guest as well thinking that may be causing high I/O. All the tools show that the<br />
VM is basically idle >with low I/O. The only way I can clear it up is to power<br /> the VM off, move the QCOW2 volume from the Gluster mount then back<br /> (basically remove and recreate it) >then power the VM back on. Once I do this<br /> process all is well again but then it happened again on the same volume/file.<br /><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #8ae234; padding-left: 1ex;"><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #fcaf3e; padding-left: 1ex;"> One additional note, I have even powered off the VM completely and the<br /></blockquote></blockquote> QCOW2 file still stays in this state.<br /><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #8ae234; padding-left: 1ex;"><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #fcaf3e; padding-left: 1ex;"> When this happens, can you share the output
of the extended attributes of<br /></blockquote></blockquote> the file in question from all the bricks of the replica in which the file resides?<br /><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #8ae234; padding-left: 1ex;"> `getfattr -d -m . -e hex /path/to/bricks/file-name`<br /><br /> Also what is the size of this VM image file?<br /><br /> Thanks,<br /> Ravi<br /><br /><br /><br /><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #fcaf3e; padding-left: 1ex;"> Is there a way to stop/abort or force the heal to finish? Any help with a<br /></blockquote></blockquote> direction would be appreciated.<br /><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #8ae234; padding-left: 1ex;"><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #fcaf3e; padding-left: 1ex;"> Thanks,<br /><br /> Richard Klein<br /> RSI<br /></blockquote><br
/><br /><br /><hr /><br /> Gluster-users mailing list<br /> Gluster-users@gluster.org<br /> <a href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a><br /><br /><hr /><br /> Gluster-users mailing list<br /> Gluster-users@gluster.org<br /> <a href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a><br /></blockquote><hr /><br /> Gluster-users mailing list<br /> Gluster-users@gluster.org<br /> <a href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a><br /></blockquote><hr /><br /> Gluster-users mailing list<br /> Gluster-users@gluster.org<br /> <a href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a><br /></blockquote><br /><br /><hr /><br />Gluster-users mailing list<br />Gluster-users@gluster.org<br /><a
href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a><br /></pre></blockquote></div><br>
-- <br>
Sent from my Android device with K-9 Mail. Please excuse my brevity.</body></html>