<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Unfortunately not:</div><div class=""><br class=""></div><div class="">Remount FS, then access test file from second client:</div><div class=""><br class=""></div><div class=""><div class="">[root@srv02 ~]# umount /mnt</div><div class="">[root@srv02 ~]# mount -t glusterfs srv01:/test01 /mnt</div><div class="">[root@srv02 ~]# ls -l /mnt/passwd </div><div class="">-rw-r--r--. 1 root root 1505 авг 16 19:59 /mnt/passwd</div><div class="">[root@srv02 ~]# ls -l /R1/test01/</div><div class="">итого 4</div><div class="">-rw-r--r--. 2 root root 1505 авг 16 19:59 passwd</div><div class="">[root@srv02 ~]# </div><div class=""><br class=""></div><div class="">Then remount FS and check if accessing the file from second node triggered self-heal on first node:</div><div class=""><br class=""></div><div class="">[root@srv01 ~]# umount /mnt</div><div class="">[root@srv01 ~]# mount -t glusterfs srv01:/test01 /mnt</div><div class="">[root@srv01 ~]# ls -l /mnt</div><div class="">итого 0</div><div class="">[root@srv01 ~]# ls -l /R1/test01/</div><div class="">итого 0</div><div class="">[root@srv01 ~]#</div></div><div class=""><br class=""></div><div class="">Nothing appeared.</div><div class=""><br class=""></div><div class=""><div class="">[root@srv01 ~]# gluster volume info test01</div><div class=""> </div><div class="">Volume Name: test01</div><div class="">Type: Replicate</div><div class="">Volume ID: 2c227085-0b06-4804-805c-ea9c1bb11d8b</div><div class="">Status: Started</div><div class="">Number of Bricks: 1 x 2 = 2</div><div class="">Transport-type: tcp</div><div class="">Bricks:</div><div class="">Brick1: srv01:/R1/test01</div><div class="">Brick2: srv02:/R1/test01</div><div class="">Options Reconfigured:</div><div class="">features.scrub-freq: hourly</div><div class="">features.scrub: Active</div><div class="">features.bitrot: on</div><div class="">transport.address-family: inet</div><div class="">performance.readdir-ahead: on</div><div class="">nfs.disable: on</div><div class="">[root@srv01 ~]# </div></div><div class=""><br class=""></div><div class=""><div class="">[root@srv01 ~]# gluster volume get test01 all | grep heal</div><div class="">cluster.background-self-heal-count 8 </div><div class="">cluster.metadata-self-heal on </div><div class="">cluster.data-self-heal on </div><div class="">cluster.entry-self-heal on </div><div class="">cluster.self-heal-daemon on </div><div class="">cluster.heal-timeout 600 </div><div class="">cluster.self-heal-window-size 1 </div><div class="">cluster.data-self-heal-algorithm (null) </div><div class="">cluster.self-heal-readdir-size 1KB </div><div class="">cluster.heal-wait-queue-length 128 </div><div class="">features.lock-heal off </div><div class="">features.lock-heal off </div><div class="">storage.health-check-interval 30 </div><div class="">features.ctr_lookupheal_link_timeout 300 </div><div class="">features.ctr_lookupheal_inode_timeout 300 </div><div class="">cluster.disperse-self-heal-daemon enable </div><div class="">disperse.background-heals 8 </div><div class="">disperse.heal-wait-qlength 128 </div><div class="">cluster.heal-timeout 600 </div><div class="">cluster.granular-entry-heal no </div><div class="">[root@srv01 ~]#</div></div><br class=""><div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">--</div><div class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class="">Dmitry Glushenok</div><div class="">Jet Infosystems</div></div></div></div></div></div></div></div></div></div></div></div>
</div>
<br class=""><div><blockquote type="cite" class=""><div class="">17 авг. 2016 г., в 11:30, Ravishankar N <<a href="mailto:ravishankar@redhat.com" class="">ravishankar@redhat.com</a>> написал(а):</div><br class="Apple-interchange-newline"><div class="">
<meta content="text/html; charset=utf-8" http-equiv="Content-Type" class="">
<div bgcolor="#FFFFFF" text="#000000" class="">
<div class="moz-cite-prefix">On 08/17/2016 01:48 PM, Дмитрий
Глушенок wrote:<br class="">
</div>
<blockquote cite="mid:FE353586-F02C-40B3-8D1C-8C31F46AB258@jet.msk.su" type="cite" class="">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" class="">
<div class="">Hello Ravi,</div>
<div class=""><br class="">
</div>
<div class="">Thank you for reply. Found bug number (for those who
will google the email) <a moz-do-not-send="true" href="https://bugzilla.redhat.com/show_bug.cgi?id=1112158" class="">https://bugzilla.redhat.com/show_bug.cgi?id=1112158</a></div>
<div class=""><br class="">
</div>
<div class="">Accessing the removed file from mount-point is not
always working because we have to find a special client which
DHT will point to the brick with removed file. Otherwise the
file will be accessed from good brick and self-healing will not
happen (just verified). Or by accessing you meant something like
touch?</div>
</blockquote>
<br class="">
Sorry should have been more explicit. I meant triggering a lookup on
that file with `stat filename`. I don't think you need a special
client. DHT sends the lookup to AFR which in turn sends to all its
children. When one of them returns ENOENT (because you removed it
from the brick), AFR will automatically trigger heal. I'm guessing
it is not always working in your case due to caching at various
levels and the lookup not coming till AFR. If you do it from a fresh
mount ,it should always work.<br class="">
-Ravi<br class="">
<br class="">
<blockquote cite="mid:FE353586-F02C-40B3-8D1C-8C31F46AB258@jet.msk.su" type="cite" class="">
<div class="">
<div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class="">
<div class="" style="word-wrap: break-word;
-webkit-nbsp-mode: space; -webkit-line-break:
after-white-space;">
<div class="" style="word-wrap: break-word;
-webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;">
<div class="" style="word-wrap: break-word;
-webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;">
<div class="">Dmitry Glushenok</div>
<div class="">Jet Infosystems</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">17 авг. 2016 г., в 4:24, Ravishankar N <<a moz-do-not-send="true" href="mailto:ravishankar@redhat.com" class="">ravishankar@redhat.com</a>>
написал(а):</div>
<br class="Apple-interchange-newline">
<div class=""><span style="font-family: Menlo-Regular;
font-size: 13px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height:
normal; orphans: auto; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows:
auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;
float: none; display: inline !important;" class="">On
08/16/2016 10:44 PM, Дмитрий Глушенок wrote:</span><br style="font-family: Menlo-Regular; font-size: 13px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<blockquote type="cite" style="font-family: Menlo-Regular;
font-size: 13px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height:
normal; orphans: auto; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows:
auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">Hello,<br class="">
<br class="">
While testing healing after bitrot error it was found that
self healing cannot heal files which were manually deleted
from brick. Gluster 3.8.1:<br class="">
<br class="">
- Create volume, mount it locally and copy test file to it<br class="">
[root@srv01 ~]# gluster volume create test01 replica 2
srv01:/R1/test01 srv02:/R1/test01<br class="">
volume create: test01: success: please start the volume to
access data<br class="">
[root@srv01 ~]# gluster volume start test01<br class="">
volume start: test01: success<br class="">
[root@srv01 ~]# mount -t glusterfs srv01:/test01 /mnt<br class="">
[root@srv01 ~]# cp /etc/passwd /mnt<br class="">
[root@srv01 ~]# ls -l /mnt<br class="">
итого 2<br class="">
-rw-r--r--. 1 root root 1505 авг 16 19:59 passwd<br class="">
<br class="">
- Then remove test file from first brick like we have to
do in case of bitrot error in the file<br class="">
</blockquote>
<br style="font-family: Menlo-Regular; font-size: 13px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Menlo-Regular; font-size: 13px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px; float:
none; display: inline !important;" class="">You also need
to remove all hard-links to the corrupted file from the
brick, including the one in the .glusterfs folder.</span><br style="font-family: Menlo-Regular; font-size: 13px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Menlo-Regular; font-size: 13px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px; float:
none; display: inline !important;" class="">There is a bug
in heal-full that prevents it from crawling all bricks of
the replica. The right way to heal the corrupted files as
of now is to access them from the mount-point like you did
after removing the hard-links. The list of files that are
corrupted can be obtained with the scrub status command.</span><br style="font-family: Menlo-Regular; font-size: 13px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br style="font-family: Menlo-Regular; font-size: 13px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Menlo-Regular; font-size: 13px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px; float:
none; display: inline !important;" class="">Hope this
helps,</span><br style="font-family: Menlo-Regular;
font-size: 13px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height:
normal; orphans: auto; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows:
auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Menlo-Regular; font-size: 13px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px; float:
none; display: inline !important;" class="">Ravi</span><br style="font-family: Menlo-Regular; font-size: 13px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br style="font-family: Menlo-Regular; font-size: 13px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<blockquote type="cite" style="font-family: Menlo-Regular;
font-size: 13px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height:
normal; orphans: auto; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows:
auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">[root@srv01 ~]# rm /R1/test01/passwd<br class="">
[root@srv01 ~]# ls -l /mnt<br class="">
итого 0<br class="">
[root@srv01 ~]#<br class="">
<br class="">
- Issue full self heal<br class="">
[root@srv01 ~]# gluster volume heal test01 full<br class="">
Launching heal operation to perform full self heal on
volume test01 has been successful<br class="">
Use heal info commands to check status<br class="">
[root@srv01 ~]# tail -2 /var/log/glusterfs/glustershd.log<br class="">
[2016-08-16 16:59:56.483767] I [MSGID: 108026]
[afr-self-heald.c:611:afr_shd_full_healer]
0-test01-replicate-0: starting full sweep on subvol
test01-client-0<br class="">
[2016-08-16 16:59:56.486560] I [MSGID: 108026]
[afr-self-heald.c:621:afr_shd_full_healer]
0-test01-replicate-0: finished full sweep on subvol
test01-client-0<br class="">
<br class="">
- Now we still see no files in mount point (it becomes
empty right after removing file from the brick)<br class="">
[root@srv01 ~]# ls -l /mnt<br class="">
итого 0<br class="">
[root@srv01 ~]#<br class="">
<br class="">
- Then try to access file by using full name
(lookup-optimize and readdir-optimize are turned off by
default). Now glusterfs shows the file!<br class="">
[root@srv01 ~]# ls -l /mnt/passwd<br class="">
-rw-r--r--. 1 root root 1505 авг 16 19:59 /mnt/passwd<br class="">
<br class="">
- And it reappeared in the brick<br class="">
[root@srv01 ~]# ls -l /R1/test01/<br class="">
итого 4<br class="">
-rw-r--r--. 2 root root 1505 авг 16 19:59 passwd<br class="">
[root@srv01 ~]#<br class="">
<br class="">
Is it a bug or we can tell self heal to scan all files on
all bricks in the volume?<br class="">
<br class="">
--<br class="">
Dmitry Glushenok<br class="">
Jet Infosystems<br class="">
<br class="">
_______________________________________________<br class="">
Gluster-users mailing list<br class="">
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class="">
<a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users" class="">http://www.gluster.org/mailman/listinfo/gluster-users</a></blockquote>
</div>
</blockquote>
</div>
<br class="">
</blockquote><p class=""><br class="">
</p>
</div>
</div></blockquote></div><br class=""></body></html>