<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Hello Ravi,</div><div class=""><br class=""></div><div class="">Thank you for reply. Found bug number (for those who will google the email) <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1112158" class="">https://bugzilla.redhat.com/show_bug.cgi?id=1112158</a></div><div class=""><br class=""></div><div class="">Accessing the removed file from mount-point is not always working because we have to find a special client which DHT will point to the brick with removed file. Otherwise the file will be accessed from good brick and self-healing will not happen (just verified). Or by accessing you meant something like touch?</div><div class=""><br class=""></div><div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">--</div><div class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class="">Dmitry Glushenok</div><div class="">Jet Infosystems</div></div></div></div></div></div></div></div></div></div></div></div>
</div>
<br class=""><div><blockquote type="cite" class=""><div class="">17 авг. 2016 г., в 4:24, Ravishankar N <<a href="mailto:ravishankar@redhat.com" class="">ravishankar@redhat.com</a>> написал(а):</div><br class="Apple-interchange-newline"><div class=""><span style="font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">On 08/16/2016 10:44 PM, Дмитрий Глушенок wrote:</span><br style="font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">Hello,<br class=""><br class="">While testing healing after bitrot error it was found that self healing cannot heal files which were manually deleted from brick. Gluster 3.8.1:<br class=""><br class="">- Create volume, mount it locally and copy test file to it<br class="">[root@srv01 ~]# gluster volume create test01 replica 2 srv01:/R1/test01 srv02:/R1/test01<br class="">volume create: test01: success: please start the volume to access data<br class="">[root@srv01 ~]# gluster volume start test01<br class="">volume start: test01: success<br class="">[root@srv01 ~]# mount -t glusterfs srv01:/test01 /mnt<br class="">[root@srv01 ~]# cp /etc/passwd /mnt<br class="">[root@srv01 ~]# ls -l /mnt<br class="">итого 2<br class="">-rw-r--r--. 1 root root 1505 авг 16 19:59 passwd<br class=""><br class="">- Then remove test file from first brick like we have to do in case of bitrot error in the file<br class=""></blockquote><br style="font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">You also need to remove all hard-links to the corrupted file from the brick, including the one in the .glusterfs folder.</span><br style="font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">There is a bug in heal-full that prevents it from crawling all bricks of the replica. The right way to heal the corrupted files as of now is to access them from the mount-point like you did after removing the hard-links. The list of files that are corrupted can be obtained with the scrub status command.</span><br style="font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Hope this helps,</span><br style="font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Ravi</span><br style="font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Menlo-Regular; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">[root@srv01 ~]# rm /R1/test01/passwd<br class="">[root@srv01 ~]# ls -l /mnt<br class="">итого 0<br class="">[root@srv01 ~]#<br class=""><br class="">- Issue full self heal<br class="">[root@srv01 ~]# gluster volume heal test01 full<br class="">Launching heal operation to perform full self heal on volume test01 has been successful<br class="">Use heal info commands to check status<br class="">[root@srv01 ~]# tail -2 /var/log/glusterfs/glustershd.log<br class="">[2016-08-16 16:59:56.483767] I [MSGID: 108026] [afr-self-heald.c:611:afr_shd_full_healer] 0-test01-replicate-0: starting full sweep on subvol test01-client-0<br class="">[2016-08-16 16:59:56.486560] I [MSGID: 108026] [afr-self-heald.c:621:afr_shd_full_healer] 0-test01-replicate-0: finished full sweep on subvol test01-client-0<br class=""><br class="">- Now we still see no files in mount point (it becomes empty right after removing file from the brick)<br class="">[root@srv01 ~]# ls -l /mnt<br class="">итого 0<br class="">[root@srv01 ~]#<br class=""><br class="">- Then try to access file by using full name (lookup-optimize and readdir-optimize are turned off by default). Now glusterfs shows the file!<br class="">[root@srv01 ~]# ls -l /mnt/passwd<br class="">-rw-r--r--. 1 root root 1505 авг 16 19:59 /mnt/passwd<br class=""><br class="">- And it reappeared in the brick<br class="">[root@srv01 ~]# ls -l /R1/test01/<br class="">итого 4<br class="">-rw-r--r--. 2 root root 1505 авг 16 19:59 passwd<br class="">[root@srv01 ~]#<br class=""><br class="">Is it a bug or we can tell self heal to scan all files on all bricks in the volume?<br class=""><br class="">--<br class="">Dmitry Glushenok<br class="">Jet Infosystems<br class=""><br class="">_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class=""><a href="http://www.gluster.org/mailman/listinfo/gluster-users" class="">http://www.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div></blockquote></div><br class=""></body></html>