<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 08/17/2016 01:48 PM, Дмитрий
Глушенок wrote:<br>
</div>
<blockquote
cite="mid:FE353586-F02C-40B3-8D1C-8C31F46AB258@jet.msk.su"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<div class="">Hello Ravi,</div>
<div class=""><br class="">
</div>
<div class="">Thank you for reply. Found bug number (for those who
will google the email) <a moz-do-not-send="true"
href="https://bugzilla.redhat.com/show_bug.cgi?id=1112158"
class="">https://bugzilla.redhat.com/show_bug.cgi?id=1112158</a></div>
<div class=""><br class="">
</div>
<div class="">Accessing the removed file from mount-point is not
always working because we have to find a special client which
DHT will point to the brick with removed file. Otherwise the
file will be accessed from good brick and self-healing will not
happen (just verified). Or by accessing you meant something like
touch?</div>
</blockquote>
<br>
Sorry should have been more explicit. I meant triggering a lookup on
that file with `stat filename`. I don't think you need a special
client. DHT sends the lookup to AFR which in turn sends to all its
children. When one of them returns ENOENT (because you removed it
from the brick), AFR will automatically trigger heal. I'm guessing
it is not always working in your case due to caching at various
levels and the lookup not coming till AFR. If you do it from a fresh
mount ,it should always work.<br>
-Ravi<br>
<br>
<blockquote
cite="mid:FE353586-F02C-40B3-8D1C-8C31F46AB258@jet.msk.su"
type="cite">
<div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap:
break-word; -webkit-nbsp-mode: space; -webkit-line-break:
after-white-space;" class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
word-wrap: break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;" class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
word-wrap: break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;" class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
word-wrap: break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;" class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows:
auto; word-spacing: 0px; -webkit-text-stroke-width:
0px; word-wrap: break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;" class="">
<div style="color: rgb(0, 0, 0); letter-spacing:
normal; orphans: auto; text-align: start;
text-indent: 0px; text-transform: none; white-space:
normal; widows: auto; word-spacing: 0px;
-webkit-text-stroke-width: 0px; word-wrap:
break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;" class="">
<div style="color: rgb(0, 0, 0); letter-spacing:
normal; orphans: auto; text-align: start;
text-indent: 0px; text-transform: none;
white-space: normal; widows: auto; word-spacing:
0px; -webkit-text-stroke-width: 0px; word-wrap:
break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;" class="">
<div class="">
<div class="" style="word-wrap: break-word;
-webkit-nbsp-mode: space; -webkit-line-break:
after-white-space;">
<div class="" style="word-wrap: break-word;
-webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;">
<div class="" style="word-wrap: break-word;
-webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;">
<div class="">Dmitry Glushenok</div>
<div class="">Jet Infosystems</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">17 авг. 2016 г., в 4:24, Ravishankar N <<a
moz-do-not-send="true"
href="mailto:ravishankar@redhat.com" class="">ravishankar@redhat.com</a>>
написал(а):</div>
<br class="Apple-interchange-newline">
<div class=""><span style="font-family: Menlo-Regular;
font-size: 13px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height:
normal; orphans: auto; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows:
auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;
float: none; display: inline !important;" class="">On
08/16/2016 10:44 PM, Дмитрий Глушенок wrote:</span><br
style="font-family: Menlo-Regular; font-size: 13px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;"
class="">
<blockquote type="cite" style="font-family: Menlo-Regular;
font-size: 13px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height:
normal; orphans: auto; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows:
auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"
class="">Hello,<br class="">
<br class="">
While testing healing after bitrot error it was found that
self healing cannot heal files which were manually deleted
from brick. Gluster 3.8.1:<br class="">
<br class="">
- Create volume, mount it locally and copy test file to it<br
class="">
[root@srv01 ~]# gluster volume create test01 replica 2
srv01:/R1/test01 srv02:/R1/test01<br class="">
volume create: test01: success: please start the volume to
access data<br class="">
[root@srv01 ~]# gluster volume start test01<br class="">
volume start: test01: success<br class="">
[root@srv01 ~]# mount -t glusterfs srv01:/test01 /mnt<br
class="">
[root@srv01 ~]# cp /etc/passwd /mnt<br class="">
[root@srv01 ~]# ls -l /mnt<br class="">
итого 2<br class="">
-rw-r--r--. 1 root root 1505 авг 16 19:59 passwd<br
class="">
<br class="">
- Then remove test file from first brick like we have to
do in case of bitrot error in the file<br class="">
</blockquote>
<br style="font-family: Menlo-Regular; font-size: 13px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;"
class="">
<span style="font-family: Menlo-Regular; font-size: 13px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px; float:
none; display: inline !important;" class="">You also need
to remove all hard-links to the corrupted file from the
brick, including the one in the .glusterfs folder.</span><br
style="font-family: Menlo-Regular; font-size: 13px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;"
class="">
<span style="font-family: Menlo-Regular; font-size: 13px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px; float:
none; display: inline !important;" class="">There is a bug
in heal-full that prevents it from crawling all bricks of
the replica. The right way to heal the corrupted files as
of now is to access them from the mount-point like you did
after removing the hard-links. The list of files that are
corrupted can be obtained with the scrub status command.</span><br
style="font-family: Menlo-Regular; font-size: 13px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;"
class="">
<br style="font-family: Menlo-Regular; font-size: 13px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;"
class="">
<span style="font-family: Menlo-Regular; font-size: 13px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px; float:
none; display: inline !important;" class="">Hope this
helps,</span><br style="font-family: Menlo-Regular;
font-size: 13px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height:
normal; orphans: auto; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows:
auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"
class="">
<span style="font-family: Menlo-Regular; font-size: 13px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px; float:
none; display: inline !important;" class="">Ravi</span><br
style="font-family: Menlo-Regular; font-size: 13px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;"
class="">
<br style="font-family: Menlo-Regular; font-size: 13px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;"
class="">
<blockquote type="cite" style="font-family: Menlo-Regular;
font-size: 13px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height:
normal; orphans: auto; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows:
auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"
class="">[root@srv01 ~]# rm /R1/test01/passwd<br class="">
[root@srv01 ~]# ls -l /mnt<br class="">
итого 0<br class="">
[root@srv01 ~]#<br class="">
<br class="">
- Issue full self heal<br class="">
[root@srv01 ~]# gluster volume heal test01 full<br
class="">
Launching heal operation to perform full self heal on
volume test01 has been successful<br class="">
Use heal info commands to check status<br class="">
[root@srv01 ~]# tail -2 /var/log/glusterfs/glustershd.log<br
class="">
[2016-08-16 16:59:56.483767] I [MSGID: 108026]
[afr-self-heald.c:611:afr_shd_full_healer]
0-test01-replicate-0: starting full sweep on subvol
test01-client-0<br class="">
[2016-08-16 16:59:56.486560] I [MSGID: 108026]
[afr-self-heald.c:621:afr_shd_full_healer]
0-test01-replicate-0: finished full sweep on subvol
test01-client-0<br class="">
<br class="">
- Now we still see no files in mount point (it becomes
empty right after removing file from the brick)<br
class="">
[root@srv01 ~]# ls -l /mnt<br class="">
итого 0<br class="">
[root@srv01 ~]#<br class="">
<br class="">
- Then try to access file by using full name
(lookup-optimize and readdir-optimize are turned off by
default). Now glusterfs shows the file!<br class="">
[root@srv01 ~]# ls -l /mnt/passwd<br class="">
-rw-r--r--. 1 root root 1505 авг 16 19:59 /mnt/passwd<br
class="">
<br class="">
- And it reappeared in the brick<br class="">
[root@srv01 ~]# ls -l /R1/test01/<br class="">
итого 4<br class="">
-rw-r--r--. 2 root root 1505 авг 16 19:59 passwd<br
class="">
[root@srv01 ~]#<br class="">
<br class="">
Is it a bug or we can tell self heal to scan all files on
all bricks in the volume?<br class="">
<br class="">
--<br class="">
Dmitry Glushenok<br class="">
Jet Infosystems<br class="">
<br class="">
_______________________________________________<br
class="">
Gluster-users mailing list<br class="">
<a moz-do-not-send="true"
href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br
class="">
<a moz-do-not-send="true"
href="http://www.gluster.org/mailman/listinfo/gluster-users"
class="">http://www.gluster.org/mailman/listinfo/gluster-users</a></blockquote>
</div>
</blockquote>
</div>
<br class="">
</blockquote>
<p><br>
</p>
</body>
</html>