<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">You are right, stat triggers self-heal. Thank you!</div><br class=""><div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">--</div><div class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class="">Dmitry Glushenok</div><div class="">Jet Infosystems</div></div></div></div></div></div></div></div></div></div></div></div>
</div>

<br class=""><div><blockquote type="cite" class=""><div class="">17 авг. 2016 г., в 13:38, Ravishankar N &lt;<a href="mailto:ravishankar@redhat.com" class="">ravishankar@redhat.com</a>&gt; написал(а):</div><br class="Apple-interchange-newline"><div class="">
  
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type" class="">
  
  <div bgcolor="#FFFFFF" text="#000000" class="">
    <div class="moz-cite-prefix">On 08/17/2016 03:48 PM, Дмитрий
      Глушенок wrote:<br class="">
    </div>
    <blockquote cite="mid:CD77111B-0A75-4DCD-9AC3-807E8E72536F@jet.msk.su" type="cite" class="">
      <meta http-equiv="Content-Type" content="text/html; charset=utf-8" class="">
      <div class="">Unfortunately not:</div>
      <div class=""><br class="">
      </div>
      <div class="">Remount FS, then access test file from second
        client:</div>
      <div class=""><br class="">
      </div>
      <div class="">
        <div class="">[root@srv02 ~]# umount /mnt</div>
        <div class="">[root@srv02 ~]# mount -t glusterfs srv01:/test01
          /mnt</div>
        <div class="">[root@srv02 ~]# ls -l /mnt/passwd&nbsp;</div>
        <div class="">-rw-r--r--. 1 root root 1505 авг 16 19:59
          /mnt/passwd</div>
        <div class="">[root@srv02 ~]# ls -l /R1/test01/</div>
        <div class="">итого 4</div>
        <div class="">-rw-r--r--. 2 root root 1505 авг 16 19:59 passwd</div>
        <div class="">[root@srv02 ~]#&nbsp;</div>
        <div class=""><br class="">
        </div>
        <div class="">Then remount FS and check if accessing the file
          from second node triggered self-heal on first node:</div>
        <div class=""><br class="">
        </div>
        <div class="">[root@srv01 ~]# umount /mnt</div>
        <div class="">[root@srv01 ~]# mount -t glusterfs srv01:/test01
          /mnt</div>
        <div class="">[root@srv01 ~]# ls -l /mnt</div>
      </div>
    </blockquote>
    <br class="">
    Can you try `stat /mnt/passwd` from this node after remounting? You
    need to explicitly lookup the file.&nbsp; `ls -l /mnt`&nbsp; is only
    triggering readdir on the parent directory.<br class="">
    If that doesn't work, is this mount connected to both clients? i.e.
    if you create a new file from here, is it getting replicated to both
    bricks?<br class="">
    <br class="">
    -Ravi<br class="">
    <br class="">
    <blockquote cite="mid:CD77111B-0A75-4DCD-9AC3-807E8E72536F@jet.msk.su" type="cite" class="">
      <div class="">
        <div class="">итого 0</div>
        <div class="">[root@srv01 ~]# ls -l /R1/test01/</div>
        <div class="">итого 0</div>
        <div class="">[root@srv01 ~]#</div>
      </div>
      <div class=""><br class="">
      </div>
      <div class="">Nothing appeared.</div>
      <div class=""><br class="">
      </div>
      <div class="">
        <div class="">[root@srv01 ~]# gluster volume info test01</div>
        <div class="">&nbsp;</div>
        <div class="">Volume Name: test01</div>
        <div class="">Type: Replicate</div>
        <div class="">Volume ID: 2c227085-0b06-4804-805c-ea9c1bb11d8b</div>
        <div class="">Status: Started</div>
        <div class="">Number of Bricks: 1 x 2 = 2</div>
        <div class="">Transport-type: tcp</div>
        <div class="">Bricks:</div>
        <div class="">Brick1: srv01:/R1/test01</div>
        <div class="">Brick2: srv02:/R1/test01</div>
        <div class="">Options Reconfigured:</div>
        <div class="">features.scrub-freq: hourly</div>
        <div class="">features.scrub: Active</div>
        <div class="">features.bitrot: on</div>
        <div class="">transport.address-family: inet</div>
        <div class="">performance.readdir-ahead: on</div>
        <div class="">nfs.disable: on</div>
        <div class="">[root@srv01 ~]#&nbsp;</div>
      </div>
      <div class=""><br class="">
      </div>
      <div class="">
        <div class="">[root@srv01 ~]# gluster volume get test01 all |
          grep heal</div>
        <div class="">cluster.background-self-heal-count &nbsp; &nbsp; &nbsp;8 &nbsp; &nbsp; &nbsp; &nbsp;
          &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</div>
        <div class="">cluster.metadata-self-heal &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;on &nbsp; &nbsp; &nbsp; &nbsp;
          &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</div>
        <div class="">cluster.data-self-heal &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;on &nbsp; &nbsp; &nbsp; &nbsp;
          &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</div>
        <div class="">cluster.entry-self-heal &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; on &nbsp; &nbsp; &nbsp; &nbsp;
          &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</div>
        <div class="">cluster.self-heal-daemon &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;on &nbsp; &nbsp; &nbsp; &nbsp;
          &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</div>
        <div class="">cluster.heal-timeout &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;600 &nbsp; &nbsp; &nbsp;
          &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</div>
        <div class="">cluster.self-heal-window-size &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 1 &nbsp; &nbsp; &nbsp; &nbsp;
          &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</div>
        <div class="">cluster.data-self-heal-algorithm &nbsp; &nbsp; &nbsp; &nbsp;(null) &nbsp; &nbsp;
          &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</div>
        <div class="">cluster.self-heal-readdir-size &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1KB &nbsp; &nbsp; &nbsp;
          &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</div>
        <div class="">cluster.heal-wait-queue-length &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;128 &nbsp; &nbsp; &nbsp;
          &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</div>
        <div class="">features.lock-heal &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;off &nbsp; &nbsp; &nbsp;
          &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</div>
        <div class="">features.lock-heal &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;off &nbsp; &nbsp; &nbsp;
          &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</div>
        <div class="">storage.health-check-interval &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 30 &nbsp; &nbsp; &nbsp; &nbsp;
          &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</div>
        <div class="">features.ctr_lookupheal_link_timeout &nbsp; &nbsp;300 &nbsp; &nbsp; &nbsp;
          &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</div>
        <div class="">features.ctr_lookupheal_inode_timeout &nbsp; 300 &nbsp; &nbsp; &nbsp;
          &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</div>
        <div class="">cluster.disperse-self-heal-daemon &nbsp; &nbsp; &nbsp; enable &nbsp; &nbsp;
          &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</div>
        <div class="">disperse.background-heals &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 8 &nbsp; &nbsp; &nbsp; &nbsp;
          &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</div>
        <div class="">disperse.heal-wait-qlength &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;128 &nbsp; &nbsp; &nbsp;
          &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</div>
        <div class="">cluster.heal-timeout &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;600 &nbsp; &nbsp; &nbsp;
          &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</div>
        <div class="">cluster.granular-entry-heal &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; no &nbsp; &nbsp; &nbsp; &nbsp;
          &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</div>
        <div class="">[root@srv01 ~]#</div>
      </div>
      <br class="">
      <div class="">
        <div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
          <div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
            <div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
              <div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
                <div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
                  <div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
                    <div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
                      <div class="">--</div>
                      <div class="">
                        <div class="" style="word-wrap: break-word;
                          -webkit-nbsp-mode: space; -webkit-line-break:
                          after-white-space;">
                          <div class="" style="word-wrap: break-word;
                            -webkit-nbsp-mode: space;
                            -webkit-line-break: after-white-space;">
                            <div class="" style="word-wrap: break-word;
                              -webkit-nbsp-mode: space;
                              -webkit-line-break: after-white-space;">
                              <div class="">Dmitry Glushenok</div>
                              <div class="">Jet Infosystems</div>
                            </div>
                          </div>
                        </div>
                      </div>
                    </div>
                  </div>
                </div>
              </div>
            </div>
          </div>
        </div>
      </div>
      <br class="">
      <div class="">
        <blockquote type="cite" class="">
          <div class="">17 авг. 2016 г., в 11:30, Ravishankar N &lt;<a moz-do-not-send="true" href="mailto:ravishankar@redhat.com" class="">ravishankar@redhat.com</a>&gt;
            написал(а):</div>
          <br class="Apple-interchange-newline">
          <div class="">
            <meta content="text/html; charset=utf-8" http-equiv="Content-Type" class="">
            <div bgcolor="#FFFFFF" text="#000000" class="">
              <div class="moz-cite-prefix">On 08/17/2016 01:48 PM,
                Дмитрий Глушенок wrote:<br class="">
              </div>
              <blockquote cite="mid:FE353586-F02C-40B3-8D1C-8C31F46AB258@jet.msk.su" type="cite" class="">
                <meta http-equiv="Content-Type" content="text/html;
                  charset=utf-8" class="">
                <div class="">Hello Ravi,</div>
                <div class=""><br class="">
                </div>
                <div class="">Thank you for reply. Found bug number (for
                  those who will google the email) <a moz-do-not-send="true" href="https://bugzilla.redhat.com/show_bug.cgi?id=1112158" class="">https://bugzilla.redhat.com/show_bug.cgi?id=1112158</a></div>
                <div class=""><br class="">
                </div>
                <div class="">Accessing the removed file from
                  mount-point is not always working because we have to
                  find a special client which DHT will point to the
                  brick with removed file. Otherwise the file will be
                  accessed from good brick and self-healing will not
                  happen (just verified). Or by accessing you meant
                  something like touch?</div>
              </blockquote>
              <br class="">
              Sorry should have been more explicit. I meant triggering a
              lookup on that file with `stat filename`. I don't think
              you need a special client. DHT sends the lookup to AFR
              which in turn sends to all its children. When one of them
              returns ENOENT (because you removed it from the brick),
              AFR will automatically trigger heal. I'm guessing it is
              not always working in your case due to caching at various
              levels and the lookup not coming till AFR. If you do it
              from a fresh mount ,it should always work.<br class="">
              -Ravi<br class="">
              <br class="">
              <blockquote cite="mid:FE353586-F02C-40B3-8D1C-8C31F46AB258@jet.msk.su" type="cite" class="">
                <div class="">
                  <div style="letter-spacing: normal; orphans: auto;
                    text-align: start; text-indent: 0px; text-transform:
                    none; white-space: normal; widows: auto;
                    word-spacing: 0px; -webkit-text-stroke-width: 0px;
                    word-wrap: break-word; -webkit-nbsp-mode: space;
                    -webkit-line-break: after-white-space;" class="">
                    <div style="letter-spacing: normal; orphans: auto;
                      text-align: start; text-indent: 0px;
                      text-transform: none; white-space: normal; widows:
                      auto; word-spacing: 0px;
                      -webkit-text-stroke-width: 0px; word-wrap:
                      break-word; -webkit-nbsp-mode: space;
                      -webkit-line-break: after-white-space;" class="">
                      <div style="letter-spacing: normal; orphans: auto;
                        text-align: start; text-indent: 0px;
                        text-transform: none; white-space: normal;
                        widows: auto; word-spacing: 0px;
                        -webkit-text-stroke-width: 0px; word-wrap:
                        break-word; -webkit-nbsp-mode: space;
                        -webkit-line-break: after-white-space;" class="">
                        <div style="letter-spacing: normal; orphans:
                          auto; text-align: start; text-indent: 0px;
                          text-transform: none; white-space: normal;
                          widows: auto; word-spacing: 0px;
                          -webkit-text-stroke-width: 0px; word-wrap:
                          break-word; -webkit-nbsp-mode: space;
                          -webkit-line-break: after-white-space;" class="">
                          <div style="letter-spacing: normal; orphans:
                            auto; text-align: start; text-indent: 0px;
                            text-transform: none; white-space: normal;
                            widows: auto; word-spacing: 0px;
                            -webkit-text-stroke-width: 0px; word-wrap:
                            break-word; -webkit-nbsp-mode: space;
                            -webkit-line-break: after-white-space;" class="">
                            <div style="letter-spacing: normal; orphans:
                              auto; text-align: start; text-indent: 0px;
                              text-transform: none; white-space: normal;
                              widows: auto; word-spacing: 0px;
                              -webkit-text-stroke-width: 0px; word-wrap:
                              break-word; -webkit-nbsp-mode: space;
                              -webkit-line-break: after-white-space;" class="">
                              <div style="letter-spacing: normal;
                                orphans: auto; text-align: start;
                                text-indent: 0px; text-transform: none;
                                white-space: normal; widows: auto;
                                word-spacing: 0px;
                                -webkit-text-stroke-width: 0px;
                                word-wrap: break-word;
                                -webkit-nbsp-mode: space;
                                -webkit-line-break: after-white-space;" class="">
                                <div class="">
                                  <div class="" style="word-wrap:
                                    break-word; -webkit-nbsp-mode:
                                    space; -webkit-line-break:
                                    after-white-space;">
                                    <div class="" style="word-wrap:
                                      break-word; -webkit-nbsp-mode:
                                      space; -webkit-line-break:
                                      after-white-space;">
                                      <div class="" style="word-wrap:
                                        break-word; -webkit-nbsp-mode:
                                        space; -webkit-line-break:
                                        after-white-space;">
                                        <div class="">Dmitry Glushenok</div>
                                        <div class="">Jet Infosystems</div>
                                      </div>
                                    </div>
                                  </div>
                                </div>
                              </div>
                            </div>
                          </div>
                        </div>
                      </div>
                    </div>
                  </div>
                </div>
                <br class="">
                <div class="">
                  <blockquote type="cite" class="">
                    <div class="">17 авг. 2016 г., в 4:24, Ravishankar N
                      &lt;<a moz-do-not-send="true" href="mailto:ravishankar@redhat.com" class="">ravishankar@redhat.com</a>&gt;
                      написал(а):</div>
                    <br class="Apple-interchange-newline">
                    <div class=""><span style="font-family:
                        Menlo-Regular; font-size: 13px; font-style:
                        normal; font-variant: normal; font-weight:
                        normal; letter-spacing: normal; line-height:
                        normal; orphans: auto; text-align: start;
                        text-indent: 0px; text-transform: none;
                        white-space: normal; widows: auto; word-spacing:
                        0px; -webkit-text-stroke-width: 0px; float:
                        none; display: inline !important;" class="">On
                        08/16/2016 10:44 PM, Дмитрий Глушенок wrote:</span><br style="font-family: Menlo-Regular; font-size:
                        13px; font-style: normal; font-variant: normal;
                        font-weight: normal; letter-spacing: normal;
                        line-height: normal; orphans: auto; text-align:
                        start; text-indent: 0px; text-transform: none;
                        white-space: normal; widows: auto; word-spacing:
                        0px; -webkit-text-stroke-width: 0px;" class="">
                      <blockquote type="cite" style="font-family:
                        Menlo-Regular; font-size: 13px; font-style:
                        normal; font-variant: normal; font-weight:
                        normal; letter-spacing: normal; line-height:
                        normal; orphans: auto; text-align: start;
                        text-indent: 0px; text-transform: none;
                        white-space: normal; widows: auto; word-spacing:
                        0px; -webkit-text-stroke-width: 0px;" class="">Hello,<br class="">
                        <br class="">
                        While testing healing after bitrot error it was
                        found that self healing cannot heal files which
                        were manually deleted from brick. Gluster 3.8.1:<br class="">
                        <br class="">
                        - Create volume, mount it locally and copy test
                        file to it<br class="">
                        [root@srv01 ~]# gluster volume create test01
                        replica 2 &nbsp;srv01:/R1/test01 srv02:/R1/test01<br class="">
                        volume create: test01: success: please start the
                        volume to access data<br class="">
                        [root@srv01 ~]# gluster volume start test01<br class="">
                        volume start: test01: success<br class="">
                        [root@srv01 ~]# mount -t glusterfs srv01:/test01
                        /mnt<br class="">
                        [root@srv01 ~]# cp /etc/passwd /mnt<br class="">
                        [root@srv01 ~]# ls -l /mnt<br class="">
                        итого 2<br class="">
                        -rw-r--r--. 1 root root 1505 авг 16 19:59 passwd<br class="">
                        <br class="">
                        - Then remove test file from first brick like we
                        have to do in case of bitrot error in the file<br class="">
                      </blockquote>
                      <br style="font-family: Menlo-Regular; font-size:
                        13px; font-style: normal; font-variant: normal;
                        font-weight: normal; letter-spacing: normal;
                        line-height: normal; orphans: auto; text-align:
                        start; text-indent: 0px; text-transform: none;
                        white-space: normal; widows: auto; word-spacing:
                        0px; -webkit-text-stroke-width: 0px;" class="">
                      <span style="font-family: Menlo-Regular;
                        font-size: 13px; font-style: normal;
                        font-variant: normal; font-weight: normal;
                        letter-spacing: normal; line-height: normal;
                        orphans: auto; text-align: start; text-indent:
                        0px; text-transform: none; white-space: normal;
                        widows: auto; word-spacing: 0px;
                        -webkit-text-stroke-width: 0px; float: none;
                        display: inline !important;" class="">You also
                        need to remove all hard-links to the corrupted
                        file from the brick, including the one in the
                        .glusterfs folder.</span><br style="font-family:
                        Menlo-Regular; font-size: 13px; font-style:
                        normal; font-variant: normal; font-weight:
                        normal; letter-spacing: normal; line-height:
                        normal; orphans: auto; text-align: start;
                        text-indent: 0px; text-transform: none;
                        white-space: normal; widows: auto; word-spacing:
                        0px; -webkit-text-stroke-width: 0px;" class="">
                      <span style="font-family: Menlo-Regular;
                        font-size: 13px; font-style: normal;
                        font-variant: normal; font-weight: normal;
                        letter-spacing: normal; line-height: normal;
                        orphans: auto; text-align: start; text-indent:
                        0px; text-transform: none; white-space: normal;
                        widows: auto; word-spacing: 0px;
                        -webkit-text-stroke-width: 0px; float: none;
                        display: inline !important;" class="">There is a
                        bug in heal-full that prevents it from crawling
                        all bricks of the replica. The right way to heal
                        the corrupted files as of now is to access them
                        from the mount-point like you did after removing
                        the hard-links. The list of files that are
                        corrupted can be obtained with the scrub status
                        command.</span><br style="font-family:
                        Menlo-Regular; font-size: 13px; font-style:
                        normal; font-variant: normal; font-weight:
                        normal; letter-spacing: normal; line-height:
                        normal; orphans: auto; text-align: start;
                        text-indent: 0px; text-transform: none;
                        white-space: normal; widows: auto; word-spacing:
                        0px; -webkit-text-stroke-width: 0px;" class="">
                      <br style="font-family: Menlo-Regular; font-size:
                        13px; font-style: normal; font-variant: normal;
                        font-weight: normal; letter-spacing: normal;
                        line-height: normal; orphans: auto; text-align:
                        start; text-indent: 0px; text-transform: none;
                        white-space: normal; widows: auto; word-spacing:
                        0px; -webkit-text-stroke-width: 0px;" class="">
                      <span style="font-family: Menlo-Regular;
                        font-size: 13px; font-style: normal;
                        font-variant: normal; font-weight: normal;
                        letter-spacing: normal; line-height: normal;
                        orphans: auto; text-align: start; text-indent:
                        0px; text-transform: none; white-space: normal;
                        widows: auto; word-spacing: 0px;
                        -webkit-text-stroke-width: 0px; float: none;
                        display: inline !important;" class="">Hope this
                        helps,</span><br style="font-family:
                        Menlo-Regular; font-size: 13px; font-style:
                        normal; font-variant: normal; font-weight:
                        normal; letter-spacing: normal; line-height:
                        normal; orphans: auto; text-align: start;
                        text-indent: 0px; text-transform: none;
                        white-space: normal; widows: auto; word-spacing:
                        0px; -webkit-text-stroke-width: 0px;" class="">
                      <span style="font-family: Menlo-Regular;
                        font-size: 13px; font-style: normal;
                        font-variant: normal; font-weight: normal;
                        letter-spacing: normal; line-height: normal;
                        orphans: auto; text-align: start; text-indent:
                        0px; text-transform: none; white-space: normal;
                        widows: auto; word-spacing: 0px;
                        -webkit-text-stroke-width: 0px; float: none;
                        display: inline !important;" class="">Ravi</span><br style="font-family: Menlo-Regular; font-size:
                        13px; font-style: normal; font-variant: normal;
                        font-weight: normal; letter-spacing: normal;
                        line-height: normal; orphans: auto; text-align:
                        start; text-indent: 0px; text-transform: none;
                        white-space: normal; widows: auto; word-spacing:
                        0px; -webkit-text-stroke-width: 0px;" class="">
                      <br style="font-family: Menlo-Regular; font-size:
                        13px; font-style: normal; font-variant: normal;
                        font-weight: normal; letter-spacing: normal;
                        line-height: normal; orphans: auto; text-align:
                        start; text-indent: 0px; text-transform: none;
                        white-space: normal; widows: auto; word-spacing:
                        0px; -webkit-text-stroke-width: 0px;" class="">
                      <blockquote type="cite" style="font-family:
                        Menlo-Regular; font-size: 13px; font-style:
                        normal; font-variant: normal; font-weight:
                        normal; letter-spacing: normal; line-height:
                        normal; orphans: auto; text-align: start;
                        text-indent: 0px; text-transform: none;
                        white-space: normal; widows: auto; word-spacing:
                        0px; -webkit-text-stroke-width: 0px;" class="">[root@srv01
                        ~]# rm /R1/test01/passwd<br class="">
                        [root@srv01 ~]# ls -l /mnt<br class="">
                        итого 0<br class="">
                        [root@srv01 ~]#<br class="">
                        <br class="">
                        - Issue full self heal<br class="">
                        [root@srv01 ~]# gluster volume heal test01 full<br class="">
                        Launching heal operation to perform full self
                        heal on volume test01 has been successful<br class="">
                        Use heal info commands to check status<br class="">
                        [root@srv01 ~]# tail -2
                        /var/log/glusterfs/glustershd.log<br class="">
                        [2016-08-16 16:59:56.483767] I [MSGID: 108026]
                        [afr-self-heald.c:611:afr_shd_full_healer]
                        0-test01-replicate-0: starting full sweep on
                        subvol test01-client-0<br class="">
                        [2016-08-16 16:59:56.486560] I [MSGID: 108026]
                        [afr-self-heald.c:621:afr_shd_full_healer]
                        0-test01-replicate-0: finished full sweep on
                        subvol test01-client-0<br class="">
                        <br class="">
                        - Now we still see no files in mount point (it
                        becomes empty right after removing file from the
                        brick)<br class="">
                        [root@srv01 ~]# ls -l /mnt<br class="">
                        итого 0<br class="">
                        [root@srv01 ~]#<br class="">
                        <br class="">
                        - Then try to access file by using full name
                        (lookup-optimize and readdir-optimize are turned
                        off by default). Now glusterfs shows the file!<br class="">
                        [root@srv01 ~]# ls -l /mnt/passwd<br class="">
                        -rw-r--r--. 1 root root 1505 авг 16 19:59
                        /mnt/passwd<br class="">
                        <br class="">
                        - And it reappeared in the brick<br class="">
                        [root@srv01 ~]# ls -l /R1/test01/<br class="">
                        итого 4<br class="">
                        -rw-r--r--. 2 root root 1505 авг 16 19:59 passwd<br class="">
                        [root@srv01 ~]#<br class="">
                        <br class="">
                        Is it a bug or we can tell self heal to scan all
                        files on all bricks in the volume?<br class="">
                        <br class="">
                        --<br class="">
                        Dmitry Glushenok<br class="">
                        Jet Infosystems<br class="">
                        <br class="">
                        _______________________________________________<br class="">
                        Gluster-users mailing list<br class="">
                        <a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class="">
                        <a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users" class="">http://www.gluster.org/mailman/listinfo/gluster-users</a></blockquote>
                    </div>
                  </blockquote>
                </div>
                <br class="">
              </blockquote><p class=""><br class="">
              </p>
            </div>
          </div>
        </blockquote>
      </div>
      <br class="">
    </blockquote><p class=""><br class="">
    </p>
  </div>

</div></blockquote></div><br class=""></body></html>