<html><head>
    <meta content="text/html; charset=windows-1252" http-equiv="Content-Type" />
  
  </head><body bgcolor="#FFFFFF" text="#000000">Shouldn&#39;t that heal with an odd-man-out strategy? Or are all three GFIDs different?<br><br><div class="gmail_quote">On January 3, 2017 10:21:31 PM PST, Ravishankar N &lt;ravishankar@redhat.com&gt; wrote:<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">

  
    <div class="moz-cite-prefix"><br />
      On 01/04/2017 09:31 AM, Michael Ward wrote:<br />
    </div>
    <blockquote cite="mid:F69C324C611AE94389095C7138A01FB27D81FD98@bne3-0002mitmbx.corp.mit" type="cite">
      <meta http-equiv="Content-Type" content="text/html;          charset=windows-1252" />
      <meta name="Generator" content="Microsoft Word 15 (filtered          medium)" />
      <style><!--
/* Font Definitions */
@font-face
 {font-family:"Cambria Math";
 panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
 {font-family:Calibri;
 panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
 {font-family:Consolas;
 panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
 {margin:0cm;
 margin-bottom:.0001pt;
 font-size:11.0pt;
 font-family:"Calibri",sans-serif;
 color:black;
 mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
 {mso-style-priority:99;
 color:#0563C1;
 text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
 {mso-style-priority:99;
 color:#954F72;
 text-decoration:underline;}
p
 {mso-style-priority:99;
 mso-margin-top-alt:auto;
 margin-right:0cm;
 mso-margin-bottom-alt:auto;
 margin-left:0cm;
 font-size:12.0pt;
 font-family:"Times New Roman",serif;
 color:black;}
pre
 {mso-style-priority:99;
 mso-style-link:"HTML Preformatted Char";
 margin:0cm;
 margin-bottom:.0001pt;
 font-size:10.0pt;
 font-family:"Courier New";
 color:black;}
span.EmailStyle17
 {mso-style-type:personal;
 font-family:"Calibri",sans-serif;
 color:windowtext;}
span.HTMLPreformattedChar
 {mso-style-name:"HTML Preformatted Char";
 mso-style-priority:99;
 mso-style-link:"HTML Preformatted";
 font-family:Consolas;
 color:black;
 mso-fareast-language:EN-US;}
span.EmailStyle21
 {mso-style-type:personal;
 font-family:"Calibri",sans-serif;
 color:#1F497D;}
span.EmailStyle22
 {mso-style-type:personal-compose;
 font-family:"Calibri",sans-serif;
 color:windowtext;}
.MsoChpDefault
 {mso-style-type:export-only;
 font-size:10.0pt;}
@page WordSection1
 {size:612.0pt 792.0pt;
 margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
 {page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
      <div class="WordSection1">
        <p class="MsoNormal"><span style="color:#1F497D">Hey,<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p>&nbsp;</p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">To give some
            more context around the initial incident.. These systems are
            hosted in AWS. The gluster brick for each instance is a
            seperate volume to the root volume. On prod-gluster01 a
            couple of nights ago we experienced massively high read iops
            on the root volume that we are unable to account for (&gt;
            200,000 iops when it usually sits between 0 - 100 iops ).
            The box became inaccessible as a result and after
            approximately 40 minutes with no sign of the iops reducing
            was rebooted through the AWS console. <p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p>&nbsp;</p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">The GFID
            mismatch problems appeared after that. There were initially
            ~50 impacted files, but I've fixed all but 1 of them now,
            which I'm leaving broken intentionally for further testing
            if required.<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p>&nbsp;</p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">If you don't
            mind, could you have a look over the information below and
            identify anything that looks like a problem, since obviously
            we did have a bunch of GFID mismatched files, which based on
            your email shouldn't happen..<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p>&nbsp;</p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">I've included
            everything I can think of, but if there is something else
            you would like to see, please let me know.<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p>&nbsp;</p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"># gluster
            volume info gv0<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p>&nbsp;</p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Volume Name:
            gv0<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Type: Replicate<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Volume ID:
            0ec7c49d-811c-4d4d-a3a9-e4ea9e83000c<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Status: Started<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Snapshot Count:
            0<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Number of
            Bricks: 1 x (2 + 1) = 3<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Transport-type:
            tcp<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Bricks:<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Brick1:
            prod-gluster01.fqdn.com:/export/glus_brick0/brick<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Brick2:
            prod-gluster02.fqdn.com:/export/glus_brick0/brick<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Brick3:
            prod-gluster03.fqdn.com:/export/glus_brick0/brick (arbiter)<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Options
            Reconfigured:<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">cluster.favorite-child-policy:
            none<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">nfs.disable: on<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">performance.readdir-ahead:
            on<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">client.event-threads:
            7<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">server.event-threads:
            3<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">performance.cache-size:
            256MB<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p>&nbsp;</p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p>&nbsp;</p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">cluster.favorite-child-policy
            is set to none because I reverted the change to majority
            when it didn't make any difference.<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p>&nbsp;</p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p>&nbsp;</p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">[root@prod-gluster01
            glusterfs]# getfattr -d -m . -e hex
            /export/glus_brick0/brick/home/user/.viminfo<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">getfattr:
            Removing leading '/' from absolute path names<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"># file:
            export/glus_brick0/brick/home/user/.viminfo<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.afr.dirty=0x000000000000000000000000<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.bit-rot.version=0x0200000000000000585756be00024333<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0x1b86a5a76e884f40be583fa33aa9a576<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p>&nbsp;</p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">[root@prod-gluster02
            glusterfs]# getfattr -d -m . -e hex
            /export/glus_brick0/brick/home/user/.viminfo<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">getfattr:
            Removing leading '/' from absolute path names<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"># file:
            export/glus_brick0/brick/home/user/.viminfo<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.afr.dirty=0x000000000000000000000000<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-0=0x000000020000000100000000<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.bit-rot.version=0x020000000000000058593aac000661fa<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0x4931b10977f34496a7cdf8f23809c372<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p>&nbsp;</p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">[root@prod-gluster03
            glusterfs]# getfattr -d -m . -e hex
            /export/glus_brick0/brick/home/user/.viminfo<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">getfattr:
            Removing leading '/' from absolute path names<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"># file:
            export/glus_brick0/brick/home/user/.viminfo<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.afr.dirty=0x000000000000000000000000<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-0=0x000000020000000100000000<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.bit-rot.version=0x020000000000000058585ed6000f2077<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0x4931b10977f34496a7cdf8f23809c372<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p>&nbsp;</p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Just in case
            it's useful, here is the getfattr for the parent directory:<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p>&nbsp;</p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">[root@prod-gluster01
            glusterfs]# getfattr -d -m . -e hex
            /export/glus_brick0/brick/home/user<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">getfattr:
            Removing leading '/' from absolute path names<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"># file:
            export/glus_brick0/brick/home/user<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.afr.dirty=0x000000000000000000000000<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-1=0x000000000000000000000000<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-2=0x000000000000000000000000<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0x0a49de7ee4f04aae9fc8a88378e0d193<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.glusterfs.dht=0x000000010000000000000000ffffffff<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p>&nbsp;</p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">[root@prod-gluster02
            glusterfs]# getfattr -d -m . -e hex
            /export/glus_brick0/brick/home/user<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">getfattr:
            Removing leading '/' from absolute path names<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"># file:
            export/glus_brick0/brick/home/user<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.afr.dirty=0x000000000000000000000000<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-0=0x000000000000000000000000<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-2=0x000000000000000000000000<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0x0a49de7ee4f04aae9fc8a88378e0d193<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.glusterfs.dht=0x000000010000000000000000ffffffff<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p>&nbsp;</p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">[root@prod-gluster03
            glusterfs]# getfattr -d -m . -e hex
            /export/glus_brick0/brick/home/user<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">getfattr:
            Removing leading '/' from absolute path names<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"># file:
            export/glus_brick0/brick/home/user<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.afr.dirty=0x000000000000000000000000<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-0=0x000000000000000000000000<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-1=0x000000000000000000000000<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-2=0x000000000000000000000000<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0x0a49de7ee4f04aae9fc8a88378e0d193<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">trusted.glusterfs.dht=0x000000010000000000000000ffffffff<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p>&nbsp;</p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">[root@prod-gluster01
            bricks]# gluster volume heal gv0 info<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Brick
            prod-gluster01.fqdn.com:/export/glus_brick0/brick<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Status:
            Connected<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Number of
            entries: 0<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p>&nbsp;</p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Brick
            prod-gluster02.fqdn.com:/export/glus_brick0/brick<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">&lt;gfid:4931b109-77f3-4496-a7cd-f8f23809c372&gt;<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Status:
            Connected<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Number of
            entries: 1<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p>&nbsp;</p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Brick
            prod-gluster03.fqdn.com:/export/glus_brick0/brick<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">&lt;gfid:4931b109-77f3-4496-a7cd-f8f23809c372&gt;<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Status:
            Connected<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Number of
            entries: 1<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p>&nbsp;</p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">[root@prod-gluster01
            bricks]# gluster volume heal gv0 info split-brain<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Brick
            prod-gluster01.fqdn.com:/export/glus_brick0/brick<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Status:
            Connected<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Number of
            entries in split-brain: 0<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p>&nbsp;</p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Brick
            prod-gluster02.fqdn.com:/export/glus_brick0/brick<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Status:
            Connected<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Number of
            entries in split-brain: 0<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p>&nbsp;</p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Brick
            prod-gluster03.fqdn.com:/export/glus_brick0/brick<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Status:
            Connected<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Number of
            entries in split-brain: 0<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p>&nbsp;</p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p>&nbsp;</p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Clients show
            this in the gluster.log:<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p>&nbsp;</p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">[2017-01-04
            03:13:40.863695] W [MSGID: 108008]
            [afr-self-heal-name.c:354:afr_selfheal_name_gfid_mismatch_check]
            0-gv0-replicate-0: GFID mismatch for
            &lt;gfid:0a49de7e-e4f0-4aae-9fc8-a88378e0d193&gt;/.viminfo
            4931b109-77f3-4496-a7cd-f8f23809c372 on gv0-client-1 and
            1b86a5a7-6e88-4f40-be58-3fa33aa9a576 on gv0-client-0<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">[2017-01-04
            03:13:40.867853] W [fuse-bridge.c:471:fuse_entry_cbk]
            0-glusterfs-fuse: 13067223: LOOKUP() /home/user/.viminfo
            =&gt; -1 (Input/output error)<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">There's no
            mention of either of the GFID's for the .viminfo file in
            /var/log/gluster/*.log or
            /var/log/gluster/brick/export-glus_brick0-brick.log file.<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D"><p>&nbsp;</p></span></p>
      </div>
    </blockquote>
    <br />
    Thanks for the details Michael. While it does look like a bug, I am
    not sure how we ended in this state. Either the afr xattrs of the
    parent directory were cleared without self-heal of .vimrc&nbsp; happening
    from <span style="color:#1F497D">gluster02 or 03 to 01</span>&nbsp; (or)
    it wasn't set in the first place when the file was recreated on 02
    and 03 when 01 was down. If you have some steps to re-create the
    issue, please raise a bug.<br />
    <br />
    Regards,<br />
    Ravi<br />
    <br />
    <br />
    <blockquote cite="mid:F69C324C611AE94389095C7138A01FB27D81FD98@bne3-0002mitmbx.corp.mit" type="cite">
      <div class="WordSection1">
        <p class="MsoNormal"><span style="color:#1F497D">Thank you very
            much for your time,<p></p></span></p>
        <p class="MsoNormal"><span style="color:#1F497D">Michael Ward<p></p></span></p>
        <p class="MsoNormal"><a moz-do-not-send="true" name="_MailEndCompose"><span style="color:#1F497D"><p>&nbsp;</p></span></a></p>
        <div>
          <div style="border:none;border-top:solid #E1E1E1              1.0pt;padding:3.0pt 0cm 0cm 0cm">
            <p class="MsoNormal"><b><span style="color:windowtext;mso-fareast-language:EN-AU" lang="EN-US">From:</span></b><span style="color:windowtext;mso-fareast-language:EN-AU" lang="EN-US"> Ravishankar N
                [<a class="moz-txt-link-freetext" href="mailto:ravishankar@redhat.com">mailto:ravishankar@redhat.com</a>]
                <br />
                <b>Sent:</b> Wednesday, 4 January 2017 12:21 PM<br />
                <b>To:</b> Michael Ward
                <a class="moz-txt-link-rfc2396E" href="mailto:Michael.Ward@melbourneit.com.au">&lt;Michael.Ward@melbourneit.com.au&gt;</a>;
                <a class="moz-txt-link-abbreviated" href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br />
                <b>Subject:</b> Re: [Gluster-users] GFID Mismatch -
                Automatic Correction ?<p></p></span></p>
          </div>
        </div>
        <p class="MsoNormal"></p><p>&nbsp;</p>
        <div>
          <p class="MsoNormal">On 01/04/2017 06:27 AM, Michael Ward
            wrote:<span style="font-size:12.0pt;mso-fareast-language:EN-AU"><p></p></span></p>
        </div>
        <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
          <p class="MsoNormal">Hi,</p><p></p>
          <p class="MsoNormal">&nbsp;</p><p></p>
          <p class="MsoNormal">We have a 2 data node plus 1 arbiter node
            replicate gluster volume running gluster 3.8.5. Clients are
            also using 3.8.5.</p><p></p>
          <p class="MsoNormal">&nbsp;</p><p></p>
          <p class="MsoNormal">One of the data nodes failed the other
            night, and whilst it was down, several files were replaced
            on the second data node / arbiter (and thus the filesystem
            path was linked to a new GFID).
            </p><p></p>
          <p class="MsoNormal">&nbsp;</p><p></p>
          <p class="MsoNormal">When the broken node was restarted, these
            files were in a gfid mismatch state. I know how to manually
            correct them, but was wondering if there is an automated way
            ?</p><p></p>
        </blockquote>
        <p class="MsoNormal"><span style="font-size:12.0pt;font-family:&quot;Times New              Roman&quot;,serif;mso-fareast-language:EN-AU">For resolving
            gfid-split-brains, there is no automated way or
            favorite-child policy. When you say 2 data+1 arbiter, you
            are using an actual arbiter volume right? (as opposed to a
            replica 2 volume + a dummy node which some people are
            referring to as arbiter for server-quourm).
            gfid-split-brains should not occur on either replica-3 or
            arbiter volumes with the steps you described.<br />
            Regards,<br />
            Ravi<br />
            <br />
            <br />
            <p></p></span></p>
        <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
          <p class="MsoNormal">&nbsp;</p><p></p>
          <p class="MsoNormal">I thought the
            cluster.favorite-child-policy volume setting of majority
            would work, but it made no difference. Clients were still
            getting Input/output error when attempting to access those
            files.</p><p></p>
          <p class="MsoNormal">&nbsp;</p><p></p>
          <p class="MsoNormal">&nbsp;</p><p></p>
          <p class="MsoNormal">Regards,</p><p></p>
          <p class="MsoNormal">Michael Ward</p><p></p>
          <p class="MsoNormal">&nbsp;</p><p></p>
          <p class="MsoNormal"><span style="font-size:12.0pt;font-family:&quot;Times New                Roman&quot;,serif;mso-fareast-language:EN-AU"><br />
              <br />
              <br />
              <p></p></span></p>
          <pre>_______________________________________________<p></p></pre>
          <pre>Gluster-users mailing list<p></p></pre>
          <pre><a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><p></p></pre>
          <pre><a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a><p></p></pre>
        </blockquote>
        <p></p><p>&nbsp;</p>
      </div>
    </blockquote>
    <p><br />
    </p>
  
</blockquote></div><br>
-- <br>
Sent from my Android device with K-9 Mail. Please excuse my brevity.</body></html>