<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;
mso-fareast-language:EN-US;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="EN-AU" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Sorry, one thing I meant to mention and forgot.. just in case it is relevant at all, this was initially a 2 node gluster volume, and the arbiter was added later.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Michael Ward.<o:p></o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="color:#1F497D"><o:p> </o:p></span></a></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="color:windowtext;mso-fareast-language:EN-AU">From:</span></b><span lang="EN-US" style="color:windowtext;mso-fareast-language:EN-AU"> Ravishankar N [mailto:ravishankar@redhat.com]
<br>
<b>Sent:</b> Wednesday, 4 January 2017 4:22 PM<br>
<b>To:</b> Michael Ward <Michael.Ward@melbourneit.com.au>; gluster-users@gluster.org<br>
<b>Subject:</b> Re: [Gluster-users] GFID Mismatch - Automatic Correction ?<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><br>
On 01/04/2017 09:31 AM, Michael Ward wrote:<span style="font-size:12.0pt;mso-fareast-language:EN-AU"><o:p></o:p></span></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="color:#1F497D">Hey,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">To give some more context around the initial incident.. These systems are hosted in AWS. The gluster brick for each instance is a seperate volume to the root volume. On prod-gluster01 a couple of nights ago we
experienced massively high read iops on the root volume that we are unable to account for (> 200,000 iops when it usually sits between 0 - 100 iops ). The box became inaccessible as a result and after approximately 40 minutes with no sign of the iops reducing
was rebooted through the AWS console. </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">The GFID mismatch problems appeared after that. There were initially ~50 impacted files, but I've fixed all but 1 of them now, which I'm leaving broken intentionally for further testing if required.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">If you don't mind, could you have a look over the information below and identify anything that looks like a problem, since obviously we did have a bunch of GFID mismatched files, which based on your email shouldn't
happen..</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">I've included everything I can think of, but if there is something else you would like to see, please let me know.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"># gluster volume info gv0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Volume Name: gv0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Type: Replicate</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Volume ID: 0ec7c49d-811c-4d4d-a3a9-e4ea9e83000c</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Status: Started</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Snapshot Count: 0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Number of Bricks: 1 x (2 + 1) = 3</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Transport-type: tcp</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Bricks:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Brick1: prod-gluster01.fqdn.com:/export/glus_brick0/brick</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Brick2: prod-gluster02.fqdn.com:/export/glus_brick0/brick</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Brick3: prod-gluster03.fqdn.com:/export/glus_brick0/brick (arbiter)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Options Reconfigured:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">cluster.favorite-child-policy: none</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">nfs.disable: on</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">performance.readdir-ahead: on</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">client.event-threads: 7</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">server.event-threads: 3</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">performance.cache-size: 256MB</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">cluster.favorite-child-policy is set to none because I reverted the change to majority when it didn't make any difference.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">[root@prod-gluster01 glusterfs]# getfattr -d -m . -e hex /export/glus_brick0/brick/home/user/.viminfo</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">getfattr: Removing leading '/' from absolute path names</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"># file: export/glus_brick0/brick/home/user/.viminfo</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.dirty=0x000000000000000000000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.bit-rot.version=0x0200000000000000585756be00024333</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0x1b86a5a76e884f40be583fa33aa9a576</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">[root@prod-gluster02 glusterfs]# getfattr -d -m . -e hex /export/glus_brick0/brick/home/user/.viminfo</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">getfattr: Removing leading '/' from absolute path names</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"># file: export/glus_brick0/brick/home/user/.viminfo</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.dirty=0x000000000000000000000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-0=0x000000020000000100000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.bit-rot.version=0x020000000000000058593aac000661fa</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0x4931b10977f34496a7cdf8f23809c372</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">[root@prod-gluster03 glusterfs]# getfattr -d -m . -e hex /export/glus_brick0/brick/home/user/.viminfo</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">getfattr: Removing leading '/' from absolute path names</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"># file: export/glus_brick0/brick/home/user/.viminfo</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.dirty=0x000000000000000000000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-0=0x000000020000000100000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.bit-rot.version=0x020000000000000058585ed6000f2077</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0x4931b10977f34496a7cdf8f23809c372</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Just in case it's useful, here is the getfattr for the parent directory:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">[root@prod-gluster01 glusterfs]# getfattr -d -m . -e hex /export/glus_brick0/brick/home/user</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">getfattr: Removing leading '/' from absolute path names</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"># file: export/glus_brick0/brick/home/user</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.dirty=0x000000000000000000000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-1=0x000000000000000000000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-2=0x000000000000000000000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0x0a49de7ee4f04aae9fc8a88378e0d193</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.glusterfs.dht=0x000000010000000000000000ffffffff</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">[root@prod-gluster02 glusterfs]# getfattr -d -m . -e hex /export/glus_brick0/brick/home/user</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">getfattr: Removing leading '/' from absolute path names</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"># file: export/glus_brick0/brick/home/user</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.dirty=0x000000000000000000000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-0=0x000000000000000000000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-2=0x000000000000000000000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0x0a49de7ee4f04aae9fc8a88378e0d193</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.glusterfs.dht=0x000000010000000000000000ffffffff</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">[root@prod-gluster03 glusterfs]# getfattr -d -m . -e hex /export/glus_brick0/brick/home/user</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">getfattr: Removing leading '/' from absolute path names</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"># file: export/glus_brick0/brick/home/user</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.dirty=0x000000000000000000000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-0=0x000000000000000000000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-1=0x000000000000000000000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-2=0x000000000000000000000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0x0a49de7ee4f04aae9fc8a88378e0d193</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.glusterfs.dht=0x000000010000000000000000ffffffff</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">[root@prod-gluster01 bricks]# gluster volume heal gv0 info</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Brick prod-gluster01.fqdn.com:/export/glus_brick0/brick</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Status: Connected</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Number of entries: 0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Brick prod-gluster02.fqdn.com:/export/glus_brick0/brick</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"><gfid:4931b109-77f3-4496-a7cd-f8f23809c372></span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Status: Connected</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Number of entries: 1</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Brick prod-gluster03.fqdn.com:/export/glus_brick0/brick</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"><gfid:4931b109-77f3-4496-a7cd-f8f23809c372></span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Status: Connected</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Number of entries: 1</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">[root@prod-gluster01 bricks]# gluster volume heal gv0 info split-brain</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Brick prod-gluster01.fqdn.com:/export/glus_brick0/brick</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Status: Connected</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Number of entries in split-brain: 0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Brick prod-gluster02.fqdn.com:/export/glus_brick0/brick</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Status: Connected</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Number of entries in split-brain: 0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Brick prod-gluster03.fqdn.com:/export/glus_brick0/brick</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Status: Connected</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Number of entries in split-brain: 0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Clients show this in the gluster.log:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">[2017-01-04 03:13:40.863695] W [MSGID: 108008] [afr-self-heal-name.c:354:afr_selfheal_name_gfid_mismatch_check] 0-gv0-replicate-0: GFID mismatch for <gfid:0a49de7e-e4f0-4aae-9fc8-a88378e0d193>/.viminfo 4931b109-77f3-4496-a7cd-f8f23809c372
on gv0-client-1 and 1b86a5a7-6e88-4f40-be58-3fa33aa9a576 on gv0-client-0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">[2017-01-04 03:13:40.867853] W [fuse-bridge.c:471:fuse_entry_cbk] 0-glusterfs-fuse: 13067223: LOOKUP() /home/user/.viminfo => -1 (Input/output error)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">There's no mention of either of the GFID's for the .viminfo file in /var/log/gluster/*.log or /var/log/gluster/brick/export-glus_brick0-brick.log file.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif;mso-fareast-language:EN-AU"><br>
Thanks for the details Michael. While it does look like a bug, I am not sure how we ended in this state. Either the afr xattrs of the parent directory were cleared without self-heal of .vimrc happening from
</span><span style="font-size:12.0pt;font-family:"Times New Roman",serif;color:#1F497D;mso-fareast-language:EN-AU">gluster02 or 03 to 01</span><span style="font-size:12.0pt;font-family:"Times New Roman",serif;mso-fareast-language:EN-AU"> (or) it wasn't set
in the first place when the file was recreated on 02 and 03 when 01 was down. If you have some steps to re-create the issue, please raise a bug.<br>
<br>
Regards,<br>
Ravi<br>
<br>
<br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="color:#1F497D">Thank you very much for your time,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Michael Ward</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="color:windowtext;mso-fareast-language:EN-AU">From:</span></b><span lang="EN-US" style="color:windowtext;mso-fareast-language:EN-AU"> Ravishankar N [<a href="mailto:ravishankar@redhat.com">mailto:ravishankar@redhat.com</a>]
<br>
<b>Sent:</b> Wednesday, 4 January 2017 12:21 PM<br>
<b>To:</b> Michael Ward <a href="mailto:Michael.Ward@melbourneit.com.au"><Michael.Ward@melbourneit.com.au></a>;
<a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] GFID Mismatch - Automatic Correction ?</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<p class="MsoNormal">On 01/04/2017 06:27 AM, Michael Ward wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">We have a 2 data node plus 1 arbiter node replicate gluster volume running gluster 3.8.5. Clients are also using 3.8.5.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">One of the data nodes failed the other night, and whilst it was down, several files were replaced on the second data node / arbiter (and thus the filesystem path was linked to a new GFID).
<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">When the broken node was restarted, these files were in a gfid mismatch state. I know how to manually correct them, but was wondering if there is an automated way ?<o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:12.0pt">For resolving gfid-split-brains, there is no automated way or favorite-child policy. When you say 2 data+1 arbiter, you are using an actual arbiter volume right? (as opposed to a replica 2 volume + a dummy
node which some people are referring to as arbiter for server-quourm). gfid-split-brains should not occur on either replica-3 or arbiter volumes with the steps you described.<br>
Regards,<br>
Ravi<br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I thought the cluster.favorite-child-policy volume setting of majority would work, but it made no difference. Clients were still getting Input/output error when attempting to access those files.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Regards,<o:p></o:p></p>
<p class="MsoNormal">Michael Ward<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:12.0pt"><br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Gluster-users mailing list<o:p></o:p></pre>
<pre><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><o:p></o:p></pre>
<pre><a href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></pre>
</blockquote>
<p> <o:p></o:p></p>
</blockquote>
<p><o:p> </o:p></p>
</div>
</body>
</html>