<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Arnold Yang,<br>
I see that the directories /export/vdb1/brick/,
/export/vdb1/brick/mpdis/ are in metadata split-brain. You can
follow the document:
<a class="moz-txt-link-freetext" href="https://github.com/gluster/glusterfs/blob/master/doc/split-brain.md">https://github.com/gluster/glusterfs/blob/master/doc/split-brain.md</a>
to fix the split-brain.<br>
<br>
export/vdb1/brick/mpdis/test.rep.00.00.00.00.dmf1, <span
style="color:#1F497D"><font color="#000000">export/vdb1/brick/mpdis/test.rep.00.00.00.00.dmf2
don't seem to be in split-brain as per the extended attributes,
could you send the stat of these two files on both the bricks?</font></span><br>
<br>
Pranith<br>
<div class="moz-cite-prefix">On 01/23/2015 02:18 PM, Arnold Yang
wrote:<br>
</div>
<blockquote
cite="mid:2B7FF152B8292E4394BBAC6105471877413ACCE4@ESGSCMB109.ericsson.se"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<meta name="Generator" content="Microsoft Word 14 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Times New Roman \, serif";
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
color:black;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";
color:black;}
span.EmailStyle24
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1577588982;
mso-list-type:hybrid;
mso-list-template-ids:1092125724 -703844236 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:27.0pt;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:63.0pt;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:99.0pt;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:135.0pt;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:171.0pt;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:207.0pt;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:243.0pt;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:279.0pt;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:315.0pt;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Hi Pranith,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">No worries!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Here is the
output of the other brick:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[root@dmf-wpst-2
~]# getfattr -d -m. -e hex /export/vdb1/brick/<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">getfattr:
Removing leading '/' from absolute path names<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"># file:
export/vdb1/brick/<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-0=0x000000000000001500000000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-1=0x000000000000000000000000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0x00000000000000000000000000000001<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.glusterfs.dht=0x000000010000000000000000ffffffff<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.glusterfs.volume-id=0x51de44c3f01e486da6b710c7b7a270d7<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[root@dmf-wpst-2
~]# getfattr -d -m. -e hex /export/vdb1/brick/mpdis/<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">getfattr:
Removing leading '/' from absolute path names<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"># file:
export/vdb1/brick/mpdis/<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-0=0x000000000000000200000000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-1=0x000000000000000000000000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0x8ff7afeb996244cd9d1bf213568398d7<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.glusterfs.dht=0x000000010000000000000000ffffffff<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[root@dmf-wpst-2
~]# getfattr -d -m. -e hex
/export/vdb1/brick/mpdis/test.rep.00.00.00.00.dmf1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">getfattr:
Removing leading '/' from absolute path names<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"># file:
export/vdb1/brick/mpdis/test.rep.00.00.00.00.dmf1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-0=0x000000000000000000000000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-1=0x000000000000000000000000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0x85ed306b179b46819d7c02eb336543b8<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-right:.5in"><span
style="color:#1F497D">[root@dmf-wpst-2 ~]# getfattr -d -m.
-e hex /export/vdb1/brick/mpdis/test.rep.00.00.00.00.dmf2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-right:.5in"><span
style="color:#1F497D">getfattr: Removing leading '/' from
absolute path names<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-right:.5in"><span
style="color:#1F497D"># file:
export/vdb1/brick/mpdis/test.rep.00.00.00.00.dmf2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-right:.5in"><span
style="color:#1F497D">security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-right:.5in"><span
style="color:#1F497D">trusted.afr.gv0-client-0=0x000000000000000000000000<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-right:.5in"><span
style="color:#1F497D">trusted.afr.gv0-client-1=0x000000000000000000000000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0xa826a389e7a042c2b5175a1acbecae9b<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">
Pranith Kumar Karampuri [<a class="moz-txt-link-freetext" href="mailto:pkarampu@redhat.com">mailto:pkarampu@redhat.com</a>]
<br>
<b>Sent:</b> Friday, January 23, 2015 3:42 PM<br>
<b>To:</b> Arnold Yang; Jifeng Li;
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] [gluster] possible
split-brain issue<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">hi Arnold,<br>
You gave the output only on one brick it seems? Could you
also provide it on other brick as well. Sorry I didn't make
that clear in my earlier mail.<br>
<br>
Pranith<o:p></o:p></p>
<div>
<p class="MsoNormal">On 01/23/2015 10:44 AM, Arnold Yang
wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="color:#1F497D">Hi Pranith,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Here is the
output for the commands provide by you, anything more you
need, please tell us!</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks!</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">[root@dmf-wpst-1
~]# getfattr -d -m. -e hex /export/vdb1/brick/</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">getfattr:
Removing leading '/' from absolute path names</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"># file:
export/vdb1/brick/</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-0=0x000000000000000000000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-1=0x000000000000001400000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0x00000000000000000000000000000001</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.glusterfs.dht=0x000000010000000000000000ffffffff</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.glusterfs.volume-id=0x51de44c3f01e486da6b710c7b7a270d7</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">[root@dmf-wpst-1
~]# getfattr -d -m. -e hex /export/vdb1/brick/mpdis/</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">getfattr:
Removing leading '/' from absolute path names</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"># file:
export/vdb1/brick/mpdis/</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-0=0x000000000000000000000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-1=0x000000000000000400000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0x8ff7afeb996244cd9d1bf213568398d7</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.glusterfs.dht=0x000000010000000000000000ffffffff</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">[root@dmf-wpst-1
~]# getfattr -d -m. -e hex
/export/vdb1/brick/mpdis/test.rep.00.00.00.00.dmf1</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">getfattr:
Removing leading '/' from absolute path names</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"># file:
export/vdb1/brick/mpdis/test.rep.00.00.00.00.dmf1</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-0=0x000000000000000000000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-1=0x000000000000000000000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0x85ed306b179b46819d7c02eb336543b8</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">[root@dmf-wpst-1
~]# getfattr -d -m. -e hex
/export/vdb1/brick/mpdis/test.rep.00.00.00.00.dmf2</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">getfattr:
Removing leading '/' from absolute path names</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"># file:
export/vdb1/brick/mpdis/test.rep.00.00.00.00.dmf2</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-0=0x000000000000000000000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-1=0x000000000000000000000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0xa826a389e7a042c2b5175a1acbecae9b</span><span
style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:0in;margin-right:.5in;margin-bottom:0in;margin-left:.5in;margin-bottom:.0001pt"><span
style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">
Pranith Kumar Karampuri [<a moz-do-not-send="true"
href="mailto:pkarampu@redhat.com">mailto:pkarampu@redhat.com</a>]
<br>
<b>Sent:</b> Thursday, January 22, 2015 12:14 AM<br>
<b>To:</b> Jifeng Li; <a moz-do-not-send="true"
href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>;
Arnold Yang<br>
<b>Subject:</b> Re: [Gluster-users] [gluster] possible
split-brain issue</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<p class="MsoNormal">On 01/14/2015 04:48 PM, Jifeng Li
wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Hi ,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span
style="background:yellow;mso-highlight:yellow">[issue]:</span>
<span style="background:yellow;mso-highlight:yellow">To
ensure the glusterFS mount point work, a script will
periodically using HTTP put a file to subdirectory under
mount point which is used as Apache DocumentRoot. But
after running some time, some errors show below:</span><o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:40.915639] E
[afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
0-gv0-replicate-0: metadata self heal failed, on
/mpdis<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:41.924584] E
[afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
0-gv0-replicate-0: Unable to self-heal contents of '/'
(possible split-brain). Please delete the file from all
but the preferred subvolume.- Pending matrix: [ [ 0 20 ]
[ 21 0 ] ]<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:41.925182] E
[afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
0-gv0-replicate-0: metadata self heal failed, on /<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:41.934827] E
[afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
0-gv0-replicate-0: Unable to self-heal contents of
'/mpdis' (possible split-brain). Please delete the file
from all but the preferred subvolume.- Pending matrix: [
[ 0 4 ] [ 2 0 ] ]<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:41.935375] E
[afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
0-gv0-replicate-0: metadata self heal failed, on
/mpdis<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:42.943742] E
[afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
0-gv0-replicate-0: Unable to self-heal contents of '/'
(possible split-brain). Please delete the file from all
but the preferred subvolume.- Pending matrix: [ [ 0 20 ]
[ 21 0 ] ]<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:42.944432] E
[afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
0-gv0-replicate-0: metadata self heal failed, on /<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:42.946664] E
[afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
0-gv0-replicate-0: Unable to self-heal contents of
'/mpdis' (possible split-brain). Please delete the file
from all but the preferred subvolume.- Pending matrix: [
[ 0 4 ] [ 2 0 ] ]<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:42.947323] E
[afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
0-gv0-replicate-0: metadata self heal failed, on
/mpdis<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:43.955929] E
[afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
0-gv0-replicate-0: Unable to self-heal contents of '/'
(possible split-brain). Please delete the file from all
but the preferred subvolume.- Pending matrix: [ [ 0 20 ]
[ 21 0 ] ]<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:43.956701] E
[afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
0-gv0-replicate-0: metadata self heal failed, on /<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:43.958874] E
[afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
0-gv0-replicate-0: Unable to self-heal contents of
'/mpdis' (possible split-brain). Please delete the file
from all but the preferred subvolume.- Pending matrix: [
[ 0 4 ] [ 2 0 ] ]<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Besides, I find Input/output error
shown below when listing the files of under mount point:<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoListParagraph">[root@dmf-wpst-2 mpdis]# ll<o:p></o:p></p>
<p class="MsoListParagraph">total 0<o:p></o:p></p>
<p class="MsoListParagraph">-rwxr-xr-x. 1 apache apache 0
Jan 14 04:21 test.rep.00.00.00.00.dmf1<o:p></o:p></p>
<p class="MsoListParagraph">-rw-r--r--. 1 apache apache 0
Jan 14 04:21 test.rep.00.00.00.00.dmf2<o:p></o:p></p>
<p class="MsoListParagraph"> <o:p></o:p></p>
<p class="MsoListParagraph">[root@dmf-wpst-2 mpdis]# ll<o:p></o:p></p>
<p class="MsoListParagraph">total 0<o:p></o:p></p>
<p class="MsoListParagraph">-rwxr-xr-x. 1 apache apache 0
Jan 14 04:21 test.rep.00.00.00.00.dmf1<o:p></o:p></p>
<p class="MsoListParagraph">-rw-r--r--. 1 apache apache 0
Jan 14 04:21 test.rep.00.00.00.00.dmf2<o:p></o:p></p>
<p class="MsoListParagraph"> <o:p></o:p></p>
<p class="MsoListParagraph">[root@dmf-wpst-2 mpdis]# ll<o:p></o:p></p>
<p class="MsoListParagraph">ls: cannot open directory .:
Input/output error<o:p></o:p></p>
<p class="MsoListParagraph"> <o:p></o:p></p>
<p class="MsoListParagraph">[root@dmf-wpst-2 mpdis]# ll<o:p></o:p></p>
<p class="MsoListParagraph"><span
style="background:red;mso-highlight:red">ls: cannot
access test.rep.00.00.00.00.dmf1: Input/output error</span><o:p></o:p></p>
<p class="MsoListParagraph"><span
style="background:red;mso-highlight:red">ls: cannot
access test.rep.00.00.00.00.dmf2: Input/output error</span><o:p></o:p></p>
<p class="MsoListParagraph">total 0<o:p></o:p></p>
<p class="MsoListParagraph">?????????? ? ? ? ? ?
test.rep.00.00.00.00.dmf1<o:p></o:p></p>
<p class="MsoListParagraph">?????????? ? ? ? ? ?
test.rep.00.00.00.00.dmf2<o:p></o:p></p>
<p class="MsoNormal"><b><span
style="font-size:12.0pt;font-family:"Arial","sans-serif""> Any
tips about debugging further or getting this fixed up
would be appreciated. </span></b><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span
style="background:yellow;mso-highlight:yellow">[version]:
3.5.3</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="background:yellow;mso-highlight:yellow">[environment]:</span>
two virtual server each has one brick :<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoListParagraph">root@dmf-wpst-2 mpdis]# gluster
volume status<o:p></o:p></p>
<p class="MsoListParagraph">Status of volume: gv0<o:p></o:p></p>
<p class="MsoListParagraph">Gluster
process
Port Online Pid<o:p></o:p></p>
<p class="MsoListParagraph">------------------------------------------------------------------------------<o:p></o:p></p>
<p class="MsoListParagraph">Brick
dmf-ha-1-glusterfs:/export/vdb1/brick
49152 Y 332<o:p></o:p></p>
<p class="MsoListParagraph">Brick
dmf-ha-2-glusterfs:/export/vdb1/brick
49154 Y 19396<o:p></o:p></p>
<p class="MsoListParagraph">Self-heal Daemon on
localhost
N/A Y 19410<o:p></o:p></p>
<p class="MsoListParagraph">Self-heal Daemon on
10.175.123.246
N/A Y 999<o:p></o:p></p>
<p class="MsoListParagraph"> <o:p></o:p></p>
<p class="MsoListParagraph">[root@dmf-wpst-1 mpdis]# gluster
volume info<o:p></o:p></p>
<p class="MsoListParagraph">Volume Name: gv0<o:p></o:p></p>
<p class="MsoListParagraph">Type: Replicate<o:p></o:p></p>
<p class="MsoListParagraph">Volume ID:
51de44c3-f01e-486d-a6b7-10c7b7a270d7<o:p></o:p></p>
<p class="MsoListParagraph">Status: Started<o:p></o:p></p>
<p class="MsoListParagraph">Number of Bricks: 1 x 2 = 2<o:p></o:p></p>
<p class="MsoListParagraph">Transport-type: tcp<o:p></o:p></p>
<p class="MsoListParagraph">Bricks:<o:p></o:p></p>
<p class="MsoListParagraph">Brick1:
dmf-ha-1-glusterfs:/export/vdb1/brick<o:p></o:p></p>
<p class="MsoListParagraph">Brick2:
dmf-ha-2-glusterfs:/export/vdb1/brick<o:p></o:p></p>
<p class="MsoListParagraph">Options Reconfigured:<o:p></o:p></p>
<p class="MsoListParagraph">nfs.disable: ON<o:p></o:p></p>
<p class="MsoListParagraph">network.ping-timeout: 2<o:p></o:p></p>
<p class="MsoListParagraph">storage.bd-aio: on<o:p></o:p></p>
<p class="MsoListParagraph">storage.linux-aio: on<o:p></o:p></p>
<p class="MsoListParagraph">cluster.eager-lock: on<o:p></o:p></p>
<p class="MsoListParagraph">performance.client-io-threads:
on<o:p></o:p></p>
<p class="MsoListParagraph">performance.cache-refresh-timeout:
60<o:p></o:p></p>
<p class="MsoListParagraph">performance.io-thread-count: 64<o:p></o:p></p>
<p class="MsoListParagraph">performance.cache-size: 8GB<o:p></o:p></p>
<p class="MsoListParagraph">cluster.server-quorum-type: none<o:p></o:p></p>
<p class="MsoListParagraph"> <o:p></o:p></p>
<p class="MsoNormal"> [mount-point info]:<o:p></o:p></p>
<p class="MsoListParagraph"
style="margin-left:27.0pt;text-indent:-.25in;mso-list:l0
level1 lfo2">
<!--[if !supportLists]--><span style="mso-list:Ignore">1.<span
style="font:7.0pt "Times New Roman"">
</span></span><!--[endif]-->mount command<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:27.0pt">glusterfs
-p /var/run/glusterfs.pid
--volfile-server=dmf-ha-1-glusterfs
--volfile-server=dmf-ha-2-glusterfs --volfile-id=gv0
/dmfcontents<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:27.0pt"> <o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:27.0pt"> <o:p></o:p></p>
<p class="MsoListParagraph"
style="margin-left:27.0pt;text-indent:-.25in;mso-list:l0
level1 lfo2">
<!--[if !supportLists]--><span style="mso-list:Ignore">2.<span
style="font:7.0pt "Times New Roman"">
</span></span><!--[endif]-->mount point directory
hierarchy <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">[root@dmf-wpst-2
/]# ls -ld /dmfcontents/<o:p></o:p></p>
<p class="MsoNormal" style="text-indent:.5in">drwxr-xr-x. 5
root root 71 Jan 14 04:39 /dmfcontents/<o:p></o:p></p>
<p class="MsoNormal"> [root@dmf-wpst-2 /]# ls
-ld /dmfcontents/mpdis/<o:p></o:p></p>
<p class="MsoNormal"> drwxr-xr-x. 2 apache
apache 89 Jan 14 04:39 /dmfcontents/mpdis/<o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New Roman
, serif","serif"">hi Jifeng Li,<br>
Sorry for the delay in response. Could you post the
output of:<br>
'getfattr -d -m. -e hex <brick-path>'<br>
'getfattr -d -m. -e hex <brick-path>/mpdis'<br>
'getfattr -d -m. -e hex <brick-path>/mpdis/<span
style="background:red;mso-highlight:red">test.rep.00.00.00.00.dmf1</span>'<br>
'getfattr -d -m. -e hex <brick-path>/mpdis/<span
style="background:red;mso-highlight:red">test.rep.00.00.00.00.dmf2</span>'<br>
<br>
Pranith<br>
<br>
<br>
</span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New Roman
, serif","serif""><br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Gluster-users mailing list<o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></pre>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New Roman
, serif","serif""> </span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman","serif""><o:p> </o:p></span></p>
</div>
</blockquote>
<br>
</body>
</html>