<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Times New Roman \, serif";
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
color:black;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";
color:black;}
span.EmailStyle24
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1577588982;
mso-list-type:hybrid;
mso-list-template-ids:1092125724 -703844236 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:27.0pt;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:63.0pt;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:99.0pt;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:135.0pt;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:171.0pt;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:207.0pt;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:243.0pt;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:279.0pt;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:315.0pt;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Hi Pranith,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">No worries!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Here is the output of the other brick:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[root@dmf-wpst-2 ~]# getfattr -d -m. -e hex /export/vdb1/brick/<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">getfattr: Removing leading '/' from absolute path names<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"># file: export/vdb1/brick/<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-0=0x000000000000001500000000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-1=0x000000000000000000000000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0x00000000000000000000000000000001<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.glusterfs.dht=0x000000010000000000000000ffffffff<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.glusterfs.volume-id=0x51de44c3f01e486da6b710c7b7a270d7<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[root@dmf-wpst-2 ~]# getfattr -d -m. -e hex /export/vdb1/brick/mpdis/<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">getfattr: Removing leading '/' from absolute path names<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"># file: export/vdb1/brick/mpdis/<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-0=0x000000000000000200000000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-1=0x000000000000000000000000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0x8ff7afeb996244cd9d1bf213568398d7<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.glusterfs.dht=0x000000010000000000000000ffffffff<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[root@dmf-wpst-2 ~]# getfattr -d -m. -e hex /export/vdb1/brick/mpdis/test.rep.00.00.00.00.dmf1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">getfattr: Removing leading '/' from absolute path names<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"># file: export/vdb1/brick/mpdis/test.rep.00.00.00.00.dmf1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-0=0x000000000000000000000000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-1=0x000000000000000000000000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0x85ed306b179b46819d7c02eb336543b8<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-right:.5in"><span style="color:#1F497D">[root@dmf-wpst-2 ~]# getfattr -d -m. -e hex /export/vdb1/brick/mpdis/test.rep.00.00.00.00.dmf2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-right:.5in"><span style="color:#1F497D">getfattr: Removing leading '/' from absolute path names<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-right:.5in"><span style="color:#1F497D"># file: export/vdb1/brick/mpdis/test.rep.00.00.00.00.dmf2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-right:.5in"><span style="color:#1F497D">security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-right:.5in"><span style="color:#1F497D">trusted.afr.gv0-client-0=0x000000000000000000000000<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-right:.5in"><span style="color:#1F497D">trusted.afr.gv0-client-1=0x000000000000000000000000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0xa826a389e7a042c2b5175a1acbecae9b<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"> Pranith Kumar Karampuri [mailto:pkarampu@redhat.com]
<br>
<b>Sent:</b> Friday, January 23, 2015 3:42 PM<br>
<b>To:</b> Arnold Yang; Jifeng Li; Gluster-users@gluster.org<br>
<b>Subject:</b> Re: [Gluster-users] [gluster] possible split-brain issue<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">hi Arnold,<br>
You gave the output only on one brick it seems? Could you also provide it on other brick as well. Sorry I didn't make that clear in my earlier mail.<br>
<br>
Pranith<o:p></o:p></p>
<div>
<p class="MsoNormal">On 01/23/2015 10:44 AM, Arnold Yang wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="color:#1F497D">Hi Pranith,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Here is the output for the commands provide by you, anything more you need, please tell us!</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks!</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">[root@dmf-wpst-1 ~]# getfattr -d -m. -e hex /export/vdb1/brick/</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">getfattr: Removing leading '/' from absolute path names</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"># file: export/vdb1/brick/</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-0=0x000000000000000000000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-1=0x000000000000001400000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0x00000000000000000000000000000001</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.glusterfs.dht=0x000000010000000000000000ffffffff</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.glusterfs.volume-id=0x51de44c3f01e486da6b710c7b7a270d7</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">[root@dmf-wpst-1 ~]# getfattr -d -m. -e hex /export/vdb1/brick/mpdis/</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">getfattr: Removing leading '/' from absolute path names</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"># file: export/vdb1/brick/mpdis/</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-0=0x000000000000000000000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-1=0x000000000000000400000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0x8ff7afeb996244cd9d1bf213568398d7</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.glusterfs.dht=0x000000010000000000000000ffffffff</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">[root@dmf-wpst-1 ~]# getfattr -d -m. -e hex /export/vdb1/brick/mpdis/test.rep.00.00.00.00.dmf1</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">getfattr: Removing leading '/' from absolute path names</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"># file: export/vdb1/brick/mpdis/test.rep.00.00.00.00.dmf1</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-0=0x000000000000000000000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-1=0x000000000000000000000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0x85ed306b179b46819d7c02eb336543b8</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">[root@dmf-wpst-1 ~]# getfattr -d -m. -e hex /export/vdb1/brick/mpdis/test.rep.00.00.00.00.dmf2</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">getfattr: Removing leading '/' from absolute path names</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"># file: export/vdb1/brick/mpdis/test.rep.00.00.00.00.dmf2</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-0=0x000000000000000000000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.afr.gv0-client-1=0x000000000000000000000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">trusted.gfid=0xa826a389e7a042c2b5175a1acbecae9b</span><span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:.5in;margin-bottom:0in;margin-left:.5in;margin-bottom:.0001pt">
<span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"> Pranith Kumar Karampuri [<a href="mailto:pkarampu@redhat.com">mailto:pkarampu@redhat.com</a>]
<br>
<b>Sent:</b> Thursday, January 22, 2015 12:14 AM<br>
<b>To:</b> Jifeng Li; <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>; Arnold Yang<br>
<b>Subject:</b> Re: [Gluster-users] [gluster] possible split-brain issue</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<p class="MsoNormal">On 01/14/2015 04:48 PM, Jifeng Li wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Hi ,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="background:yellow;mso-highlight:yellow">[issue]:</span> <span style="background:yellow;mso-highlight:yellow">To ensure the glusterFS mount point work, a script will periodically using HTTP put a file to subdirectory under
mount point which is used as Apache DocumentRoot. But after running some time, some errors show below:</span><o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:40.915639] E [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] 0-gv0-replicate-0: metadata self heal failed, on /mpdis<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:41.924584] E [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-gv0-replicate-0: Unable to self-heal contents of '/' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending
matrix: [ [ 0 20 ] [ 21 0 ] ]<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:41.925182] E [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] 0-gv0-replicate-0: metadata self heal failed, on /<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:41.934827] E [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-gv0-replicate-0: Unable to self-heal contents of '/mpdis' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending
matrix: [ [ 0 4 ] [ 2 0 ] ]<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:41.935375] E [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] 0-gv0-replicate-0: metadata self heal failed, on /mpdis<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:42.943742] E [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-gv0-replicate-0: Unable to self-heal contents of '/' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending
matrix: [ [ 0 20 ] [ 21 0 ] ]<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:42.944432] E [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] 0-gv0-replicate-0: metadata self heal failed, on /<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:42.946664] E [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-gv0-replicate-0: Unable to self-heal contents of '/mpdis' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending
matrix: [ [ 0 4 ] [ 2 0 ] ]<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:42.947323] E [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] 0-gv0-replicate-0: metadata self heal failed, on /mpdis<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:43.955929] E [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-gv0-replicate-0: Unable to self-heal contents of '/' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending
matrix: [ [ 0 20 ] [ 21 0 ] ]<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:43.956701] E [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] 0-gv0-replicate-0: metadata self heal failed, on /<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:43.958874] E [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-gv0-replicate-0: Unable to self-heal contents of '/mpdis' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending
matrix: [ [ 0 4 ] [ 2 0 ] ]<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Besides, I find Input/output error shown below when listing the files of under mount point:<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoListParagraph">[root@dmf-wpst-2 mpdis]# ll<o:p></o:p></p>
<p class="MsoListParagraph">total 0<o:p></o:p></p>
<p class="MsoListParagraph">-rwxr-xr-x. 1 apache apache 0 Jan 14 04:21 test.rep.00.00.00.00.dmf1<o:p></o:p></p>
<p class="MsoListParagraph">-rw-r--r--. 1 apache apache 0 Jan 14 04:21 test.rep.00.00.00.00.dmf2<o:p></o:p></p>
<p class="MsoListParagraph"> <o:p></o:p></p>
<p class="MsoListParagraph">[root@dmf-wpst-2 mpdis]# ll<o:p></o:p></p>
<p class="MsoListParagraph">total 0<o:p></o:p></p>
<p class="MsoListParagraph">-rwxr-xr-x. 1 apache apache 0 Jan 14 04:21 test.rep.00.00.00.00.dmf1<o:p></o:p></p>
<p class="MsoListParagraph">-rw-r--r--. 1 apache apache 0 Jan 14 04:21 test.rep.00.00.00.00.dmf2<o:p></o:p></p>
<p class="MsoListParagraph"> <o:p></o:p></p>
<p class="MsoListParagraph">[root@dmf-wpst-2 mpdis]# ll<o:p></o:p></p>
<p class="MsoListParagraph">ls: cannot open directory .: Input/output error<o:p></o:p></p>
<p class="MsoListParagraph"> <o:p></o:p></p>
<p class="MsoListParagraph">[root@dmf-wpst-2 mpdis]# ll<o:p></o:p></p>
<p class="MsoListParagraph"><span style="background:red;mso-highlight:red">ls: cannot access test.rep.00.00.00.00.dmf1: Input/output error</span><o:p></o:p></p>
<p class="MsoListParagraph"><span style="background:red;mso-highlight:red">ls: cannot access test.rep.00.00.00.00.dmf2: Input/output error</span><o:p></o:p></p>
<p class="MsoListParagraph">total 0<o:p></o:p></p>
<p class="MsoListParagraph">?????????? ? ? ? ? ? test.rep.00.00.00.00.dmf1<o:p></o:p></p>
<p class="MsoListParagraph">?????????? ? ? ? ? ? test.rep.00.00.00.00.dmf2<o:p></o:p></p>
<p class="MsoNormal"><b><span style="font-size:12.0pt;font-family:"Arial","sans-serif""> Any tips about debugging further or getting this fixed up would be appreciated. </span></b><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="background:yellow;mso-highlight:yellow">[version]: 3.5.3</span><o:p></o:p></p>
<p class="MsoNormal"><span style="background:yellow;mso-highlight:yellow">[environment]:</span> two virtual server each has one brick :<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoListParagraph">root@dmf-wpst-2 mpdis]# gluster volume status<o:p></o:p></p>
<p class="MsoListParagraph">Status of volume: gv0<o:p></o:p></p>
<p class="MsoListParagraph">Gluster process Port Online Pid<o:p></o:p></p>
<p class="MsoListParagraph">------------------------------------------------------------------------------<o:p></o:p></p>
<p class="MsoListParagraph">Brick dmf-ha-1-glusterfs:/export/vdb1/brick 49152 Y 332<o:p></o:p></p>
<p class="MsoListParagraph">Brick dmf-ha-2-glusterfs:/export/vdb1/brick 49154 Y 19396<o:p></o:p></p>
<p class="MsoListParagraph">Self-heal Daemon on localhost N/A Y 19410<o:p></o:p></p>
<p class="MsoListParagraph">Self-heal Daemon on 10.175.123.246 N/A Y 999<o:p></o:p></p>
<p class="MsoListParagraph"> <o:p></o:p></p>
<p class="MsoListParagraph">[root@dmf-wpst-1 mpdis]# gluster volume info<o:p></o:p></p>
<p class="MsoListParagraph">Volume Name: gv0<o:p></o:p></p>
<p class="MsoListParagraph">Type: Replicate<o:p></o:p></p>
<p class="MsoListParagraph">Volume ID: 51de44c3-f01e-486d-a6b7-10c7b7a270d7<o:p></o:p></p>
<p class="MsoListParagraph">Status: Started<o:p></o:p></p>
<p class="MsoListParagraph">Number of Bricks: 1 x 2 = 2<o:p></o:p></p>
<p class="MsoListParagraph">Transport-type: tcp<o:p></o:p></p>
<p class="MsoListParagraph">Bricks:<o:p></o:p></p>
<p class="MsoListParagraph">Brick1: dmf-ha-1-glusterfs:/export/vdb1/brick<o:p></o:p></p>
<p class="MsoListParagraph">Brick2: dmf-ha-2-glusterfs:/export/vdb1/brick<o:p></o:p></p>
<p class="MsoListParagraph">Options Reconfigured:<o:p></o:p></p>
<p class="MsoListParagraph">nfs.disable: ON<o:p></o:p></p>
<p class="MsoListParagraph">network.ping-timeout: 2<o:p></o:p></p>
<p class="MsoListParagraph">storage.bd-aio: on<o:p></o:p></p>
<p class="MsoListParagraph">storage.linux-aio: on<o:p></o:p></p>
<p class="MsoListParagraph">cluster.eager-lock: on<o:p></o:p></p>
<p class="MsoListParagraph">performance.client-io-threads: on<o:p></o:p></p>
<p class="MsoListParagraph">performance.cache-refresh-timeout: 60<o:p></o:p></p>
<p class="MsoListParagraph">performance.io-thread-count: 64<o:p></o:p></p>
<p class="MsoListParagraph">performance.cache-size: 8GB<o:p></o:p></p>
<p class="MsoListParagraph">cluster.server-quorum-type: none<o:p></o:p></p>
<p class="MsoListParagraph"> <o:p></o:p></p>
<p class="MsoNormal"> [mount-point info]:<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:27.0pt;text-indent:-.25in;mso-list:l0 level1 lfo2">
<![if !supportLists]><span style="mso-list:Ignore">1.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>mount command<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:27.0pt">glusterfs -p /var/run/glusterfs.pid --volfile-server=dmf-ha-1-glusterfs --volfile-server=dmf-ha-2-glusterfs --volfile-id=gv0 /dmfcontents<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:27.0pt"> <o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:27.0pt"> <o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:27.0pt;text-indent:-.25in;mso-list:l0 level1 lfo2">
<![if !supportLists]><span style="mso-list:Ignore">2.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>mount point directory hierarchy <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">[root@dmf-wpst-2 /]# ls -ld /dmfcontents/<o:p></o:p></p>
<p class="MsoNormal" style="text-indent:.5in">drwxr-xr-x. 5 root root 71 Jan 14 04:39 /dmfcontents/<o:p></o:p></p>
<p class="MsoNormal"> [root@dmf-wpst-2 /]# ls -ld /dmfcontents/mpdis/<o:p></o:p></p>
<p class="MsoNormal"> drwxr-xr-x. 2 apache apache 89 Jan 14 04:39 /dmfcontents/mpdis/<o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif"">hi Jifeng Li,<br>
Sorry for the delay in response. Could you post the output of:<br>
'getfattr -d -m. -e hex <brick-path>'<br>
'getfattr -d -m. -e hex <brick-path>/mpdis'<br>
'getfattr -d -m. -e hex <brick-path>/mpdis/<span style="background:red;mso-highlight:red">test.rep.00.00.00.00.dmf1</span>'<br>
'getfattr -d -m. -e hex <brick-path>/mpdis/<span style="background:red;mso-highlight:red">test.rep.00.00.00.00.dmf2</span>'<br>
<br>
Pranith<br>
<br>
<br>
</span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif""><br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Gluster-users mailing list<o:p></o:p></pre>
<pre><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><o:p></o:p></pre>
<pre><a href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></pre>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif""> </span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><o:p> </o:p></span></p>
</div>
</body>
</html>