<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1577588982;
mso-list-type:hybrid;
mso-list-template-ids:1092125724 -703844236 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:27.0pt;
text-indent:-18.0pt;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:63.0pt;
text-indent:-18.0pt;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:99.0pt;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:135.0pt;
text-indent:-18.0pt;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:171.0pt;
text-indent:-18.0pt;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:207.0pt;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:243.0pt;
text-indent:-18.0pt;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:279.0pt;
text-indent:-18.0pt;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:315.0pt;
text-indent:-9.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi ,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="background:yellow;mso-highlight:yellow">[issue]:</span> <span style="background:yellow;mso-highlight:yellow">To ensure the glusterFS mount point work, a script will periodically using HTTP put a file to subdirectory under
mount point which is used as Apache DocumentRoot. But after running some time, some errors show below:</span><o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:40.915639] E [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] 0-gv0-replicate-0: metadata self heal failed, on /mpdis<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:41.924584] E [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-gv0-replicate-0: Unable to self-heal contents of '/' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending
matrix: [ [ 0 20 ] [ 21 0 ] ]<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:41.925182] E [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] 0-gv0-replicate-0: metadata self heal failed, on /<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:41.934827] E [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-gv0-replicate-0: Unable to self-heal contents of '/mpdis' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending
matrix: [ [ 0 4 ] [ 2 0 ] ]<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:41.935375] E [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] 0-gv0-replicate-0: metadata self heal failed, on /mpdis<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:42.943742] E [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-gv0-replicate-0: Unable to self-heal contents of '/' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending
matrix: [ [ 0 20 ] [ 21 0 ] ]<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:42.944432] E [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] 0-gv0-replicate-0: metadata self heal failed, on /<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:42.946664] E [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-gv0-replicate-0: Unable to self-heal contents of '/mpdis' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending
matrix: [ [ 0 4 ] [ 2 0 ] ]<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:42.947323] E [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] 0-gv0-replicate-0: metadata self heal failed, on /mpdis<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:43.955929] E [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-gv0-replicate-0: Unable to self-heal contents of '/' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending
matrix: [ [ 0 20 ] [ 21 0 ] ]<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:43.956701] E [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] 0-gv0-replicate-0: metadata self heal failed, on /<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:43.958874] E [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-gv0-replicate-0: Unable to self-heal contents of '/mpdis' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending
matrix: [ [ 0 4 ] [ 2 0 ] ]<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Besides, I find Input/output error shown below when listing the files of under mount point:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoListParagraph">[root@dmf-wpst-2 mpdis]# ll<o:p></o:p></p>
<p class="MsoListParagraph">total 0<o:p></o:p></p>
<p class="MsoListParagraph">-rwxr-xr-x. 1 apache apache 0 Jan 14 04:21 test.rep.00.00.00.00.dmf1<o:p></o:p></p>
<p class="MsoListParagraph">-rw-r--r--. 1 apache apache 0 Jan 14 04:21 test.rep.00.00.00.00.dmf2<o:p></o:p></p>
<p class="MsoListParagraph"><o:p> </o:p></p>
<p class="MsoListParagraph">[root@dmf-wpst-2 mpdis]# ll<o:p></o:p></p>
<p class="MsoListParagraph">total 0<o:p></o:p></p>
<p class="MsoListParagraph">-rwxr-xr-x. 1 apache apache 0 Jan 14 04:21 test.rep.00.00.00.00.dmf1<o:p></o:p></p>
<p class="MsoListParagraph">-rw-r--r--. 1 apache apache 0 Jan 14 04:21 test.rep.00.00.00.00.dmf2<o:p></o:p></p>
<p class="MsoListParagraph"><o:p> </o:p></p>
<p class="MsoListParagraph">[root@dmf-wpst-2 mpdis]# ll<o:p></o:p></p>
<p class="MsoListParagraph">ls: cannot open directory .: Input/output error<o:p></o:p></p>
<p class="MsoListParagraph"><o:p> </o:p></p>
<p class="MsoListParagraph">[root@dmf-wpst-2 mpdis]# ll<o:p></o:p></p>
<p class="MsoListParagraph"><span style="background:red;mso-highlight:red">ls: cannot access test.rep.00.00.00.00.dmf1: Input/output error<o:p></o:p></span></p>
<p class="MsoListParagraph"><span style="background:red;mso-highlight:red">ls: cannot access test.rep.00.00.00.00.dmf2: Input/output error</span><o:p></o:p></p>
<p class="MsoListParagraph">total 0<o:p></o:p></p>
<p class="MsoListParagraph">?????????? ? ? ? ? ? test.rep.00.00.00.00.dmf1<o:p></o:p></p>
<p class="MsoListParagraph">?????????? ? ? ? ? ? test.rep.00.00.00.00.dmf2<o:p></o:p></p>
<p class="MsoNormal"><o:p></o:p></p>
<p class="MsoNormal"><b><span style="font-size:12.0pt;font-family:"Arial","sans-serif";color:black"> Any tips about debugging further or getting this fixed up would be appreciated. <o:p></o:p></span></b></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="background:yellow;mso-highlight:yellow">[version]: 3.5.3</span><o:p></o:p></p>
<p class="MsoNormal"><span style="background:yellow;mso-highlight:yellow">[environment]:</span> two virtual server each has one brick :<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoListParagraph">root@dmf-wpst-2 mpdis]# gluster volume status<o:p></o:p></p>
<p class="MsoListParagraph">Status of volume: gv0<o:p></o:p></p>
<p class="MsoListParagraph">Gluster process Port Online Pid<o:p></o:p></p>
<p class="MsoListParagraph">------------------------------------------------------------------------------<o:p></o:p></p>
<p class="MsoListParagraph">Brick dmf-ha-1-glusterfs:/export/vdb1/brick 49152 Y 332<o:p></o:p></p>
<p class="MsoListParagraph">Brick dmf-ha-2-glusterfs:/export/vdb1/brick 49154 Y 19396<o:p></o:p></p>
<p class="MsoListParagraph">Self-heal Daemon on localhost N/A Y 19410<o:p></o:p></p>
<p class="MsoListParagraph">Self-heal Daemon on 10.175.123.246 N/A Y 999<o:p></o:p></p>
<p class="MsoListParagraph"><o:p> </o:p></p>
<p class="MsoListParagraph">[root@dmf-wpst-1 mpdis]# gluster volume info<o:p></o:p></p>
<p class="MsoListParagraph">Volume Name: gv0<o:p></o:p></p>
<p class="MsoListParagraph">Type: Replicate<o:p></o:p></p>
<p class="MsoListParagraph">Volume ID: 51de44c3-f01e-486d-a6b7-10c7b7a270d7<o:p></o:p></p>
<p class="MsoListParagraph">Status: Started<o:p></o:p></p>
<p class="MsoListParagraph">Number of Bricks: 1 x 2 = 2<o:p></o:p></p>
<p class="MsoListParagraph">Transport-type: tcp<o:p></o:p></p>
<p class="MsoListParagraph">Bricks:<o:p></o:p></p>
<p class="MsoListParagraph">Brick1: dmf-ha-1-glusterfs:/export/vdb1/brick<o:p></o:p></p>
<p class="MsoListParagraph">Brick2: dmf-ha-2-glusterfs:/export/vdb1/brick<o:p></o:p></p>
<p class="MsoListParagraph">Options Reconfigured:<o:p></o:p></p>
<p class="MsoListParagraph">nfs.disable: ON<o:p></o:p></p>
<p class="MsoListParagraph">network.ping-timeout: 2<o:p></o:p></p>
<p class="MsoListParagraph">storage.bd-aio: on<o:p></o:p></p>
<p class="MsoListParagraph">storage.linux-aio: on<o:p></o:p></p>
<p class="MsoListParagraph">cluster.eager-lock: on<o:p></o:p></p>
<p class="MsoListParagraph">performance.client-io-threads: on<o:p></o:p></p>
<p class="MsoListParagraph">performance.cache-refresh-timeout: 60<o:p></o:p></p>
<p class="MsoListParagraph">performance.io-thread-count: 64<o:p></o:p></p>
<p class="MsoListParagraph">performance.cache-size: 8GB<o:p></o:p></p>
<p class="MsoListParagraph">cluster.server-quorum-type: none<o:p></o:p></p>
<p class="MsoListParagraph"><o:p> </o:p></p>
<p class="MsoNormal"> [mount-point info]:<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:27.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo1">
<![if !supportLists]><span style="mso-list:Ignore">1.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>mount command<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:27.0pt">glusterfs -p /var/run/glusterfs.pid --volfile-server=dmf-ha-1-glusterfs --volfile-server=dmf-ha-2-glusterfs --volfile-id=gv0 /dmfcontents<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:27.0pt"><o:p> </o:p></p>
<p class="MsoListParagraph" style="margin-left:27.0pt"><o:p> </o:p></p>
<p class="MsoListParagraph" style="margin-left:27.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo1">
<![if !supportLists]><span style="mso-list:Ignore">2.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>mount point directory hierarchy <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">[root@dmf-wpst-2 /]# ls -ld /dmfcontents/<o:p></o:p></p>
<p class="MsoNormal" style="text-indent:36.0pt">drwxr-xr-x. 5 root root 71 Jan 14 04:39 /dmfcontents/<o:p></o:p></p>
<p class="MsoNormal"><o:p></o:p></p>
<p class="MsoNormal"> [root@dmf-wpst-2 /]# ls -ld /dmfcontents/mpdis/<o:p></o:p></p>
<p class="MsoNormal"> drwxr-xr-x. 2 apache apache 89 Jan 14 04:39 /dmfcontents/mpdis/<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>