<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
<div class="moz-cite-prefix">On 01/14/2015 04:48 PM, Jifeng Li
wrote:<br>
</div>
<blockquote
cite="mid:C9A3299D5F8F4542822D6A171931C17519949578@ESGSCMB101.ericsson.se"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<meta name="Generator" content="Microsoft Word 14 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1577588982;
mso-list-type:hybrid;
mso-list-template-ids:1092125724 -703844236 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:27.0pt;
text-indent:-18.0pt;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:63.0pt;
text-indent:-18.0pt;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:99.0pt;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:135.0pt;
text-indent:-18.0pt;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:171.0pt;
text-indent:-18.0pt;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:207.0pt;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:243.0pt;
text-indent:-18.0pt;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:279.0pt;
text-indent:-18.0pt;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:315.0pt;
text-indent:-9.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal">Hi ,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span
style="background:yellow;mso-highlight:yellow">[issue]:</span>
<span style="background:yellow;mso-highlight:yellow">To
ensure the glusterFS mount point work, a script will
periodically using HTTP put a file to subdirectory under
mount point which is used as Apache DocumentRoot. But after
running some time, some errors show below:</span><o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:40.915639] E
[afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
0-gv0-replicate-0: metadata self heal failed, on /mpdis<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:41.924584] E
[afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
0-gv0-replicate-0: Unable to self-heal contents of '/'
(possible split-brain). Please delete the file from all but
the preferred subvolume.- Pending matrix: [ [ 0 20 ] [ 21 0 ]
]<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:41.925182] E
[afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
0-gv0-replicate-0: metadata self heal failed, on /<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:41.934827] E
[afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
0-gv0-replicate-0: Unable to self-heal contents of '/mpdis'
(possible split-brain). Please delete the file from all but
the preferred subvolume.- Pending matrix: [ [ 0 4 ] [ 2 0 ] ]<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:41.935375] E
[afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
0-gv0-replicate-0: metadata self heal failed, on /mpdis<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:42.943742] E
[afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
0-gv0-replicate-0: Unable to self-heal contents of '/'
(possible split-brain). Please delete the file from all but
the preferred subvolume.- Pending matrix: [ [ 0 20 ] [ 21 0 ]
]<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:42.944432] E
[afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
0-gv0-replicate-0: metadata self heal failed, on /<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:42.946664] E
[afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
0-gv0-replicate-0: Unable to self-heal contents of '/mpdis'
(possible split-brain). Please delete the file from all but
the preferred subvolume.- Pending matrix: [ [ 0 4 ] [ 2 0 ] ]<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:42.947323] E
[afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
0-gv0-replicate-0: metadata self heal failed, on /mpdis<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:43.955929] E
[afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
0-gv0-replicate-0: Unable to self-heal contents of '/'
(possible split-brain). Please delete the file from all but
the preferred subvolume.- Pending matrix: [ [ 0 20 ] [ 21 0 ]
]<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:43.956701] E
[afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
0-gv0-replicate-0: metadata self heal failed, on /<o:p></o:p></p>
<p class="MsoNormal">[2015-01-14 09:18:43.958874] E
[afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
0-gv0-replicate-0: Unable to self-heal contents of '/mpdis'
(possible split-brain). Please delete the file from all but
the preferred subvolume.- Pending matrix: [ [ 0 4 ] [ 2 0 ] ]<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Besides, I find Input/output error shown
below when listing the files of under mount point:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoListParagraph">[root@dmf-wpst-2 mpdis]# ll<o:p></o:p></p>
<p class="MsoListParagraph">total 0<o:p></o:p></p>
<p class="MsoListParagraph">-rwxr-xr-x. 1 apache apache 0 Jan 14
04:21 test.rep.00.00.00.00.dmf1<o:p></o:p></p>
<p class="MsoListParagraph">-rw-r--r--. 1 apache apache 0 Jan 14
04:21 test.rep.00.00.00.00.dmf2<o:p></o:p></p>
<p class="MsoListParagraph"><o:p> </o:p></p>
<p class="MsoListParagraph">[root@dmf-wpst-2 mpdis]# ll<o:p></o:p></p>
<p class="MsoListParagraph">total 0<o:p></o:p></p>
<p class="MsoListParagraph">-rwxr-xr-x. 1 apache apache 0 Jan 14
04:21 test.rep.00.00.00.00.dmf1<o:p></o:p></p>
<p class="MsoListParagraph">-rw-r--r--. 1 apache apache 0 Jan 14
04:21 test.rep.00.00.00.00.dmf2<o:p></o:p></p>
<p class="MsoListParagraph"><o:p> </o:p></p>
<p class="MsoListParagraph">[root@dmf-wpst-2 mpdis]# ll<o:p></o:p></p>
<p class="MsoListParagraph">ls: cannot open directory .:
Input/output error<o:p></o:p></p>
<p class="MsoListParagraph"><o:p> </o:p></p>
<p class="MsoListParagraph">[root@dmf-wpst-2 mpdis]# ll<o:p></o:p></p>
<p class="MsoListParagraph"><span
style="background:red;mso-highlight:red">ls: cannot access
test.rep.00.00.00.00.dmf1: Input/output error<o:p></o:p></span></p>
<p class="MsoListParagraph"><span
style="background:red;mso-highlight:red">ls: cannot access
test.rep.00.00.00.00.dmf2: Input/output error</span><o:p></o:p></p>
<p class="MsoListParagraph">total 0<o:p></o:p></p>
<p class="MsoListParagraph">?????????? ? ? ? ? ?
test.rep.00.00.00.00.dmf1<o:p></o:p></p>
<p class="MsoListParagraph">?????????? ? ? ? ? ?
test.rep.00.00.00.00.dmf2<o:p></o:p></p>
<p class="MsoNormal"><o:p></o:p></p>
<p class="MsoNormal"><b><span
style="font-size:12.0pt;font-family:"Arial","sans-serif";color:black"> Any
tips about debugging further or getting this fixed up
would be appreciated. <o:p></o:p></span></b></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span
style="background:yellow;mso-highlight:yellow">[version]:
3.5.3</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="background:yellow;mso-highlight:yellow">[environment]:</span>
two virtual server each has one brick :<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoListParagraph">root@dmf-wpst-2 mpdis]# gluster
volume status<o:p></o:p></p>
<p class="MsoListParagraph">Status of volume: gv0<o:p></o:p></p>
<p class="MsoListParagraph">Gluster
process
Port Online Pid<o:p></o:p></p>
<p class="MsoListParagraph">------------------------------------------------------------------------------<o:p></o:p></p>
<p class="MsoListParagraph">Brick
dmf-ha-1-glusterfs:/export/vdb1/brick
49152 Y 332<o:p></o:p></p>
<p class="MsoListParagraph">Brick
dmf-ha-2-glusterfs:/export/vdb1/brick
49154 Y 19396<o:p></o:p></p>
<p class="MsoListParagraph">Self-heal Daemon on
localhost
N/A Y 19410<o:p></o:p></p>
<p class="MsoListParagraph">Self-heal Daemon on
10.175.123.246
N/A Y 999<o:p></o:p></p>
<p class="MsoListParagraph"><o:p> </o:p></p>
<p class="MsoListParagraph">[root@dmf-wpst-1 mpdis]# gluster
volume info<o:p></o:p></p>
<p class="MsoListParagraph">Volume Name: gv0<o:p></o:p></p>
<p class="MsoListParagraph">Type: Replicate<o:p></o:p></p>
<p class="MsoListParagraph">Volume ID:
51de44c3-f01e-486d-a6b7-10c7b7a270d7<o:p></o:p></p>
<p class="MsoListParagraph">Status: Started<o:p></o:p></p>
<p class="MsoListParagraph">Number of Bricks: 1 x 2 = 2<o:p></o:p></p>
<p class="MsoListParagraph">Transport-type: tcp<o:p></o:p></p>
<p class="MsoListParagraph">Bricks:<o:p></o:p></p>
<p class="MsoListParagraph">Brick1:
dmf-ha-1-glusterfs:/export/vdb1/brick<o:p></o:p></p>
<p class="MsoListParagraph">Brick2:
dmf-ha-2-glusterfs:/export/vdb1/brick<o:p></o:p></p>
<p class="MsoListParagraph">Options Reconfigured:<o:p></o:p></p>
<p class="MsoListParagraph">nfs.disable: ON<o:p></o:p></p>
<p class="MsoListParagraph">network.ping-timeout: 2<o:p></o:p></p>
<p class="MsoListParagraph">storage.bd-aio: on<o:p></o:p></p>
<p class="MsoListParagraph">storage.linux-aio: on<o:p></o:p></p>
<p class="MsoListParagraph">cluster.eager-lock: on<o:p></o:p></p>
<p class="MsoListParagraph">performance.client-io-threads: on<o:p></o:p></p>
<p class="MsoListParagraph">performance.cache-refresh-timeout:
60<o:p></o:p></p>
<p class="MsoListParagraph">performance.io-thread-count: 64<o:p></o:p></p>
<p class="MsoListParagraph">performance.cache-size: 8GB<o:p></o:p></p>
<p class="MsoListParagraph">cluster.server-quorum-type: none<o:p></o:p></p>
<p class="MsoListParagraph"><o:p> </o:p></p>
<p class="MsoNormal"> [mount-point info]:<o:p></o:p></p>
<p class="MsoListParagraph"
style="margin-left:27.0pt;text-indent:-18.0pt;mso-list:l0
level1 lfo1">
<!--[if !supportLists]--><span style="mso-list:Ignore">1.<span
style="font:7.0pt "Times New Roman"">
</span></span><!--[endif]-->mount command<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:27.0pt">glusterfs
-p /var/run/glusterfs.pid --volfile-server=dmf-ha-1-glusterfs
--volfile-server=dmf-ha-2-glusterfs --volfile-id=gv0
/dmfcontents<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:27.0pt"><o:p> </o:p></p>
<p class="MsoListParagraph" style="margin-left:27.0pt"><o:p> </o:p></p>
<p class="MsoListParagraph"
style="margin-left:27.0pt;text-indent:-18.0pt;mso-list:l0
level1 lfo1">
<!--[if !supportLists]--><span style="mso-list:Ignore">2.<span
style="font:7.0pt "Times New Roman"">
</span></span><!--[endif]-->mount point directory
hierarchy <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">[root@dmf-wpst-2
/]# ls -ld /dmfcontents/<o:p></o:p></p>
<p class="MsoNormal" style="text-indent:36.0pt">drwxr-xr-x. 5
root root 71 Jan 14 04:39 /dmfcontents/<o:p></o:p></p>
<p class="MsoNormal"><o:p></o:p></p>
<p class="MsoNormal"> [root@dmf-wpst-2 /]# ls -ld
/dmfcontents/mpdis/<o:p></o:p></p>
<p class="MsoNormal"> drwxr-xr-x. 2 apache apache
89 Jan 14 04:39 /dmfcontents/mpdis/<o:p></o:p></p>
</div>
</blockquote>
hi Jifeng Li,<br>
Sorry for the delay in response. Could you post the output of:<br>
'getfattr -d -m. -e hex <brick-path>'<br>
'getfattr -d -m. -e hex <brick-path>/mpdis'<br>
'getfattr -d -m. -e hex <brick-path>/mpdis/<span
style="background:red;mso-highlight:red">test.rep.00.00.00.00.dmf1</span>'<br>
'getfattr -d -m. -e hex <brick-path>/mpdis/<span
style="background:red;mso-highlight:red">test.rep.00.00.00.00.dmf2</span>'<br>
<br>
Pranith<br>
<blockquote
cite="mid:C9A3299D5F8F4542822D6A171931C17519949578@ESGSCMB101.ericsson.se"
type="cite">
<div class="WordSection1">
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</body>
</html>