<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
<div class="moz-cite-prefix">On 03/19/2015 10:16 PM, Jonathan Heese
wrote:<br>
</div>
<blockquote
cite="mid:5d7e04f2d9d3496980428d7d8bf8a825@int-exch6.int.inetu.net"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:Georgia;
panose-1:2 4 5 2 5 4 5 2 3 3;}
@font-face
{font-family:o365IconsIE8;}
@font-face
{font-family:o365IconsMouse;}
@font-face
{font-family:"Times New Roman \,serif";
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p
{mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
p.ms-cui-menu, li.ms-cui-menu, div.ms-cui-menu
{mso-style-name:ms-cui-menu;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
background:white;
font-size:10.0pt;
font-family:"Segoe UI",sans-serif;
color:#333333;}
p.ms-cui-menusection-title, li.ms-cui-menusection-title, div.ms-cui-menusection-title
{mso-style-name:ms-cui-menusection-title;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;
display:none;}
p.ms-cui-ctl, li.ms-cui-ctl, div.ms-cui-ctl
{mso-style-name:ms-cui-ctl;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:#333333;}
p.ms-cui-ctl-on, li.ms-cui-ctl-on, div.ms-cui-ctl-on
{mso-style-name:ms-cui-ctl-on;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
background:#DFEDFA;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
p.ms-cui-img-cont-float, li.ms-cui-img-cont-float, div.ms-cui-img-cont-float
{mso-style-name:ms-cui-img-cont-float;
mso-style-priority:99;
margin-top:1.5pt;
margin-right:0in;
margin-bottom:0in;
margin-left:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
p.ms-cui-smenu-inner, li.ms-cui-smenu-inner, div.ms-cui-smenu-inner
{mso-style-name:ms-cui-smenu-inner;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
p.ms-owa-paste-option-icon, li.ms-owa-paste-option-icon, div.ms-owa-paste-option-icon
{mso-style-name:ms-owa-paste-option-icon;
mso-style-priority:99;
margin-top:1.5pt;
margin-right:3.0pt;
margin-bottom:0in;
margin-left:3.0pt;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;
vertical-align:sub;}
p.ms-rtepasteflyout-option, li.ms-rtepasteflyout-option, div.ms-rtepasteflyout-option
{mso-style-name:ms-rtepasteflyout-option;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
p.ms-cui-menusection, li.ms-cui-menusection, div.ms-cui-menusection
{mso-style-name:ms-cui-menusection;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
p.wf, li.wf, div.wf
{mso-style-name:wf;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
p.wf-family-owa, li.wf-family-owa, div.wf-family-owa
{mso-style-name:wf-family-owa;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:o365IconsMouse;
color:black;}
p.msochpdefault, li.msochpdefault, div.msochpdefault
{mso-style-name:msochpdefault;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;
color:black;}
p.wf-owa-play-large, li.wf-owa-play-large, div.wf-owa-play-large
{mso-style-name:wf-owa-play-large;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
p.wf-size-play-large, li.wf-size-play-large, div.wf-size-play-large
{mso-style-name:wf-size-play-large;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
p.wf-family-owa1, li.wf-family-owa1, div.wf-family-owa1
{mso-style-name:wf-family-owa1;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:o365IconsIE8;
color:black;}
p.wf-owa-play-large1, li.wf-owa-play-large1, div.wf-owa-play-large1
{mso-style-name:wf-owa-play-large1;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:white;}
p.wf-owa-play-large2, li.wf-owa-play-large2, div.wf-owa-play-large2
{mso-style-name:wf-owa-play-large2;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
text-align:center;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:white;}
p.wf-size-play-large1, li.wf-size-play-large1, div.wf-size-play-large1
{mso-style-name:wf-size-play-large1;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:22.5pt;
font-family:"Times New Roman",serif;
color:black;}
p.wf-size-play-large2, li.wf-size-play-large2, div.wf-size-play-large2
{mso-style-name:wf-size-play-large2;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:22.5pt;
font-family:"Times New Roman",serif;
color:black;}
p.wf-family-owa2, li.wf-family-owa2, div.wf-family-owa2
{mso-style-name:wf-family-owa2;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:o365IconsIE8;
color:black;}
p.wf-owa-play-large3, li.wf-owa-play-large3, div.wf-owa-play-large3
{mso-style-name:wf-owa-play-large3;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:white;}
p.wf-owa-play-large4, li.wf-owa-play-large4, div.wf-owa-play-large4
{mso-style-name:wf-owa-play-large4;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
text-align:center;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:white;}
p.wf-size-play-large3, li.wf-size-play-large3, div.wf-size-play-large3
{mso-style-name:wf-size-play-large3;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:22.5pt;
font-family:"Times New Roman",serif;
color:black;}
p.wf-size-play-large4, li.wf-size-play-large4, div.wf-size-play-large4
{mso-style-name:wf-size-play-large4;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:22.5pt;
font-family:"Times New Roman",serif;
color:black;}
span.emailstyle17
{mso-style-name:emailstyle17;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle45
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle46
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle47
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><a moz-do-not-send="true"
name="_MailEndCompose"><span style="color:#1F497D">Hello
all,<o:p></o:p></span></a></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Does anyone
else have any further suggestions for troubleshooting this?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">To sum up: I
have a 2 node 2 brick replicated volume, which holds a
handful of iSCSI image files which are mounted and served up
by tgtd (CentOS 6) to a handful of devices on a dedicated
iSCSI network. The most important iSCSI clients
(initiators) are four VMware ESXi 5.5 hosts that use the
iSCSI volumes as backing for their datastores for virtual
machine storage.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">After a few
minutes of sustained writing to the volume, I am seeing a
massive flood (over 1500 per second at times) of this error
in /var/log/glusterfs/mnt-gluster-disk.log:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2015-03-16
02:24:07.582801] W [fuse-bridge.c:2242:fuse_writev_cbk]
0-glusterfs-fuse: 635358: WRITE => -1 (Input/output
error)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">When this
happens, the ESXi box fails its write operation and returns
an error to the effect of “Unable to write data to
datastore”. I don’t see anything else in the supporting
logs to explain the root cause of the i/o errors.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Any and all
suggestions are appreciated. Thanks.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
</div>
</blockquote>
<br>
From the mount logs, i assume that your volume transport type is
rdma. There are some known issues for rdma in 3.5.3, and the patch
for to address those issues are already send to upstream [1]. From
the logs, I'm not sure and it is hard to tell you whether this
problem is something related to rdma transport or not. To make sure
that the tcp transport is works well in this scenario, if possible
can you try to reproduce the same using tcp type volumes. You can
change the transport type of volume by doing the following step (
not recommended in normal use case).<br>
<br>
1) unmount every client<br>
2) stop the volume<br>
3) run gluster volume set volname config.transport tcp<br>
4) start the volume again<br>
5) mount the clients<br>
<br>
[1] : <a class="moz-txt-link-freetext" href="http://goo.gl/2PTL61">http://goo.gl/2PTL61</a><br>
<br>
Regards<br>
Rafi KC<br>
<br>
<blockquote
cite="mid:5d7e04f2d9d3496980428d7d8bf8a825@int-exch6.int.inetu.net"
type="cite">
<div class="WordSection1">
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span
style="font-size:16.0pt;font-family:"Georgia",serif;color:#0F5789">Jon
Heese</span></i><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif;color:#1F497D"><br>
</span><i><span style="color:#333333">Systems Engineer</span></i><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif;color:#1F497D"><br>
</span><b><span style="color:#333333">INetU Managed Hosting</span></b><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif;color:#1F497D"><br>
</span><span style="color:#333333">P: 610.266.7441 x 261</span><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif;color:#1F497D"><br>
</span><span style="color:#333333">F: 610.266.7434</span><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif;color:#1F497D"><br>
</span><a moz-do-not-send="true"
href="https://www.inetu.net/"><span style="color:blue">www.inetu.net</span></a><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><i><span
style="font-size:8.0pt;color:#333333">** This message
contains confidential information, which also may be
privileged, and is intended only for the person(s)
addressed above. Any unauthorized use, distribution,
copying or disclosure of confidential and/or privileged
information is strictly prohibited. If you have received
this communication in error, please erase all copies of
the message and its attachments and notify the sender
immediately via reply e-mail. **</span></i><span
style="color:#1F497D"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span
style="color:windowtext"> Jonathan Heese
<br>
<b>Sent:</b> Tuesday, March 17, 2015 12:36 PM<br>
<b>To:</b> 'Ravishankar N'; <a class="moz-txt-link-abbreviated" href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br>
<b>Subject:</b> RE: [Gluster-users] I/O error on
replicated volume<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Ravi,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">The last lines
in the mount log before the massive vomit of I/O errors are
from 22 minutes prior, and seem innocuous to me:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2015-03-16
01:37:07.126340] E
[client-handshake.c:1760:client_query_portmap_cbk]
0-gluster_disk-client-0: failed to get the port number for
remote subvolume. Please run 'gluster volume status' on
server to see if brick process is running.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2015-03-16
01:37:07.126587] W [rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a]))) 0-gluster_disk-client-0: disconnect
called (peer:10.10.10.1:24008)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2015-03-16
01:37:07.126687] E
[client-handshake.c:1760:client_query_portmap_cbk]
0-gluster_disk-client-1: failed to get the port number for
remote subvolume. Please run 'gluster volume status' on
server to see if brick process is running.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2015-03-16
01:37:07.126737] W [rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a]))) 0-gluster_disk-client-1: disconnect
called (peer:10.10.10.2:24008)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2015-03-16
01:37:10.730165] I [rpc-clnt.c:1729:rpc_clnt_reconfig]
0-gluster_disk-client-0: changing port to 49152 (from 0)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2015-03-16
01:37:10.730276] W [rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a]))) 0-gluster_disk-client-0: disconnect
called (peer:10.10.10.1:24008)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2015-03-16
01:37:10.739500] I [rpc-clnt.c:1729:rpc_clnt_reconfig]
0-gluster_disk-client-1: changing port to 49152 (from 0)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2015-03-16
01:37:10.739560] W [rdma.c:4273:gf_rdma_disconnect]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f)
[0x7fd9c557bccf]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)
[0x7fd9c557a995]
(-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea)
[0x7fd9c0d8fb9a]))) 0-gluster_disk-client-1: disconnect
called (peer:10.10.10.2:24008)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2015-03-16
01:37:10.741883] I
[client-handshake.c:1677:select_server_supported_programs]
0-gluster_disk-client-0: Using Program GlusterFS 3.3, Num
(1298437), Version (330)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2015-03-16
01:37:10.744524] I
[client-handshake.c:1462:client_setvolume_cbk]
0-gluster_disk-client-0: Connected to 10.10.10.1:49152,
attached to remote volume '/bricks/brick1'.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2015-03-16
01:37:10.744537] I
[client-handshake.c:1474:client_setvolume_cbk]
0-gluster_disk-client-0: Server and Client lk-version
numbers are not same, reopening the fds<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2015-03-16
01:37:10.744566] I [afr-common.c:4267:afr_notify]
0-gluster_disk-replicate-0: Subvolume
'gluster_disk-client-0' came back up; going online.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2015-03-16
01:37:10.744627] I
[client-handshake.c:450:client_set_lk_version_cbk]
0-gluster_disk-client-0: Server lk version = 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2015-03-16
01:37:10.753037] I
[client-handshake.c:1677:select_server_supported_programs]
0-gluster_disk-client-1: Using Program GlusterFS 3.3, Num
(1298437), Version (330)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2015-03-16
01:37:10.755657] I
[client-handshake.c:1462:client_setvolume_cbk]
0-gluster_disk-client-1: Connected to 10.10.10.2:49152,
attached to remote volume '/bricks/brick1'.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2015-03-16
01:37:10.755676] I
[client-handshake.c:1474:client_setvolume_cbk]
0-gluster_disk-client-1: Server and Client lk-version
numbers are not same, reopening the fds<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2015-03-16
01:37:10.761945] I [fuse-bridge.c:5016:fuse_graph_setup]
0-fuse: switched to graph 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2015-03-16
01:37:10.762144] I
[client-handshake.c:450:client_set_lk_version_cbk]
0-gluster_disk-client-1: Server lk version = 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[<b>2015-03-16
01:37:10.762279</b>] I [fuse-bridge.c:3953:fuse_init]
0-glusterfs-fuse: FUSE inited with protocol versions:
glusterfs 7.22 kernel 7.14<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[<b>2015-03-16
01:59:26.098670</b>] W
[fuse-bridge.c:2242:fuse_writev_cbk] 0-glusterfs-fuse:
292084: WRITE => -1 (Input/output error)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">…<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I’ve seen no
indication of split-brain on any files at any point in this
(ever since downdating from 3.6.2 to 3.5.3, which is when
this particular issue started):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[root@duke
gfapi-module-for-linux-target-driver-]# gluster v heal
gluster_disk info<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Brick
duke.jonheese.local:/bricks/brick1/<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Number of
entries: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Brick
duchess.jonheese.local:/bricks/brick1/<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Number of
entries: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span
style="font-size:16.0pt;font-family:"Georgia",serif;color:#0F5789">Jon
Heese</span></i><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif;color:#1F497D"><br>
</span><i><span style="color:#333333">Systems Engineer</span></i><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif;color:#1F497D"><br>
</span><b><span style="color:#333333">INetU Managed Hosting</span></b><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif;color:#1F497D"><br>
</span><span style="color:#333333">P: 610.266.7441 x 261</span><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif;color:#1F497D"><br>
</span><span style="color:#333333">F: 610.266.7434</span><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif;color:#1F497D"><br>
</span><a moz-do-not-send="true"
href="https://www.inetu.net/"><span style="color:blue">www.inetu.net</span></a><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><i><span
style="font-size:8.0pt;color:#333333">** This message
contains confidential information, which also may be
privileged, and is intended only for the person(s)
addressed above. Any unauthorized use, distribution,
copying or disclosure of confidential and/or privileged
information is strictly prohibited. If you have received
this communication in error, please erase all copies of
the message and its attachments and notify the sender
immediately via reply e-mail. **</span></i><span
style="color:#1F497D"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span
style="color:windowtext"> Ravishankar N [</span><a
moz-do-not-send="true"
href="mailto:ravishankar@redhat.com">mailto:ravishankar@redhat.com</a><span
style="color:windowtext">]
<br>
<b>Sent:</b> Tuesday, March 17, 2015 12:35 AM<br>
<b>To:</b> Jonathan Heese; </span><a
moz-do-not-send="true"
href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><span
style="color:windowtext"><br>
<b>Subject:</b> Re: [Gluster-users] I/O error on
replicated volume<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:12.0pt"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal">On 03/17/2015 02:14 AM, Jonathan Heese
wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="background:white"><span
style="font-size:12.0pt">Hello,<br>
<br>
So I resolved my previous issue with split-brains and
the lack of self-healing by dropping my installed
glusterfs* packages from 3.6.2 to 3.5.3, but now I've
picked up a new issue, which actually makes normal use
of the volume practically impossible.<br>
<br>
A little background for those not already paying close
attention:<br>
I have a 2 node 2 brick replicating volume whose
purpose in life is to hold iSCSI target files,
primarily for use to provide datastores to a VMware
ESXi cluster. The plan is to put a handful of image
files on the Gluster volume, mount them locally on
both Gluster nodes, and run tgtd on both, pointed to
the image files on the mounted gluster volume. Then
the ESXi boxes will use multipath (active/passive)
iSCSI to connect to the nodes, with automatic failover
in case of planned or unplanned downtime of the
Gluster nodes.<br>
<br>
In my most recent round of testing with 3.5.3, I'm
seeing a massive failure to write data to the volume
after about 5-10 minutes, so I've simplified the
scenario a bit (to minimize the variables) to: both
Gluster nodes up, only one node (duke) mounted and
running tgtd, and just regular (single path) iSCSI
from a single ESXi server.<br>
<br>
About 5-10 minutes into migration a VM onto the test
datastore, /var/log/messages on duke gets blasted with
a ton of messages exactly like this:<o:p></o:p></span></p>
<p class="MsoNormal" style="background:white">Mar 15
22:24:06 duke tgtd: bs_rdwr_request(180) io error
0x1781e00 2a -1 512 22971904, Input/output error<o:p></o:p></p>
<p class="MsoNormal" style="background:white"><o:p> </o:p></p>
<p class="MsoNormal" style="background:white">And
/var/log/glusterfs/mnt-gluster_disk.log gets blased with
a ton of messages exactly like this:<o:p></o:p></p>
<p class="MsoNormal" style="background:white">[2015-03-16
02:24:07.572279] W [fuse-bridge.c:2242:fuse_writev_cbk]
0-glusterfs-fuse: 635299: WRITE => -1 (Input/output
error)<o:p></o:p></p>
<p class="MsoNormal" style="background:white"><o:p> </o:p></p>
</div>
</div>
</blockquote>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><br>
Are there any messages in the mount log from AFR about
split-brain just before the above line appears?<br>
Does `gluster v heal <VOLNAME> info` show any files?
Performing I/O on files that are in split-brain fail with
EIO.<br>
<br>
-Ravi<br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="background:white">And the
write operation from VMware's side fails as soon as
these messages start.<o:p></o:p></p>
<p class="MsoNormal" style="background:white"><o:p> </o:p></p>
<p class="MsoNormal" style="background:white">I don't see
any other errors (in the log files I know of) indicating
the root cause of these i/o errors. I'm sure that this
is not enough information to tell what's going on, but
can anyone help me figure out what to look at next to
figure this out?<o:p></o:p></p>
<p class="MsoNormal" style="background:white"><o:p> </o:p></p>
<p class="MsoNormal" style="background:white">I've also
considered using Dan Lambright's libgfapi gluster module
for tgtd (or something similar) to avoid going through
FUSE, but I'm not sure whether that would be irrelevant
to this problem, since I'm not 100% sure if it lies in
FUSE or elsewhere.<o:p></o:p></p>
<p class="MsoNormal" style="background:white"><o:p> </o:p></p>
<p class="MsoNormal" style="background:white">Thanks!<o:p></o:p></p>
<p class="MsoNormal" style="background:white"><o:p> </o:p></p>
<p class="MsoNormal" style="background:white"><i><span
style="font-size:16.0pt;font-family:"Georgia",serif;color:#0F5789">Jon
Heese</span></i><span
style="font-size:12.0pt;font-family:"Times New
Roman ,serif",serif"><br>
</span><i><span style="color:#333333">Systems Engineer</span></i><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><br>
</span><b><span style="color:#333333">INetU Managed
Hosting</span></b><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><br>
</span><span style="color:#333333">P: 610.266.7441 x 261</span><span
style="font-size:12.0pt;font-family:"Times New
Roman ,serif",serif"><br>
</span><span style="color:#333333">F: 610.266.7434</span><span
style="font-size:12.0pt;font-family:"Times New
Roman ,serif",serif"><br>
</span><a moz-do-not-send="true"
href="https://www.inetu.net/"><span style="color:blue">www.inetu.net</span></a><o:p></o:p></p>
<p class="MsoNormal" style="background:white"><i><span
style="font-size:8.0pt;color:#333333">** This
message contains confidential information, which
also may be privileged, and is intended only for the
person(s) addressed above. Any unauthorized use,
distribution, copying or disclosure of confidential
and/or privileged information is strictly
prohibited. If you have received this communication
in error, please erase all copies of the message and
its attachments and notify the sender immediately
via reply e-mail. **</span></i><o:p></o:p></p>
<p class="MsoNormal" style="background:white"> <o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><br>
<br>
<o:p></o:p></span></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Gluster-users mailing list<o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p> </o:p></span></p>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</body>
</html>