<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
<div class="moz-cite-prefix">On 03/17/2015 02:14 AM, Jonathan Heese
wrote:<br>
</div>
<blockquote
cite="mid:79aaa9ff350749d6a313601caf3c2d6b@int-exch6.int.inetu.net"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<style type="text/css" style="display:none"><!--P{margin-top:0;margin-bottom:0;} .ms-cui-menu {background-color:#ffffff;border:1px rgb(171, 171, 171) solid;font-family:"Segoe UI WPC","Segoe UI",Tahoma,"Microsoft Sans Serif",Verdana,sans-serif;font-size:10pt;color:rgb(51, 51, 51);} .ms-cui-menusection-title {display:none;} .ms-cui-ctl {vertical-align:text-top;text-decoration:none;color:rgb(51, 51, 51);} .ms-cui-ctl-on {background-color:rgb(223, 237, 250);opacity: 0.8;} .ms-cui-img-cont-float {display:inline-block;margin-top:2px} .ms-cui-smenu-inner {padding-top:0px;} .ms-owa-paste-option-icon {margin: 2px 4px 0px 4px;vertical-align:sub;padding-bottom: 2px;display:inline-block;} .ms-rtePasteFlyout-option:hover {background-color:rgb(223, 237, 250) !important;opacity:1 !important;} .ms-rtePasteFlyout-option {padding:8px 4px 8px 4px;outline:none;} .ms-cui-menusection {float:left; width:85px;height:24px;overflow:hidden}.wf {speak:none; font-weight:normal; font-variant:normal!
; text-tra
nsform:none; -webkit-font-smoothing:antialiased; vertical-align:middle; display:inline-block;}.wf-family-owa {font-family:'o365Icons'}@font-face { font-family:'o365IconsIE8'; src:url('prem/15.0.913.22/resources/styles/office365icons.ie8.eot?#iefix') format('embedded-opentype'), url('prem/15.0.913.22/resources/styles/office365icons.ie8.woff') format('woff'), url('prem/15.0.913.22/resources/styles/office365icons.ie8.ttf') format('truetype'); font-weight:normal; font-style:normal;}@font-face { font-family:'o365IconsMouse'; src:url('prem/15.0.913.22/resources/styles/office365icons.mouse.eot?#iefix') format('embedded-opentype'), url('prem/15.0.913.22/resources/styles/office365icons.mouse.woff') format('woff'), url('prem/15.0.913.22/resources/styles/office365icons.mouse.ttf') format('truetype'); font-weight:normal; font-style:normal;}.wf-family-owa {font-family:'o365IconsMouse'}.ie8 .wf-family-owa {font-family:'o365IconsIE8'}.ie8 .wf-owa-pla!
y-large:be
fore {content:'\e254';}.notIE8 .wf-owa-play-large:before {content:'\e054';}.ie8 .wf-owa-play-large {color:#FFFFFF/*$WFWhiteColor*/;}.notIE8 .wf-owa-play-large {border-color:#FFFFFF/*$WFWhiteColor*/; width:1.4em; height:1.4em; border-width:.1em; border-style:solid; border-radius:.8em; text-align:center; box-sizing:border-box; -moz-box-sizing:border-box; padding:0.1em; color:#FFFFFF/*$WFWhiteColor*/;}.ie8 .wf-size-play-large {width:40px; height:40px; font-size:30px}.notIE8 .wf-size-play-large {width:40px; height:40px; font-size:30px}
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
@font-face
{font-family:Georgia}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
a:link, span.MsoHyperlink
{color:#0563C1;
text-decoration:underline}
a:visited, span.MsoHyperlinkFollowed
{color:#954F72;
text-decoration:underline}
span.EmailStyle17
{font-family:"Calibri",sans-serif;
color:windowtext}
span.SpellE
{}
.MsoChpDefault
{font-family:"Calibri",sans-serif}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.WordSection1
{}
-->
--></style>
<div
style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<link rel="File-List" href="cid:filelist.xml@01D05FE8.DB8A26E0">
<link rel="themeData" href="%7E%7Ethemedata%7E%7E">
<link rel="colorSchemeMapping"
href="%7E%7Ecolorschememapping%7E%7E">
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
@font-face
{font-family:Georgia}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
a:link, span.MsoHyperlink
{color:#0563C1;
text-decoration:underline}
a:visited, span.MsoHyperlinkFollowed
{color:#954F72;
text-decoration:underline}
span.EmailStyle17
{font-family:"Calibri",sans-serif;
color:windowtext}
span.SpellE
{}
.MsoChpDefault
{font-family:"Calibri",sans-serif}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.WordSection1
{}
-->
</style>
<div class="WordSection1">Hello,<br>
<br>
So I resolved my previous issue with split-brains and the lack
of self-healing by dropping my installed glusterfs* packages
from 3.6.2 to 3.5.3, but now I've picked up a new issue, which
actually makes normal use of the volume practically
impossible.<br>
<br>
A little background for those not already paying close
attention:<br>
I have a 2 node 2 brick replicating volume whose purpose in
life is to hold iSCSI target files, primarily for use to
provide datastores to a VMware ESXi cluster. The plan is to
put a handful of image files on the Gluster volume, mount them
locally on both Gluster nodes, and run tgtd on both, pointed
to the image files on the mounted gluster volume. Then the
ESXi boxes will use multipath (active/passive) iSCSI to
connect to the nodes, with automatic failover in case of
planned or unplanned downtime of the Gluster nodes.<br>
<br>
In my most recent round of testing with 3.5.3, I'm seeing a
massive failure to write data to the volume after about 5-10
minutes, so I've simplified the scenario a bit (to minimize
the variables) to: both Gluster nodes up, only one node (duke)
mounted and running tgtd, and just regular (single path) iSCSI
from a single ESXi server.<br>
<br>
About 5-10 minutes into migration a VM onto the test
datastore, /var/log/messages on duke gets blasted with a ton
of messages exactly like this:<br>
<p class="MsoNormal">Mar 15 22:24:06 duke <span
class="SpellE">tgtd</span>: <span class="SpellE">
bs_rdwr_request</span>(180) <span class="SpellE">io</span>
error 0x1781e00 2a -1 512 22971904, Input/output error</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal">And
/var/log/glusterfs/mnt-gluster_disk.log gets blased with a
ton of messages exactly like this:<br>
</p>
<p class="MsoNormal">[2015-03-16 02:24:07.572279] W
[fuse-bridge.c:2242:fuse_writev_cbk] 0-glusterfs-fuse:
635299: WRITE => -1 (Input/output error)</p>
<p class="MsoNormal"><br>
</p>
</div>
</div>
</blockquote>
<br>
Are there any messages in the mount log from AFR about split-brain
just before the above line appears?<br>
Does `gluster v heal <VOLNAME> info` show any files?
Performing I/O on files that are in split-brain fail with EIO.<br>
<br>
-Ravi<br>
<br>
<blockquote
cite="mid:79aaa9ff350749d6a313601caf3c2d6b@int-exch6.int.inetu.net"
type="cite">
<div
style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<div class="WordSection1">
<p class="MsoNormal">
</p>
<p class="MsoNormal">And the write operation from VMware's
side fails as soon as these messages start.</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal">I don't see any other errors (in the log
files I know of) indicating the root cause of these i/o
errors. I'm sure that this is not enough information to
tell what's going on, but can anyone help me figure out what
to look at next to figure this out?</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal">I've also considered using Dan
Lambright's libgfapi gluster module for tgtd (or something
similar) to avoid going through FUSE, but I'm not sure
whether that would be irrelevant to this problem, since I'm
not 100% sure if it lies in FUSE or elsewhere.</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal">Thanks!<br>
</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal" style=""><a moz-do-not-send="true"
name="_MailAutoSig"><i><span style="font-size:16.0pt;
font-family:"Georgia",serif; color:#0F5789">Jon
Heese</span></i></a><span style=""><span
style="font-size:12.0pt; font-family:"Times New
Roman",serif"><br>
</span></span><span style=""><i><span
style="color:#333333">Systems Engineer</span></i></span><span
style=""><span style="font-size:12.0pt;
font-family:"Times New Roman",serif"><br>
</span></span><span style=""><b><span
style="color:#333333">INetU Managed Hosting</span></b></span><span
style=""><span style="font-size:12.0pt;
font-family:"Times New Roman",serif"><br>
</span></span><span style=""><span style="color:#333333">P:
610.266.7441 x 261</span></span><span style=""><span
style="font-size:12.0pt; font-family:"Times New
Roman",serif"><br>
</span></span><span style=""><span style="color:#333333">F:
610.266.7434</span></span><span style=""><span
style="font-size:12.0pt; font-family:"Times New
Roman",serif"><br>
</span></span><span style=""></span><a
moz-do-not-send="true" href="https://www.inetu.net/"><span
style=""><span style="color:blue">www.inetu.net</span></span><span
style=""></span></a><span style=""><span
style="font-size:12.0pt; font-family:"Times New
Roman",serif"></span></span></p>
<p class="MsoNormal"><span style=""><i><span
style="font-size:8.0pt; color:#333333">** This message
contains confidential information, which also may be
privileged, and is intended only for the person(s)
addressed above. Any unauthorized use, distribution,
copying or disclosure of confidential and/or
privileged information is strictly prohibited. If you
have received this communication in error, please
erase all copies of the message and its attachments
and notify the sender immediately via reply e-mail. **</span></i></span><span
style=""><span style=""></span></span></p>
<span style=""></span>
<p class="MsoNormal"> </p>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</body>
</html>