<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none"><!--P{margin-top:0;margin-bottom:0;} .ms-cui-menu {background-color:#ffffff;border:1px rgb(171, 171, 171) solid;font-family:"Segoe UI WPC","Segoe UI",Tahoma,"Microsoft Sans Serif",Verdana,sans-serif;font-size:10pt;color:rgb(51, 51, 51);} .ms-cui-menusection-title {display:none;} .ms-cui-ctl {vertical-align:text-top;text-decoration:none;color:rgb(51, 51, 51);} .ms-cui-ctl-on {background-color:rgb(223, 237, 250);opacity: 0.8;} .ms-cui-img-cont-float {display:inline-block;margin-top:2px} .ms-cui-smenu-inner {padding-top:0px;} .ms-owa-paste-option-icon {margin: 2px 4px 0px 4px;vertical-align:sub;padding-bottom: 2px;display:inline-block;} .ms-rtePasteFlyout-option:hover {background-color:rgb(223, 237, 250) !important;opacity:1 !important;} .ms-rtePasteFlyout-option {padding:8px 4px 8px 4px;outline:none;} .ms-cui-menusection {float:left; width:85px;height:24px;overflow:hidden}.wf {speak:none; font-weight:normal; font-variant:normal; text-transform:none; -webkit-font-smoothing:antialiased; vertical-align:middle; display:inline-block;}.wf-family-owa {font-family:'o365Icons'}@font-face { font-family:'o365IconsIE8'; src:url('prem/15.0.913.22/resources/styles/office365icons.ie8.eot?#iefix') format('embedded-opentype'), url('prem/15.0.913.22/resources/styles/office365icons.ie8.woff') format('woff'), url('prem/15.0.913.22/resources/styles/office365icons.ie8.ttf') format('truetype'); font-weight:normal; font-style:normal;}@font-face { font-family:'o365IconsMouse'; src:url('prem/15.0.913.22/resources/styles/office365icons.mouse.eot?#iefix') format('embedded-opentype'), url('prem/15.0.913.22/resources/styles/office365icons.mouse.woff') format('woff'), url('prem/15.0.913.22/resources/styles/office365icons.mouse.ttf') format('truetype'); font-weight:normal; font-style:normal;}.wf-family-owa {font-family:'o365IconsMouse'}.ie8 .wf-family-owa {font-family:'o365IconsIE8'}.ie8 .wf-owa-play-large:before {content:'\e254';}.notIE8 .wf-owa-play-large:before {content:'\e054';}.ie8 .wf-owa-play-large {color:#FFFFFF/*$WFWhiteColor*/;}.notIE8 .wf-owa-play-large {border-color:#FFFFFF/*$WFWhiteColor*/; width:1.4em; height:1.4em; border-width:.1em; border-style:solid; border-radius:.8em; text-align:center; box-sizing:border-box; -moz-box-sizing:border-box; padding:0.1em; color:#FFFFFF/*$WFWhiteColor*/;}.ie8 .wf-size-play-large {width:40px; height:40px; font-size:30px}.notIE8 .wf-size-play-large {width:40px; height:40px; font-size:30px}
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
@font-face
{font-family:Georgia}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
a:link, span.MsoHyperlink
{color:#0563C1;
text-decoration:underline}
a:visited, span.MsoHyperlinkFollowed
{color:#954F72;
text-decoration:underline}
span.EmailStyle17
{font-family:"Calibri",sans-serif;
color:windowtext}
span.SpellE
{}
.MsoChpDefault
{font-family:"Calibri",sans-serif}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.WordSection1
{}
-->
--></style>
</head>
<body dir="ltr">
<div style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<link rel="File-List" href="cid:filelist.xml@01D05FE8.DB8A26E0"><link rel="themeData" href="~~themedata~~"><link rel="colorSchemeMapping" href="~~colorschememapping~~"><style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
@font-face
{font-family:Georgia}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
a:link, span.MsoHyperlink
{color:#0563C1;
text-decoration:underline}
a:visited, span.MsoHyperlinkFollowed
{color:#954F72;
text-decoration:underline}
span.EmailStyle17
{font-family:"Calibri",sans-serif;
color:windowtext}
span.SpellE
{}
.MsoChpDefault
{font-family:"Calibri",sans-serif}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.WordSection1
{}
-->
</style>
<div class="WordSection1">Hello,<br>
<br>
So I resolved my previous issue with split-brains and the lack of self-healing by dropping my installed glusterfs* packages from 3.6.2 to 3.5.3, but now I've picked up a new issue, which actually makes normal use of the volume practically impossible.<br>
<br>
A little background for those not already paying close attention:<br>
I have a 2 node 2 brick replicating volume whose purpose in life is to hold iSCSI target files, primarily for use to provide datastores to a VMware ESXi cluster. The plan is to put a handful of image files on the Gluster volume, mount them locally on both
Gluster nodes, and run tgtd on both, pointed to the image files on the mounted gluster volume. Then the ESXi boxes will use multipath (active/passive) iSCSI to connect to the nodes, with automatic failover in case of planned or unplanned downtime of the Gluster
nodes.<br>
<br>
In my most recent round of testing with 3.5.3, I'm seeing a massive failure to write data to the volume after about 5-10 minutes, so I've simplified the scenario a bit (to minimize the variables) to: both Gluster nodes up, only one node (duke) mounted and running
tgtd, and just regular (single path) iSCSI from a single ESXi server.<br>
<br>
About 5-10 minutes into migration a VM onto the test datastore, /var/log/messages on duke gets blasted with a ton of messages exactly like this:<br>
<p class="MsoNormal">Mar 15 22:24:06 duke <span class="SpellE">tgtd</span>: <span class="SpellE">
bs_rdwr_request</span>(180) <span class="SpellE">io</span> error 0x1781e00 2a -1 512 22971904, Input/output error</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal">And /var/log/glusterfs/mnt-gluster_disk.log gets blased with a ton of messages exactly like this:<br>
</p>
<p class="MsoNormal">[2015-03-16 02:24:07.572279] W [fuse-bridge.c:2242:fuse_writev_cbk] 0-glusterfs-fuse: 635299: WRITE => -1 (Input/output error)</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal">And the write operation from VMware's side fails as soon as these messages start.</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal">I don't see any other errors (in the log files I know of) indicating the root cause of these i/o errors. I'm sure that this is not enough information to tell what's going on, but can anyone help me figure out what to look at next to figure
this out?</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal">I've also considered using Dan Lambright's libgfapi gluster module for tgtd (or something similar) to avoid going through FUSE, but I'm not sure whether that would be irrelevant to this problem, since I'm not 100% sure if it lies in FUSE
or elsewhere.</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal">Thanks!<br>
</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal" style=""><a name="_MailAutoSig"><i><span style="font-size:16.0pt; font-family:"Georgia",serif; color:#0F5789">Jon Heese</span></i></a><span style=""><span style="font-size:12.0pt; font-family:"Times New Roman",serif"><br>
</span></span><span style=""><i><span style="color:#333333">Systems Engineer</span></i></span><span style=""><span style="font-size:12.0pt; font-family:"Times New Roman",serif"><br>
</span></span><span style=""><b><span style="color:#333333">INetU Managed Hosting</span></b></span><span style=""><span style="font-size:12.0pt; font-family:"Times New Roman",serif"><br>
</span></span><span style=""><span style="color:#333333">P: 610.266.7441 x 261</span></span><span style=""><span style="font-size:12.0pt; font-family:"Times New Roman",serif"><br>
</span></span><span style=""><span style="color:#333333">F: 610.266.7434</span></span><span style=""><span style="font-size:12.0pt; font-family:"Times New Roman",serif"><br>
</span></span><span style=""></span><a href="https://www.inetu.net/"><span style=""><span style="color:blue">www.inetu.net</span></span><span style=""></span></a><span style=""><span style="font-size:12.0pt; font-family:"Times New Roman",serif"></span></span></p>
<p class="MsoNormal"><span style=""><i><span style="font-size:8.0pt; color:#333333">** This message contains confidential information, which also may be privileged, and is intended only for the person(s) addressed above. Any unauthorized use, distribution,
copying or disclosure of confidential and/or privileged information is strictly prohibited. If you have received this communication in error, please erase all copies of the message and its attachments and notify the sender immediately via reply e-mail. **</span></i></span><span style=""><span style=""></span></span></p>
<span style=""></span>
<p class="MsoNormal"> </p>
</div>
</div>
</body>
</html>