<div dir="ltr"><br><div class="gmail_extra"><div><div class="gmail_signature"><div dir="ltr"><div><span><font color="#888888"><font><font size="1"><span style="color:rgb(51,51,51)"><span style="color:rgb(153,153,153)"><a value="+17086132426"><font color="#888888"><font size="1"><br></font></font></a></span></span></font></font></font></span></div></div></div></div>
<br><div class="gmail_quote">On Sun, Mar 13, 2016 at 8:16 AM, Mahdi Adnan <span dir="ltr"><<a href="mailto:mahdi.adnan@earthlinktele.com" target="_blank">mahdi.adnan@earthlinktele.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
Okay so i have enabled shard in my test volume and it did not help,
stupidly enough, i have enabled it in a production volume
"Distributed-Replicate" and it currpted half of my VMs.<br>
I have updated Gluster to the latest and nothing seems to be changed
in my situation.<br>
below the info of my volume;<br></div></blockquote><div><br></div><div>I was pointing at the settings in that email as an example for corruption fixing. I wouldn't recommend enabling sharding if you haven't gotten the base working yet on that cluster. What HBA's are you using and what is layout of filesystem for bricks? </div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF">
<br>
Number of Bricks: 3 x 2 = 6<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: gfs001:/bricks/b001/vmware<br>
Brick2: gfs002:/bricks/b004/vmware<br>
Brick3: gfs001:/bricks/b002/vmware<br>
Brick4: gfs002:/bricks/b005/vmware<br>
Brick5: gfs001:/bricks/b003/vmware<br>
Brick6: gfs002:/bricks/b006/vmware<br>
Options Reconfigured:<br>
performance.strict-write-ordering: on<br>
cluster.server-quorum-type: server<br>
cluster.quorum-type: auto<br>
network.remote-dio: enable<br>
performance.stat-prefetch: disable<br>
performance.io-cache: off<br>
performance.read-ahead: off<br>
performance.quick-read: off<br>
cluster.eager-lock: enable<br>
features.shard-block-size: 16MB<br>
features.shard: on<br>
performance.readdir-ahead: off<br>
<br>
<br>
<div>On 03/12/2016 08:11 PM, David Gossage
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sat, Mar 12, 2016 at 10:21 AM,
Mahdi Adnan <span dir="ltr"><<a href="mailto:mahdi.adnan@earthlinktele.com" target="_blank">mahdi.adnan@earthlinktele.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> Both servers have
HBA no RAIDs and i can setup a replicated or dispensers
without any issues.<br>
Logs are clean and when i tried to migrate a vm and got
the error, nothing showed up in the logs.<br>
i tried mounting the volume into my laptop and it
mounted fine but, if i use dd to create a data file it
just hang and i cant cancel it, and i cant unmount it or
anything, i just have to reboot.<br>
The same servers have another volume on other bricks in
a distributed replicas, works fine.<br>
I have even tried the same setup in a virtual
environment (created two vms and install gluster and
created a replicated striped) and again same thing, data
corruption.<br>
</div>
</blockquote>
<div><br>
</div>
<div>I'd look through mail archives for a topic "Shard in
Production" I think it's called. The shard portion may
not be relevant but it does discuss certain settings that
had to be applied with regards to avoiding corruption with
VM's. You may want to try and disable the
performance.readdir-ahead also.</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> <br>
<br>
<div>On 03/12/2016 07:02 PM, David Gossage wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div>
<div>
<div dir="ltr">
<div><span><font color="#888888"><font><font size="1"><span style="color:rgb(51,51,51)"><span style="color:rgb(153,153,153)"><a value="+17086132426"><font color="#888888"><font size="1"><br>
</font></font></a></span></span></font></font></font></span></div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On Sat, Mar 12, 2016 at
9:51 AM, Mahdi Adnan <span dir="ltr"><<a href="mailto:mahdi.adnan@earthlinktele.com" target="_blank"></a><a href="mailto:mahdi.adnan@earthlinktele.com" target="_blank">mahdi.adnan@earthlinktele.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> Thanks
David,<br>
<br>
My settings are all defaults, i have just
created the pool and started it.<br>
I have set the settings as your
recommendation and it seems to be the same
issue;<br>
<br>
Type: Striped-Replicate<br>
Volume ID:
44adfd8c-2ed1-4aa5-b256-d12b64f7fc14<br>
Status: Started<br>
Number of Bricks: 1 x 2 x 2 = 4<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: gfs001:/bricks/t1/s<br>
Brick2: gfs002:/bricks/t1/s<br>
Brick3: gfs001:/bricks/t2/s<br>
Brick4: gfs002:/bricks/t2/s<br>
Options Reconfigured:<br>
performance.stat-prefetch: off<br>
network.remote-dio: on<br>
cluster.eager-lock: enable<br>
performance.io-cache: off<br>
performance.read-ahead: off<br>
performance.quick-read: off<br>
performance.readdir-ahead: on<br>
</div>
</blockquote>
<div> </div>
<div><br>
</div>
<div>Is their a raid controller perhaps doing
any caching? <br>
</div>
<div><br>
</div>
<div>In the gluster logs any errors being
reported during migration process?</div>
<div>Since they aren't in use yet have you
tested making just mirrored bricks using
different pairings of servers two at a time to
see if problem follows certain machine or
network ports?</div>
<div><br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> <br>
<br>
<br>
<br>
<br>
<div>On 03/12/2016 03:25 PM, David Gossage
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sat, Mar
12, 2016 at 1:55 AM, Mahdi Adnan <span dir="ltr"><<a href="mailto:mahdi.adnan@earthlinktele.com" target="_blank"></a><a href="mailto:mahdi.adnan@earthlinktele.com" target="_blank">mahdi.adnan@earthlinktele.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<p dir="ltr">Dears,</p>
<p dir="ltr">I have created a
replicated striped volume with
two bricks and two servers but I
can't use it because when I
mount it in ESXi and try to
migrate a VM to it, the data get
corrupted.<br>
Is any one have any idea why is
this happening ?</p>
<p dir="ltr">Dell 2950 x2<br>
Seagate 15k 600GB<br>
CentOS 7.2<br>
Gluster 3.7.8</p>
<p dir="ltr">Appreciate your help.</p>
</blockquote>
<div><br>
</div>
<div>Most reports of this I have
seen end up being settings
related. Post gluster volume
info. Below is what I have seen as
most common recommended settings.</div>
<div>I'd hazard a guess you may have
some the read ahead cache or
prefetch on.</div>
<div><br>
</div>
<div>quick-read=off</div>
<div>read-ahead=off</div>
<div>io-cache=off</div>
<div>stat-prefetch=off</div>
<div>eager-lock=enable</div>
<div>remote-dio=on</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<p dir="ltr"><span><font color="#888888"><br>
Mahdi Adnan<br>
System Admin<br>
<br>
</font></span></p>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote></div><br></div></div>