<div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Mar 13, 2016 at 11:07 AM, Mahdi Adnan <span dir="ltr"><<a href="mailto:mahdi.adnan@earthlinktele.com" target="_blank">mahdi.adnan@earthlinktele.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
My HBAs are LSISAS1068E, and the filesystem is XFS.<br>
I tried EXT4 and it did not help.<br>
I have created a stripted volume in one server with two bricks, same
issue.<br>
and i tried a replicated volume with just "sharding enabled" same
issue, as soon as i disable the sharding it works just fine, niether
sharding nor striping works for me.<br>
i did follow up with some of threads in the mailing list and tried
some of the fixes that worked with the others, none worked for me.
:(<br></div></blockquote><div><br></div><div>Is it possible the LSI has write-cache enabled? </div><div><br></div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF">
<br>
<div>On 03/13/2016 06:54 PM, David Gossage
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra">
<div>
<div>
<div dir="ltr">
<div><span><font color="#888888"><font><font size="1"><span style="color:rgb(51,51,51)"><span style="color:rgb(153,153,153)"><a value="+17086132426"><font color="#888888"><font size="1"><br>
</font></font></a></span></span></font></font></font></span></div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On Sun, Mar 13, 2016 at 8:16 AM,
Mahdi Adnan <span dir="ltr"><<a href="mailto:mahdi.adnan@earthlinktele.com" target="_blank">mahdi.adnan@earthlinktele.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> Okay so i have
enabled shard in my test volume and it did not help,
stupidly enough, i have enabled it in a production
volume "Distributed-Replicate" and it currpted half of
my VMs.<br>
I have updated Gluster to the latest and nothing seems
to be changed in my situation.<br>
below the info of my volume;<br>
</div>
</blockquote>
<div><br>
</div>
<div>I was pointing at the settings in that email as an
example for corruption fixing. I wouldn't recommend
enabling sharding if you haven't gotten the base working
yet on that cluster. What HBA's are you using and what is
layout of filesystem for bricks? </div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> <br>
Number of Bricks: 3 x 2 = 6<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: gfs001:/bricks/b001/vmware<br>
Brick2: gfs002:/bricks/b004/vmware<br>
Brick3: gfs001:/bricks/b002/vmware<br>
Brick4: gfs002:/bricks/b005/vmware<br>
Brick5: gfs001:/bricks/b003/vmware<br>
Brick6: gfs002:/bricks/b006/vmware<br>
Options Reconfigured:<br>
performance.strict-write-ordering: on<br>
cluster.server-quorum-type: server<br>
cluster.quorum-type: auto<br>
network.remote-dio: enable<br>
performance.stat-prefetch: disable<br>
performance.io-cache: off<br>
performance.read-ahead: off<br>
performance.quick-read: off<br>
cluster.eager-lock: enable<br>
features.shard-block-size: 16MB<br>
features.shard: on<br>
performance.readdir-ahead: off<br>
<br>
<br>
<div>On 03/12/2016 08:11 PM, David Gossage wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sat, Mar 12, 2016 at
10:21 AM, Mahdi Adnan <span dir="ltr"><<a href="mailto:mahdi.adnan@earthlinktele.com" target="_blank"></a><a href="mailto:mahdi.adnan@earthlinktele.com" target="_blank">mahdi.adnan@earthlinktele.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> Both
servers have HBA no RAIDs and i can setup a
replicated or dispensers without any issues.<br>
Logs are clean and when i tried to migrate a
vm and got the error, nothing showed up in
the logs.<br>
i tried mounting the volume into my laptop
and it mounted fine but, if i use dd to
create a data file it just hang and i cant
cancel it, and i cant unmount it or
anything, i just have to reboot.<br>
The same servers have another volume on
other bricks in a distributed replicas,
works fine.<br>
I have even tried the same setup in a
virtual environment (created two vms and
install gluster and created a replicated
striped) and again same thing, data
corruption.<br>
</div>
</blockquote>
<div><br>
</div>
<div>I'd look through mail archives for a topic
"Shard in Production" I think it's called.
The shard portion may not be relevant but it
does discuss certain settings that had to be
applied with regards to avoiding corruption
with VM's. You may want to try and disable
the performance.readdir-ahead also.</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> <br>
<br>
<div>On 03/12/2016 07:02 PM, David Gossage
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div>
<div>
<div dir="ltr">
<div><span><font color="#888888"><font><font size="1"><span style="color:rgb(51,51,51)"><span style="color:rgb(153,153,153)"><a value="+17086132426"><font color="#888888"><font size="1"><br>
</font></font></a></span></span></font></font></font></span></div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On Sat, Mar
12, 2016 at 9:51 AM, Mahdi Adnan <span dir="ltr"><<a href="mailto:mahdi.adnan@earthlinktele.com" target="_blank"></a><a href="mailto:mahdi.adnan@earthlinktele.com" target="_blank">mahdi.adnan@earthlinktele.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> Thanks David,<br>
<br>
My settings are all defaults, i
have just created the pool and
started it.<br>
I have set the settings as your
recommendation and it seems to
be the same issue;<br>
<br>
Type: Striped-Replicate<br>
Volume ID:
44adfd8c-2ed1-4aa5-b256-d12b64f7fc14<br>
Status: Started<br>
Number of Bricks: 1 x 2 x 2 = 4<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: gfs001:/bricks/t1/s<br>
Brick2: gfs002:/bricks/t1/s<br>
Brick3: gfs001:/bricks/t2/s<br>
Brick4: gfs002:/bricks/t2/s<br>
Options Reconfigured:<br>
performance.stat-prefetch: off<br>
network.remote-dio: on<br>
cluster.eager-lock: enable<br>
performance.io-cache: off<br>
performance.read-ahead: off<br>
performance.quick-read: off<br>
performance.readdir-ahead: on<br>
</div>
</blockquote>
<div> </div>
<div><br>
</div>
<div>Is their a raid controller
perhaps doing any caching? <br>
</div>
<div><br>
</div>
<div>In the gluster logs any errors
being reported during migration
process?</div>
<div>Since they aren't in use yet
have you tested making just
mirrored bricks using different
pairings of servers two at a time
to see if problem follows certain
machine or network ports?</div>
<div><br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> <br>
<br>
<br>
<br>
<br>
<div>On 03/12/2016 03:25 PM,
David Gossage wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Sat, Mar 12, 2016 at
1:55 AM, Mahdi Adnan <span dir="ltr"><<a href="mailto:mahdi.adnan@earthlinktele.com" target="_blank"></a><a href="mailto:mahdi.adnan@earthlinktele.com" target="_blank">mahdi.adnan@earthlinktele.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<p dir="ltr">Dears,</p>
<p dir="ltr">I have
created a replicated
striped volume with
two bricks and two
servers but I can't
use it because when
I mount it in ESXi
and try to migrate a
VM to it, the data
get corrupted.<br>
Is any one have any
idea why is this
happening ?</p>
<p dir="ltr">Dell 2950
x2<br>
Seagate 15k 600GB<br>
CentOS 7.2<br>
Gluster 3.7.8</p>
<p dir="ltr">Appreciate
your help.</p>
</blockquote>
<div><br>
</div>
<div>Most reports of
this I have seen end
up being settings
related. Post gluster
volume info. Below is
what I have seen as
most common
recommended settings.</div>
<div>I'd hazard a guess
you may have some the
read ahead cache or
prefetch on.</div>
<div><br>
</div>
<div>quick-read=off</div>
<div>read-ahead=off</div>
<div>io-cache=off</div>
<div>stat-prefetch=off</div>
<div>eager-lock=enable</div>
<div>remote-dio=on</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<p dir="ltr"><span><font color="#888888"><br>
Mahdi Adnan<br>
System Admin<br>
<br>
</font></span></p>
<br>
_______________________________________________<br>
Gluster-users mailing
list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank"></a><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank"></a><a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote></div><br></div></div>