<div dir="ltr"><div>Glad the fixes worked for you. Thanks for that update!<br><br></div>-Krutika<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 2, 2016 at 7:31 PM, David Gossage <span dir="ltr"><<a href="mailto:dgossage@carouselchecks.com" target="_blank">dgossage@carouselchecks.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">So far both dd commands that failed previously worked fine on 3.7.14<div><br></div><div>Once I deleted old content from test volume it mounted to oVirt via storage add when previously it would error out. I am now creating a test VM with default disk caching settings (pretty sure oVirt is defaulting to none rather than writeback/through). So far all shards are being created properly.</div><div><br></div><div>Load is sky rocketing but I have all 3 gluster bricks running off 1 hard drive on test box so I would expect horrible io/load issues with that.</div><div><br></div><div>Very promising so far. Thank you developers for your help in working through this.</div><div><br></div><div>Once I have the VM installed and running will test for a few days and make sure it doesn't have any freeze or locking issues then will roll this out to working cluster.</div><div><br></div><div><br></div></div><div class="gmail_extra"><span class=""><br clear="all"><div><div data-smartmail="gmail_signature"><div dir="ltr"><span><font color="#888888"><span style="color:rgb(0,0,0)"><b><i>David Gossage</i></b></span><font><i><span style="color:rgb(51,51,51)"><b><br>
</b></span></i></font></font></span><div><span><font color="#888888"><font><i><span style="color:rgb(51,51,51)"></span></i><font size="1"><b style="color:rgb(153,0,0)">Carousel Checks Inc.<span style="color:rgb(204,204,204)"> | System Administrator</span></b></font></font><font style="color:rgb(153,153,153)"><font size="1"><br>
</font></font><font><font size="1"><span style="color:rgb(51,51,51)"><b style="color:rgb(153,153,153)">Office</b><span style="color:rgb(153,153,153)"> <a value="+17086132426">708.613.2284<font color="#888888"><font size="1"><br></font></font></a></span></span></font></font></font></span></div></div></div></div>
<br></span><div><div class="h5"><div class="gmail_quote">On Wed, Jul 27, 2016 at 8:37 AM, David Gossage <span dir="ltr"><<a href="mailto:dgossage@carouselchecks.com" target="_blank">dgossage@carouselchecks.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div><div><div dir="ltr"><div>On Tue, Jul 26, 2016 at 9:38 PM, Krutika Dhananjay <span dir="ltr"><<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>></span> wrote:<br></div></div></div></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Yes please, could you file a bug against glusterfs for this issue?<br></div></div></blockquote><div><br></div><div><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1360785" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1360785</a></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br><br></div>-Krutika</div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jul 27, 2016 at 1:39 AM, David Gossage <span dir="ltr"><<a href="mailto:dgossage@carouselchecks.com" target="_blank">dgossage@carouselchecks.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Has a bug report been filed for this issue or should l I create one with the logs and results provided so far?</div><div class="gmail_extra"><span><br clear="all"><div><div><div dir="ltr"><span><font color="#888888"><span style="color:rgb(0,0,0)"><b><i>David Gossage</i></b></span><font><i><span style="color:rgb(51,51,51)"><b><br>
</b></span></i></font></font></span><div><span><font color="#888888"><font><i><span style="color:rgb(51,51,51)"></span></i><font size="1"><b style="color:rgb(153,0,0)">Carousel Checks Inc.<span style="color:rgb(204,204,204)"> | System Administrator</span></b></font></font><font style="color:rgb(153,153,153)"><font size="1"><br>
</font></font><font><font size="1"><span style="color:rgb(51,51,51)"><b style="color:rgb(153,153,153)">Office</b><span style="color:rgb(153,153,153)"> <a value="+17086132426">708.613.2284<font color="#888888"><font size="1"><br></font></font></a></span></span></font></font></font></span></div></div></div></div>
<br></span><div><div><div class="gmail_quote">On Fri, Jul 22, 2016 at 12:53 PM, David Gossage <span dir="ltr"><<a href="mailto:dgossage@carouselchecks.com" target="_blank">dgossage@carouselchecks.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><br clear="all"><div><div><div dir="ltr"><br></div></div></div>
<br><div class="gmail_quote">On Fri, Jul 22, 2016 at 9:37 AM, Vijay Bellur <span dir="ltr"><<a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">On Fri, Jul 22, 2016 at 10:03 AM, Samuli Heinonen <<a href="mailto:samppah@neutraali.net" target="_blank">samppah@neutraali.net</a>> wrote:<br>
> Here is a quick way how to test this:<br>
> GlusterFS 3.7.13 volume with default settings with brick on ZFS dataset. gluster-test1 is server and gluster-test2 is client mounting with FUSE.<br>
><br>
> Writing file with oflag=direct is not ok:<br>
> [root@gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct count=1 bs=1024000<br>
> dd: failed to open ‘file’: Invalid argument<br>
><br>
> Enable network.remote-dio on Gluster Volume:<br>
> [root@gluster-test1 gluster]# gluster volume set gluster network.remote-dio enable<br>
> volume set: success<br>
><br>
> Writing small file with oflag=direct is ok:<br>
> [root@gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct count=1 bs=1024000<br>
> 1+0 records in<br>
> 1+0 records out<br>
> 1024000 bytes (1.0 MB) copied, 0.0103793 s, 98.7 MB/s<br>
><br>
> Writing bigger file with oflag=direct is ok:<br>
> [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct count=100 bs=1M<br>
> 100+0 records in<br>
> 100+0 records out<br>
> 104857600 bytes (105 MB) copied, 1.10583 s, 94.8 MB/s<br>
><br>
> Enable Sharding on Gluster Volume:<br>
> [root@gluster-test1 gluster]# gluster volume set gluster features.shard enable<br>
> volume set: success<br>
><br>
> Writing small file with oflag=direct is ok:<br>
> [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct count=1 bs=1M<br>
> 1+0 records in<br>
> 1+0 records out<br>
> 1048576 bytes (1.0 MB) copied, 0.0115247 s, 91.0 MB/s<br>
><br>
> Writing bigger file with oflag=direct is not ok:<br>
> [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct count=100 bs=1M<br>
> dd: error writing ‘file3’: Operation not permitted<br>
> dd: closing output file ‘file3’: Operation not permitted<br>
><br>
<br>
<br>
Thank you for these tests! would it be possible to share the brick and<br>
client logs?<br></blockquote><div><br></div><div>Not sure if his tests are same as my setup but here is what I end up with</div><div><br></div><div><div>Volume Name: glustershard</div><div>Type: Replicate</div><div>Volume ID: 0cc4efb6-3836-4caa-b24a-b3afb6e407c3</div><div>Status: Started</div><div>Number of Bricks: 1 x 3 = 3</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: 192.168.71.10:/gluster1/shard1/1</div><div>Brick2: 192.168.71.11:/gluster1/shard2/1</div><div>Brick3: 192.168.71.12:/gluster1/shard3/1</div><div>Options Reconfigured:</div><div>features.shard-block-size: 64MB</div><div>features.shard: on</div><div>server.allow-insecure: on</div><div>storage.owner-uid: 36</div><div>storage.owner-gid: 36</div><div>cluster.server-quorum-type: server</div><div>cluster.quorum-type: auto</div><div>network.remote-dio: enable</div><div>cluster.eager-lock: enable</div><div>performance.stat-prefetch: off</div><div>performance.io-cache: off</div><div>performance.quick-read: off</div><div>cluster.self-heal-window-size: 1024</div><div>cluster.background-self-heal-count: 16</div><div>nfs.enable-ino32: off</div><div>nfs.addr-namelookup: off</div><div>nfs.disable: on</div><div>performance.read-ahead: off</div><div>performance.readdir-ahead: on</div></div><div><br></div><div><br></div><div><br></div><div> dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/<a href="http://192.168.71.11" target="_blank">192.168.71.11</a>\:_glustershard/ oflag=direct count=100 bs=1M</div><div>81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/ __DIRECT_IO_TEST__ .trashcan/ </div><div>[root@ccengine2 ~]# dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/<a href="http://192.168.71.11" target="_blank">192.168.71.11</a>\:_glustershard/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test oflag=direct count=100 bs=1M</div><div>dd: error writing ‘/rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test’: Operation not permitted</div><div><br></div><div>creates the 64M file in expected location then the shard is 0</div><div><br></div><div><div># file: gluster1/shard1/1/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test</div><div>security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000</div><div>trusted.afr.dirty=0x000000000000000000000000</div><div>trusted.bit-rot.version=0x0200000000000000579231f3000e16e7</div><div>trusted.gfid=0xec6de302b35f427985639ca3e25d9df0</div><div>trusted.glusterfs.shard.block-size=0x0000000004000000</div><div>trusted.glusterfs.shard.file-size=0x0000000004000000000000000000000000000000000000010000000000000000</div></div><div><br></div><div><br></div><div><div># file: gluster1/shard1/1/.shard/ec6de302-b35f-4279-8563-9ca3e25d9df0.1</div><div>security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000</div><div>trusted.afr.dirty=0x000000000000000000000000</div><div>trusted.gfid=0x2bfd3cc8a727489b9a0474241548fe80</div></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<br>
Regards,<br>
Vijay<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div><br></div></div>
</blockquote></div><br></div></div></div>
<br>_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br></blockquote></div><br></div>
</blockquote></div><br></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div>