<div dir="ltr"><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><br></div></div></div>
<br><div class="gmail_quote">On Fri, Jul 22, 2016 at 9:37 AM, Vijay Bellur <span dir="ltr"><<a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">On Fri, Jul 22, 2016 at 10:03 AM, Samuli Heinonen <<a href="mailto:samppah@neutraali.net">samppah@neutraali.net</a>> wrote:<br>
> Here is a quick way how to test this:<br>
> GlusterFS 3.7.13 volume with default settings with brick on ZFS dataset. gluster-test1 is server and gluster-test2 is client mounting with FUSE.<br>
><br>
> Writing file with oflag=direct is not ok:<br>
> [root@gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct count=1 bs=1024000<br>
> dd: failed to open ‘file’: Invalid argument<br>
><br>
> Enable network.remote-dio on Gluster Volume:<br>
> [root@gluster-test1 gluster]# gluster volume set gluster network.remote-dio enable<br>
> volume set: success<br>
><br>
> Writing small file with oflag=direct is ok:<br>
> [root@gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct count=1 bs=1024000<br>
> 1+0 records in<br>
> 1+0 records out<br>
> 1024000 bytes (1.0 MB) copied, 0.0103793 s, 98.7 MB/s<br>
><br>
> Writing bigger file with oflag=direct is ok:<br>
> [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct count=100 bs=1M<br>
> 100+0 records in<br>
> 100+0 records out<br>
> 104857600 bytes (105 MB) copied, 1.10583 s, 94.8 MB/s<br>
><br>
> Enable Sharding on Gluster Volume:<br>
> [root@gluster-test1 gluster]# gluster volume set gluster features.shard enable<br>
> volume set: success<br>
><br>
> Writing small file with oflag=direct is ok:<br>
> [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct count=1 bs=1M<br>
> 1+0 records in<br>
> 1+0 records out<br>
> 1048576 bytes (1.0 MB) copied, 0.0115247 s, 91.0 MB/s<br>
><br>
> Writing bigger file with oflag=direct is not ok:<br>
> [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct count=100 bs=1M<br>
> dd: error writing ‘file3’: Operation not permitted<br>
> dd: closing output file ‘file3’: Operation not permitted<br>
><br>
<br>
<br>
Thank you for these tests! would it be possible to share the brick and<br>
client logs?<br></blockquote><div><br></div><div>Not sure if his tests are same as my setup but here is what I end up with</div><div><br></div><div><div>Volume Name: glustershard</div><div>Type: Replicate</div><div>Volume ID: 0cc4efb6-3836-4caa-b24a-b3afb6e407c3</div><div>Status: Started</div><div>Number of Bricks: 1 x 3 = 3</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: 192.168.71.10:/gluster1/shard1/1</div><div>Brick2: 192.168.71.11:/gluster1/shard2/1</div><div>Brick3: 192.168.71.12:/gluster1/shard3/1</div><div>Options Reconfigured:</div><div>features.shard-block-size: 64MB</div><div>features.shard: on</div><div>server.allow-insecure: on</div><div>storage.owner-uid: 36</div><div>storage.owner-gid: 36</div><div>cluster.server-quorum-type: server</div><div>cluster.quorum-type: auto</div><div>network.remote-dio: enable</div><div>cluster.eager-lock: enable</div><div>performance.stat-prefetch: off</div><div>performance.io-cache: off</div><div>performance.quick-read: off</div><div>cluster.self-heal-window-size: 1024</div><div>cluster.background-self-heal-count: 16</div><div>nfs.enable-ino32: off</div><div>nfs.addr-namelookup: off</div><div>nfs.disable: on</div><div>performance.read-ahead: off</div><div>performance.readdir-ahead: on</div></div><div><br></div><div><br></div><div><br></div><div> dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/<a href="http://192.168.71.11">192.168.71.11</a>\:_glustershard/ oflag=direct count=100 bs=1M</div><div>81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/ __DIRECT_IO_TEST__ .trashcan/ </div><div>[root@ccengine2 ~]# dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/<a href="http://192.168.71.11">192.168.71.11</a>\:_glustershard/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test oflag=direct count=100 bs=1M</div><div>dd: error writing ‘/rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test’: Operation not permitted</div><div><br></div><div>creates the 64M file in expected location then the shard is 0</div><div><br></div><div><div># file: gluster1/shard1/1/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test</div><div>security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000</div><div>trusted.afr.dirty=0x000000000000000000000000</div><div>trusted.bit-rot.version=0x0200000000000000579231f3000e16e7</div><div>trusted.gfid=0xec6de302b35f427985639ca3e25d9df0</div><div>trusted.glusterfs.shard.block-size=0x0000000004000000</div><div>trusted.glusterfs.shard.file-size=0x0000000004000000000000000000000000000000000000010000000000000000</div></div><div><br></div><div><br></div><div><div># file: gluster1/shard1/1/.shard/ec6de302-b35f-4279-8563-9ca3e25d9df0.1</div><div>security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000</div><div>trusted.afr.dirty=0x000000000000000000000000</div><div>trusted.gfid=0x2bfd3cc8a727489b9a0474241548fe80</div></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<br>
Regards,<br>
Vijay<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div><br></div></div>