<div dir="ltr">Hi, <div><br></div><div>In my case, block size of underlying filesystem is 4K. </div><div><br></div><div>So, I tested again using bs=768K (multiples of 12KB).</div><div>Your advice worked! </div><div><br></div><div><div>10gnode1:disp on /mnt/gluster type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)</div><div>root@node1:~/gluster_ec# dd if=/dev/zero of=/mnt/gluster/test.io.1 bs=768k count=1024</div><div>1024+0 records in</div><div>1024+0 records out</div><div>805306368 bytes (805 MB) copied, 9.5404 s, 84.4 MB/s</div><div>root@node1:~/gluster_ec#</div></div><div><br></div><div>Thank you for your advice.</div><div><br></div><div>Regards.</div><div><br></div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">2015-09-20 6:54 GMT+09:00 Xavier Hernandez <span dir="ltr"><<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi,<br>
<br>
what filesystem are you using on the bricks and what block size is it using ?<br>
<br>
Direct I/O requires strict alignment of buffer offsets and block sizes. The disperse volume writes fragments multiples of 512 bytes in the bricks. If the underlying filesystem requires an alignment different than 512 bytes, it could cause the problem you are seeing.<br>
<br>
For example if the required alignment of the underlying filesystem is 4KB, you will need to write to the disperse volume in multiples of 12KB (you are using 3 data bricks).<br>
<br>
Xavi<span class=""><br>
<br>
El 18/09/2015 a las 3:00, 이태화 escribió:<br>
</span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span class="">
Hi, I have tested Disperse volume.<br>
<br>
so I perform a test using "dd"<br>
<br>
<br>
I'm using 4 nodes. and nodes are connected through 10g network.<br>
<br></span><span class="">
*Volume creation & mount command:*<br></span><span class="">
root@node1:~# gluster vol create disp_vol disperse 4 redundancy 1<br>
10gnode1:/node/dispv 10gnode2:/node/dispv 10gnode3:/node/dispv<br>
10gnode4:/node/dispv<br>
This configuration is not optimal on most workloads. Do you want to use<br>
it ? (y/n) y<br>
volume create: disp_vol: success: please start the volume to access data<br>
root@node1:~# gluster vol start disp_vol<br>
volume start: disp_vol: success<br>
root@node1:~# mount -t glusterfs 10gnode1:/disp_vol /mnt/gluster<br>
<br></span><span class="">
*Test command & error message:*<br></span><span class="">
root@node1:~# dd if=/dev/zero of=/mnt/gluster/test.io.1 oflag=direct<br>
bs=1024k count=1024<br>
dd: error writing ‘/mnt/gluster/test.io.1’: Invalid argument<br>
dd: closing output file ‘/mnt/gluster/test.io.1’: Invalid argument<br>
<br>
<br>
To recognize why, I saw logs. but I didn't know reasons.<br>
<br></span>
*Warning messages in /var/log/glusterfs/mnt-gluster.log*<div><div class="h5"><br>
[2015-09-18 00:39:38.460320] W [MSGID: 114031]<br>
[client-rpc-fops.c:904:client3_3_writev_cbk] 0-disp_vol-client-2: remote<br>
operation failed [Invalid argument]<br>
[2015-09-18 00:39:38.460320] W [MSGID: 114031]<br>
[client-rpc-fops.c:904:client3_3_writev_cbk] 0-disp_vol-client-1: remote<br>
operation failed [Invalid argument]<br>
[2015-09-18 00:39:38.464081] W [MSGID: 114031]<br>
[client-rpc-fops.c:904:client3_3_writev_cbk] 0-disp_vol-client-3: remote<br>
operation failed [Invalid argument]<br>
[2015-09-18 00:39:38.464297] W [MSGID: 114031]<br>
[client-rpc-fops.c:904:client3_3_writev_cbk] 0-disp_vol-client-0: remote<br>
operation failed [Invalid argument]<br>
[2015-09-18 00:39:38.464459] W [fuse-bridge.c:2240:fuse_writev_cbk]<br>
0-glusterfs-fuse: 18: WRITE => -1 (Invalid argument)<br>
[2015-09-18 00:39:38.464643] W [fuse-bridge.c:2240:fuse_writev_cbk]<br>
0-glusterfs-fuse: 19: WRITE => -1 (Invalid argument)<br>
[2015-09-18 00:39:38.464801] W [fuse-bridge.c:2240:fuse_writev_cbk]<br>
0-glusterfs-fuse: 20: WRITE => -1 (Invalid argument)<br>
[2015-09-18 00:39:38.464958] W [fuse-bridge.c:2240:fuse_writev_cbk]<br>
0-glusterfs-fuse: 21: WRITE => -1 (Invalid argument)<br>
[2015-09-18 00:39:38.465114] W [fuse-bridge.c:2240:fuse_writev_cbk]<br>
0-glusterfs-fuse: 22: WRITE => -1 (Invalid argument)<br>
[2015-09-18 00:39:38.465278] W [fuse-bridge.c:2240:fuse_writev_cbk]<br>
0-glusterfs-fuse: 23: WRITE => -1 (Invalid argument)<br>
[2015-09-18 00:39:38.465433] W [fuse-bridge.c:2240:fuse_writev_cbk]<br>
0-glusterfs-fuse: 24: WRITE => -1 (Invalid argument)<br>
[2015-09-18 00:39:38.465585] W [fuse-bridge.c:2240:fuse_writev_cbk]<br>
0-glusterfs-fuse: 25: WRITE => -1 (Invalid argument)<br>
[2015-09-18 00:39:38.465529] W [MSGID: 114031]<br>
[client-rpc-fops.c:904:client3_3_writev_cbk] 0-disp_vol-client-0: remote<br>
operation failed [Invalid argument]<br>
[2015-09-18 00:39:38.465863] W [MSGID: 114031]<br>
[client-rpc-fops.c:904:client3_3_writev_cbk] 0-disp_vol-client-1: remote<br>
operation failed [Invalid argument]<br>
[2015-09-18 00:39:38.465890] W [fuse-bridge.c:690:fuse_truncate_cbk]<br>
0-glusterfs-fuse: 26: FTRUNCATE() ERR => -1 (Invalid argument)<br>
[2015-09-18 00:39:38.466192] W [MSGID: 114031]<br>
[client-rpc-fops.c:904:client3_3_writev_cbk] 0-disp_vol-client-2: remote<br>
operation failed [Invalid argument]<br>
[2015-09-18 00:39:38.466353] W [MSGID: 114031]<br>
[client-rpc-fops.c:904:client3_3_writev_cbk] 0-disp_vol-client-3: remote<br>
operation failed [Invalid argument]<br>
[2015-09-18 00:39:38.467464] W [MSGID: 114031]<br>
[client-rpc-fops.c:904:client3_3_writev_cbk] 0-disp_vol-client-0: remote<br>
operation failed [Invalid argument]<br>
[2015-09-18 00:39:38.467848] W [MSGID: 114031]<br>
[client-rpc-fops.c:904:client3_3_writev_cbk] 0-disp_vol-client-1: remote<br>
operation failed [Invalid argument]<br>
---- repeated ----<br>
[2015-09-18 00:39:38.473542] W [fuse-bridge.c:1230:fuse_err_cbk]<br>
0-glusterfs-fuse: 27: FLUSH() ERR => -1 (Invalid argument)<br>
<br>
<br>
I need your help. anything about this is fine.<br>
<br>
If you need some environment informations, I'm ready to inform that.<br>
<br>
Thanks<br>
<br>
<br></div></div><span class="">
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
<br>
</span></blockquote>
</blockquote></div><br></div></div>