<div dir="ltr">Hello!<div><br></div><div>I experience very slow performance on gluster 3.7.8 on CentOS 7.2 when using small block sizes with dd. </div><div><br></div><div>I run 2 gluster servers with 2 bricks each (each located on separate SSDs) and 2 gluster clients (which will run virtual machines). It's a completely fresh setup, so no load on the systems. The systems are connected via 1 GBit network. iperf3 shows 940 MBit throughput, ping shows less than 1ms latency, so network should be fine.</div><div><br></div><div>The volume is a stripe 2 replica 2 volume. I've only set the volume options for ping-timeout and owner-uid/gid (as required by qemu). The volume is mounted via fuse client. </div><div><br></div><div>He're my stats as seen from a client (370MB test file). 'storage1' is one of the glusterfs servers which also has a fuse mount for these tests:</div><div><br></div><div><br></div><div><font face="monospace, monospace"># read (always done echo 3 > /proc/sys/vm/drop_caches before running tests)</font></div><div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">$ dd if=/glustermount/testfile of=/localdisk/testfile bs=4K</font></div><div><font face="monospace, monospace">69.7 MB/s</font></div></div><div><font face="monospace, monospace"><br></font></div><div><div><div><font face="monospace, monospace">$ dd if=/glustermount/testfile of=/localdisk/testfile bs=8K</font></div><div><font face="monospace, monospace">78.8 MB/s</font></div></div></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">$ scp storage1:/glustermount/testfile /localdisk/testfile</font></div><div><font face="monospace, monospace">83.3 MB/s</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">$ ssh storage1 'dd if=/glustermount/testfile bs=8K' | dd of=/localdisk/testfile bs=8K<br></font></div><div><font face="monospace, monospace">93.3 MB/s</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"># write</font></div><div><font face="monospace, monospace"><br></font></div><div><div><font face="monospace, monospace">$ dd if=/localdisk/testfile of=/glustermount/testfile bs=4K</font></div><div><font face="monospace, monospace">3.8 MB/s</font></div></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">$ dd if=/localdisk/testfile of=/glustermount/testfile bs=8K</font></div><div><font face="monospace, monospace">6.9 MB/s</font></div><div><font face="monospace, monospace"><br></font></div><div><div><font face="monospace, monospace">$ dd if=/localdisk/testfile of=/glustermount/testfile bs=8M</font></div><div><font face="monospace, monospace">58.8 MB/s</font></div></div><div><font face="monospace, monospace"><br></font></div><div><div><font face="monospace, monospace">$ dd if=/localdisk/testfile of=/glustermount/testfile bs=64M</font></div><div><font face="monospace, monospace">58.8 MB/s</font></div></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">$ scp /localdisk/testfile storage1:/localdisk/testfile</font></div><div><font face="monospace, monospace">94 MB/s</font></div><div><font face="monospace, monospace"><br></font></div><div><div><font face="monospace, monospace">$ scp /localdisk/testfile storage1:/glustermount/testfile</font></div><div><font face="monospace, monospace">980 KB/s</font></div></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"># immediately ran after the previous one. what happened?!?!<br></font></div><div><div><font face="monospace, monospace">$ scp /localdisk/testfile storage1:/glustermount/testfile </font></div><div><font face="monospace, monospace">75 MB/s</font></div></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">$ dd if=/localdisk/testfile bs=8K | ssh storage1 'dd of=/localdisk/testfile bs=8K' </font></div><div><font face="monospace, monospace">96.6 MB/s</font></div><div><font face="monospace, monospace"><br></font></div><div><div><font face="monospace, monospace">$ dd if=/localdisk/testfile bs=8K | ssh storage1 'dd of=/glustermount/testfile bs=8K' </font></div><div><font face="monospace, monospace">225 KB/s</font></div></div><div><br></div><div>Is this really as good as it gets with gluster? I've tried the following setups in order to improve write speed for small block sizes:</div><div><br></div><div>- tuning stripe-size. smaller (16KB) and larger (4MB) values</div><div>- 'replica 2' only setup (i.e. no striping)</div><div>- write-behind on/off</div><div>- larger write-behind-window-size </div><div>- larger cache-size</div><div>- more io-threads</div><div><br></div><div>None of these helped or even changed the numbers much. I've also tried a few sysctl tweaks, but still not much change.</div><div><br></div><div>So I'd like to know:</div><div><br></div><div>- is this normal speed/behaviour?</div><div>- what are the reasons for this?</div><div>- can anything be done to improve performance? </div><div>- is running VMs on top of gluster a supported/intended use case at all?</div><div><br></div><div>Thanks a lot,</div><div>Andreas</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br><div><br></div><div><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="font-family:arial;font-size:small"><br></div><div style="font-family:arial;font-size:small"><br></div></div></div></div></div></div></div></div></div>
</div></div></div>