<div dir="ltr">Just a quick follow up - I had to somehow miss the updated thread count because fio (client side) and brick show appropriate number of threads if I raise count above 16. Seems like minimal thread count is always spawned at least on the server which probably confused me. Unfortunately the connection count to the brick is still the same (2). <div><br></div><div>However the read perfomance is still the same. strace on fio threads shows reads (same on the server side), so the workload is somehow distributed between them, but performance is very different from running multiple jobs (numjobs > 1). Does anyone know how fio resp. gfapi ioengine uses multiple threads?</div><div><br></div><div>The 200MB/s I get on client vs 1.5GB/s if the same test is run locally on the server so far seems to be caused by the network latency, but could the client be advised to open multiple connections (maybe one per thread)? netstat reports 2 connections per fio process and raising numjobs results in more connections and maxes out brick utilization. </div><div><br></div><div>My goal is to max out IO performance of QEMU/KVM guests. So far only approximately 200MB/s are possible with for certain block sizes.</div><div><br></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature">-ps</div></div>
<br><div class="gmail_quote">On Mon, Nov 7, 2016 at 4:47 PM, Pavel Szalbot <span dir="ltr"><<a href="mailto:pavel.szalbot@gmail.com" target="_blank">pavel.szalbot@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi everybody, </div><div><br></div><div>I am trying to benchmark my cluster with fio's gfapi ioengine and evaluate effect of various volume options on performance. I have so far observed following:</div><div><br></div><div>1) *thread* options do not affect performance or thread count - htop always show 2 threads on client, there are always 16 glusterfsd threads on server</div><div>2) running the same test locally (on the brick) shows up to 5x better throughput than over 10GBe (MTU 9000, iperfed, pinged with DF set, no drops on switches or cards, tcpdumped to check network issues)</div><div>3) performance.cache-size value has no effect on performance (32MB or 1GB)</div><div><br></div><div>I would expect raising client threads number leading to more TCP connections, higher disk utilization and throughput. If I run multiple fio jobs (numjobs=8), I am able to saturate the network link. </div><div><br></div><div>Is this normal or I am missing something really badly?</div><div><br></div><div>fio config:</div><div><br></div><div><div>[global]</div><div>name=gfapi test</div><div>create_on_open=1</div><div>volume=test3-vol</div><div>brick=gfs-3.san</div><div>ioengine=gfapi</div><div>direct=1</div><div>bs=256k</div><div>rw=read</div><div>iodepth=1</div><div>numjobs=1</div><div>size=8192m</div><div>loops=1</div><div>refill_buffers=1</div><div>[job1]</div></div><div><br></div>reconfigured volume options:<div><div><div>performance.client-io-threads: on</div><div>performance.cache-size: 1GB</div><div>performance.read-ahead: off</div><div>server.outstanding-rpc-limit: 128</div><div>performance.io-thread-count: 16</div><div>server.event-threads: 16</div><div>client.event-threads: 16</div><div>nfs.disable: on</div><div>transport.address-family: inet</div><div>performance.readdir-ahead: on</div></div><span class="HOEnZb"><font color="#888888"><div><br></div><div><div class="m_2152898283589291878gmail_signature">-ps</div></div>
</font></span></div></div>
</blockquote></div><br></div>