<div dir="ltr"><div>Hi everybody, </div><div><br></div><div>I am trying to benchmark my cluster with fio's gfapi ioengine and evaluate effect of various volume options on performance. I have so far observed following:</div><div><br></div><div>1) *thread* options do not affect performance or thread count - htop always show 2 threads on client, there are always 16 glusterfsd threads on server</div><div>2) running the same test locally (on the brick) shows up to 5x better throughput than over 10GBe (MTU 9000, iperfed, pinged with DF set, no drops on switches or cards, tcpdumped to check network issues)</div><div>3) performance.cache-size value has no effect on performance (32MB or 1GB)</div><div><br></div><div>I would expect raising client threads number leading to more TCP connections, higher disk utilization and throughput. If I run multiple fio jobs (numjobs=8), I am able to saturate the network link. </div><div><br></div><div>Is this normal or I am missing something really badly?</div><div><br></div><div>fio config:</div><div><br></div><div><div>[global]</div><div>name=gfapi test</div><div>create_on_open=1</div><div>volume=test3-vol</div><div>brick=gfs-3.san</div><div>ioengine=gfapi</div><div>direct=1</div><div>bs=256k</div><div>rw=read</div><div>iodepth=1</div><div>numjobs=1</div><div>size=8192m</div><div>loops=1</div><div>refill_buffers=1</div><div>[job1]</div></div><div><br></div>reconfigured volume options:<div><div><div>performance.client-io-threads: on</div><div>performance.cache-size: 1GB</div><div>performance.read-ahead: off</div><div>server.outstanding-rpc-limit: 128</div><div>performance.io-thread-count: 16</div><div>server.event-threads: 16</div><div>client.event-threads: 16</div><div>nfs.disable: on</div><div>transport.address-family: inet</div><div>performance.readdir-ahead: on</div></div><div><br></div><div><div class="gmail_signature">-ps</div></div>
</div></div>