<div dir="ltr">I run alot of random IO tests with gluster and it has really come a long way in the 3.7 release. What version are you running on? I have a couple of suggestions:<br><div><br></div><div>-Run on 3.7 if you are not already.</div><div>-Run the IOZone test you are running on the back end without gluster to verify that your HW meets your perf needs. SSDs really scream with random IO if your current HW won't meet your needs.</div><div>-When you are running tests watch top -H on both clients and servers, look for any threads hitting 100%</div><div>-If you see hot threads bump up the server.event-threads and/or client.event-threads from the default of 2</div><div><br></div><div>HTH!</div><div><br></div><div>-b</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jul 23, 2015 at 3:04 AM, Susant Palai <span dir="ltr"><<a href="mailto:spalai@redhat.com" target="_blank">spalai@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">++CCing gluster-devel to have more eye on this problem.<br>
<br>
Susant<br>
<br>
----- Original Message -----<br>
> From: "Subrata Ghosh" <<a href="mailto:subrata.ghosh@ericsson.com">subrata.ghosh@ericsson.com</a>><br>
> To: "Susant Palai <<a href="mailto:spalai@redhat.com">spalai@redhat.com</a>> (<a href="mailto:spalai@redhat.com">spalai@redhat.com</a>)" <<a href="mailto:spalai@redhat.com">spalai@redhat.com</a>>, "Vijay Bellur <<a href="mailto:vbellur@redhat.com">vbellur@redhat.com</a>><br>
> (<a href="mailto:vbellur@redhat.com">vbellur@redhat.com</a>)" <<a href="mailto:vbellur@redhat.com">vbellur@redhat.com</a>><br>
> Cc: "Subrata Ghosh" <<a href="mailto:subrata.ghosh@ericsson.com">subrata.ghosh@ericsson.com</a>><br>
> Sent: Sunday, 19 July, 2015 7:57:28 PM<br>
> Subject: Iusses with Random read/write<br>
><br>
> Hi Vijay/Prashant,<br>
><br>
> How you are you :).<br>
><br>
> We need your immediate help / suggestion to meet our random I/IO<br>
> performance metrics.<br>
> Currently we have performance issues with random read/write - our basic<br>
> requirement 20 MB/sec for random I/O.<br>
><br>
> We tried with both "iozone" and "fio", received almost same ( random I/O)<br>
> performance which is not meeting our fundamental I/IO requirements.<br>
><br>
> Our use case is as below.<br>
><br>
> "Application running on different cards Writes/Reads (random) continuous<br>
> files to the volume comprising with storage belonging from different cards<br>
> in the distributed system, where replica presence across cards and<br>
> applications are using non-local storages."<br>
> We have verified and identified the bottleneck mostly on Gluster Client side<br>
> inside the application, however gluster server to server I/O speed looks<br>
> enough good. Performance tuning on gluster server side would not expected to<br>
> help.<br>
><br>
> We also cross verified checked using NFS client we are getting far better<br>
> performance, but we cannot use NFS client /libgfapi because of use case<br>
> limitation ( brick failures cases etc..)<br>
><br>
> Please throw some lights or thoughts to improve gluster client to achieve ><br>
> 20 MB/Secs<br>
><br>
> Observations:<br>
><br>
> Fio:<br>
><br>
><br>
> lease find the test results of Random write & read in 2 APPs scenarios.<br>
><br>
> Scenario<br>
><br>
> APP_1<br>
><br>
> APP_2<br>
><br>
> File size<br>
><br>
> No of AMC's<br>
><br>
> Random-Write<br>
><br>
> 3.06 MB/s<br>
><br>
> 3.02 MB/s<br>
><br>
> 100 MB<br>
><br>
> 4<br>
><br>
> Random-Read<br>
><br>
> 8.1 MB/s<br>
><br>
> 8.4 MB/s<br>
><br>
> 100 MB<br>
><br>
> 4<br>
><br>
><br>
><br>
> Iozone:<br>
><br>
> ./iozone -R -l 1 -u 1 -r 4k -s 2G -F /home/cdr/f1 | tee -a<br>
> /tmp/iozone_results.txt &<br>
><br>
><br>
> APP 1<br>
><br>
> APP2<br>
><br>
> File Size : 2GB<br>
><br>
> File size : 2GB<br>
><br>
> Record size = 4 Kbytes<br>
><br>
> Record size = 4 Kbytes<br>
><br>
> Output is in Kbytes/sec<br>
><br>
> Output is in Kbytes/sec<br>
><br>
><br>
><br>
><br>
> Initial write 41061.78<br>
><br>
> Initial write 41167.36<br>
><br>
><br>
> Rewrite 40395.64<br>
><br>
> Rewrite 40810.41<br>
><br>
><br>
> Read 262685.69<br>
><br>
> Read 269644.62<br>
><br>
><br>
> Re-read 263751.66<br>
><br>
> Re-read 270760.62<br>
><br>
><br>
> Reverse Read 27715.72<br>
><br>
> Reverse Read 28604.22<br>
><br>
><br>
> Stride read 83776.44<br>
><br>
> Stride read 84347.88<br>
><br>
><br>
> Random read 16239.74 (15.8 MB/s )<br>
><br>
> Random read 15815.94 (15.4 MB/s )<br>
><br>
><br>
> Mixed workload 16260.95<br>
><br>
> Mixed workload 15787.55<br>
><br>
><br>
> Random write 3356.57 (3.3 MB/s )<br>
><br>
> Random write 3365.17 ( 3.3 MB/s)<br>
><br>
><br>
> Pwrite 40914.55<br>
><br>
> Pwrite 40692.34<br>
><br>
><br>
> Pread 260613.83<br>
><br>
> Pread 269850.59<br>
><br>
><br>
> Fwrite 40412.40<br>
><br>
> Fwrite 40369.78<br>
><br>
><br>
> Fread 261506.61<br>
><br>
> Fread 267142.41<br>
><br>
><br>
><br>
> Some of the info on performance testing is at<br>
> <a href="http://www.gluster.org/community/documentation/index.php/Performance_Testing" rel="noreferrer" target="_blank">http://www.gluster.org/community/documentation/index.php/Performance_Testing</a><br>
> Also pls check iozone limitations listed there.<br>
><br>
> "WARNING: random I/O testing in iozone is very restricted by iozone<br>
> constraint that it must randomly read then randomly write the entire file!<br>
> This is not what we want - instead it should randomly read/write for some<br>
> fraction of file size or time duration, allowing us to spread out more on<br>
> the disk while not waiting too long for test to finish. This is why fio<br>
> (below) is the preferred test tool for random I/O workloads."<br>
><br>
><br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
</blockquote></div><br></div>