<div dir="ltr">Hi Ben,<div><br></div><div>Yes...i am using 2*10G (Bonding LACP)... <br><div><br></div><div><p class="MsoNormal"><span style="color:rgb(31,73,125)">[root@cpu02 ~]# /usr/bin/iperf3
-c 10.10.0.10</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Connecting to host 10.10.0.10,
port 5201</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">[ 4] local 10.10.0.11 port
45135 connected to 10.10.0.10 port 5201</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">[ ID]
Interval
Transfer Bandwidth
Retr Cwnd</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">[ 4]
0.00-1.00 sec 1.15 GBytes 9.85 Gbits/sec
13 1.30 MBytes</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">[ 4]
1.00-2.00 sec 1.15 GBytes 9.91
Gbits/sec 0 1.89 MBytes</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">[ 4]
2.00-3.00 sec 1.15 GBytes 9.90
Gbits/sec 0 2.33 MBytes</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">[ 4]
3.00-4.00 sec 1.15 GBytes 9.89
Gbits/sec 1 2.41 MBytes</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">[ 4]
4.00-5.00 sec 1.15 GBytes 9.90
Gbits/sec 0 2.42 MBytes</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">[ 4]
5.00-6.00 sec 1.15 GBytes 9.90
Gbits/sec 0 2.53 MBytes</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">[ 4]
6.00-7.00 sec 1.15 GBytes 9.90
Gbits/sec 1 2.53 MBytes</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">[ 4]
7.00-8.00 sec 1.15 GBytes 9.90
Gbits/sec 0 2.68 MBytes</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">[ 4] 8.00-9.00
sec 1.15 GBytes 9.90 Gbits/sec
0 2.76 MBytes</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">[ 4]
9.00-10.00 sec 1.15 GBytes 9.90 Gbits/sec
0 2.88 MBytes</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">- - - - - - - - - - - - - - - -
- - - - - - - - -</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">[ ID]
Interval
Transfer Bandwidth
Retr</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">[ 4]
0.00-10.00 sec 11.5 GBytes 9.89 Gbits/sec
15
sender</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">[ 4]
0.00-10.00 sec 11.5 GBytes 9.89
Gbits/sec
receiver</span></p><p class="MsoNormal"><span style="color:rgb(31,73,125)"><br></span></p><p class="MsoNormal"><span style="color:rgb(31,73,125)"><br></span></p><p class="MsoNormal"><font color="#1f497d">[root@cpu01 ~]# time `dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/<a href="http://10.10.0.14">10.10.0.14</a>\:_ds01/myfile bs=1024k count=1000; sync`</font></p><p class="MsoNormal"><font color="#1f497d">1000+0 records in</font></p><p class="MsoNormal"><font color="#1f497d">1000+0 records out</font></p><p class="MsoNormal"><font color="#1f497d">1048576000 bytes (1.0 GB) copied, 2.70556 s, 388 MB/s</font></p><p class="MsoNormal"><font color="#1f497d"><br></font></p><p class="MsoNormal"><font color="#1f497d">real 0m2.815s</font></p><p class="MsoNormal"><font color="#1f497d">user 0m0.002s</font></p><p class="MsoNormal"></p><p class="MsoNormal"><font color="#1f497d">sys 0m0.822s</font></p><p class="MsoNormal"><font color="#1f497d"><br></font></p><p class="MsoNormal"><font color="#1f497d">[root@cpu01 ~]# time `dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/<a href="http://10.10.0.14">10.10.0.14</a>\:_ds01/e732a82f-bae9-4368-8b98-dedc1c3814de/images/myfile bs=1024k count=1000; sync`</font></p><p class="MsoNormal"><font color="#1f497d">1000+0 records in</font></p><p class="MsoNormal"><font color="#1f497d">1000+0 records out</font></p><p class="MsoNormal"><font color="#1f497d">1048576000 bytes (1.0 GB) copied, 1.93056 s, 543 MB/s</font></p><p class="MsoNormal"><font color="#1f497d"><br></font></p><p class="MsoNormal"><font color="#1f497d">real 0m2.077s</font></p><p class="MsoNormal"><font color="#1f497d">user 0m0.002s</font></p><p class="MsoNormal"><font color="#1f497d">sys 0m0.795s</font></p><p class="MsoNormal"><font color="#1f497d"></font></p><p class="MsoNormal"><font color="#1f497d">[root@cpu01 ~]#</font></p><p class="MsoNormal"><font color="#1f497d"><br></font></p><p class="MsoNormal"><span style="color:rgb(31,73,125)">Ban@ how i will check those things :- </span></p><p class="MsoNormal"><span style="color:rgb(31,73,125)"><br></span></p><p class="MsoNormal"><span style="font-size:12.8000001907349px">A couple things to check with your SSDs:</span><br style="font-size:12.8000001907349px"><br style="font-size:12.8000001907349px"><span style="font-size:12.8000001907349px">-Scheduler {noop or deadline }</span><br style="font-size:12.8000001907349px"><span style="font-size:12.8000001907349px">-No read ahead!</span><br style="font-size:12.8000001907349px"><span style="font-size:12.8000001907349px">-No RAID!</span><br style="font-size:12.8000001907349px"><span style="font-size:12.8000001907349px">-Make sure the kernel seems them as SSDs</span><span style="color:rgb(31,73,125)"><br></span></p><p class="MsoNormal"><span style="font-size:12.8000001907349px"><br></span></p><p class="MsoNormal"><span style="font-size:12.8000001907349px">Thanks,</span></p><p class="MsoNormal"><span style="font-size:12.8000001907349px">Punit</span></p>
</div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Apr 9, 2015 at 2:55 AM, Ben Turner <span dir="ltr"><<a href="mailto:bturner@redhat.com" target="_blank">bturner@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">----- Original Message -----<br>
> From: "Vijay Bellur" <<a href="mailto:vbellur@redhat.com">vbellur@redhat.com</a>><br>
> To: "Punit Dambiwal" <<a href="mailto:hypunit@gmail.com">hypunit@gmail.com</a>>, <a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br>
> Sent: Wednesday, April 8, 2015 6:44:42 AM<br>
> Subject: Re: [Gluster-users] Glusterfs performance tweaks<br>
><br>
> On 04/08/2015 02:57 PM, Punit Dambiwal wrote:<br>
> > Hi,<br>
> ><br>
> > I am getting very slow throughput in the glusterfs (dead slow...even<br>
> > SATA is better) ... i am using all SSD in my environment.....<br>
> ><br>
> > I have the following setup :-<br>
> > A. 4* host machine with Centos 7(Glusterfs 3.6.2 | Distributed<br>
> > Replicated | replica=2)<br>
> > B. Each server has 24 SSD as bricks…(Without HW Raid | JBOD)<br>
> > C. Each server has 2 Additional ssd for OS…<br>
> > D. Network 2*10G with bonding…(2*E5 CPU and 64GB RAM)<br>
> ><br>
> > Note :- Performance/Throughput slower then Normal SATA 7200 RPM…even i<br>
> > am using all SSD in my ENV..<br>
> ><br>
> > Gluster Volume options :-<br>
> ><br>
> > +++++++++++++++<br>
> > Options Reconfigured:<br>
> > performance.nfs.write-behind-window-size: 1024MB<br>
> > performance.io-thread-count: 32<br>
> > performance.cache-size: 1024MB<br>
> > cluster.quorum-type: auto<br>
> > cluster.server-quorum-type: server<br>
> > diagnostics.count-fop-hits: on<br>
> > diagnostics.latency-measurement: on<br>
> > nfs.disable: on<br>
> > user.cifs: enable<br>
> > auth.allow: *<br>
> > performance.quick-read: off<br>
> > performance.read-ahead: off<br>
> > performance.io-cache: off<br>
> > performance.stat-prefetch: off<br>
> > cluster.eager-lock: enable<br>
> > network.remote-dio: enable<br>
> > storage.owner-uid: 36<br>
> > storage.owner-gid: 36<br>
> > server.allow-insecure: on<br>
> > network.ping-timeout: 0<br>
> > diagnostics.brick-log-level: INFO<br>
> > +++++++++++++++++++<br>
> ><br>
> > Test with SATA and Glusterfs SSD….<br>
> > ———————<br>
> > Dell EQL (SATA disk 7200 RPM)<br>
> > —-<br>
> > [root@mirror ~]#<br>
> > 4096+0 records in<br>
> > 4096+0 records out<br>
> > 268435456 bytes (268 MB) copied, 20.7763 s, 12.9 MB/s<br>
> > [root@mirror ~]# dd if=/dev/zero of=test bs=64k count=4k oflag=dsync<br>
> > 4096+0 records in<br>
> > 4096+0 records out<br>
> > 268435456 bytes (268 MB) copied, 23.5947 s, 11.4 MB/s<br>
> ><br>
> > GlsuterFS SSD<br>
> > —<br>
> > [root@sv-VPN1 ~]# dd if=/dev/zero of=test bs=64k count=4k oflag=dsync<br>
> > 4096+0 records in<br>
> > 4096+0 records out<br>
> > 268435456 bytes (268 MB) copied, 66.2572 s, 4.1 MB/s<br>
> > [root@sv-VPN1 ~]# dd if=/dev/zero of=test bs=64k count=4k oflag=dsync<br>
> > 4096+0 records in<br>
> > 4096+0 records out<br>
> > 268435456 bytes (268 MB) copied, 62.6922 s, 4.3 MB/s<br>
> > ————————<br>
> ><br>
> > Please let me know what i should do to improve the performance of my<br>
> > glusterfs…<br>
><br>
><br>
> What is the throughput that you get when you run these commands on the<br>
> disks directly without gluster in the picture?<br>
><br>
> By running dd with dsync you are ensuring that there is no buffering<br>
> anywhere in the stack and that is the reason why low throughput is being<br>
> observed.<br>
<br>
</div></div>This is slow for the env you described. Are you sure you are using your 10G NICs? What do you see with iperf between the client and server? In my env with 12 spinning disks in a RAID 6 + single 10G NIC I get:<br>
<br>
[root@gqac025 gluster-mount]# dd if=/dev/zero of=test bs=64k count=4k oflag=dsync<br>
<span class="">4096+0 records in<br>
4096+0 records out<br>
</span>268435456 bytes (268 MB) copied, 9.88752 s, 27.1 MB/s<br>
<br>
A couple things to check with your SSDs:<br>
<br>
-Scheduler {noop or deadline }<br>
-No read ahead!<br>
-No RAID!<br>
-Make sure the kernel seems them as SSDs<br>
<br>
As Vijay said you will see WAY better throughput if you get rid of the dsync flag. Maybe try something like:<br>
<br>
$ time `dd if=/dev/zero of=/gluster-mount/myfile bs=1024k count=1000; sync`<br>
<br>
That will give you an idea of what it takes to write to RAM then sync the dirty pages to disk.<br>
<br>
-b<br>
<br>
> -Vijay<br>
><br>
> -Vijay<br>
><br>
> _______________________________________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div><br></div>