<div dir="ltr">Hi Vijay,<div><br></div><div>If i run the same command directly on the brick...</div><div><br></div><div><div>[root@cpu01 1]# dd if=/dev/zero of=test bs=64k count=4k oflag=dsync</div><div>4096+0 records in</div><div>4096+0 records out</div><div>268435456 bytes (268 MB) copied, 16.8022 s, 16.0 MB/s</div><div>[root@cpu01 1]# pwd</div><div>/bricks/1</div><div>[root@cpu01 1]#</div></div><div><br></div><div><img src="cid:ii_14c9be30e8016695" alt="Inline image 1" width="552" height="121"><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Apr 8, 2015 at 6:44 PM, Vijay Bellur <span dir="ltr"><<a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On 04/08/2015 02:57 PM, Punit Dambiwal wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
Hi,<br>
<br>
I am getting very slow throughput in the glusterfs (dead slow...even<br>
SATA is better) ... i am using all SSD in my environment.....<br>
<br>
I have the following setup :-<br>
A. 4* host machine with Centos 7(Glusterfs 3.6.2 | Distributed<br>
Replicated | replica=2)<br>
B. Each server has 24 SSD as bricks…(Without HW Raid | JBOD)<br>
C. Each server has 2 Additional ssd for OS…<br>
D. Network 2*10G with bonding…(2*E5 CPU and 64GB RAM)<br>
<br>
Note :- Performance/Throughput slower then Normal SATA 7200 RPM…even i<br>
am using all SSD in my ENV..<br>
<br>
Gluster Volume options :-<br>
<br>
+++++++++++++++<br>
Options Reconfigured:<br>
performance.nfs.write-behind-<u></u>window-size: 1024MB<br>
performance.io-thread-count: 32<br>
performance.cache-size: 1024MB<br>
cluster.quorum-type: auto<br>
cluster.server-quorum-type: server<br>
diagnostics.count-fop-hits: on<br>
diagnostics.latency-<u></u>measurement: on<br>
nfs.disable: on<br>
user.cifs: enable<br>
auth.allow: *<br>
performance.quick-read: off<br>
performance.read-ahead: off<br>
performance.io-cache: off<br>
performance.stat-prefetch: off<br>
cluster.eager-lock: enable<br>
network.remote-dio: enable<br>
storage.owner-uid: 36<br>
storage.owner-gid: 36<br>
server.allow-insecure: on<br>
network.ping-timeout: 0<br>
diagnostics.brick-log-level: INFO<br>
+++++++++++++++++++<br>
<br>
Test with SATA and Glusterfs SSD….<br>
———————<br>
Dell EQL (SATA disk 7200 RPM)<br>
—-<br>
[root@mirror ~]#<br></div></div><span class="">
4096+0 records in<br>
4096+0 records out<br>
268435456 bytes (268 MB) copied, 20.7763 s, 12.9 MB/s<br>
[root@mirror ~]# dd if=/dev/zero of=test bs=64k count=4k oflag=dsync<br>
4096+0 records in<br>
4096+0 records out<br>
268435456 bytes (268 MB) copied, 23.5947 s, 11.4 MB/s<br>
<br>
GlsuterFS SSD<br>
—<br>
[root@sv-VPN1 ~]# dd if=/dev/zero of=test bs=64k count=4k oflag=dsync<br>
4096+0 records in<br>
4096+0 records out<br>
268435456 bytes (268 MB) copied, 66.2572 s, 4.1 MB/s<br>
[root@sv-VPN1 ~]# dd if=/dev/zero of=test bs=64k count=4k oflag=dsync<br>
4096+0 records in<br>
4096+0 records out<br>
268435456 bytes (268 MB) copied, 62.6922 s, 4.3 MB/s<br>
————————<br>
<br>
Please let me know what i should do to improve the performance of my<br>
glusterfs…<br>
</span></blockquote>
<br>
<br>
What is the throughput that you get when you run these commands on the disks directly without gluster in the picture?<br>
<br>
By running dd with dsync you are ensuring that there is no buffering anywhere in the stack and that is the reason why low throughput is being observed.<br>
<br>
-Vijay<span class="HOEnZb"><font color="#888888"><br>
<br>
-Vijay<br>
<br>
</font></span></blockquote></div><br></div>