<div dir="ltr">Hi Ben,<div><br></div><div>That means if i will not attach the SSD in to brick...even not install glusterfs on the server...it gives me throughput about 300mb/s but once i will install glusterfs and add this ssd in to glusterfs volume it gives me 16 mb/s...</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Apr 10, 2015 at 9:32 PM, Ben Turner <span dir="ltr"><<a href="mailto:bturner@redhat.com" target="_blank">bturner@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">----- Original Message -----<br>
> From: "Punit Dambiwal" <<a href="mailto:hypunit@gmail.com">hypunit@gmail.com</a>><br>
</span><div><div class="h5">> To: "Ben Turner" <<a href="mailto:bturner@redhat.com">bturner@redhat.com</a>><br>
> Cc: "Vijay Bellur" <<a href="mailto:vbellur@redhat.com">vbellur@redhat.com</a>>, <a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br>
> Sent: Thursday, April 9, 2015 9:36:59 PM<br>
> Subject: Re: [Gluster-users] Glusterfs performance tweaks<br>
><br>
> Hi Ben,<br>
><br>
> But without glusterfs if i run the same command with dsync on the same<br>
> ssd...it gives me good throughput...all setup (CPU,RAM,Network are same)<br>
> the only difference is no glusterfs...<br>
><br>
> [root@cpu09 mnt]# dd if=/dev/zero of=test bs=64k count=4k oflag=dsync<br>
> 4096+0 records in<br>
> 4096+0 records out<br>
> 268435456 bytes (268 MB) copied, 0.935646 s, 287 MB/s<br>
> [root@cpu09 mnt]#<br>
><br>
> [image: Inline image 1]<br>
><br>
> But on the top of the glusterfs it gives too slow performance....i run the<br>
> ssd trim every night to clean the garbage collection...i think there is<br>
> something need to do from gluster or OS side to improve the<br>
> performance....otherwise no use to use the ALL SSD with gluster because<br>
> with all SSD you will get the performance slower then SATA....<br>
><br>
><br>
><br>
> On Fri, Apr 10, 2015 at 2:12 AM, Ben Turner <<a href="mailto:bturner@redhat.com">bturner@redhat.com</a>> wrote:<br>
><br>
> > ----- Original Message -----<br>
> > > From: "Punit Dambiwal" <<a href="mailto:hypunit@gmail.com">hypunit@gmail.com</a>><br>
> > > To: "Vijay Bellur" <<a href="mailto:vbellur@redhat.com">vbellur@redhat.com</a>><br>
> > > Cc: <a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br>
> > > Sent: Wednesday, April 8, 2015 9:55:38 PM<br>
> > > Subject: Re: [Gluster-users] Glusterfs performance tweaks<br>
> > ><br>
> > > Hi Vijay,<br>
> > ><br>
> > > If i run the same command directly on the brick...<br>
<br>
</div></div>What does this mean then? Running directly on the brick to me means running directly on the SSD. The command below is the same thing as above, what changed?<br>
<span class="HOEnZb"><font color="#888888"><br>
-b<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
> > ><br>
> > > [root@cpu01 1]# dd if=/dev/zero of=test bs=64k count=4k oflag=dsync<br>
> > > 4096+0 records in<br>
> > > 4096+0 records out<br>
> > > 268435456 bytes (268 MB) copied, 16.8022 s, 16.0 MB/s<br>
> > > [root@cpu01 1]# pwd<br>
> > > /bricks/1<br>
> > > [root@cpu01 1]#<br>
> > ><br>
> ><br>
> > This is your problem. Gluster is only as fast as its slowest piece, and<br>
> > here your storage is the bottleneck. Being that you get 16 MB to the brick<br>
> > and 12 to gluster that works out to about 25% overhead which is what I<br>
> > would expect with a single thread, single brick, single client scenario.<br>
> > This may have something to do with the way SSDs write? On my SSD at my<br>
> > desk I only get 11.4 MB / sec when I run that DD command:<br>
> ><br>
> > # dd if=/dev/zero of=test bs=64k count=4k oflag=dsync<br>
> > 4096+0 records in<br>
> > 4096+0 records out<br>
> > 268435456 bytes (268 MB) copied, 23.065 s, 11.4 MB/s<br>
> ><br>
> > My thought is that maybe using dsync is forcing the SSD to clean the data<br>
> > or something else before writing to it:<br>
> ><br>
> > <a href="http://www.blog.solidstatediskshop.com/2012/how-does-an-ssd-write/" target="_blank">http://www.blog.solidstatediskshop.com/2012/how-does-an-ssd-write/</a><br>
> ><br>
> > Do your drives support fstrim? It may be worth it to trim before you run<br>
> > and see what results you get. Other than tuning the SSD / OS to perform<br>
> > better on the back end there isn't much we can do from the gluster<br>
> > perspective on that specific DD w/ the dsync flag.<br>
> ><br>
> > -b<br>
> ><br>
> > ><br>
> > > On Wed, Apr 8, 2015 at 6:44 PM, Vijay Bellur < <a href="mailto:vbellur@redhat.com">vbellur@redhat.com</a> ><br>
> > wrote:<br>
> > ><br>
> > ><br>
> > ><br>
> > > On 04/08/2015 02:57 PM, Punit Dambiwal wrote:<br>
> > ><br>
> > ><br>
> > ><br>
> > > Hi,<br>
> > ><br>
> > > I am getting very slow throughput in the glusterfs (dead slow...even<br>
> > > SATA is better) ... i am using all SSD in my environment.....<br>
> > ><br>
> > > I have the following setup :-<br>
> > > A. 4* host machine with Centos 7(Glusterfs 3.6.2 | Distributed<br>
> > > Replicated | replica=2)<br>
> > > B. Each server has 24 SSD as bricks…(Without HW Raid | JBOD)<br>
> > > C. Each server has 2 Additional ssd for OS…<br>
> > > D. Network 2*10G with bonding…(2*E5 CPU and 64GB RAM)<br>
> > ><br>
> > > Note :- Performance/Throughput slower then Normal SATA 7200 RPM…even i<br>
> > > am using all SSD in my ENV..<br>
> > ><br>
> > > Gluster Volume options :-<br>
> > ><br>
> > > +++++++++++++++<br>
> > > Options Reconfigured:<br>
> > > performance.nfs.write-behind- window-size: 1024MB<br>
> > > performance.io-thread-count: 32<br>
> > > performance.cache-size: 1024MB<br>
> > > cluster.quorum-type: auto<br>
> > > cluster.server-quorum-type: server<br>
> > > diagnostics.count-fop-hits: on<br>
> > > diagnostics.latency- measurement: on<br>
> > > nfs.disable: on<br>
> > > user.cifs: enable<br>
> > > auth.allow: *<br>
> > > performance.quick-read: off<br>
> > > performance.read-ahead: off<br>
> > > performance.io-cache: off<br>
> > > performance.stat-prefetch: off<br>
> > > cluster.eager-lock: enable<br>
> > > network.remote-dio: enable<br>
> > > storage.owner-uid: 36<br>
> > > storage.owner-gid: 36<br>
> > > server.allow-insecure: on<br>
> > > network.ping-timeout: 0<br>
> > > diagnostics.brick-log-level: INFO<br>
> > > +++++++++++++++++++<br>
> > ><br>
> > > Test with SATA and Glusterfs SSD….<br>
> > > ———————<br>
> > > Dell EQL (SATA disk 7200 RPM)<br>
> > > —-<br>
> > > [root@mirror ~]#<br>
> > > 4096+0 records in<br>
> > > 4096+0 records out<br>
> > > 268435456 bytes (268 MB) copied, 20.7763 s, 12.9 MB/s<br>
> > > [root@mirror ~]# dd if=/dev/zero of=test bs=64k count=4k oflag=dsync<br>
> > > 4096+0 records in<br>
> > > 4096+0 records out<br>
> > > 268435456 bytes (268 MB) copied, 23.5947 s, 11.4 MB/s<br>
> > ><br>
> > > GlsuterFS SSD<br>
> > > —<br>
> > > [root@sv-VPN1 ~]# dd if=/dev/zero of=test bs=64k count=4k oflag=dsync<br>
> > > 4096+0 records in<br>
> > > 4096+0 records out<br>
> > > 268435456 bytes (268 MB) copied, 66.2572 s, 4.1 MB/s<br>
> > > [root@sv-VPN1 ~]# dd if=/dev/zero of=test bs=64k count=4k oflag=dsync<br>
> > > 4096+0 records in<br>
> > > 4096+0 records out<br>
> > > 268435456 bytes (268 MB) copied, 62.6922 s, 4.3 MB/s<br>
> > > ————————<br>
> > ><br>
> > > Please let me know what i should do to improve the performance of my<br>
> > > glusterfs…<br>
> > ><br>
> > ><br>
> > > What is the throughput that you get when you run these commands on the<br>
> > disks<br>
> > > directly without gluster in the picture?<br>
> > ><br>
> > > By running dd with dsync you are ensuring that there is no buffering<br>
> > anywhere<br>
> > > in the stack and that is the reason why low throughput is being observed.<br>
> > ><br>
> > > -Vijay<br>
> > ><br>
> > > -Vijay<br>
> > ><br>
> > ><br>
> > ><br>
> > > _______________________________________________<br>
> > > Gluster-users mailing list<br>
> > > <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> > > <a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
> ><br>
><br>
</div></div></blockquote></div><br></div>