<div dir="ltr"><div>I did other tests , but the performance does not improve . </div><div>Please , can someone give me a good hint , I do not know what else to do . </div><div>Thank you very much.</div><div><br></div><div>####################################################################</div><div><br></div><div>(how I build the volume) </div><div>root@store-1:~# gluster volume create cinder-distributed-replicated-striped stripe 2 replica 2 transport tcp 172.16.155.21:/data/cdrs1 172.16.155.21:/data/cdrs2 172.16.155.22:/data/cdrs3 172.16.155.22:/data/cdrs4 172.16.155.23:/data/cdrs5 172.16.155.23:/data/cdrs6 172.16.155.24:/data/cdrs7 172.16.155.24:/data/cdrs8</div><div><br></div><div>(and the test repeated three times) </div><div>root@nodo-3:/mnt# time sh -c "dd if=/dev/zero of=/mnt/cinder-distributed-replicated-striped/test.tmp bs=4k count=2000000 && sync"</div><div><br></div><div>8192000000 byte (8,2 GB) copiati, 112,437 s, 72,9 MB/s</div><div><br></div><div>8192000000 byte (8,2 GB) copiati, 95,0725 s, 86,2 MB/s</div><div><br></div><div>8192000000 byte (8,2 GB) copiati, 92,0918 s, 89,0 MB/s</div><div><br></div><div><br></div><div>####################################################################</div><div><br></div><div>(how I build the volume) </div><div>root@store-1:~# gluster volume create cinder-replica replica 2 172.16.155.21:/data/nova1 172.16.155.22:/data/nova2 172.16.155.22:/data/nova1 172.16.155.23:/data/nova2 172.16.155.23:/data/nova1 172.16.155.24:/data/nova2 172.16.155.24:/data/nova1 172.16.155.21:/data/nova2</div><div><br></div><div>(and the test repeated three times) </div><div>root@nodo-3:/mnt# time sh -c "dd if=/dev/zero of=/mnt/cinder-replica/test.tmp bs=4k count=2000000 && sync"</div><div><br></div><div>8192000000 byte (8,2 GB) copiati, 86,7926 s, 94,4 MB/s</div><div><br></div><div>8192000000 byte (8,2 GB) copiati, 93,6442 s, 87,5 MB/s</div><div><br></div><div>8192000000 byte (8,2 GB) copiati, 90,5922 s, 90,4 MB/s</div><div><br></div><div><br></div><div>####################################################################</div><div><br></div><div>(how I build the volume) </div><div>root@store-1:~# gluster volume create cinder-replicated-striped stripe 2 replica 2 transport tcp 172.16.155.21:/crs1 172.16.155.22:/crs2 172.16.155.23:/crs3 172.16.155.24:/crs4</div><div><br></div><div>(and the test repeated three times) </div><div>root@nodo-3:/mnt# time sh -c "dd if=/dev/zero of=/mnt/cinder-replicated-striped/test.tmp bs=4k count=2000000 && sync"</div><div><br></div><div>8192000000 byte (8,2 GB) copiati, 111,163 s, 73,7 MB/s</div><div><br></div><div>8192000000 byte (8,2 GB) copiati, 93,9147 s, 87,2 MB/s</div><div><br></div><div>8192000000 byte (8,2 GB) copiati, 88,9809 s, 92,1 MB/s</div><div><br></div><div>####################################################################</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">2015-04-11 18:39 GMT+02:00 marianna cattani <span dir="ltr"><<a href="mailto:marianna.cattani@gmail.com" target="_blank">marianna.cattani@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>I have a similar behaviour.</div><div><br></div><div>All systems are identical</div><div><br></div><div>This is my setup:</div><div><br></div><div><br></div><div>############################################################</div><div>root@store-1:~# uname -a</div><div>Linux store-1 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt7-1 (2015-03-01) x86_64 GNU/Linux</div><div>root@store-1:~# glusterfsd --version</div><div>glusterfs 3.6.2 built on Jan 21 2015 14:23:41</div><div>Repository revision: git://<a href="http://git.gluster.com/glusterfs.git" target="_blank">git.gluster.com/glusterfs.git</a></div><div>Copyright (c) 2006-2013 Red Hat, Inc. <<a href="http://www.redhat.com/" target="_blank">http://www.redhat.com/</a>></div><div>GlusterFS comes with ABSOLUTELY NO WARRANTY.</div><div>It is licensed to you under your choice of the GNU Lesser</div><div>General Public License, version 3 or any later version (LGPLv3</div><div>or later), or the GNU General Public License, version 2 (GPLv2),</div><div>in all cases as published by the Free Software Foundation.</div><div>root@store-1:~# gluster peer status</div><div>Number of Peers: 3</div><div><br></div><div>Hostname: 172.16.155.23</div><div>Uuid: 094c8ec4-f472-4d32-b3cb-705458cca949</div><div>State: Peer in Cluster (Connected)</div><div><br></div><div>Hostname: 172.16.155.22</div><div>Uuid: 00042351-cc66-4528-ac66-75ed7ea01b44</div><div>State: Peer in Cluster (Connected)</div><div><br></div><div>Hostname: 172.16.155.24</div><div>Uuid: 113307a5-79c8-47c3-a6fa-8b725a56f807</div><div>State: Peer in Cluster (Connected)</div><div>root@store-1:~# gluster volume info</div><div> </div><div>Volume Name: cinder-disperse</div><div>Type: Disperse</div><div>Volume ID: a70cf0c4-9320-4e7a-8e6b-a9b6242d151e</div><div>Status: Started</div><div>Number of Bricks: 1 x (3 + 1) = 4</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: 172.16.155.21:/data/cidi</div><div>Brick2: 172.16.155.22:/data/cidi</div><div>Brick3: 172.16.155.23:/data/cidi</div><div>Brick4: 172.16.155.24:/data/cidi</div><div>Options Reconfigured:</div><div>storage.owner-gid: 114</div><div>storage.owner-uid: 108</div><div>performance.cache-size: 24576MB</div><div>performance.io-thread-count: 64</div><div>cluster.data-self-heal-algorithm: full</div><div>cluster.server-quorum-type: none</div><div>cluster.quorum-type: none</div><div>network.remote-dio: disable</div><div>cluster.eager-lock: enable</div><div>performance.stat-prefetch: off</div><div>performance.io-cache: off</div><div>performance.read-ahead: off</div><div>performance.quick-read: off</div><div>server.allow-insecure: on</div><div>root@store-1:~# iperf -c 172.16.155.22</div><div>------------------------------------------------------------</div><div>Client connecting to 172.16.155.22, TCP port 5001</div><div>TCP window size: 85.0 KByte (default)</div><div>------------------------------------------------------------</div><div>[ 3] local 172.16.155.21 port 46843 connected with 172.16.155.22 port 5001</div><div>[ ID] Interval Transfer Bandwidth</div><div>[ 3] 0.0-10.0 sec 7.92 GBytes 6.80 Gbits/sec</div><div>root@store-1:~# iperf -c 172.16.155.23</div><div>------------------------------------------------------------</div><div>Client connecting to 172.16.155.23, TCP port 5001</div><div>TCP window size: 85.0 KByte (default)</div><div>------------------------------------------------------------</div><div>[ 3] local 172.16.155.21 port 59368 connected with 172.16.155.23 port 5001</div><div>[ ID] Interval Transfer Bandwidth</div><div>[ 3] 0.0-10.0 sec 7.94 GBytes 6.82 Gbits/sec</div><div>root@store-1:~# iperf -c 172.16.155.24</div><div>------------------------------------------------------------</div><div>Client connecting to 172.16.155.24, TCP port 5001</div><div>TCP window size: 85.0 KByte (default)</div><div>------------------------------------------------------------</div><div>[ 3] local 172.16.155.21 port 33656 connected with 172.16.155.24 port 5001</div><div>[ ID] Interval Transfer Bandwidth</div><div>[ 3] 0.0-10.0 sec 7.94 GBytes 6.82 Gbits/sec</div><div>root@store-1:~# </div><div>root@store-1:~# time sh -c "dd if=/dev/zero of=/data/cidi/test.tmp bs=4k count=2000000 && sync"</div><div>2000000+0 record dentro</div><div>2000000+0 record fuori</div><div><a href="tel:8192000000" value="+18192000000" target="_blank">8192000000</a> byte (8,2 GB) copiati, 8,58092 s, 955 MB/s</div><div><br></div><div>real<span style="white-space:pre-wrap"> </span>0m20.749s</div><div>user<span style="white-space:pre-wrap"> </span>0m0.276s</div><div>sys<span style="white-space:pre-wrap"> </span>0m6.304s</div><div>root@store-1:/data# </div><div><br></div><div>############################################################</div><div><br></div><div>root@store-2:~# time sh -c "dd if=/dev/zero of=/data/cidi/test.tmp bs=4k count=2000000 && sync"</div><div>2000000+0 record dentro</div><div>2000000+0 record fuori</div><div><a href="tel:8192000000" value="+18192000000" target="_blank">8192000000</a> byte (8,2 GB) copiati, 9,60108 s, 853 MB/s</div><div><br></div><div>real<span style="white-space:pre-wrap"> </span>0m20.751s</div><div>user<span style="white-space:pre-wrap"> </span>0m0.264s</div><div>sys<span style="white-space:pre-wrap"> </span>0m6.128s</div><div><br></div><div>############################################################</div><div><br></div><div>root@store-3:~# time sh -c "dd if=/dev/zero of=/data/cidi/test.tmp bs=4k count=2000000 && sync"</div><div>2000000+0 record dentro</div><div>2000000+0 record fuori</div><div><a href="tel:8192000000" value="+18192000000" target="_blank">8192000000</a> byte (8,2 GB) copiati, 9,27243 s, 883 MB/s</div><div><br></div><div>real<span style="white-space:pre-wrap"> </span>0m20.979s</div><div>user<span style="white-space:pre-wrap"> </span>0m0.284s</div><div>sys<span style="white-space:pre-wrap"> </span>0m6.168s</div><div><br></div><div>############################################################</div><div><br></div><div>root@store-4:~# time sh -c "dd if=/dev/zero of=/data/cidi/test.tmp bs=4k count=2000000 && sync"</div><div>2000000+0 record dentro</div><div>2000000+0 record fuori</div><div><a href="tel:8192000000" value="+18192000000" target="_blank">8192000000</a> byte (8,2 GB) copiati, 9,12428 s, 898 MB/s</div><div><br></div><div>real<span style="white-space:pre-wrap"> </span>0m20.067s</div><div>user<span style="white-space:pre-wrap"> </span>0m0.244s</div><div>sys<span style="white-space:pre-wrap"> </span>0m6.304s</div><div><br></div><div>############################################################</div><div><br></div><div>I try to mount from another node</div><div><br></div><div>############################################################</div><div><br></div><div>root@nodo-3:~# iperf -c 172.16.155.21</div><div>------------------------------------------------------------</div><div>Client connecting to 172.16.155.21, TCP port 5001</div><div>TCP window size: 85.0 KByte (default)</div><div>------------------------------------------------------------</div><div>[ 3] local 172.16.155.13 port 59299 connected with 172.16.155.21 port 5001</div><div>[ ID] Interval Transfer Bandwidth</div><div>[ 3] 0.0-10.0 sec 7.08 GBytes 6.08 Gbits/sec</div><div>root@nodo-3:~# mount -t glusterfs 172.16.155.21:/cinder-disperse /mnt/cinder-disperse</div><div>root@nodo-3:~# time sh -c "dd if=/dev/zero of=/mnt/cinder-disperse/test.tmp bs=4k count=2000000 && sync"</div><div>2000000+0 record dentro</div><div>2000000+0 record fuori</div><div><a href="tel:8192000000" value="+18192000000" target="_blank">8192000000</a> byte (8,2 GB) copiati, 72,6074 s, 113 MB/s</div><div><br></div><div>real<span style="white-space:pre-wrap"> </span>1m13.842s</div><div>user<span style="white-space:pre-wrap"> </span>0m0.584s</div><div>sys<span style="white-space:pre-wrap"> </span>0m19.868s</div><div>root@nodo-3:~# </div><div><br></div><div>############################################################</div><div><br></div><div>Is this the write speed that I can expect ?</div><div>Can I improve it in some way ?</div><div><br></div><div>Thanks a lot. </div><span class="HOEnZb"><font color="#888888"><div><br></div><div>M.</div></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">2015-04-11 6:19 GMT+02:00 Prasun Gera <span dir="ltr"><<a href="mailto:prasun.gera@gmail.com" target="_blank">prasun.gera@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">There is something that's not clear in what you are describing. Gluster doesn't come into play until you access your data through the gulsterfs mount. You can even stop your gluster volume and stop the glusterfs daemon to confirm that it is not really interfering with your writes to the brick in any way. What you are describing sounds like an issue with the way you have partitioned your drive or set up the filesystem, which is probably xfs in case of glusterfs if you are using defaults. Are you comparing the same file system in both your cases ? </div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Apr 10, 2015 at 11:45 AM, Punit Dambiwal <span dir="ltr"><<a href="mailto:hypunit@gmail.com" target="_blank">hypunit@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Ben,<div><br></div><div>That means if i will not attach the SSD in to brick...even not install glusterfs on the server...it gives me throughput about 300mb/s but once i will install glusterfs and add this ssd in to glusterfs volume it gives me 16 mb/s...</div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Apr 10, 2015 at 9:32 PM, Ben Turner <span dir="ltr"><<a href="mailto:bturner@redhat.com" target="_blank">bturner@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>----- Original Message -----<br>
> From: "Punit Dambiwal" <<a href="mailto:hypunit@gmail.com" target="_blank">hypunit@gmail.com</a>><br>
</span><div><div>> To: "Ben Turner" <<a href="mailto:bturner@redhat.com" target="_blank">bturner@redhat.com</a>><br>
> Cc: "Vijay Bellur" <<a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>>, <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
> Sent: Thursday, April 9, 2015 9:36:59 PM<br>
> Subject: Re: [Gluster-users] Glusterfs performance tweaks<br>
><br>
> Hi Ben,<br>
><br>
> But without glusterfs if i run the same command with dsync on the same<br>
> ssd...it gives me good throughput...all setup (CPU,RAM,Network are same)<br>
> the only difference is no glusterfs...<br>
><br>
> [root@cpu09 mnt]# dd if=/dev/zero of=test bs=64k count=4k oflag=dsync<br>
> 4096+0 records in<br>
> 4096+0 records out<br>
> 268435456 bytes (268 MB) copied, 0.935646 s, 287 MB/s<br>
> [root@cpu09 mnt]#<br>
><br>
> [image: Inline image 1]<br>
><br>
> But on the top of the glusterfs it gives too slow performance....i run the<br>
> ssd trim every night to clean the garbage collection...i think there is<br>
> something need to do from gluster or OS side to improve the<br>
> performance....otherwise no use to use the ALL SSD with gluster because<br>
> with all SSD you will get the performance slower then SATA....<br>
><br>
><br>
><br>
> On Fri, Apr 10, 2015 at 2:12 AM, Ben Turner <<a href="mailto:bturner@redhat.com" target="_blank">bturner@redhat.com</a>> wrote:<br>
><br>
> > ----- Original Message -----<br>
> > > From: "Punit Dambiwal" <<a href="mailto:hypunit@gmail.com" target="_blank">hypunit@gmail.com</a>><br>
> > > To: "Vijay Bellur" <<a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>><br>
> > > Cc: <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
> > > Sent: Wednesday, April 8, 2015 9:55:38 PM<br>
> > > Subject: Re: [Gluster-users] Glusterfs performance tweaks<br>
> > ><br>
> > > Hi Vijay,<br>
> > ><br>
> > > If i run the same command directly on the brick...<br>
<br>
</div></div>What does this mean then? Running directly on the brick to me means running directly on the SSD. The command below is the same thing as above, what changed?<br>
<span><font color="#888888"><br>
-b<br>
</font></span><div><div><br>
> > ><br>
> > > [root@cpu01 1]# dd if=/dev/zero of=test bs=64k count=4k oflag=dsync<br>
> > > 4096+0 records in<br>
> > > 4096+0 records out<br>
> > > 268435456 bytes (268 MB) copied, 16.8022 s, 16.0 MB/s<br>
> > > [root@cpu01 1]# pwd<br>
> > > /bricks/1<br>
> > > [root@cpu01 1]#<br>
> > ><br>
> ><br>
> > This is your problem. Gluster is only as fast as its slowest piece, and<br>
> > here your storage is the bottleneck. Being that you get 16 MB to the brick<br>
> > and 12 to gluster that works out to about 25% overhead which is what I<br>
> > would expect with a single thread, single brick, single client scenario.<br>
> > This may have something to do with the way SSDs write? On my SSD at my<br>
> > desk I only get 11.4 MB / sec when I run that DD command:<br>
> ><br>
> > # dd if=/dev/zero of=test bs=64k count=4k oflag=dsync<br>
> > 4096+0 records in<br>
> > 4096+0 records out<br>
> > 268435456 bytes (268 MB) copied, 23.065 s, 11.4 MB/s<br>
> ><br>
> > My thought is that maybe using dsync is forcing the SSD to clean the data<br>
> > or something else before writing to it:<br>
> ><br>
> > <a href="http://www.blog.solidstatediskshop.com/2012/how-does-an-ssd-write/" target="_blank">http://www.blog.solidstatediskshop.com/2012/how-does-an-ssd-write/</a><br>
> ><br>
> > Do your drives support fstrim? It may be worth it to trim before you run<br>
> > and see what results you get. Other than tuning the SSD / OS to perform<br>
> > better on the back end there isn't much we can do from the gluster<br>
> > perspective on that specific DD w/ the dsync flag.<br>
> ><br>
> > -b<br>
> ><br>
> > ><br>
> > > On Wed, Apr 8, 2015 at 6:44 PM, Vijay Bellur < <a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a> ><br>
> > wrote:<br>
> > ><br>
> > ><br>
> > ><br>
> > > On 04/08/2015 02:57 PM, Punit Dambiwal wrote:<br>
> > ><br>
> > ><br>
> > ><br>
> > > Hi,<br>
> > ><br>
> > > I am getting very slow throughput in the glusterfs (dead slow...even<br>
> > > SATA is better) ... i am using all SSD in my environment.....<br>
> > ><br>
> > > I have the following setup :-<br>
> > > A. 4* host machine with Centos 7(Glusterfs 3.6.2 | Distributed<br>
> > > Replicated | replica=2)<br>
> > > B. Each server has 24 SSD as bricks…(Without HW Raid | JBOD)<br>
> > > C. Each server has 2 Additional ssd for OS…<br>
> > > D. Network 2*10G with bonding…(2*E5 CPU and 64GB RAM)<br>
> > ><br>
> > > Note :- Performance/Throughput slower then Normal SATA 7200 RPM…even i<br>
> > > am using all SSD in my ENV..<br>
> > ><br>
> > > Gluster Volume options :-<br>
> > ><br>
> > > +++++++++++++++<br>
> > > Options Reconfigured:<br>
> > > performance.nfs.write-behind- window-size: 1024MB<br>
> > > performance.io-thread-count: 32<br>
> > > performance.cache-size: 1024MB<br>
> > > cluster.quorum-type: auto<br>
> > > cluster.server-quorum-type: server<br>
> > > diagnostics.count-fop-hits: on<br>
> > > diagnostics.latency- measurement: on<br>
> > > nfs.disable: on<br>
> > > user.cifs: enable<br>
> > > auth.allow: *<br>
> > > performance.quick-read: off<br>
> > > performance.read-ahead: off<br>
> > > performance.io-cache: off<br>
> > > performance.stat-prefetch: off<br>
> > > cluster.eager-lock: enable<br>
> > > network.remote-dio: enable<br>
> > > storage.owner-uid: 36<br>
> > > storage.owner-gid: 36<br>
> > > server.allow-insecure: on<br>
> > > network.ping-timeout: 0<br>
> > > diagnostics.brick-log-level: INFO<br>
> > > +++++++++++++++++++<br>
> > ><br>
> > > Test with SATA and Glusterfs SSD….<br>
> > > ———————<br>
> > > Dell EQL (SATA disk 7200 RPM)<br>
> > > —-<br>
> > > [root@mirror ~]#<br>
> > > 4096+0 records in<br>
> > > 4096+0 records out<br>
> > > 268435456 bytes (268 MB) copied, 20.7763 s, 12.9 MB/s<br>
> > > [root@mirror ~]# dd if=/dev/zero of=test bs=64k count=4k oflag=dsync<br>
> > > 4096+0 records in<br>
> > > 4096+0 records out<br>
> > > 268435456 bytes (268 MB) copied, 23.5947 s, 11.4 MB/s<br>
> > ><br>
> > > GlsuterFS SSD<br>
> > > —<br>
> > > [root@sv-VPN1 ~]# dd if=/dev/zero of=test bs=64k count=4k oflag=dsync<br>
> > > 4096+0 records in<br>
> > > 4096+0 records out<br>
> > > 268435456 bytes (268 MB) copied, 66.2572 s, 4.1 MB/s<br>
> > > [root@sv-VPN1 ~]# dd if=/dev/zero of=test bs=64k count=4k oflag=dsync<br>
> > > 4096+0 records in<br>
> > > 4096+0 records out<br>
> > > 268435456 bytes (268 MB) copied, 62.6922 s, 4.3 MB/s<br>
> > > ————————<br>
> > ><br>
> > > Please let me know what i should do to improve the performance of my<br>
> > > glusterfs…<br>
> > ><br>
> > ><br>
> > > What is the throughput that you get when you run these commands on the<br>
> > disks<br>
> > > directly without gluster in the picture?<br>
> > ><br>
> > > By running dd with dsync you are ensuring that there is no buffering<br>
> > anywhere<br>
> > > in the stack and that is the reason why low throughput is being observed.<br>
> > ><br>
> > > -Vijay<br>
> > ><br>
> > > -Vijay<br>
> > ><br>
> > ><br>
> > ><br>
> > > _______________________________________________<br>
> > > Gluster-users mailing list<br>
> > > <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
> > > <a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
> ><br>
><br>
</div></div></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>