<div dir="ltr">Hello,<div><br></div><div>is it for small files with lot of them, or mainly big ones?</div><div><br></div><div>When mounting from client, i add <font face="monospace, monospace">direct-io-mode=disable</font> in fstab</div><div><br></div><div>sysctl setting I use (I don't have 10G network, may not be enough numbers) (on clients and servers):</div><div><div><font face="monospace, monospace">vm.swappiness=0</font></div><div><font face="monospace, monospace">net.core.rmem_max=67108864</font></div><div><font face="monospace, monospace">net.core.wmem_max=67108864</font></div><div><font face="monospace, monospace"># increase Linux autotuning TCP buffer limit to 32MB</font></div><div><font face="monospace, monospace">net.ipv4.tcp_rmem=4096 87380 33554432</font></div><div><font face="monospace, monospace">net.ipv4.tcp_wmem=4096 65536 33554432</font></div><div><font face="monospace, monospace"># increase the length of the processor input queue</font></div><div><font face="monospace, monospace">net.core.netdev_max_backlog=30000</font></div><div><font face="monospace, monospace"># recommended default congestion control is htcp</font></div><div><font face="monospace, monospace">net.ipv4.tcp_congestion_control=htcp</font></div></div><div><br></div><div>Options I have reconfigured for small files tuning:</div><div><div><font face="monospace, monospace">performance.cache-size: 1GB</font></div><div><font face="monospace, monospace">nfs.disable: on</font></div><div><font face="monospace, monospace">performance.client-io-threads: on</font></div><div><font face="monospace, monospace">performance.io-cache: on</font></div><div><font face="monospace, monospace">performance.io-thread-count: 16</font></div><div><font face="monospace, monospace">performance.readdir-ahead: enable</font></div><div><font face="monospace, monospace">performance.read-ahead: disable</font></div><div><font face="monospace, monospace">server.allow-insecure: on</font></div><div><font face="monospace, monospace">cluster.lookup-optimize: on</font></div><div><font face="monospace, monospace">client.event-threads: 4</font></div><div><font face="monospace, monospace">server.event-threads: 4</font></div><div><font face="monospace, monospace">cluster.readdir-optimize: on</font></div><div><font face="monospace, monospace">performance.write-behind-window-size: 1MB</font></div></div><div><br></div><div>Again, you should try but even with bigger number based on your hardware specs.</div><div><br></div><div>Did you set MTU to 9000 ? (jumbo frame)</div><div>Did you set in bios power settings to high static mode (name depend on vendor) ?</div><div><br></div><div><br></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature">Cordialement,<br>Mathieu CHATEAU<br><a href="http://www.lotp.fr" target="_blank">http://www.lotp.fr</a></div></div>
<br><div class="gmail_quote">2015-11-25 15:44 GMT+01:00 Rao, Uthra R. (GSFC-672.0)[ADNET SYSTEMS INC] <span dir="ltr"><<a href="mailto:uthra.r.rao@nasa.gov" target="_blank">uthra.r.rao@nasa.gov</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal">Thank you all for taking the time to reply to my email:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Here is some more information on our setup:<u></u><u></u></p>
<p class="MsoNormal">- Number of Nodes <span style="font-family:Wingdings">à</span> 2 Gluster servers and 1 client for testing. After testing we will mount the GlusterFS volume on 3 clients.<u></u><u></u></p>
<p class="MsoNormal">- CPU & RAM on Each Node <span style="font-family:Wingdings">
à</span> 2 CPUs 3.4MHz, 384GB RAM on each Gluster Server<u></u><u></u></p>
<p class="MsoNormal">- What else is running on the nodes <span style="font-family:Wingdings">
à</span> Nothing it is only our data server<u></u><u></u></p>
<p class="MsoNormal">- Number of bricks <span style="font-family:Wingdings">à</span> Two<u></u><u></u></p><span class="">
<p class="MsoNormal">- output of "gluster volume info" & "gluster volume status"<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</span><p class="MsoNormal"><b>Storage server1:<u></u><u></u></b></p>
<p class="MsoNormal"># <b>gluster volume info gtower</b><u></u><u></u></p>
<p class="MsoNormal">Volume Name: gtower<u></u><u></u></p>
<p class="MsoNormal">Type: Replicate<u></u><u></u></p>
<p class="MsoNormal">Volume ID: 838ab806-06d9-45c5-8d88-2a905c167dba<u></u><u></u></p>
<p class="MsoNormal">Status: Started<u></u><u></u></p>
<p class="MsoNormal">Number of Bricks: 1 x 2 = 2<u></u><u></u></p>
<p class="MsoNormal">Transport-type: tcp<u></u><u></u></p>
<p class="MsoNormal">Bricks:<u></u><u></u></p>
<p class="MsoNormal">Brick1: storage1.sci.gsfc.nasa.gov:/tower7/gluster1/brick<u></u><u></u></p>
<p class="MsoNormal">Brick2: storage2.sci.gsfc.nasa.gov:/tower8/gluster2/brick<u></u><u></u></p>
<p class="MsoNormal">Options Reconfigured:<u></u><u></u></p>
<p class="MsoNormal">nfs.export-volumes: off<u></u><u></u></p>
<p class="MsoNormal">nfs.addr-namelookup: off<u></u><u></u></p>
<p class="MsoNormal">performance.readdir-ahead: on<u></u><u></u></p>
<p class="MsoNormal">performance.cache-size: 2GB<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">-----------------------------------------------------------<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><b>Storage server 2:<u></u><u></u></b></p>
<p class="MsoNormal"># <b>gluster volume info gtower</b><u></u><u></u></p>
<p class="MsoNormal">Volume Name: gtower<u></u><u></u></p>
<p class="MsoNormal">Type: Replicate<u></u><u></u></p>
<p class="MsoNormal">Volume ID: 838ab806-06d9-45c5-8d88-2a905c167dba<u></u><u></u></p>
<p class="MsoNormal">Status: Started<u></u><u></u></p>
<p class="MsoNormal">Number of Bricks: 1 x 2 = 2<u></u><u></u></p>
<p class="MsoNormal">Transport-type: tcp<u></u><u></u></p>
<p class="MsoNormal">Bricks:<u></u><u></u></p>
<p class="MsoNormal">Brick1: storage1.sci.gsfc.nasa.gov:/tower7/gluster1/brick<u></u><u></u></p>
<p class="MsoNormal">Brick2: storage2.sci.gsfc.nasa.gov:/tower8/gluster2/brick<u></u><u></u></p>
<p class="MsoNormal">Options Reconfigured:<u></u><u></u></p>
<p class="MsoNormal">nfs.export-volumes: off<u></u><u></u></p>
<p class="MsoNormal">nfs.addr-namelookup: off<u></u><u></u></p>
<p class="MsoNormal">performance.readdir-ahead: on<u></u><u></u></p>
<p class="MsoNormal">performance.cache-size: 2GB<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">-------------------------------------------------------------------------<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">We have made a raidz3 consisting of 6 vdevs each consisting of 12 (6TB) drives and assigned one 200GB SSD drive for ZFS caching.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Our attached storage has 60 (6TB) drives for which I have done multipathing. We are also using 12 drives in the server for which I have set-up vdevs. So we are using 60+12 = 72 drives for ZFS (raidz3)<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">If you have any other suggestions based on our configuration please let me know.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thank you.<u></u><u></u></p>
<p class="MsoNormal">Uthra<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><br>
<br>
<span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Gmail [mailto:<a href="mailto:b.s.mikhael@gmail.com" target="_blank">b.s.mikhael@gmail.com</a>]
<br>
<b>Sent:</b> Tuesday, November 24, 2015 4:50 PM<br>
<b>To:</b> Pierre MGP Pro<br>
<b>Cc:</b> Lindsay Mathieson; Rao, Uthra R. (GSFC-672.0)[ADNET SYSTEMS INC]; <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] GlusterFS Performance tuning<u></u><u></u></span></p>
</div>
</div><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">you can do the following:<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="color:#454545"># gluster volume set $vol performance.o-thread-count 64<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#454545">Today’s CPU are powerful enough to handle 64 threads per volume.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#454545"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#454545"># gluster volume set $vol client.event-threads XX<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#454545">XX depend on the number of connections from the FUSE client to the server, you can get this number by running netstat and grep on the server IP and count the number of connections.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#454545"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#454545"># gluster volume set $vol server.event-threads XX<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#454545"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#454545">XX depend on the number of connections from the server to the client(s), you can get this number by running netstat and grep on “gluster" and count the number of connections.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#454545"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#454545">also, you can follow the instructions in the following page:<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#454545"><a href="http://gluster.readthedocs.org/en/release-3.7.0/Administrator%20Guide/Linux%20Kernel%20Tuning/" target="_blank">http://gluster.readthedocs.org/en/release-3.7.0/Administrator%20Guide/Linux%20Kernel%20Tuning/</a></span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#454545"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#454545">-Bishoy<u></u><u></u></span></p>
</div>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Nov 24, 2015, at 1:31 PM, Pierre MGP Pro <<a href="mailto:pierre-mgp-jouy.inra@agsic.net" target="_blank">pierre-mgp-jouy.inra@agsic.net</a>> wrote:<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hi Lindsay Mathieson and all,<br>
<br>
<u></u><u></u></p>
<div>
<p class="MsoNormal">Le 24/11/2015 21:09, Lindsay Mathieson a écrit :<u></u><u></u></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">More details on your setup[ would be useful:<u></u><u></u></p>
</div>
<p class="MsoNormal">- Number of Nodes<u></u><u></u></p>
</div>
<p class="MsoNormal">- CPU & RAM on Each Node<u></u><u></u></p>
</div>
<p class="MsoNormal">- What else is running on the nodes<u></u><u></u></p>
</div>
<p class="MsoNormal">- Number of bricks<u></u><u></u></p>
</div>
<p class="MsoNormal">- output of "gluster volume info" & "gluster volume status"<u></u><u></u></p>
</div>
<p class="MsoNormal"><br>
- ZFS config for each Node<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * number of disks and rai arrangement<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * log and cache SSD?<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"> * zpool status<u></u><u></u></p>
</div>
</blockquote>
<p class="MsoNormal">OK I have tested that kind of configuration, and the result depend of what you are waiting for :<u></u><u></u></p>
<ul type="disc">
<li class="MsoNormal">
zfsonlinux is now efficient, but you will not have access to the ACL ;<u></u><u></u></li><li class="MsoNormal">
on a volume with seven disk we get the maximum of the PCI Express bandwidth ;<u></u><u></u></li><li class="MsoNormal">
so you can mount a distributed gluster volume with your zfsonlinux nodes. The bandwidth will depend of the kind of glusterfs volume you want to build : distributed, stripped, replicated ;<u></u><u></u></li></ul>
<ul type="disc">
<ul type="circle">
<li class="MsoNormal">
replicated : bad, because of the synchronism write for the files replication ;<u></u><u></u></li><li class="MsoNormal">
striped is the best because it allow you to get an average bandwidth on a file whatever the node you R/W the file ;<u></u><u></u></li></ul>
</ul>
<ul type="disc">
<li class="MsoNormal">
then the last, for me, is the Ethernet access between each nodes. If you have 1Gb, get back to your sand box, At the year you need 10Gbs and as the minimal Ethernet access is two port you need to bound them ;<u></u><u></u></li><li class="MsoNormal">
have 10 Gbs Ethernet switch ;<u></u><u></u></li></ul>
<p class="MsoNormal">That the expressions of the needs for the now and future necessities.<u></u><u></u></p>
<p class="MsoNormal">sincerely<u></u><u></u></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Pierre Léonard<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><u></u><u></u></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div></div></div>
</div>
<br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>