<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1699353734;
mso-list-template-ids:76714162;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Thank you all for taking the time to reply to my email:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Here is some more information on our setup:<o:p></o:p></p>
<p class="MsoNormal">- Number of Nodes <span style="font-family:Wingdings">à</span> 2 Gluster servers and 1 client for testing. After testing we will mount the GlusterFS volume on 3 clients.<o:p></o:p></p>
<p class="MsoNormal">- CPU & RAM on Each Node <span style="font-family:Wingdings">
à</span> 2 CPUs 3.4MHz, 384GB RAM on each Gluster Server<o:p></o:p></p>
<p class="MsoNormal">- What else is running on the nodes <span style="font-family:Wingdings">
à</span> Nothing it is only our data server<o:p></o:p></p>
<p class="MsoNormal">- Number of bricks <span style="font-family:Wingdings">à</span> Two<o:p></o:p></p>
<p class="MsoNormal">- output of "gluster volume info" & "gluster volume status"<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>Storage server1:<o:p></o:p></b></p>
<p class="MsoNormal"># <b>gluster volume info gtower</b><o:p></o:p></p>
<p class="MsoNormal">Volume Name: gtower<o:p></o:p></p>
<p class="MsoNormal">Type: Replicate<o:p></o:p></p>
<p class="MsoNormal">Volume ID: 838ab806-06d9-45c5-8d88-2a905c167dba<o:p></o:p></p>
<p class="MsoNormal">Status: Started<o:p></o:p></p>
<p class="MsoNormal">Number of Bricks: 1 x 2 = 2<o:p></o:p></p>
<p class="MsoNormal">Transport-type: tcp<o:p></o:p></p>
<p class="MsoNormal">Bricks:<o:p></o:p></p>
<p class="MsoNormal">Brick1: storage1.sci.gsfc.nasa.gov:/tower7/gluster1/brick<o:p></o:p></p>
<p class="MsoNormal">Brick2: storage2.sci.gsfc.nasa.gov:/tower8/gluster2/brick<o:p></o:p></p>
<p class="MsoNormal">Options Reconfigured:<o:p></o:p></p>
<p class="MsoNormal">nfs.export-volumes: off<o:p></o:p></p>
<p class="MsoNormal">nfs.addr-namelookup: off<o:p></o:p></p>
<p class="MsoNormal">performance.readdir-ahead: on<o:p></o:p></p>
<p class="MsoNormal">performance.cache-size: 2GB<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-----------------------------------------------------------<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>Storage server 2:<o:p></o:p></b></p>
<p class="MsoNormal"># <b>gluster volume info gtower</b><o:p></o:p></p>
<p class="MsoNormal">Volume Name: gtower<o:p></o:p></p>
<p class="MsoNormal">Type: Replicate<o:p></o:p></p>
<p class="MsoNormal">Volume ID: 838ab806-06d9-45c5-8d88-2a905c167dba<o:p></o:p></p>
<p class="MsoNormal">Status: Started<o:p></o:p></p>
<p class="MsoNormal">Number of Bricks: 1 x 2 = 2<o:p></o:p></p>
<p class="MsoNormal">Transport-type: tcp<o:p></o:p></p>
<p class="MsoNormal">Bricks:<o:p></o:p></p>
<p class="MsoNormal">Brick1: storage1.sci.gsfc.nasa.gov:/tower7/gluster1/brick<o:p></o:p></p>
<p class="MsoNormal">Brick2: storage2.sci.gsfc.nasa.gov:/tower8/gluster2/brick<o:p></o:p></p>
<p class="MsoNormal">Options Reconfigured:<o:p></o:p></p>
<p class="MsoNormal">nfs.export-volumes: off<o:p></o:p></p>
<p class="MsoNormal">nfs.addr-namelookup: off<o:p></o:p></p>
<p class="MsoNormal">performance.readdir-ahead: on<o:p></o:p></p>
<p class="MsoNormal">performance.cache-size: 2GB<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-------------------------------------------------------------------------<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We have made a raidz3 consisting of 6 vdevs each consisting of 12 (6TB) drives and assigned one 200GB SSD drive for ZFS caching.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Our attached storage has 60 (6TB) drives for which I have done multipathing. We are also using 12 drives in the server for which I have set-up vdevs. So we are using 60+12 = 72 drives for ZFS (raidz3)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">If you have any other suggestions based on our configuration please let me know.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you.<o:p></o:p></p>
<p class="MsoNormal">Uthra<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><br>
<br>
<span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Gmail [mailto:b.s.mikhael@gmail.com]
<br>
<b>Sent:</b> Tuesday, November 24, 2015 4:50 PM<br>
<b>To:</b> Pierre MGP Pro<br>
<b>Cc:</b> Lindsay Mathieson; Rao, Uthra R. (GSFC-672.0)[ADNET SYSTEMS INC]; gluster-users@gluster.org<br>
<b>Subject:</b> Re: [Gluster-users] GlusterFS Performance tuning<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">you can do the following:<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="color:#454545"># gluster volume set $vol performance.o-thread-count 64<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#454545">Today’s CPU are powerful enough to handle 64 threads per volume.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#454545"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#454545"># gluster volume set $vol client.event-threads XX<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#454545">XX depend on the number of connections from the FUSE client to the server, you can get this number by running netstat and grep on the server IP and count the number of connections.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#454545"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#454545"># gluster volume set $vol server.event-threads XX<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#454545"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#454545">XX depend on the number of connections from the server to the client(s), you can get this number by running netstat and grep on “gluster" and count the number of connections.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#454545"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#454545">also, you can follow the instructions in the following page:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#454545"><a href="http://gluster.readthedocs.org/en/release-3.7.0/Administrator%20Guide/Linux%20Kernel%20Tuning/">http://gluster.readthedocs.org/en/release-3.7.0/Administrator%20Guide/Linux%20Kernel%20Tuning/</a></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#454545"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#454545">-Bishoy<o:p></o:p></span></p>
</div>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Nov 24, 2015, at 1:31 PM, Pierre MGP Pro <<a href="mailto:pierre-mgp-jouy.inra@agsic.net">pierre-mgp-jouy.inra@agsic.net</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hi Lindsay Mathieson and all,<br>
<br>
<o:p></o:p></p>
<div>
<p class="MsoNormal">Le 24/11/2015 21:09, Lindsay Mathieson a écrit :<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">More details on your setup[ would be useful:<o:p></o:p></p>
</div>
<p class="MsoNormal">- Number of Nodes<o:p></o:p></p>
</div>
<p class="MsoNormal">- CPU & RAM on Each Node<o:p></o:p></p>
</div>
<p class="MsoNormal">- What else is running on the nodes<o:p></o:p></p>
</div>
<p class="MsoNormal">- Number of bricks<o:p></o:p></p>
</div>
<p class="MsoNormal">- output of "gluster volume info" & "gluster volume status"<o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
- ZFS config for each Node<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> * number of disks and rai arrangement<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> * log and cache SSD?<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"> * zpool status<o:p></o:p></p>
</div>
</blockquote>
<p class="MsoNormal">OK I have tested that kind of configuration, and the result depend of what you are waiting for :<o:p></o:p></p>
<ul type="disc">
<li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
zfsonlinux is now efficient, but you will not have access to the ACL ;<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
on a volume with seven disk we get the maximum of the PCI Express bandwidth ;<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
so you can mount a distributed gluster volume with your zfsonlinux nodes. The bandwidth will depend of the kind of glusterfs volume you want to build : distributed, stripped, replicated ;<o:p></o:p></li></ul>
<ul type="disc">
<ul type="circle">
<li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level2 lfo1">
replicated : bad, because of the synchronism write for the files replication ;<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level2 lfo1">
striped is the best because it allow you to get an average bandwidth on a file whatever the node you R/W the file ;<o:p></o:p></li></ul>
</ul>
<ul type="disc">
<li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
then the last, for me, is the Ethernet access between each nodes. If you have 1Gb, get back to your sand box, At the year you need 10Gbs and as the minimal Ethernet access is two port you need to bound them ;<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
have 10 Gbs Ethernet switch ;<o:p></o:p></li></ul>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">That the expressions of the needs for the now and future necessities.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">sincerely<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Pierre Léonard<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>