<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Lucida Console";
panose-1:2 11 6 9 4 5 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoPlainText">Hi Dan,<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Thank you for the response.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">The Gluster is only access by the application and in the startup of the application time, I see a huge difference.<o:p></o:p></p>
<p class="MsoPlainText">I you see the statistics, you see that the application is writing a 2184 small files.<o:p></o:p></p>
<p class="MsoPlainText">You point me in the direction of the cache options. Do you have an explanation for the cache, why it would take 120min instead of 2min to start the application ?<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">Block Size: 1b+ 16b+ 32b+<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">No. of Reads: 0 0 0<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">No. of Writes: 342 25
<span style="background:yellow;mso-highlight:yellow">575</span><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> Block Size: 64b+ 128b+ 256b+<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">No. of Reads: 0 0 0<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">No. of Writes: 143
<span style="background:yellow;mso-highlight:yellow">898</span> 118<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> Block Size: 512b+ 1024b+ 2048b+<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">No. of Reads: 1 4 11<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">No. of Writes: 82 0 0<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> Block Size: 4096b+ 8192b+ 16384b+<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">No. of Reads: 11 31 39<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">No. of Writes: 0 0 0<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> Block Size: 32768b+ 65536b+ 131072b+<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">No. of Reads: 59 148
<span style="background:yellow;mso-highlight:yellow">555</span><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">No. of Writes: 0 0 0<o:p></o:p></span></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="NL-BE" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#666666">Vriendelijke groeten,<br>
Cordialement,<br>
Best regards,</span><b><span lang="NL-BE" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#666666"><o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#666666">Stijn Van Renterghem</span></b><span style="font-size:9.0pt;color:#1F497D">
</span><span lang="NL-BE" style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL-BE"><o:p> </o:p></span></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Message: 10<o:p></o:p></p>
<p class="MsoPlainText">Date: Thu, 18 Feb 2016 10:14:59 +1000<o:p></o:p></p>
<p class="MsoPlainText">From: Dan Mons <dmons@cuttingedge.com.au><o:p></o:p></p>
<p class="MsoPlainText">To: Stefan Jakobs <stefan@localside.net><o:p></o:p></p>
<p class="MsoPlainText">Cc: gluster-users <gluster-users@gluster.org><o:p></o:p></p>
<p class="MsoPlainText">Subject: Re: [Gluster-users] FW: Performance with Gluster+Fuse is 60x<o:p></o:p></p>
<p class="MsoPlainText"> slower then Gluster+NFS ?<o:p></o:p></p>
<p class="MsoPlainText">Message-ID:<o:p></o:p></p>
<p class="MsoPlainText"><span lang="NL-BE"><CACa6Tydras9yd8=VmGE1hXzhKJVa6ipVat10Ee3pfr=GjTUbOA@mail.gmail.com><o:p></o:p></span></p>
<p class="MsoPlainText">Content-Type: text/plain; charset=UTF-8<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Without knowing the details, I'm putting my money on cache.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Choosing how to mount Gluster is workload dependent. If you're doing<o:p></o:p></p>
<p class="MsoPlainText">a lot of small files with single threaded writes, I suggest NFS. Your<o:p></o:p></p>
<p class="MsoPlainText">client's nfscache will dramatically improve performance from the<o:p></o:p></p>
<p class="MsoPlainText">end-user's point of view.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">If you're doing heavy multi-threaded reads and writes, and you have<o:p></o:p></p>
<p class="MsoPlainText">very good bandwidth from your client (e.g.: 10GbE) FUSE+GlusterFS is<o:p></o:p></p>
<p class="MsoPlainText">better, as it allows your client to talk to all Gluster nodes.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">If you are using FUSE+GlusterFS, on the gluster nodes themselves,<o:p></o:p></p>
<p class="MsoPlainText">experiment with the "performance.write-behind-window-size" and<o:p></o:p></p>
<p class="MsoPlainText">"performance.cache-size" options. Note that these will affect the<o:p></o:p></p>
<p class="MsoPlainText">cache used by the clients, so don't set them so high as to exhaust the<o:p></o:p></p>
<p class="MsoPlainText">RAM of any client connecting (or, for low-memory clients, use NFS<o:p></o:p></p>
<p class="MsoPlainText">instead).<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Gluster ships with conservative defaults for cache, which is a good<o:p></o:p></p>
<p class="MsoPlainText">thing. It's up to the user to tweak for their optimal needs.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">There's no right or wrong answer here. Experiment with NFS and<o:p></o:p></p>
<p class="MsoPlainText">various cache allocations with FUSE+GlusterFS, and see how you go.<o:p></o:p></p>
<p class="MsoPlainText">And again, consider your workloads, and whether or not they're taking<o:p></o:p></p>
<p class="MsoPlainText">full advantage of the FUSE client's ability to deal with highly<o:p></o:p></p>
<p class="MsoPlainText"><span lang="NL-BE">parallel workloads.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="NL-BE"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="NL-BE">-Dan<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="NL-BE">----------------<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="NL-BE">Dan Mons - VFX Sysadmin<o:p></o:p></span></p>
<p class="MsoPlainText">Cutting Edge<o:p></o:p></p>
<p class="MsoPlainText"><a href="http://cuttingedge.com.au">http://cuttingedge.com.au</a><o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="NL-BE" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b>From:</b> Van Renterghem Stijn <br>
<b>Sent:</b> Wednesday, February 17, 2016 4:20 PM<br>
<b>To:</b> 'gluster-users@gluster.org' <gluster-users@gluster.org><br>
<b>Subject:</b> Performance with Gluster+Fuse is 60x slower then Gluster+NFS ?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I have setup a server with a new installation of Gluster.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">The volume type is ‘Replicate’.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">1)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I mounted the volume with Fuse<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">IP1:/app /srv/data glusterfs defaults,_netdev,backupvolfile-server=IP2,fetch-attempts=2 0 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">When I start my application, it takes 2h until the application is started<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Below you can see the stats after the application is started. I can see a very high LOOKUP value.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Can you explain this high value ? The volume type is replicate, so I should think, I shouldn’t have LOOKUPs ?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">Interval2<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">Block Size: 1b+ 16b+ 32b+<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">No. of Reads: 0 0 0<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">No. of Writes: 342 25
<span style="background:yellow;mso-highlight:yellow">575</span><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> Block Size: 64b+ 128b+ 256b+<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">No. of Reads: 0 0 0<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">No. of Writes: 143
<span style="background:yellow;mso-highlight:yellow">898</span> 118<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> Block Size: 512b+ 1024b+ 2048b+<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">No. of Reads: 1 4 11<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">No. of Writes: 82 0 0<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> Block Size: 4096b+ 8192b+ 16384b+<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">No. of Reads: 11 31 39<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">No. of Writes: 0 0 0<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> Block Size: 32768b+ 65536b+ 131072b+<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">No. of Reads: 59 148
<span style="background:yellow;mso-highlight:yellow">555</span><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">No. of Writes: 0 0 0<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">--------- ----------- ----------- ----------- ------------ ----<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.00 0.00 us 0.00 us 0.00 us 1 FORGET<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.00 0.00 us 0.00 us 0.00 us 201 RELEASE<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.00 0.00 us 0.00 us 0.00 us 54549 RELEASEDIR<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.00 47.00 us 47.00 us 47.00 us 1 REMOVEXATTR<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.00 94.00 us 74.00 us 114.00 us 2 XATTROP<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.00 191.00 us 191.00 us 191.00 us 1 TRUNCATE<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.00 53.50 us 35.00 us 74.00 us 4 STATFS<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.00 79.67 us 70.00 us 91.00 us 3 RENAME<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.00 37.33 us 27.00 us 68.00 us 15 INODELK<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.00 190.67 us 116.00 us 252.00 us 3 UNLINK<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.00 28.83 us 8.00 us 99.00 us 30 ENTRYLK<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.00 146.33 us 117.00 us 188.00 us 6 CREATE<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.00 37.63 us 12.00 us 73.00 us 84 READDIR<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.00 23.75 us 8.00 us 75.00 us 198 FLUSH<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.00 65.33 us 42.00 us 141.00 us 204 OPEN<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.01 45.78 us 11.00 us 191.00 us 944 FINODELK<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.01 80.34 us 31.00 us 211.00 us 859 READ<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.02 96.74 us 50.00 us 188.00 us 944 FXATTROP<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.02 55.84 us 24.00 us 140.00 us 1707 FSTAT<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.02 52.89 us 21.00 us 175.00 us 2183 WRITE<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.02 59.69 us 11.00 us 235.00 us 2312 GETXATTR<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.03 51.18 us 8.00 us 142.00 us 3091 STAT<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.46 48.66 us 1.00 us 179.00 us 54549 OPENDIR<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 1.13 135.93 us 18.00 us 16362.00 us 48124 READDIRP<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">
<span style="background:yellow;mso-highlight:yellow">98.29</span> 70.46 us 16.00 us 2903.00 us
<span style="background:yellow;mso-highlight:yellow">8104385</span> <span style="background:yellow;mso-highlight:yellow">
LOOKUP</span><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> Duration: 7560 seconds<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> Data Read: 91208567 bytes
<span style="background:yellow;mso-highlight:yellow">= 91MB</span><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">Data Written: 292007 bytes
<span style="background:yellow;mso-highlight:yellow">= 0,292MB</span><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I have tried some tuning options, but that didn’t changed anything:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">#gluster volume info app<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">Volume Name: app<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">Type: Replicate<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">Volume ID: f1b59aec-adf8-41f8-ad95-839ace247041<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">Status: Started<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">Number of Bricks: 1 x 2 = 2<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span lang="NL-BE" style="font-size:8.0pt;font-family:"Lucida Console"">Transport-type: tcp<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span lang="NL-BE" style="font-size:8.0pt;font-family:"Lucida Console"">Bricks:<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">Brick1: IP1:/exports/app/app<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">Brick2: IP2:/exports/app/app<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">Options Reconfigured:<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">cluster.readdir-optimize: on<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">server.event-threads: 8<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">client.event-threads: 8<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">cluster.lookup-optimize: on<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">diagnostics.count-fop-hits: on<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">diagnostics.latency-measurement: on<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">auth.allow: client1,client2<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">nfs.rpc-auth-allow: client1,client2<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">nfs.export-volumes: on<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">nfs.addr-namelookup: off<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">nfs.disable: off<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">performance.readdir-ahead: on<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">performance.io-thread-count: 64<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">3)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I then have enabled NFS support.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I stopped the application and unmounted the volume. I then mounted it again with nfs:<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">IP1:/app /srv/data nfs rsize=4096,wsize=4096,hard,intr 0 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I started the application again and it was running within 3minutes.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">The stats with NFS where very different then with Fuse. It seems that they are almost not logged.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">Interval 11 Stats:<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> Block Size: 128b+ 256b+ 512b+<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">No. of Reads: 0 0 0<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">No. of Writes: 9 1 1<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> Block Size: 1024b+ 2048b+ 4096b+<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">No. of Reads: 0 0 0<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">No. of Writes: 1 5 8<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">--------- ----------- ----------- ----------- ------------ ----<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.00 0.00 us 0.00 us 0.00 us 2 RELEASE<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.00 0.00 us 0.00 us 0.00 us 1 RELEASEDIR<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.02 2.00 us 2.00 us 2.00 us 1 OPENDIR<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.57 34.00 us 19.00 us 49.00 us 2 READDIR<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 0.81 96.00 us 96.00 us 96.00 us 1 SETATTR<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 1.06 62.50 us 61.00 us 64.00 us 2 OPEN<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 1.39 164.00 us 164.00 us 164.00 us 1 TRUNCATE<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 1.39 41.25 us 30.00 us 52.00 us 4 GETXATTR<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 1.54 91.00 us 86.00 us 96.00 us 2 XATTROP<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 2.72 80.50 us 29.00 us 122.00 us 4
<span style="background:yellow;mso-highlight:yellow">LOOKUP</span><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 2.81 33.30 us 17.00 us 56.00 us 10 INODELK<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 10.36 76.69 us 26.00 us 133.00 us 16 FLUSH<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 15.83 75.00 us 61.00 us 105.00 us 25 WRITE<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 17.22 48.55 us 13.00 us 78.00 us 42 FINODELK<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> 44.28 124.83 us 62.00 us 161.00 us 42 FXATTROP<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> Duration: 580 seconds<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console""> Data Read: 0 bytes<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;font-family:"Lucida Console"">Data Written: 60839 bytes<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="color:#1F497D"><o:p> </o:p></span></b></p>
<p class="MsoNormal"><b>What is wrong with the Fuse client ?<o:p></o:p></b></p>
<p class="MsoNormal"><b>Why does my application start in 120min with Gluster+Fuse and in 3min with Gluster+NFS ?<o:p></o:p></b></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="NL-BE">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL-BE">Stijn<o:p></o:p></span></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>