<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hi,<br>
<br>
Comments inline.<br>
<br>
<div class="moz-cite-prefix">On 24/06/15 14:39, 莊尚豪 wrote:<br>
</div>
<blockquote cite="mid:014101d0ae5d$76cc81d0$64658570$@iii.org.tw"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 14 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:新細明體;
panose-1:2 2 5 0 0 0 0 0 0 0;}
@font-face
{font-family:新細明體;
panose-1:2 2 5 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@新細明體";
panose-1:2 2 5 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
/* Page Definitions */
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Hi all,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I test some perfomance
from pnfs (gluster-3.7.1 + ganesha-2.2) in Fedora 22.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">There are 4 glusterfs
nodes with ganesha.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I reference from
<a class="moz-txt-link-freetext" href="https://gluster.readthedocs.org/en/latest/Features/mount_gluster_volume_using_pnfs/">https://gluster.readthedocs.org/en/latest/Features/mount_gluster_volume_using_pnfs/</a><o:p></o:p></span></p>
</div>
</blockquote>
<br>
<br>
Can u check whether ganesha is running on every nodes(M.D.S and
D.Ses) , <br>
# service nfs-ganesha status <br>
or try<br>
#ps ax | grep ganesha<br>
<br>
and also checks whether volume is exported in every node<br>
# showmount -e local host<br>
<br>
<blockquote cite="mid:014101d0ae5d$76cc81d0$64658570$@iii.org.tw"
type="cite">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">The clients(Fedora 21)
are fine to mount and commit some small files to gluster.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">However, when dd the
bigger files(dd 600MB file), client will be suspend on
layout commit protocol.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</blockquote>
<br>
I had tested dd command (two to three months back) upto file size
10GB , I didn't notice this issue<br>
<br>
<br>
<blockquote cite="mid:014101d0ae5d$76cc81d0$64658570$@iii.org.tw"
type="cite">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">There are some tshark
information from client<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">--------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331036 68.713945
192.168.100.12 -> 192.168.100.16 NFS 182 V4 Reply (Call
In 331012) WRITE<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331037 68.718067
192.168.100.16 -> 192.168.100.12 NFS 286 V4 Call COMMIT
FH: 0x67571bfb Offset: 0 Len: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331038 68.718999
192.168.100.12 -> 192.168.100.16 NFS 174 V4 Reply (Call
In 331037) COMMIT<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331039 68.740898
192.168.100.16 -> 192.168.100.10 NFS 334 V4 Call
LAYOUTCOMMIT<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331040 68.741619
192.168.100.10 -> 192.168.100.16 NFS 114 V4 Reply (Call
In 331039) SEQUENCE Status: NFS4ERR_BADSESSION<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331041 68.741684
192.168.100.16 -> 192.168.100.10 TCP 66 908</span><span
style="font-family:"新細明體","serif"">→</span><span
lang="EN-US">2049 [ACK] Seq=8561 Ack=6417 Win=942 Len=0
TSval=8509746 TSecr=108629619<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331042 68.742060
192.168.100.16 -> 192.168.100.10 NFS 186 V4 Call
DESTROY_SESSION<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331043 68.742686
192.168.100.10 -> 192.168.100.16 NFS 114 V4 Reply (Call
In 331042) DESTROY_SESSION Status: NFS4ERR_BADSESSION<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331044 68.742819
192.168.100.16 -> 192.168.100.10 NFS 298 V4 Call
CREATE_SESSION<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331045 68.743371
192.168.100.10 -> 192.168.100.16 NFS 114 V4 Reply (Call
In 331044) CREATE_SESSION Status: NFS4ERR_STALE_CLIENTID<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331046 68.743529
192.168.100.16 -> 192.168.100.10 NFS 334 V4 Call
EXCHANGE_ID<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331047 68.744174
192.168.100.10 -> 192.168.100.16 NFS 182 V4 Reply (Call
In 331046) EXCHANGE_ID<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331048 68.744317
192.168.100.16 -> 192.168.100.10 NFS 298 V4 Call
CREATE_SESSION<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331049 68.756698
192.168.100.10 -> 192.168.100.16 NFS 154 V1 CB_NULL Call<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331050 68.756825
192.168.100.16 -> 192.168.100.10 NFS 94 V1 CB_NULL Reply
(Call In 331049)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331051 68.757543
192.168.100.10 -> 192.168.100.16 NFS 194 V4 Reply (Call
In 331048) CREATE_SESSION<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331052 68.757655
192.168.100.16 -> 192.168.100.10 NFS 218 V4 Call
PUTROOTFH | GETATTR<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331053 68.758289
192.168.100.10 -> 192.168.100.16 NFS 182 V4 Reply (Call
In 331052) PUTROOTFH | GETATTR<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331054 68.758329
192.168.100.16 -> 192.168.100.12 TCP 66 980</span><span
style="font-family:"新細明體","serif"">→</span><span
lang="EN-US">2049 [ACK] Seq=1574831169 Ack=697393 Win=31360
Len=0 TSval=8509763 TSecr=109327597<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331055 68.761509
192.168.100.16 -> 192.168.100.10 NFS 338 V4 Call OPEN DH:
0x1dfddbb4/<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331056 68.762148
192.168.100.10 -> 192.168.100.16 NFS 166 V4 Reply (Call
In 331055) OPEN Status: NFS4ERR_NO_GRACE<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331057 68.762324
192.168.100.16 -> 192.168.100.10 NFS 210 V4 Call
RECLAIM_COMPLETE<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331058 68.762969
192.168.100.10 -> 192.168.100.16 NFS 158 V4 Reply (Call
In 331057) RECLAIM_COMPLETE<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331059 68.763135
192.168.100.16 -> 192.168.100.10 NFS 338 V4 Call OPEN DH:
0x1dfddbb4/<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331060 68.763844
192.168.100.10 -> 192.168.100.16 NFS 398 V4 Reply (Call
In 331059) OPEN StateID: 0x9d75<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331061 68.764080
192.168.100.16 -> 192.168.100.10 NFS 334 V4 Call
LAYOUTCOMMIT<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331062 68.764720
192.168.100.10 -> 192.168.100.16 NFS 166 V4 Reply (Call
In 331061) LAYOUTCOMMIT Status: NFS4ERR_EXPIRED<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331063 68.765075
192.168.100.16 -> 192.168.100.10 NFS 202 V4 Call SEQUENCE<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331064 68.765699
192.168.100.10 -> 192.168.100.16 NFS 150 V4 Reply (Call
In 331063) SEQUENCE<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331065 68.765906
192.168.100.16 -> 192.168.100.10 NFS 334 V4 Call
LAYOUTCOMMIT<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331066 68.766555
192.168.100.10 -> 192.168.100.16 NFS 166 V4 Reply (Call
In 331065) LAYOUTCOMMIT Status: NFS4ERR_EXPIRED<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331067 68.766855
192.168.100.16 -> 192.168.100.10 NFS 202 V4 Call SEQUENCE<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331068 68.767493
192.168.100.10 -> 192.168.100.16 NFS 150 V4 Reply (Call
In 331067) SEQUENCE<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">331069 68.767697
192.168.100.16 -> 192.168.100.10 NFS 334 V4 Call
LAYOUTCOMMIT<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">--------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Meanwhile, the ganesha
server appears the log like these.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">There are some server
logs when client are failed to commit.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">----------------------------------------------------------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">24/06/2015 16:08:49 :
epoch 558a6555 : gluster1 : nfs-ganesha-20876[reaper]
nfs_in_grace :STATE :EVENT :NFS Server Now NOT IN GRACE<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">24/06/2015 16:09:56 :
epoch 558a6555 : gluster1 : nfs-ganesha-20876[work-10]
nfs4_op_lookup :EXPORT :MAJ :PSEUDO FS JUNCTION TRAVERSAL:
Failed to get FSAL credentials for /ganesha, id=1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">24/06/2015 16:09:56 :
epoch 558a6555 : gluster1 : nfs-ganesha-20876[work-13]
nfs4_op_lookup :EXPORT :MAJ :PSEUDO FS JUNCTION TRAVERSAL:
Failed to get FSAL credentials for /ganesha, id=1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">----------------------------------------------------------------------------------------------------<o:p></o:p></span></p>
</div>
</blockquote>
<br>
I suspect logs says lookup fails to find the export entry.<br>
<br>
Some modification might need in the configuration :<br>
<br>
<blockquote cite="mid:014101d0ae5d$76cc81d0$64658570$@iii.org.tw"
type="cite">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">The following is my
glusterfs nodes configuration. There are the same
confiuration for all node.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">/etc/ganesha/gluster.conf<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">--------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">EXPORT{<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Export_Id = 1;</span></p>
</div>
</blockquote>
<br>
It is better to use a different export_id other than one and zero.
As far as I know export_id = 1 will be used by pseudo_fs ('/')<br>
<br>
<blockquote cite="mid:014101d0ae5d$76cc81d0$64658570$@iii.org.tw"
type="cite">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Path = /ganesha; #Is the
path attribute useless in the configuration?</span></p>
</div>
</blockquote>
<br>
It should be Path=/<volname>, in your case it should be
/gluster <br>
<blockquote cite="mid:014101d0ae5d$76cc81d0$64658570$@iii.org.tw"
type="cite">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">FSAL {<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">name = GLUSTER;<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">hostname = "localhost";<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">volume = "gluster";<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">}<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Access_type = RW;<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Squash = No_Root_Squash;<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Disable_ACL = TRUE;<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Pseudo = /ganesha;<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Protocols = "4";<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Transports = "TCP";<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">SecType = sys, krb5,
krb5i, krb5p;<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">}<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">--------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</blockquote>
<br>
You should include following block in conf file of M.D.S (due to the
latest changes) :<br>
<br>
GLUSTER<br>
{<br>
<span class="blob-code-inner"><span class="pl-s1"> PNFS_MDS = <span
class="pl-c1">true;</span></span></span><br>
}<br>
<br>
<blockquote cite="mid:014101d0ae5d$76cc81d0$64658570$@iii.org.tw"
type="cite">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">gluster volume info<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">--------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Volume Name: gluster<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Type: Distribute<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Volume ID:
8a5afe82-41fe-456e-935f-3361edce1995<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Status: Started<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Number of Bricks: 4<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Transport-type: tcp<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Bricks:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Brick1:
192.168.100.10:/volume1/brick1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Brick2:
192.168.100.11:/volume1/brick1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Brick3:
192.168.100.12:/volume1/brick1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Brick4:
192.168.100.13:/volume1/brick1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Options Reconfigured:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">nfs.disable: ON<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">performance.readdir-ahead:
on</span></p>
</div>
</blockquote>
<br>
also turn on cache-invalidation feature for the volume.<br>
<span class="blob-code-inner"><br>
gluster v set <volname> features.cache-invalidation on<br>
<br>
</span>
<blockquote cite="mid:014101d0ae5d$76cc81d0$64658570$@iii.org.tw"
type="cite">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">--------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Are this problem from
ganesha server or client?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">or it would be fixed in
the ganesha 2.3 version? <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Many thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Ben<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</blockquote>
<br>
Regards,<br>
Jiffin<br>
<blockquote cite="mid:014101d0ae5d$76cc81d0$64658570$@iii.org.tw"
type="cite">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</body>
</html>