<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p><br>
</p>
<p>Using the gluster client rather than NFS seems to fix the problem<br>
</p>
<br>
<div class="moz-cite-prefix">On 09/01/2016 02:35 PM, Pat Haley
wrote:<br>
</div>
<blockquote cite="mid:2e7917f4-ca43-07e6-c912-d0e6505379f7@mit.edu"
type="cite">
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
<p><br>
</p>
<p>Hi Pranith,</p>
<p>In attached file capture.pcap<br>
</p>
<br>
<div class="moz-cite-prefix">On 09/01/2016 01:01 PM, Pranith Kumar
Karampuri wrote:<br>
</div>
<blockquote
cite="mid:CAOgeEnbHo1f7AoKwBVEVLdbT4SaVqz0zDwri4fTDobmnVhxY1w@mail.gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<div dir="ltr">You need to capture the file so that we can see
the tcpdump in wireshark to inspect the uid/gid etc that are
going out the wire.<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Sep 1, 2016 at 10:04 PM, Pat
Haley <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p><br>
</p>
<p>Hi Pranith,</p>
<p>Here is the output when I'm trying a touch command
that fails with "Permission denied"<br>
</p>
<p><tt>[root@compute-11-10 ~]# tcpdump -nnSs 0 host
10.1.1.4</tt><tt><br>
</tt><tt>tcpdump: verbose output suppressed, use -v or
-vv for full protocol decode</tt><tt><br>
</tt><tt>listening on eth1, link-type EN10MB
(Ethernet), capture size 65535 bytes</tt><tt><br>
</tt><tt>12:30:46.248293 IP 10.255.255.124.4215828946
> 10.1.1.4.2049: 208 getattr fh 0,0/22</tt><tt><br>
</tt><tt>12:30:46.252509 IP 10.1.1.4.2049 >
10.255.255.124.4215828946: reply ok 240 getattr NON
3 ids 0/3 sz 0</tt><tt><br>
</tt><tt>12:30:46.252596 IP 10.255.255.124.<a
moz-do-not-send="true" href="tel:4232606162"
value="+14232606162" target="_blank">4232606162</a>
> 10.1.1.4.2049: 300 getattr fh 0,0/22</tt><tt><br>
</tt><tt>12:30:46.253308 IP 10.1.1.4.2049 >
10.255.255.124.4232606162: reply ok 52 getattr
ERROR: Permission denied</tt><tt><br>
</tt><tt>12:30:46.253358 IP 10.255.255.124.<a
moz-do-not-send="true" href="tel:4249383378"
value="+14249383378" target="_blank">4249383378</a>
> 10.1.1.4.2049: 216 getattr fh 0,0/22</tt><tt><br>
</tt><tt>12:30:46.260347 IP 10.1.1.4.2049 >
10.255.255.124.4249383378: reply ok 52 getattr
ERROR: No such file or directory</tt><tt><br>
</tt><tt>12:30:46.300306 IP 10.255.255.124.931 >
10.1.1.4.2049: Flags [.], ack 1979284005, win 501,
options [nop,nop,TS val 490628016 ecr 75449144],
length 0</tt><tt><br>
</tt><tt>^C</tt><tt><br>
</tt><tt>7 packets captured</tt><tt><br>
</tt><tt>7 packets received by filter</tt><tt><br>
</tt><tt>0 packets dropped by kernel</tt><tt><br>
</tt><br>
</p>
<div>
<div class="h5"> <br>
<div>On 09/01/2016 03:31 AM, Pranith Kumar Karampuri
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">hi Pat,<br>
I think the other thing we should
probably look for would be to see the tcp dump
of what uid/gid parameters are sent over network
when this command is executed.<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Sep 1, 2016 at
7:14 AM, Pat Haley <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:phaley@mit.edu"
target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>--<span><br>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>hi Pat,<br>
</div>
Are you seeing this issue
only after migration or even
before? May be we should look at
the gid numbers on the disk and
the ones that are coming from
client for the given user to see
if they match or not?<br>
</div>
</div>
</blockquote>
</span> ------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-------<br>
This issue was not being seen before the
migration. We have copied the /etc/passwd
and /etc/group files from the front-end
machine (the client) to the data server,
so they all match<br>
------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-------<span><br>
<blockquote type="cite">
<div dir="ltr">Could you give stat
output of the directory in question
from both the brick and the nfs
client<br>
<div><br>
</div>
</div>
</blockquote>
</span> ------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>--------<br>
From the server for gluster:<br>
<tt>[root@mseas-data2 ~]# stat
/gdata/projects/nsf_alpha</tt><tt><br>
</tt><tt> File:
`/gdata/projects/nsf_alpha'</tt><tt><br>
</tt><tt> Size: 4096 Blocks:
8 IO Block: 131072 directory</tt><tt><br>
</tt><tt>Device: 13h/19d Inode:
13094773206281819436 Links: 13</tt><tt><br>
</tt><tt>Access: (2775/drwxrwsr-x) Uid:
( 0/ root) Gid: (
598/nsf_alpha)</tt><tt><br>
</tt><tt>Access: 2016-08-31
19:08:59.735990904 -0400</tt><tt><br>
</tt><tt>Modify: 2016-08-31
16:37:09.048997167 -0400</tt><tt><br>
</tt><tt>Change: 2016-08-31
16:37:41.315997148 -0400</tt><tt><br>
</tt><tt><br>
From the server for first underlying
brick<br>
[root@mseas-data2 ~]# stat
/mnt/brick1/projects/nsf_alpha<wbr>/<br>
File: `/mnt/brick1/projects/nsf_alph<wbr>a/'<br>
Size: 4096 Blocks: 8
IO Block: 4096 directory<br>
Device: 800h/2048d Inode: 185630
Links: 13<br>
Access: (2775/drwxrwsr-x) Uid: (
0/ root) Gid: ( 598/nsf_alpha)<br>
Access: 2016-08-31 19:08:59.669990907
-0400<br>
Modify: 2016-08-31 16:37:09.048997167
-0400<br>
Change: 2016-08-31 16:37:41.315997148
-0400<br>
</tt><br>
<tt>From the server for second underlying
brick<br>
[root@mseas-data2 ~]# stat
/mnt/brick2/projects/nsf_alpha<wbr>/<br>
File: `/mnt/brick2/projects/nsf_alph<wbr>a/'<br>
Size: 4096 Blocks: 8
IO Block: 4096 directory<br>
Device: 810h/2064d Inode: 24085468
Links: 13<br>
Access: (2775/drwxrwsr-x) Uid: (
0/ root) Gid: ( 598/nsf_alpha)<br>
Access: 2016-08-31 19:08:59.735990904
-0400<br>
Modify: 2016-08-03 14:01:52.000000000
-0400<br>
Change: 2016-08-31 16:37:41.315997148
-0400<br>
<br>
From the client<br>
[root@mseas FixOwn]# stat
/gdata/projects/nsf_alpha<br>
File: `/gdata/projects/nsf_alpha'<br>
Size: 4096 Blocks: 8
IO Block: 1048576 directory<br>
Device: 23h/35d Inode:
13094773206281819436 Links: 13<br>
Access: (2775/drwxrwsr-x) Uid: (
0/ root) Gid: ( 598/nsf_alpha)<br>
Access: 2016-08-31 19:08:59.735990904
-0400<br>
Modify: 2016-08-31 16:37:09.048997167
-0400<br>
Change: 2016-08-31 16:37:41.315997148
-0400<br>
<br>
</tt><tt> </tt>
<blockquote type="cite"> </blockquote>
------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------<span><br>
<blockquote type="cite"> </blockquote>
<blockquote type="cite">
<div dir="ltr">
<div>Could you also let us know
version of gluster you are using</div>
</div>
</blockquote>
</span>
<blockquote type="cite">
<div dir="ltr">
<div>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-------<br>
</div>
</div>
</blockquote>
<br>
<blockquote type="cite">
<div dir="ltr">
<div> <tt>[root@mseas-data2 ~]#
gluster --version</tt><tt><br>
</tt><tt>glusterfs 3.7.11 built on
Apr 27 2016 14:09:22</tt><tt><br>
</tt><br>
</div>
</div>
</blockquote>
<tt>[root@mseas-data2 ~]# gluster volume
info</tt><tt><br>
</tt><tt> </tt><tt><br>
</tt><tt>Volume Name: data-volume</tt><tt><br>
</tt><tt>Type: Distribute</tt><tt><br>
</tt><tt>Volume ID:
c162161e-2a2d-4dac-b015-f31fd8<wbr>9ceb18</tt><tt><br>
</tt><tt>Status: Started</tt><tt><br>
</tt><tt>Number of Bricks: 2</tt><tt><br>
</tt><tt>Transport-type: tcp</tt><tt><br>
</tt><tt>Bricks:</tt><tt><br>
</tt><tt>Brick1: mseas-data2:/mnt/brick1</tt><tt><br>
</tt><tt>Brick2: mseas-data2:/mnt/brick2</tt><tt><br>
</tt><tt>Options Reconfigured:</tt><tt><br>
</tt><tt>performance.readdir-ahead: on</tt><tt><br>
</tt><tt>nfs.disable: on</tt><tt><br>
</tt><tt>nfs.export-volumes: off</tt><tt><br>
</tt><br>
<tt>[root@mseas-data2 ~]# gluster volume
status </tt><tt><br>
</tt><tt>Status of volume: data-volume</tt><tt><br>
</tt><tt>Gluster
process <wbr>
TCP Port RDMA Port Online Pid</tt><tt><br>
</tt><tt>------------------------------<wbr>------------------------------<wbr>------------------</tt><tt><br>
</tt><tt>Brick
mseas-data2:/mnt/brick1 <wbr>
49154 0 Y 5005 </tt><tt><br>
</tt><tt>Brick
mseas-data2:/mnt/brick2 <wbr>
49155 0 Y 5010 </tt><tt><br>
</tt><tt> </tt><tt><br>
</tt><tt>Task Status of Volume data-volume</tt><tt><br>
</tt><tt>------------------------------<wbr>------------------------------<wbr>------------------</tt><tt><br>
</tt><tt>Task :
Rebalance </tt><tt><br>
</tt><tt>ID :
892d9e3a-b38c-4971-b96a-8e4a49<wbr>6685ba</tt><tt><br>
</tt><tt>Status :
completed <br>
<br>
</tt><tt><br>
</tt><tt>[root@mseas-data2 ~]# gluster
peer status</tt><tt><br>
</tt><tt>Number of Peers: 0</tt><tt><br>
</tt><br>
<blockquote type="cite">
<div class="gmail_extra"><br>
<div class="gmail_quote">------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-------
<div>
<div><br>
On Thu, Sep 1, 2016 at 2:46 AM,
Pat Haley <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:phaley@mit.edu"
target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF"
text="#000000">
<p><br>
</p>
<p>Hi,</p>
<p>Another piece of data.
There are 2 distinct
volumes on the file server</p>
<ol>
<li>a straight nfs
partition</li>
<li>a gluster volume
(served over nfs)<br>
</li>
</ol>
The straight nfs partition
does respect the group write
permissions, while the
gluster volume does not.
Any suggestions on how to
debug this or what
additional information would
be helpful would be greatly
appreciated<br>
<br>
Thanks<span><br>
<br>
<div>On 08/30/2016 06:01
PM, Pat Haley wrote:<br>
</div>
<blockquote type="cite"> <br>
Hi <br>
<br>
We have just migrated
our data to a new file
server (more space, old
server was showing its
age). We have a volume
for collaborative use,
based on group
membership. In our new
server, the group write
permissions are not
being respected (e.g.
the owner of a directory
can still write to that
directory but any other
member of the associated
group cannot, even
though the directory
clearly has group write
permissions set). This
is occurring regardless
of how many groups the
user is a member of
(i.e. users that are
members of fewer then 16
groups are still
affected). <br>
<br>
the relevant fstab line
from the server looks
like <br>
localhost:/data-volume
/gdata
glusterfs defaults
0 0 <br>
<br>
and for a client: <br>
mseas-data2:/gdata
/gdata nfs
defaults 0 0 <br>
<br>
Any help would be
greatly appreciated. <br>
<br>
Thanks <br>
<br>
</blockquote>
<br>
<pre cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a moz-do-not-send="true" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: <a moz-do-not-send="true" href="tel:%28617%29%20253-6824" value="+16172536824" target="_blank">(617) 253-6824</a>
Dept. of Mechanical Engineering Fax: <a moz-do-not-send="true" href="tel:%28617%29%20253-8125" value="+16172538125" target="_blank">(617) 253-8125</a>
MIT, Room 5-213 <a moz-do-not-send="true" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</span></div>
______________________________<wbr>_________________
Gluster-users mailing list
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a>
</blockquote></div></div></div><span><font color="#888888">
--
<div data-smartmail="gmail_signature"><div dir="ltr">Pranith
</div></div>
</font></span></div>
</blockquote><div><div>
<pre cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a moz-do-not-send="true" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: <a moz-do-not-send="true" href="tel:%28617%29%20253-6824" value="+16172536824" target="_blank">(617) 253-6824</a>
Dept. of Mechanical Engineering Fax: <a moz-do-not-send="true" href="tel:%28617%29%20253-8125" value="+16172538125" target="_blank">(617) 253-8125</a>
MIT, Room 5-213 <a moz-do-not-send="true" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre></div></div></div>
______________________________<wbr>_________________
Gluster-users mailing list
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a>
</blockquote></div>
--
<div data-smartmail="gmail_signature"><div dir="ltr">Pranith
</div></div>
</div>
</blockquote>
<pre cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a moz-do-not-send="true" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: <a moz-do-not-send="true" href="tel:%28617%29%20253-6824" value="+16172536824" target="_blank">(617) 253-6824</a>
Dept. of Mechanical Engineering Fax: <a moz-do-not-send="true" href="tel:%28617%29%20253-8125" value="+16172538125" target="_blank">(617) 253-8125</a>
MIT, Room 5-213 <a moz-do-not-send="true" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre></div></div></div>
______________________________<wbr>_________________
Gluster-users mailing list
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-users</a>
</blockquote></div>
--
<div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith
</div></div>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:phaley@mit.edu">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: <a class="moz-txt-link-abbreviated" href="mailto:phaley@mit.edu">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre></body></html>