<html>
  <head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <p><br>
    </p>
    <p>Using the gluster client rather than NFS seems to fix the problem<br>
    </p>
    <br>
    <div class="moz-cite-prefix">On 09/01/2016 02:35 PM, Pat Haley
      wrote:<br>
    </div>
    <blockquote cite="mid:2e7917f4-ca43-07e6-c912-d0e6505379f7@mit.edu"
      type="cite">
      <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
      <p><br>
      </p>
      <p>Hi Pranith,</p>
      <p>In attached file capture.pcap<br>
      </p>
      <br>
      <div class="moz-cite-prefix">On 09/01/2016 01:01 PM, Pranith Kumar
        Karampuri wrote:<br>
      </div>
      <blockquote
cite="mid:CAOgeEnbHo1f7AoKwBVEVLdbT4SaVqz0zDwri4fTDobmnVhxY1w@mail.gmail.com"
        type="cite">
        <meta http-equiv="Content-Type" content="text/html;
          charset=utf-8">
        <div dir="ltr">You need to capture the file so that we can see
          the tcpdump in wireshark to inspect the uid/gid etc that are
          going out the wire.<br>
        </div>
        <div class="gmail_extra"><br>
          <div class="gmail_quote">On Thu, Sep 1, 2016 at 10:04 PM, Pat
            Haley <span dir="ltr">&lt;<a moz-do-not-send="true"
                href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span>
            wrote:<br>
            <blockquote class="gmail_quote" style="margin:0 0 0
              .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div bgcolor="#FFFFFF" text="#000000">
                <p><br>
                </p>
                <p>Hi Pranith,</p>
                <p>Here is the output when I'm trying a touch command
                  that fails with "Permission denied"<br>
                </p>
                <p><tt>[root@compute-11-10 ~]# tcpdump -nnSs 0 host
                    10.1.1.4</tt><tt><br>
                  </tt><tt>tcpdump: verbose output suppressed, use -v or
                    -vv for full protocol decode</tt><tt><br>
                  </tt><tt>listening on eth1, link-type EN10MB
                    (Ethernet), capture size 65535 bytes</tt><tt><br>
                  </tt><tt>12:30:46.248293 IP 10.255.255.124.4215828946
                    &gt; 10.1.1.4.2049: 208 getattr fh 0,0/22</tt><tt><br>
                  </tt><tt>12:30:46.252509 IP 10.1.1.4.2049 &gt;
                    10.255.255.124.4215828946: reply ok 240 getattr NON
                    3 ids 0/3 sz 0</tt><tt><br>
                  </tt><tt>12:30:46.252596 IP 10.255.255.124.<a
                      moz-do-not-send="true" href="tel:4232606162"
                      value="+14232606162" target="_blank">4232606162</a>
                    &gt; 10.1.1.4.2049: 300 getattr fh 0,0/22</tt><tt><br>
                  </tt><tt>12:30:46.253308 IP 10.1.1.4.2049 &gt;
                    10.255.255.124.4232606162: reply ok 52 getattr
                    ERROR: Permission denied</tt><tt><br>
                  </tt><tt>12:30:46.253358 IP 10.255.255.124.<a
                      moz-do-not-send="true" href="tel:4249383378"
                      value="+14249383378" target="_blank">4249383378</a>
                    &gt; 10.1.1.4.2049: 216 getattr fh 0,0/22</tt><tt><br>
                  </tt><tt>12:30:46.260347 IP 10.1.1.4.2049 &gt;
                    10.255.255.124.4249383378: reply ok 52 getattr
                    ERROR: No such file or directory</tt><tt><br>
                  </tt><tt>12:30:46.300306 IP 10.255.255.124.931 &gt;
                    10.1.1.4.2049: Flags [.], ack 1979284005, win 501,
                    options [nop,nop,TS val 490628016 ecr 75449144],
                    length 0</tt><tt><br>
                  </tt><tt>^C</tt><tt><br>
                  </tt><tt>7 packets captured</tt><tt><br>
                  </tt><tt>7 packets received by filter</tt><tt><br>
                  </tt><tt>0 packets dropped by kernel</tt><tt><br>
                  </tt><br>
                </p>
                <div>
                  <div class="h5"> <br>
                    <div>On 09/01/2016 03:31 AM, Pranith Kumar Karampuri
                      wrote:<br>
                    </div>
                    <blockquote type="cite">
                      <div dir="ltr">hi Pat,<br>
                               I think the other thing we should
                        probably look for would be to see the tcp dump
                        of what uid/gid parameters are sent over network
                        when this command is executed.<br>
                      </div>
                      <div class="gmail_extra"><br>
                        <div class="gmail_quote">On Thu, Sep 1, 2016 at
                          7:14 AM, Pat Haley <span dir="ltr">&lt;<a
                              moz-do-not-send="true"
                              href="mailto:phaley@mit.edu"
                              target="_blank">phaley@mit.edu</a>&gt;</span>
                          wrote:<br>
                          <blockquote class="gmail_quote"
                            style="margin:0 0 0 .8ex;border-left:1px
                            #ccc solid;padding-left:1ex">
                            <div bgcolor="#FFFFFF" text="#000000">
                              ------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>--<span><br>
                                <blockquote type="cite">
                                  <div dir="ltr">
                                    <div>
                                      <div>hi Pat,<br>
                                      </div>
                                            Are you seeing this issue
                                      only after migration or even
                                      before? May be we should look at
                                      the gid numbers on the disk and
                                      the ones that are coming from
                                      client for the given user to see
                                      if they match or not?<br>
                                    </div>
                                  </div>
                                </blockquote>
                              </span> ------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-------<br>
                              This issue was not being seen before the
                              migration.  We have copied the /etc/passwd
                              and /etc/group files from the front-end
                              machine (the client) to the data server,
                              so they all match<br>
                              ------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-------<span><br>
                                <blockquote type="cite">
                                  <div dir="ltr">Could you give stat
                                    output of the directory in question
                                    from both the brick and the nfs
                                    client<br>
                                    <div><br>
                                    </div>
                                  </div>
                                </blockquote>
                              </span> ------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>--------<br>
                              From the server for gluster:<br>
                              <tt>[root@mseas-data2 ~]# stat
                                /gdata/projects/nsf_alpha</tt><tt><br>
                              </tt><tt>  File:
                                `/gdata/projects/nsf_alpha'</tt><tt><br>
                              </tt><tt>  Size: 4096          Blocks:
                                8          IO Block: 131072 directory</tt><tt><br>
                              </tt><tt>Device: 13h/19d    Inode:
                                13094773206281819436  Links: 13</tt><tt><br>
                              </tt><tt>Access: (2775/drwxrwsr-x)  Uid:
                                (    0/    root)   Gid: ( 
                                598/nsf_alpha)</tt><tt><br>
                              </tt><tt>Access: 2016-08-31
                                19:08:59.735990904 -0400</tt><tt><br>
                              </tt><tt>Modify: 2016-08-31
                                16:37:09.048997167 -0400</tt><tt><br>
                              </tt><tt>Change: 2016-08-31
                                16:37:41.315997148 -0400</tt><tt><br>
                              </tt><tt><br>
                                From the server for first underlying
                                brick<br>
                                [root@mseas-data2 ~]# stat
                                /mnt/brick1/projects/nsf_alpha<wbr>/<br>
                                  File: `/mnt/brick1/projects/nsf_alph<wbr>a/'<br>
                                  Size: 4096          Blocks: 8         
                                IO Block: 4096   directory<br>
                                Device: 800h/2048d    Inode: 185630     
                                Links: 13<br>
                                Access: (2775/drwxrwsr-x)  Uid: (   
                                0/    root)   Gid: (  598/nsf_alpha)<br>
                                Access: 2016-08-31 19:08:59.669990907
                                -0400<br>
                                Modify: 2016-08-31 16:37:09.048997167
                                -0400<br>
                                Change: 2016-08-31 16:37:41.315997148
                                -0400<br>
                              </tt><br>
                              <tt>From the server for second underlying
                                brick<br>
                                [root@mseas-data2 ~]# stat
                                /mnt/brick2/projects/nsf_alpha<wbr>/<br>
                                  File: `/mnt/brick2/projects/nsf_alph<wbr>a/'<br>
                                  Size: 4096          Blocks: 8         
                                IO Block: 4096   directory<br>
                                Device: 810h/2064d    Inode: 24085468   
                                Links: 13<br>
                                Access: (2775/drwxrwsr-x)  Uid: (   
                                0/    root)   Gid: (  598/nsf_alpha)<br>
                                Access: 2016-08-31 19:08:59.735990904
                                -0400<br>
                                Modify: 2016-08-03 14:01:52.000000000
                                -0400<br>
                                Change: 2016-08-31 16:37:41.315997148
                                -0400<br>
                                <br>
                                From the client<br>
                                [root@mseas FixOwn]# stat
                                /gdata/projects/nsf_alpha<br>
                                  File: `/gdata/projects/nsf_alpha'<br>
                                  Size: 4096          Blocks: 8         
                                IO Block: 1048576 directory<br>
                                Device: 23h/35d    Inode:
                                13094773206281819436  Links: 13<br>
                                Access: (2775/drwxrwsr-x)  Uid: (   
                                0/    root)   Gid: (  598/nsf_alpha)<br>
                                Access: 2016-08-31 19:08:59.735990904
                                -0400<br>
                                Modify: 2016-08-31 16:37:09.048997167
                                -0400<br>
                                Change: 2016-08-31 16:37:41.315997148
                                -0400<br>
                                <br>
                              </tt><tt> </tt>
                              <blockquote type="cite"> </blockquote>
                              ------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------<span><br>
                                <blockquote type="cite"> </blockquote>
                                <blockquote type="cite">
                                  <div dir="ltr">
                                    <div>Could you also let us know
                                      version of gluster you are using</div>
                                  </div>
                                </blockquote>
                              </span>
                              <blockquote type="cite">
                                <div dir="ltr">
                                  <div>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-------<br>
                                  </div>
                                </div>
                              </blockquote>
                              <br>
                              <blockquote type="cite">
                                <div dir="ltr">
                                  <div> <tt>[root@mseas-data2 ~]#
                                      gluster --version</tt><tt><br>
                                    </tt><tt>glusterfs 3.7.11 built on
                                      Apr 27 2016 14:09:22</tt><tt><br>
                                    </tt><br>
                                  </div>
                                </div>
                              </blockquote>
                              <tt>[root@mseas-data2 ~]# gluster volume
                                info</tt><tt><br>
                              </tt><tt> </tt><tt><br>
                              </tt><tt>Volume Name: data-volume</tt><tt><br>
                              </tt><tt>Type: Distribute</tt><tt><br>
                              </tt><tt>Volume ID:
                                c162161e-2a2d-4dac-b015-f31fd8<wbr>9ceb18</tt><tt><br>
                              </tt><tt>Status: Started</tt><tt><br>
                              </tt><tt>Number of Bricks: 2</tt><tt><br>
                              </tt><tt>Transport-type: tcp</tt><tt><br>
                              </tt><tt>Bricks:</tt><tt><br>
                              </tt><tt>Brick1: mseas-data2:/mnt/brick1</tt><tt><br>
                              </tt><tt>Brick2: mseas-data2:/mnt/brick2</tt><tt><br>
                              </tt><tt>Options Reconfigured:</tt><tt><br>
                              </tt><tt>performance.readdir-ahead: on</tt><tt><br>
                              </tt><tt>nfs.disable: on</tt><tt><br>
                              </tt><tt>nfs.export-volumes: off</tt><tt><br>
                              </tt><br>
                              <tt>[root@mseas-data2 ~]# gluster volume
                                status </tt><tt><br>
                              </tt><tt>Status of volume: data-volume</tt><tt><br>
                              </tt><tt>Gluster
                                process                       <wbr>     
                                TCP Port  RDMA Port  Online  Pid</tt><tt><br>
                              </tt><tt>------------------------------<wbr>------------------------------<wbr>------------------</tt><tt><br>
                              </tt><tt>Brick
                                mseas-data2:/mnt/brick1       <wbr>       
                                49154     0          Y       5005 </tt><tt><br>
                              </tt><tt>Brick
                                mseas-data2:/mnt/brick2       <wbr>       
                                49155     0          Y       5010 </tt><tt><br>
                              </tt><tt> </tt><tt><br>
                              </tt><tt>Task Status of Volume data-volume</tt><tt><br>
                              </tt><tt>------------------------------<wbr>------------------------------<wbr>------------------</tt><tt><br>
                              </tt><tt>Task                 :
                                Rebalance           </tt><tt><br>
                              </tt><tt>ID                   :
                                892d9e3a-b38c-4971-b96a-8e4a49<wbr>6685ba</tt><tt><br>
                              </tt><tt>Status               :
                                completed    <br>
                                <br>
                                       </tt><tt><br>
                              </tt><tt>[root@mseas-data2 ~]# gluster
                                peer status</tt><tt><br>
                              </tt><tt>Number of Peers: 0</tt><tt><br>
                              </tt><br>
                              <blockquote type="cite">
                                <div class="gmail_extra"><br>
                                  <div class="gmail_quote">------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-------
                                    <div>
                                      <div><br>
                                        On Thu, Sep 1, 2016 at 2:46 AM,
                                        Pat Haley <span dir="ltr">&lt;<a
                                            moz-do-not-send="true"
                                            href="mailto:phaley@mit.edu"
                                            target="_blank">phaley@mit.edu</a>&gt;</span>
                                        wrote:<br>
                                        <blockquote class="gmail_quote"
                                          style="margin:0 0 0
                                          .8ex;border-left:1px #ccc
                                          solid;padding-left:1ex">
                                          <div bgcolor="#FFFFFF"
                                            text="#000000">
                                            <p><br>
                                            </p>
                                            <p>Hi,</p>
                                            <p>Another piece of data. 
                                              There are 2 distinct
                                              volumes on the file server</p>
                                            <ol>
                                              <li>a straight nfs
                                                partition</li>
                                              <li>a gluster volume
                                                (served over nfs)<br>
                                              </li>
                                            </ol>
                                            The straight nfs partition
                                            does respect the group write
                                            permissions, while the
                                            gluster volume does not. 
                                            Any suggestions on how to
                                            debug this or what
                                            additional information would
                                            be helpful would be greatly
                                            appreciated<br>
                                            <br>
                                            Thanks<span><br>
                                              <br>
                                              <div>On 08/30/2016 06:01
                                                PM, Pat Haley wrote:<br>
                                              </div>
                                              <blockquote type="cite"> <br>
                                                Hi <br>
                                                <br>
                                                We have just migrated
                                                our data to a new file
                                                server (more space, old
                                                server was showing its
                                                age). We have a volume
                                                for collaborative use,
                                                based on group
                                                membership.  In our new
                                                server, the group write
                                                permissions are not
                                                being respected (e.g. 
                                                the owner of a directory
                                                can still write to that
                                                directory but any other
                                                member of the associated
                                                group cannot, even
                                                though the directory
                                                clearly has group write
                                                permissions set).  This
                                                is occurring regardless
                                                of how many groups the
                                                user is a member of
                                                (i.e. users that are
                                                members of fewer then 16
                                                groups are still
                                                affected). <br>
                                                <br>
                                                the relevant fstab line
                                                from the server looks
                                                like <br>
                                                localhost:/data-volume
                                                /gdata   
                                                glusterfs       defaults
                                                0 0 <br>
                                                <br>
                                                and for a client: <br>
                                                mseas-data2:/gdata      
                                                /gdata      nfs    
                                                defaults        0 0 <br>
                                                <br>
                                                Any help would be
                                                greatly appreciated. <br>
                                                <br>
                                                Thanks <br>
                                                <br>
                                              </blockquote>
                                              <br>
                                              <pre cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a moz-do-not-send="true" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  <a moz-do-not-send="true" href="tel:%28617%29%20253-6824" value="+16172536824" target="_blank">(617) 253-6824</a>
Dept. of Mechanical Engineering    Fax:    <a moz-do-not-send="true" href="tel:%28617%29%20253-8125" value="+16172538125" target="_blank">(617) 253-8125</a>
MIT, Room 5-213                    <a moz-do-not-send="true" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
  </span></div>


______________________________<wbr>_________________

Gluster-users mailing list

<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>

<a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a>
</blockquote></div></div></div><span><font color="#888888">


-- 
<div data-smartmail="gmail_signature"><div dir="ltr">Pranith
</div></div>
</font></span></div>



</blockquote><div><div>
<pre cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a moz-do-not-send="true" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  <a moz-do-not-send="true" href="tel:%28617%29%20253-6824" value="+16172536824" target="_blank">(617) 253-6824</a>
Dept. of Mechanical Engineering    Fax:    <a moz-do-not-send="true" href="tel:%28617%29%20253-8125" value="+16172538125" target="_blank">(617) 253-8125</a>
MIT, Room 5-213                    <a moz-do-not-send="true" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre></div></div></div>
______________________________<wbr>_________________

Gluster-users mailing list

<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>

<a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a>
</blockquote></div>


-- 
<div data-smartmail="gmail_signature"><div dir="ltr">Pranith
</div></div>
</div>



</blockquote>
<pre cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a moz-do-not-send="true" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  <a moz-do-not-send="true" href="tel:%28617%29%20253-6824" value="+16172536824" target="_blank">(617) 253-6824</a>
Dept. of Mechanical Engineering    Fax:    <a moz-do-not-send="true" href="tel:%28617%29%20253-8125" value="+16172538125" target="_blank">(617) 253-8125</a>
MIT, Room 5-213                    <a moz-do-not-send="true" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre></div></div></div>
______________________________<wbr>_________________

Gluster-users mailing list

<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>

<a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-users</a>
</blockquote></div>


-- 
<div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith
</div></div>
</div>



</blockquote>
<pre class="moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:phaley@mit.edu">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>


</blockquote>
<pre class="moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  <a class="moz-txt-link-abbreviated" href="mailto:phaley@mit.edu">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre></body></html>