<div dir="ltr">A discussion thread on this list from last May 2014, <a href="http://www.gluster.org/pipermail/gluster-users.old/2014-May/017283.html">http://www.gluster.org/pipermail/gluster-users.old/2014-May/017283.html</a>, discussed how Gluster is limited to 32 groups, due to FUSE, or maybe to 96 groups, due to the AUTH header size in the RPC library being used, unless the administrator enables the server.manage-gids option, which causes user/group membership to be resolved on the server instead of on the client, thus requiring the same user/group membership information to be maintained across all clients and servers.<div><br></div><div><div>As far as I can tell, Gluster 3.0 did not have this limitation. What changed?</div><div><br></div><div>Here is my evidence: I have created a Gluster 3.0 and 3.4 installation on separate clients and servers, mounted at /mnt/gfs in each case. I create >32 groups and add one user (bjaspan) to all of them. I create a directory named for the group and owned by root:<group> mode 0750. Then, as the user, I try to ls all of them. On Gluster 3.0, it works fine:</div><div><br></div><div><div>bjaspan@web-47:~$ glusterfs --version | head -n 1</div><div>glusterfs 3.0.0git built on Oct 28 2013 16:38:44</div><div>bjaspan@web-47:~$ groups<br></div><div>bjaspan www-data ahops g30000 g30001 g30002 g30003 g30004 g30005 g30006 g30007 g30008 g30009 g30010 g30011 g30012 g30013 g30014 g30015 g30016 g30017 g30018 g30019 g30020 g30021 g30022 g30023 g30024 g30025 g30026 g30027 g30028 g30029 g30030 g30031 g30032 g30033 g30034 g30035 g30036 g30037 g30038 g30039 g30040 g30041 g30042 g30043 g30044 g30045 g30046 g30047 g30048 g30049 g30050 g30051 g30052 g30053 g30054 g30055 g30056 g30057 g30058 g30059 g30060 g30061 g30062 g30063 g30064 g30065 g30066 g30067 g30068 g30069 g30070 g30071 g30072 g30073 g30074 g30075 g30076 g30077 g30078 g30079 g30080 g30081 g30082 g30083 g30084 g30085 g30086 g30087 g30088 g30089 g30090 g30091 g30092 g30093 g30094 g30095 g30096 g30097 g30098 g30099 g30100</div><div>bjaspan@web-47:~$ groups | wc</div><div> 1 104 730</div><div>bjaspan@web-47:~$ ls -ld /mnt/gfs/g* | head<br></div><div>drwxr-x--- 2 root g30000 6 Feb 3 02:41 /mnt/gfs/g30000</div><div>drwxr-x--- 2 root g30001 6 Feb 3 02:41 /mnt/gfs/g30001</div><div>drwxr-x--- 2 root g30002 6 Feb 3 02:41 /mnt/gfs/g30002</div><div>drwxr-x--- 2 root g30003 6 Feb 3 02:41 /mnt/gfs/g30003</div><div>drwxr-x--- 2 root g30004 6 Feb 3 02:41 /mnt/gfs/g30004</div><div>drwxr-x--- 2 root g30005 6 Feb 3 02:41 /mnt/gfs/g30005</div><div>drwxr-x--- 2 root g30006 6 Feb 3 02:41 /mnt/gfs/g30006</div><div>drwxr-x--- 2 root g30007 6 Feb 3 02:41 /mnt/gfs/g30007</div><div>drwxr-x--- 2 root g30008 6 Feb 3 02:41 /mnt/gfs/g30008</div><div>drwxr-x--- 2 root g30009 6 Feb 3 02:41 /mnt/gfs/g30009</div><div><div>bjaspan@web-47:~$ ls -l /mnt/gfs/g*/. > /dev/null</div></div><div>bjaspan@web-47:~$</div></div><div><br></div><div>However, the same test on Gluster 3.4 gives Permission Denied errors (notice that it gives exactly 18 such errors, and my user is in 50 groups, 50-18=32):</div><div><br></div><div><div>bjaspan@web-28:~$ glusterfs --version | head -n 1</div><div>glusterfs 3.4git built on Nov 21 2013 14:10:04</div><div>bjaspan@web-28:~$ groups</div><div>bjaspan www-data ahops test0003shared693 test0040shared000 remotealias795 test1010shared000 test1011shared576 test1011shared981 g30000 g30001 g30002 g30003 g30004 g30005 g30006 g30007 g30008 g30009 g30010 g30011 g30012 g30013 g30014 g30015 g30016 g30017 g30018 g30019 g30020 g30021 g30022 g30023 g30024 g30025 g30026 g30027 g30028 g30029 g30030 g30031 g30032 g30033 g30034 g30035 g30036 g30037 g30038 g30039 g30040</div><div>bjaspan@web-28:~$ groups | wc</div><div> 1 50 415</div><div>bjaspan@web-28:~$ ls -ld /mnt/gfs/g* | head</div><div>drwxr-x--- 2 root g30000 6 Feb 2 21:52 /mnt/gfs/g30000</div><div>drwxr-x--- 2 root g30001 6 Feb 2 21:52 /mnt/gfs/g30001</div><div>drwxr-x--- 2 root g30002 6 Feb 2 21:52 /mnt/gfs/g30002</div><div>drwxr-x--- 2 root g30003 6 Feb 2 21:52 /mnt/gfs/g30003</div><div>drwxr-x--- 2 root g30004 6 Feb 2 21:52 /mnt/gfs/g30004</div><div>drwxr-x--- 2 root g30005 6 Feb 2 21:52 /mnt/gfs/g30005</div><div>drwxr-x--- 2 root g30006 6 Feb 2 21:52 /mnt/gfs/g30006</div><div>drwxr-x--- 2 root g30007 6 Feb 2 21:52 /mnt/gfs/g30007</div><div>drwxr-x--- 2 root g30008 6 Feb 2 21:52 /mnt/gfs/g30008</div><div>drwxr-x--- 2 root g30009 6 Feb 2 21:52 /mnt/gfs/g30009</div><div>bjaspan@web-28:~$ ls -l /mnt/gfs/g*/. > /dev/null</div><div>ls: cannot open directory /mnt/gfs/g30023/.: Permission denied</div><div>ls: cannot open directory /mnt/gfs/g30024/.: Permission denied</div><div>ls: cannot open directory /mnt/gfs/g30025/.: Permission denied</div><div>ls: cannot open directory /mnt/gfs/g30026/.: Permission denied</div><div>ls: cannot open directory /mnt/gfs/g30027/.: Permission denied</div><div>ls: cannot open directory /mnt/gfs/g30028/.: Permission denied</div><div>ls: cannot open directory /mnt/gfs/g30029/.: Permission denied</div><div>ls: cannot open directory /mnt/gfs/g30030/.: Permission denied</div><div>ls: cannot open directory /mnt/gfs/g30031/.: Permission denied</div><div>ls: cannot open directory /mnt/gfs/g30032/.: Permission denied</div><div>ls: cannot open directory /mnt/gfs/g30033/.: Permission denied</div><div>ls: cannot open directory /mnt/gfs/g30034/.: Permission denied</div><div>ls: cannot open directory /mnt/gfs/g30035/.: Permission denied</div><div>ls: cannot open directory /mnt/gfs/g30036/.: Permission denied</div><div>ls: cannot open directory /mnt/gfs/g30037/.: Permission denied</div><div>ls: cannot open directory /mnt/gfs/g30038/.: Permission denied</div><div>ls: cannot open directory /mnt/gfs/g30039/.: Permission denied</div><div>ls: cannot open directory /mnt/gfs/g30040/.: Permission denied</div><div>bjaspan@web-28:~$</div></div><div><br></div><div>Both of these systems are Ubuntu 12.04. I note that in the previous thread, someone asserted that /proc/$$/status only contains the first 32 groups for a user, and FUSE is using that. On Ubuntu 10.04, /proc/$$/status only contained 32 groups (and Gluster 3.0 worked fine with >32 anyway), but on Ubuntu 12.04, it contains all of the user's groups:</div><div><br></div><div><div>bjaspan@web-28:~$ grep Groups: /proc/$$/status</div><div>Groups:<span class="" style="white-space:pre"> </span>33 2000 3004 10847 10859 10905 11017 11029 11031 30000 30001 30002 30003 30004 30005 30006 30007 30008 30009 30010 30011 30012 30013 30014 30015 30016 30017 30018 30019 30020 30021 30022 30023 30024 30025 30026 30027 30028 30029 30030 30031 30032 30033 30034 30035 30036 30037 30038 30039 30040</div></div><div><br></div><div>So this rules out /proc/$$/status as a factor. My understanding is that Gluster 3.0 uses FUSE, which rules out FUSE as a factor.</div><div><br></div><div>The one factor mentioned in the thread linked above is the RPC AUTH header structure being limited to 400 bytes. So one guess is that the wire protocol changed between Gluster 3.0 and 3.4 changed; perhaps 3.0 did not use Sun/ONC RPC and so is not limited to the 400-byte AUTH header. I haven't checked the code, though, so that is just a guess.</div><div><br></div><div>Does anyone know *for sure* why Gluster 3.4+ is limited to 32 groups and Gluster 3.0 was not?</div><div><br></div><div>Thanks,</div><div><br></div><div>Barry</div><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Barry Jaspan<br><font size="1"><span style="color:rgb(102,102,102)">Chief Software Architect | </span><a style="color:rgb(102,102,102)" href="http://acquia.com" target="_blank">Acquia</a><br style="color:rgb(102,102,102)"><a style="color:rgb(102,102,102)" href="mailto:barry.jaspan@acquia.com" target="_blank">barry.jaspan@acquia.com</a><br style="color:rgb(102,102,102)"></font><br></div></div></div></div></div>
</div></div>