<div dir="ltr"><div>My intention to initiate the email was more from how to prevent users to hit this problem with a proper work around captured in the release note. We can fork a separate thread on the approach of fixing this issue. Now on the fix, my take is given that glusterd coming up with upgrade mode is just to regenerate the volfiles, we don't need to call gsyncd --version with upgrade=ON and it should solve this specific issue. However from long term perspective we do need to think about versioning the other libraries as pointed out by Kaushal/Niels.<br><br></div><div>The BZ is not yet filed upstream, I think Kotresh would be taking care of that.<br></div><div><br></div>~Atin <br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jul 11, 2016 at 1:19 PM, Niels de Vos <span dir="ltr"><<a href="mailto:ndevos@redhat.com" target="_blank">ndevos@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On Mon, Jul 11, 2016 at 12:56:24PM +0530, Kaushal M wrote:<br>
> On Sat, Jul 9, 2016 at 10:02 PM, Atin Mukherjee <<a href="mailto:amukherj@redhat.com">amukherj@redhat.com</a>> wrote:<br>
> > We have hit a bug 1347250 in downstream (applicable upstream too) where it<br>
> > was seen that glusterd didnt regenerate the volfiles when it was interimly<br>
> > brought up with upgrade mode by yum. Log file captured that gsyncd --version<br>
> > failed to execute and hence glusterd init couldnt proceed till the volfile<br>
> > regeneration. Since the ret code is not handled here in spec file users<br>
> > wouldnt come to know about this and going forward this is going to cause<br>
> > major issues in healing and all and finally it exploits the possibility of<br>
> > split brains at its best.<br>
> ><br>
> > Further analysis by Kotresh & Raghavendra Talur reveals that gsyncd failed<br>
> > here because of the compatibility issue where gsyncd was still not upgraded<br>
> > where as glusterfs-server was and this failure was mainly because of change<br>
> > in the mem type enum. We have seen a similar issue for RDMA as well<br>
> > (probably a year back). So to be very generic this can happen in any upgrade<br>
> > path from one version to another where new mem type is introduced. We have<br>
> > seen this from 3.7.8 to 3.7.12 and 3.8. People upgrading from 3.6 to 3.7/3.8<br>
> > will also experience this issue.<br>
> ><br>
> > Till we work on this fix, I suggest all the release managers to highlight<br>
> > this in the release note of the latest releases with the following work<br>
> > around after yum update:<br>
> ><br>
> > 1. grep -irns "geo-replication module not working as desired"<br>
> > /var/log/glusterfs/etc-glusterfs-glusterd.vol.log | wc -l<br>
> ><br>
> > If the output is non-zero, then go to step 2 else follow the rest of the<br>
> > steps as per the guide.<br>
> ><br>
> > 2.Check if glusterd instance is running or not by 'ps aux | grep glusterd',<br>
> > if it is, then stop the glusterd service.<br>
> ><br>
> > 3. glusterd --xlator-option *.upgrade=on -N<br>
> ><br>
> > and then proceed ahead with rest of the steps as per the guide.<br>
> ><br>
> > Thoughts?<br>
><br>
> Proper .so versioning of libglusterfs should help with problems like<br>
> this. I don't know how to do this though.<br>
<br>
</div></div>We could provde the 'current' version of libglusterfs with the same<br>
number as the op-version. For 3.7.13 it would be 030713, dropping the<br>
prefixed 0 makes that 30713, so libglusterfs.so.30713. The same should<br>
probably be done for all other internal libraries.<br>
<br>
Some more details about library versioning can be found here:<br>
<a href="https://github.com/gluster/glusterfs/blob/master/doc/developer-guide/versioning.md" rel="noreferrer" target="_blank">https://github.com/gluster/glusterfs/blob/master/doc/developer-guide/versioning.md</a><br>
<br>
Note that libgfapi uses symbol versioning, that is a more fine-grained<br>
solution. It prevents the need for applications using the library to get<br>
re-compiled. Details about that, and the more involved changes to get<br>
that to work correctly are in this document:<br>
<a href="https://github.com/gluster/glusterfs/blob/master/doc/developer-guide/gfapi-symbol-versions.md" rel="noreferrer" target="_blank">https://github.com/gluster/glusterfs/blob/master/doc/developer-guide/gfapi-symbol-versions.md</a><br>
<br>
Is there already a bug filed to get this fixed?<br>
<br>
Thanks,<br>
Niels<br>
</blockquote></div><br></div>