<div dir="ltr"><div><div><div>I still see the release notes for 3.8.1 & 3.7.13 not reflecting this change. <br><br></div>Niels, Kaushal,<br><br></div>Shouldn't we highlight this as early as possible to the users given release note is the best possible medium to capture all the known issues and the work around?<br><br></div><br><div><div>~Atin<br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jul 9, 2016 at 10:02 PM, Atin Mukherjee <span dir="ltr"><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">We have hit a bug 1347250 in downstream (applicable upstream too) where it was seen that glusterd didnt regenerate the volfiles when it was interimly brought up with upgrade mode by yum. Log file captured that gsyncd --version failed to execute and hence glusterd init couldnt proceed till the volfile regeneration. Since the ret code is not handled here in spec file users wouldnt come to know about this and going forward this is going to cause major issues in healing and all and finally it exploits the possibility of split brains at its best.<div><br></div><div>Further analysis by Kotresh & Raghavendra Talur reveals that gsyncd failed here because of the compatibility issue where gsyncd was still not upgraded where as glusterfs-server was and this failure was mainly because of change in the mem type enum. We have seen a similar issue for RDMA as well (probably a year back). So to be very generic this can happen in any upgrade path from one version to another where new mem type is introduced. We have seen this from 3.7.8 to 3.7.12 and 3.8. People upgrading from 3.6 to 3.7/3.8 will also experience this issue.<span></span></div><div><br></div><div>Till we work on this fix, I suggest all the release managers to highlight this in the release note of the latest releases with the following work around after yum update:</div><div><pre style="word-wrap:break-word;width:50em"><font face="Helvetica Neue, Helvetica, Arial, sans-serif" size="3"><span style="white-space:normal;background-color:rgba(255,255,255,0)">1. grep -irns "geo-replication module not working as desired" /var/log/glusterfs/etc-glusterfs-glusterd.vol.log | wc -l </span></font></pre><pre style="word-wrap:break-word;width:50em"><font face="Helvetica Neue, Helvetica, Arial, sans-serif" size="3"><span style="white-space:normal;background-color:rgba(255,255,255,0)"> If the output is non-zero, then go to step 2 else follow the rest of the steps as per the guide.</span></font></pre><pre style="word-wrap:break-word;width:50em"><font face="Helvetica Neue, Helvetica, Arial, sans-serif" size="3"><span style="white-space:normal;background-color:rgba(255,255,255,0)">2.Check if glusterd instance is running or not by 'ps aux | grep glusterd', if it is, then stop the glusterd service. </span></font></pre><pre style="word-wrap:break-word;width:50em"><font face="Helvetica Neue, Helvetica, Arial, sans-serif" size="3"><span style="white-space:normal;background-color:rgba(255,255,255,0)"> 3. glusterd --xlator-option *.upgrade=on -N
and then proceed ahead with rest of the steps as per the guide.</span></font></pre></div><div>Thoughts?</div><div><br></div><div>P.S : this email is limited to maintainers till we decide on the approach to highlight this issues to the users</div><span class="HOEnZb"><font color="#888888"><br><br>-- <br>Atin<br>Sent from iPhone<br>
</font></span></blockquote></div><br></div></div>