<div dir="ltr">I have replicated my upgradable environment in a testing lab with the following configuration:<br><br>Distributed gluster (one brick per node)<br>- Node gluster-1: glusterfs version 3.7.4<br>- Node gluster-2: glusterfs version 3.7.4<br>- Node gluster-3: glusterfs version 3.7.4<br><br>I began by upgrading only the first node to newest version (3.7.5). <br><br>root@gluster-1 ~]# gluster --version<div>glusterfs 3.7.5 built on Oct 7 2015 16:27:05<br><br>When I tried to request the status of gluster volume, I got these error messages:<br><br><div>[root@gluster-1 ~]# gluster volume status</div><div>Staging failed on gluster-2. Please check log file for details.</div><div>Staging failed on gluster-3. Please check log file for details.<br><br>In node gluster-2, tailed messages from /var/log/glusterfs/etc-glusterfs-glusterd.vol.log:<br><br><div>[2015-10-26 10:50:16.378672] E [MSGID: 106062] [glusterd-volume-ops.c:1796:glusterd_op_stage_heal_volume] 0-glusterd: Unable to get volume name</div><div>[2015-10-26 10:50:16.378735] E [MSGID: 106301] [glusterd-op-sm.c:5171:glusterd_op_ac_stage_op] 0-management: Stage failed on operation 'Volume Heal', Status : -2<br><br><br>On the other hand, if I upgrade all the nodes at the same time, everything seems working fine!<br><br>The issue may be when nodes have different versions (3.7.4 and 3.7.5).<br>Is this a normal behavior? It is needed to stop the entire cluster?<br><br></div></div><br></div><div class="gmail_extra">Regards,</div><div class="gmail_extra"><div><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><p style="margin:0cm 0cm 0.0001pt;font-size:12pt"><font color="#351c75" face="arial, helvetica, sans-serif"><span lang="EN-US" style="font-size:11pt">.....................................................................</span></font></p><div style="font-size:12.8000001907349px"><p style="margin:0cm 0cm 0.0001pt;font-size:12pt"><font face="arial, helvetica, sans-serif"><span style="font-size:11pt"><font color="#666666">Juan Francisco Rodríguez Cardoso</font></span><br></font></p><p style="margin:0cm 0cm 0.0001pt;font-size:12pt"><font face="arial, helvetica, sans-serif"><span style="font-size:11pt;color:rgb(153,153,153)"><a href="mailto:jfrodriguez@keedio.com" style="color:rgb(17,85,204)" target="_blank">jfrodriguez@keedio.com</a> | </span><a value="+34656452646" style="color:rgb(17,85,204);font-size:14.6666669845581px">+34 636 69 26 91</a></font></p></div><p style="margin:0cm 0cm 0.0001pt;font-size:12pt"><span style="font-size:11pt;color:rgb(153,153,153)"><a href="http://www.keedio.com/" style="color:rgb(17,85,204)" target="_blank"><font face="arial, helvetica, sans-serif">www.keedio.com</font></a></span></p><p style="margin:0cm 0cm 0.0001pt;font-size:12pt"><span style="font-size:11pt"><font color="#351c75" face="arial, helvetica, sans-serif">.....................................................................</font></span></p></div></div></div></div></div></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On 26 October 2015 at 11:48, Alan Orth <span dir="ltr"><<a href="mailto:alan.orth@gmail.com" target="_blank">alan.orth@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>We're debating updating from 3.5.x to 3.7.x soon on our 2x2 replica set and these upgrade issues are a bit worrying. Can I hear a few voices from people who have had positive experiences? :)</div><div><br></div><div>Thanks,</div><div><br></div><div>Alan</div></div><div class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">On Fri, Oct 23, 2015 at 6:32 PM, JuanFra Rodríguez Cardoso <span dir="ltr"><<a href="mailto:jfrodriguez@keedio.com" target="_blank">jfrodriguez@keedio.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I had that problem too, but I'm not able to fix it. I was forced to downgrade to 3.7.4 to continue running my gluster volumes.<br><br>The upgrading process (3.7.4 -> 3.7.5) does not seem fully reliable.<div class="gmail_extra"><br>Best.<br clear="all"><div><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><p style="margin:0cm 0cm 0.0001pt;font-size:12pt"><font color="#351c75" face="arial, helvetica, sans-serif"><span lang="EN-US" style="font-size:11pt">.....................................................................</span></font></p><div style="font-size:12.8000001907349px"><p style="margin:0cm 0cm 0.0001pt;font-size:12pt"><font face="arial, helvetica, sans-serif"><span style="font-size:11pt"><font color="#666666">Juan Francisco Rodríguez Cardoso</font></span><br></font></p><p style="margin:0cm 0cm 0.0001pt;font-size:12pt"><font face="arial, helvetica, sans-serif"><span style="font-size:11pt;color:rgb(153,153,153)"><a href="mailto:jfrodriguez@keedio.com" style="color:rgb(17,85,204)" target="_blank">jfrodriguez@keedio.com</a> | </span><a value="+34656452646" style="color:rgb(17,85,204);font-size:14.6666669845581px">+34 636 69 26 91</a></font></p></div><p style="margin:0cm 0cm 0.0001pt;font-size:12pt"><span style="font-size:11pt;color:rgb(153,153,153)"><a href="http://www.keedio.com/" style="color:rgb(17,85,204)" target="_blank"><font face="arial, helvetica, sans-serif">www.keedio.com</font></a></span></p><p style="margin:0cm 0cm 0.0001pt;font-size:12pt"><span style="font-size:11pt"><font color="#351c75" face="arial, helvetica, sans-serif">.....................................................................</font></span></p></div></div></div></div></div></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote"><div><div>On 16 October 2015 at 15:24, David Robinson <span dir="ltr"><<a href="mailto:david.robinson@corvidtec.com" target="_blank">david.robinson@corvidtec.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>
<div bgcolor="#ffffff">
<div>That log was the frick one, which is the node that I upgraded. The frack one is attached. One thing I did notice was the errors below in the etc log file. The /usr/lib64/glusterfs/3.7.5 directory doesn't exist yet on frack. </div>
<div> </div>
<div>+------------------------------------------------------------------------------+<br>[2015-10-16 12:04:06.235993] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2<br>[2015-10-16 12:04:06.236036] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1<br>[2015-10-16 12:04:06.236099] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2<br>[2015-10-16 12:04:09.242413] E [socket.c:2278:socket_connect_finish] 0-management: connection to <a href="http://10.200.82.1:24007" target="_blank">10.200.82.1:24007</a> failed (No route to host)<br>[2015-10-16 12:04:09.242504] I [MSGID: 106004] [glusterd-handler.c:5056:__glusterd_peer_rpc_notify] 0-management: Peer <<a href="http://frackib01.corvidtec.com" target="_blank">frackib01.corvidtec.com</a>> (<8ab9a966-d536-4bd1-828a-64b2d72c47ca>), in state <Peer in Cluster>, has disconnected from glusterd.<br>[2015-10-16 12:04:09.726895] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 14, Invalid argument<br>[2015-10-16 12:04:09.726918] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument<br>[2015-10-16 12:04:09.902756] W [MSGID: 101095] [xlator.c:143:xlator_volopt_dynload] 0-xlator: <strong><em>/usr/lib64/glusterfs/3.7.5/xlator/rpc-transport/socket.so:</em></strong> cannot open shared object file: No such file or directory<br></div><div><div>
<div> </div>
<div> </div>
<div>------ Original Message ------</div>
<div>From: "Mohammed Rafi K C" <<a href="mailto:rkavunga@redhat.com" target="_blank">rkavunga@redhat.com</a>></div>
<div>To: "David Robinson" <<a href="mailto:drobinson@corvidtec.com" target="_blank">drobinson@corvidtec.com</a>>; "<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>" <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>>; "Gluster Devel" <<a href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a>></div>
<div>Sent: 10/16/2015 8:43:21 AM</div>
<div>Subject: Re: [Gluster-devel] 3.7.5 upgrade issues</div>
<div> </div>
<div style="COLOR:#000000">
<blockquote cite="http://5620F0E9.6010007@redhat.com" type="cite">Hi David,<br><br>The logs you attached, are they from node "<a href="http://frackib01.corvidtec.com" target="_blank">frackib01.corvidtec.com</a>", if not can you attach logs from the node "<a href="http://frackib01.corvidtec.com" target="_blank">frackib01.corvidtec.com</a>" ?<br><br>Regards<br>Rafi KC<br>
<div>On 10/16/2015 05:46 PM, David Robinson wrote:<br></div>
<blockquote type="cite">
<div>I have a replica pair setup that I was trying to upgrade from 3.7.4 to 3.7.5. </div>
<div>After upgrading the rpm packages (rpm -Uvh *.rpm) and rebooting one of the nodes, I am now receiving the following:</div>
<div> </div>
<div>[root@frick01 log]# gluster volume status<br>Staging failed on <a href="http://frackib01.corvidtec.com" target="_blank">frackib01.corvidtec.com</a>. Please check log file for details.</div>
<div> </div>
<div> </div>
<div> </div>
<div>The logs are attached and my setup is shown below. Can anyone help?</div>
<div> </div>
<div>[root@frick01 log]# gluster volume info<br> <br>Volume Name: gfs<br>Type: Replicate<br>Volume ID: abc63b5c-bed7-4e3d-9057-00930a2d85d3<br>Status: Started<br>Number of Bricks: 1 x 2 = 2<br>Transport-type: tcp,rdma<br>Bricks:<br>Brick1: frickib01.corvidtec.com:/data/brick01/gfs<br>Brick2: frackib01.corvidtec.com:/data/brick01/gfs<br>Options Reconfigured:<br>storage.owner-gid: 100<br>server.allow-insecure: on<br>performance.readdir-ahead: on<br>server.event-threads: 4<br>client.event-threads: 4<br></div>
<div>David</div>
<div> </div><br>
<fieldset></fieldset> <br><pre>_______________________________________________
Gluster-devel mailing list
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a></pre></blockquote><br></blockquote></div></div></div></div><br></div></div>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div></div>
<br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br><br clear="all"><div><br></div></div></div><span class="HOEnZb"><font color="#888888">-- <br><div><div dir="ltr">Alan Orth<br><a href="mailto:alan.orth@gmail.com" target="_blank">alan.orth@gmail.com</a><br><a href="https://alaninkenya.org" target="_blank">https://alaninkenya.org</a><br><a href="https://mjanja.ch" target="_blank">https://mjanja.ch</a><br>"In heaven all the interesting people are missing." -Friedrich Nietzsche<div>GPG public key ID: 0x8cb0d0acb5cd81ec209c6cdfbd1a0e09c2f836c0<br></div></div></div>
</font></span></div>
</blockquote></div><br></div></div>