<div dir="ltr">Hi guys,<div><br></div><div>I managed to get gluster running but im having a couple of issues with my setup 1) my peer status is rejected but connected 2) my self heal daemon is not running on one server and im getting split brain files.</div><div>My setup is two gluster volumes (gfs1 and gfs2) on replicate each with a brick</div><div><br></div><div>1) My peer status doesnt go into Peer in Cluster. running a peer status command gives me State:Peer Rejected (Connected) . At this point, the brick on gfs2 does not go online and i get this output</div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><br></div><div>#gluster volume status</div></blockquote></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>Status of volume: gfsvolume</div></blockquote></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>Gluster process<span class="" style="white-space:pre"> </span>Port<span class="" style="white-space:pre"> </span>Online<span class="" style="white-space:pre"> </span>Pid</div></blockquote></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>------------------------------------------------------------------------------</div></blockquote></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>Brick gfs1:/export/sda/brick<span class="" style="white-space:pre"> </span>49153<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>15025</div></blockquote></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>NFS Server on localhost<span class="" style="white-space:pre"> </span>2049<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>15039</div></blockquote></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>Self-heal Daemon on localhost<span class="" style="white-space:pre"> </span>N/A<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>15044</div></blockquote></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div> </div></blockquote></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>Task Status of Volume gfsvolume</div></blockquote></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>------------------------------------------------------------------------------</div></blockquote></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>There are no active volume tasks</div></blockquote></blockquote></blockquote><div><br></div><div><br></div><div>I have followed the methods used in one of the threads and performed the following</div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div> a) stop glusterd</div><div> b) rm all files in /var/lib/glusterd/ except for <a href="http://glusterd.info">glusterd.info</a></div><div> c) start glusterd and probe gfs1 from gfs2 and peer status which gives me</div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><br></div></blockquote></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div># gluster peer status</div></blockquote></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>Number of Peers: 1</div></div></blockquote></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><br></div></div></blockquote></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>Hostname: gfs1</div></div></blockquote></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>Uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070</div></div></blockquote></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>State: Sent and Received peer request (Connected)</div></div></blockquote></div></blockquote></blockquote><div><br></div><div>the same thread mentioned that changing the status of the peer in /var/lib/glusterd/peer/{UUID} from status=5 to status=3 fixes this and on restart of gfs1 the peer status goes to</div><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>#gluster peer status</div></div></blockquote></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>Number of Peers: 1</div></div></blockquote></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><br></div></div></blockquote></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>Hostname: gfs1</div></div></blockquote></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>Uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070</div></div></blockquote></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>State: Peer in Cluster (Connected)</div></div><div><br></div></blockquote></blockquote></blockquote>This fixes the connection between the peers and the volume status shows<blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><br></div></blockquote></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>Status of volume: gfsvolume</div></div></blockquote></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>Gluster process<span class="" style="white-space:pre"> </span>Port<span class="" style="white-space:pre"> </span>Online<span class="" style="white-space:pre"> </span>Pid</div></div></blockquote></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>------------------------------------------------------------------------------</div></div></blockquote></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>Brick gfs1:/export/sda/brick<span class="" style="white-space:pre"> </span>49153<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>10852</div></div></blockquote></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>Brick gfs2:/export/sda/brick<span class="" style="white-space:pre"> </span>49152<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>17024</div></div></blockquote></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>NFS Server on localhost<span class="" style="white-space:pre"> </span>N/A<span class="" style="white-space:pre"> </span>N<span class="" style="white-space:pre"> </span>N/A</div></div></blockquote></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>Self-heal Daemon on localhost<span class="" style="white-space:pre"> </span>N/A<span class="" style="white-space:pre"> </span>N<span class="" style="white-space:pre"> </span>N/A</div></div></blockquote></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>NFS Server on gfs2<span class="" style="white-space:pre"> </span>N/A<span class="" style="white-space:pre"> </span>N<span class="" style="white-space:pre"> </span>N/A</div></div></blockquote></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>Self-heal Daemon on gfs2<span class="" style="white-space:pre"> </span>N/A<span class="" style="white-space:pre"> </span>N<span class="" style="white-space:pre"> </span>N/A</div></div></blockquote></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div> </div></div></blockquote></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>Task Status of Volume gfsvolume</div></div></blockquote></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>------------------------------------------------------------------------------</div></div></blockquote></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>There are no active volume tasks</div></div><div><br></div><div><br></div></blockquote></div></blockquote></blockquote><div>Which brings us to problem 2</div><div><br></div><div>2) My self-heal demon is not alive</div><div><br></div><div>I fixed the self heal on gfs1 by running </div><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div> #find <gluster-mount> -noleaf -print0 | xargs --null stat >/dev/null 2>/var/log/gluster/<gluster-mount>-selfheal.log</div></div><div><br></div></blockquote></blockquote></blockquote>and running a volume status command gives me<div><br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div># gluster volume status</div></div></blockquote></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div>Status of volume: gfsvolume</div></div></blockquote></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div>Gluster process<span class="" style="white-space:pre"> </span>Port<span class="" style="white-space:pre"> </span>Online<span class="" style="white-space:pre"> </span>Pid</div></div></blockquote></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div>------------------------------------------------------------------------------</div></div></blockquote></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div>Brick gfs1:/export/sda/brick<span class="" style="white-space:pre"> </span>49152<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>16660</div></div></blockquote></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div>Brick gfs2:/export/sda/brick<span class="" style="white-space:pre"> </span>49152<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>21582</div></div></blockquote></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div>NFS Server on localhost<span class="" style="white-space:pre"> </span>2049<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>16674</div></div></blockquote></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div>Self-heal Daemon on localhost<span class="" style="white-space:pre"> </span>N/A<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>16679</div></div></blockquote></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div>NFS Server on gfs2<span class="" style="white-space:pre"> </span>N/A<span class="" style="white-space:pre"> </span>N<span class="" style="white-space:pre"> </span>21596</div></div></blockquote></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div>Self-heal Daemon on gfs2<span class="" style="white-space:pre"> </span>N/A<span class="" style="white-space:pre"> </span>N<span class="" style="white-space:pre"> </span>21600</div></div></blockquote></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div> </div></div></blockquote></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div>Task Status of Volume gfsvolume</div></div></blockquote></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div>------------------------------------------------------------------------------</div></div></blockquote></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div>There are no active volume tasks</div></div></blockquote></blockquote></blockquote><div><div><br></div><div><div> </div></div><div>However, running this on gfs2 doesnt fix the daemon.</div><div><br></div><div>Restarting the gfs2 server brings me back to problem 1 and the cycle continues.. </div><div><br></div><div><div>Can anyone assist me with this issue(s).. thank you.</div><div><br></div><div><div><div class="gmail_signature"><div dir="ltr"><div>Thank You Kindly,</div><div>Kaamesh</div><div><br></div></div></div></div>
</div></div></div></div>