<div dir="ltr">Hi everyone,<div><br></div><div>I managed to fix my problem. It was an old process of gluster which was using up the ports.. i manually killed all the processes and the corresponding /var/run file used by that process before starting a new instance of gluster and it worked fine. </div><div><br></div><div>What i did was : </div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><br></div></blockquote></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div>service glusterfs-server stop</div></div></blockquote></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><br></div></div></blockquote></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div>kill -9 `ps -ef | grep gl | grep -v grep | awk '{print $2}'`</div></div></blockquote></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><br></div></div></blockquote></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div>rm -r /var/lib/glusterd/geo-replication /var/lib/glusterd/glustershd /var/lib/glusterd/groups /var/lib/glusterd/hooks /var/lib/glusterd/nfs /var/lib/glusterd/options /var/lib/glusterd/peers /var/lib/glusterd/quotad /var/lib/glusterd/vols</div></div></blockquote></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><br></div></div></blockquote></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div>service glusterfs-server start</div></div></blockquote></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><br></div></div></blockquote></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div>gluster peer probe gfs1</div></div></blockquote></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><br></div></div></blockquote></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div>service glusterfs-server restart</div></div></blockquote></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><br></div></div></blockquote></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><br></div></div></blockquote></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div>gluster volume sync gfs1 all</div></div></blockquote></div></blockquote></blockquote><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><div>Thank You Kindly,</div><div>Kaamesh</div><div>Bioinformatician</div><div>Novocraft Technologies Sdn Bhd</div><div><div>C-23A-05, 3 Two Square, Section 19, 46300 Petaling Jaya</div><div>Selangor Darul Ehsan</div><div>Malaysia</div></div><div>Mobile: <a value="+60176562635" style="color:rgb(17,85,204)">+60176562635</a></div><div>Ph: <a value="+60379600541" style="color:rgb(17,85,204)">+60379600541</a></div><div>Fax: <a value="+60379600540" style="color:rgb(17,85,204)">+60379600540</a></div></div></div></div>
<br><div class="gmail_quote">On Fri, Feb 27, 2015 at 8:51 AM, Kaamesh Kamalaaharan <span dir="ltr"><<a href="mailto:kaamesh@novocraft.com" target="_blank">kaamesh@novocraft.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi atin,<div><br></div><div>I have tried to flush the iptables and this time i managed to get the peer into cluster. However, the self heal daemon is still offline and im unable to bring the daemon back online on gfs2. Doing a heal on either server gives me a succesful output but when i check the heal info i am getting many split brain errors on gfs2</div><div><br><div class="gmail_extra"><div><div><div dir="ltr"><div>Thank You Kindly,</div><div>Kaamesh</div><div><br></div></div></div></div><div><div class="h5">
<br><div class="gmail_quote">On Thu, Feb 26, 2015 at 5:40 PM, Atin Mukherjee <span dir="ltr"><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Could you check the N/W firewall setting? Flush iptable setting using<br>
iptables -F and retry.<br>
<br>
~Atin<br>
<div><div><br>
On 02/26/2015 02:55 PM, Kaamesh Kamalaaharan wrote:<br>
> Hi guys,<br>
><br>
> I managed to get gluster running but im having a couple of issues with my<br>
> setup 1) my peer status is rejected but connected 2) my self heal daemon is<br>
> not running on one server and im getting split brain files.<br>
> My setup is two gluster volumes (gfs1 and gfs2) on replicate each with a<br>
> brick<br>
><br>
> 1) My peer status doesnt go into Peer in Cluster. running a peer status<br>
> command gives me State:Peer Rejected (Connected) . At this point, the brick<br>
> on gfs2 does not go online and i get this output<br>
><br>
><br>
> #gluster volume status<br>
><br>
> Status of volume: gfsvolume<br>
><br>
> Gluster process Port Online Pid<br>
><br>
> ------------------------------------------------------------------------------<br>
><br>
> Brick gfs1:/export/sda/brick 49153 Y 15025<br>
><br>
> NFS Server on localhost 2049 Y 15039<br>
><br>
> Self-heal Daemon on localhost N/A Y 15044<br>
><br>
><br>
><br>
> Task Status of Volume gfsvolume<br>
><br>
> ------------------------------------------------------------------------------<br>
><br>
> There are no active volume tasks<br>
><br>
><br>
><br>
> I have followed the methods used in one of the threads and performed the<br>
> following<br>
><br>
> a) stop glusterd<br>
> b) rm all files in /var/lib/glusterd/ except for <a href="http://glusterd.info" target="_blank">glusterd.info</a><br>
> c) start glusterd and probe gfs1 from gfs2 and peer status which gives me<br>
><br>
><br>
> # gluster peer status<br>
><br>
> Number of Peers: 1<br>
><br>
><br>
> Hostname: gfs1<br>
><br>
> Uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070<br>
><br>
> State: Sent and Received peer request (Connected)<br>
><br>
><br>
> the same thread mentioned that changing the status of the peer in<br>
> /var/lib/glusterd/peer/{UUID} from status=5 to status=3 fixes this and on<br>
> restart of gfs1 the peer status goes to<br>
><br>
> #gluster peer status<br>
><br>
> Number of Peers: 1<br>
><br>
><br>
> Hostname: gfs1<br>
><br>
> Uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070<br>
><br>
> State: Peer in Cluster (Connected)<br>
><br>
> This fixes the connection between the peers and the volume status shows<br>
><br>
><br>
> Status of volume: gfsvolume<br>
><br>
> Gluster process Port Online Pid<br>
><br>
> ------------------------------------------------------------------------------<br>
><br>
> Brick gfs1:/export/sda/brick 49153 Y 10852<br>
><br>
> Brick gfs2:/export/sda/brick 49152 Y 17024<br>
><br>
> NFS Server on localhost N/A N N/A<br>
><br>
> Self-heal Daemon on localhost N/A N N/A<br>
><br>
> NFS Server on gfs2 N/A N N/A<br>
><br>
> Self-heal Daemon on gfs2 N/A N N/A<br>
><br>
><br>
><br>
> Task Status of Volume gfsvolume<br>
><br>
> ------------------------------------------------------------------------------<br>
><br>
> There are no active volume tasks<br>
><br>
><br>
> Which brings us to problem 2<br>
><br>
> 2) My self-heal demon is not alive<br>
><br>
> I fixed the self heal on gfs1 by running<br>
><br>
> #find <gluster-mount> -noleaf -print0 | xargs --null stat >/dev/null<br>
> 2>/var/log/gluster/<gluster-mount>-selfheal.log<br>
><br>
> and running a volume status command gives me<br>
><br>
> # gluster volume status<br>
><br>
> Status of volume: gfsvolume<br>
><br>
> Gluster process Port Online Pid<br>
><br>
> ------------------------------------------------------------------------------<br>
><br>
> Brick gfs1:/export/sda/brick 49152 Y 16660<br>
><br>
> Brick gfs2:/export/sda/brick 49152 Y 21582<br>
><br>
> NFS Server on localhost 2049 Y 16674<br>
><br>
> Self-heal Daemon on localhost N/A Y 16679<br>
><br>
> NFS Server on gfs2 N/A N 21596<br>
><br>
> Self-heal Daemon on gfs2 N/A N 21600<br>
><br>
><br>
><br>
> Task Status of Volume gfsvolume<br>
><br>
> ------------------------------------------------------------------------------<br>
><br>
> There are no active volume tasks<br>
><br>
><br>
><br>
> However, running this on gfs2 doesnt fix the daemon.<br>
><br>
> Restarting the gfs2 server brings me back to problem 1 and the cycle<br>
> continues..<br>
><br>
> Can anyone assist me with this issue(s).. thank you.<br>
><br>
> Thank You Kindly,<br>
> Kaamesh<br>
><br>
><br>
><br>
</div></div>> _______________________________________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
><br>
<span><font color="#888888"><br>
--<br>
~Atin<br>
</font></span></blockquote></div><br></div></div></div></div></div>
</blockquote></div><br></div>