<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 4, 2016 at 6:36 PM, Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span class="">
<div>On 03/04/2016 06:23 PM, ABHISHEK
PALIWAL wrote:<br>
</div>
<blockquote type="cite">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><font size="2"><br>
Ok, just to confirm, glusterd and other brick processes are
running after this node rebooted? <br>
When you run the above command, you need to check
/var/log/glusterfs/glfsheal-volname.log logs errros. Setting
client-log-level to DEBUG would give you a more verbose
message<br>
<br>
</font></div>
</blockquote>
<div>Yes, glusterd and other brick processes running fine. I have
check the <font size="2">
/var/log/glusterfs/glfsheal-volname.log file without the
log-level= DEBUG. Here is the logs from that file<br>
<br>
[2016-03-02 13:51:39.059440] I [MSGID: 101190]
[event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll:
Started thread with index 1<br>
[2016-03-02 13:51:39.072172] W [MSGID: 101012]
[common-utils.c:2776:gf_get_reserved_ports] 0-glusterfs: could
not open the file /proc/sys/net/ipv4/ip_local_reserved_ports
for getting reserved ports info [No such file or directory]<br>
[2016-03-02 13:51:39.072228] W [MSGID: 101081]
[common-utils.c:2810:gf_process_reserved_ports] 0-glusterfs:
Not able to get reserved ports, hence there is a possibility
that glusterfs may consume reserved port<br>
[2016-03-02 13:51:39.072583] E
[socket.c:2278:socket_connect_finish] 0-gfapi: connection to <a href="http://127.0.0.1:24007" target="_blank">127.0.0.1:24007</a>
failed (Connection refused)<br>
</font></div>
</blockquote>
<br>
</span><font size="2">Not sure why ^^ occurs. You could try flushing
iptables (iptables -F), restart glusterd and run the heal info
command again .<br></font></div></blockquote><div><br></div><div>No hint from the logs? I'll try your suggestion. <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF"><font size="2">
<br>
</font><span class="">
<blockquote type="cite">
<div><font size="2">[2016-03-02 13:51:39.072663] E [MSGID: 104024]
[glfs-mgmt.c:738:mgmt_rpc_notify] 0-glfs-mgmt: failed to
connect with remote-host: localhost (Transport endpoint is not
connected) [Transport endpoint is not connected]<br>
[2016-03-02 13:51:39.072700] I [MSGID: 104025]
[glfs-mgmt.c:744:mgmt_rpc_notify] 0-glfs-mgmt: Exhausted all
volfile servers [Transport endpoint is not connected]<br>
</font> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><font size="2"> </font><span>
<blockquote type="cite">
<div dir="ltr">
<div><span><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2"># gluster
volume heal c_glusterfs info split-brain</font>
<br>
<font face="sans-serif" size="2">c_glusterfs:
Not able to fetch volfile from glusterd</font>
<br>
<font face="sans-serif" size="2">Volume heal
failed.</font></span></font></span></font></span></div>
</div>
</blockquote>
<font size="2"><br>
</font>
<blockquote type="cite">
<div dir="ltr">
<div><span><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"> <br>
<br>
</span></font></span></font></span></div>
<span><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times
New Roman" size="2"><span style="font-size:16px">And
based on the your observation I understood
that this is not the problem of split-brain
but <b>is there any way through which can
find out the file which is not in
split-brain as well as not in sync?</b><br>
</span></font></span></font></span></div>
</blockquote>
</span><span><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New
Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2"><br>
`gluster volume heal c_glusterfs info
split-brain` should give you files that need
heal.<br>
</font></span></font></span></font></span></div>
</blockquote>
</blockquote>
</span><font size="2"><br>
Sorry I meant '</font><span><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2">gluster volume heal
c_glusterfs info' should give you the files that need
heal and '</font></span></font></span></font></span><span><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2">gluster volume heal
c_glusterfs info split-brain' the list of files in split-brain.<br>
The commands are detailed in
<a href="https://github.com/gluster/glusterfs-specs/blob/master/done/Features/heal-info-and-split-brain-resolution.md" target="_blank">https://github.com/gluster/glusterfs-specs/blob/master/done/Features/heal-info-and-split-brain-resolution.md</a><br></font></span></font></span></font></span></div></blockquote><div><br></div><div>Yes, I have tried this as well It is also giving Number of entries : 0 means no healing is required but the file <font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2">/opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml</font> is not in sync both of brick showing the different version of this file.<br><br></span></font></span></font></div><div><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px">You can see it in the getfattr command outcome as well.<br></span></font></span></font> </div><div><br><br><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2"># getfattr -m . -d -e hex /opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml</font> <br>
<font face="sans-serif" size="2">getfattr: Removing leading '/' from absolute path names</font> <br>
<font face="sans-serif" size="2"># file: opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml</font> <br>
<font face="sans-serif" size="2">trusted.afr.c_glusterfs-client-0=0x000000000000000000000000</font> <br>
<font face="sans-serif" size="2">trusted.afr.c_glusterfs-client-2=0x000000000000000000000000</font> <br>
<font face="sans-serif" size="2">trusted.afr.c_glusterfs-client-4=0x000000000000000000000000</font> <br>
<font face="sans-serif" size="2">trusted.afr.c_glusterfs-client-6=0x000000000000000000000000</font> <br>
<font face="sans-serif" size="2">trusted.afr.c_glusterfs-client-8=<b>0x000000060000000000000000</b></font><b> //because client8 is the latest client in our case and starting 8 digits </b></span></font></span></font><b><br></b><div><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2"><b>00000006....are saying like there is something in changelog data.<br></b></font></span></font></span></font></span></font></span></font></div><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"></span></font></span></font><font face="sans-serif" size="2">trusted.afr.dirty=0x000000000000000000000000</font> <br>
<font face="sans-serif" size="2">trusted.bit-rot.version=0x000000000000001356d86c0c000217fd</font> <br>
<font face="sans-serif" size="2">trusted.gfid=0x9f5e354ecfda40149ddce7d5ffe760ae</font> <br>
<br>
<font face="sans-serif" size="2"># lhsh 002500 getfattr -m . -d -e hex /opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml</font> <br>
<font face="sans-serif" size="2">getfattr: Removing leading '/' from absolute path names</font> <br>
<font face="sans-serif" size="2"># file: opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml</font> <br>
<font face="sans-serif" size="2">trusted.afr.c_glusterfs-client-1=<b>0x000000000000000000000000</b></font><b> // and here we can say that there is no split brain but the file is out of sync</b><br>
<font face="sans-serif" size="2">trusted.afr.dirty=0x000000000000000000000000</font> <br>
<font face="sans-serif" size="2">trusted.bit-rot.version=0x000000000000001156d86c290005735c</font> <br>
<font face="sans-serif" size="2">trusted.gfid=0x9f5e354ecfda40149ddce7d5ffe760ae</font> </span></font></span></font><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF"><span><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2">
Regards,<br></font></span></font></span></font></span></div></blockquote><div> Abhishek <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF"><span><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2">
</font></span></font></span></font></span></div></blockquote></div><br></div></div>