<div dir="ltr"><div><div><div><div><div>Hi Ravishankar,<br><br></div>I just want to inform that this file have some different properties from other files like this is the file which having the fixed size and when there is no space in file the next data will start wrapping from the top of the file. <br><br></div>Means in this file we are doing the wrapping of the data as well. <br><br></div>So, I just want to know is this feature of file will effect gluster to identify the split-brain or xattr attributes?<br><br></div>Regards,<br></div>Abhishek<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 4, 2016 at 7:00 PM, ABHISHEK PALIWAL <span dir="ltr"><<a href="mailto:abhishpaliwal@gmail.com" target="_blank">abhishpaliwal@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Fri, Mar 4, 2016 at 6:36 PM, Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span>
<div>On 03/04/2016 06:23 PM, ABHISHEK
PALIWAL wrote:<br>
</div>
<blockquote type="cite">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><font size="2"><br>
Ok, just to confirm, glusterd and other brick processes are
running after this node rebooted? <br>
When you run the above command, you need to check
/var/log/glusterfs/glfsheal-volname.log logs errros. Setting
client-log-level to DEBUG would give you a more verbose
message<br>
<br>
</font></div>
</blockquote>
<div>Yes, glusterd and other brick processes running fine. I have
check the <font size="2">
/var/log/glusterfs/glfsheal-volname.log file without the
log-level= DEBUG. Here is the logs from that file<br>
<br>
[2016-03-02 13:51:39.059440] I [MSGID: 101190]
[event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll:
Started thread with index 1<br>
[2016-03-02 13:51:39.072172] W [MSGID: 101012]
[common-utils.c:2776:gf_get_reserved_ports] 0-glusterfs: could
not open the file /proc/sys/net/ipv4/ip_local_reserved_ports
for getting reserved ports info [No such file or directory]<br>
[2016-03-02 13:51:39.072228] W [MSGID: 101081]
[common-utils.c:2810:gf_process_reserved_ports] 0-glusterfs:
Not able to get reserved ports, hence there is a possibility
that glusterfs may consume reserved port<br>
[2016-03-02 13:51:39.072583] E
[socket.c:2278:socket_connect_finish] 0-gfapi: connection to <a href="http://127.0.0.1:24007" target="_blank">127.0.0.1:24007</a>
failed (Connection refused)<br>
</font></div>
</blockquote>
<br>
</span><font size="2">Not sure why ^^ occurs. You could try flushing
iptables (iptables -F), restart glusterd and run the heal info
command again .<br></font></div></blockquote><div><br></div></span><div>No hint from the logs? I'll try your suggestion. <br></div><span class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF"><font size="2">
<br>
</font><span>
<blockquote type="cite">
<div><font size="2">[2016-03-02 13:51:39.072663] E [MSGID: 104024]
[glfs-mgmt.c:738:mgmt_rpc_notify] 0-glfs-mgmt: failed to
connect with remote-host: localhost (Transport endpoint is not
connected) [Transport endpoint is not connected]<br>
[2016-03-02 13:51:39.072700] I [MSGID: 104025]
[glfs-mgmt.c:744:mgmt_rpc_notify] 0-glfs-mgmt: Exhausted all
volfile servers [Transport endpoint is not connected]<br>
</font> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><font size="2"> </font><span>
<blockquote type="cite">
<div dir="ltr">
<div><span><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2"># gluster
volume heal c_glusterfs info split-brain</font>
<br>
<font face="sans-serif" size="2">c_glusterfs:
Not able to fetch volfile from glusterd</font>
<br>
<font face="sans-serif" size="2">Volume heal
failed.</font></span></font></span></font></span></div>
</div>
</blockquote>
<font size="2"><br>
</font>
<blockquote type="cite">
<div dir="ltr">
<div><span><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"> <br>
<br>
</span></font></span></font></span></div>
<span><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times
New Roman" size="2"><span style="font-size:16px">And
based on the your observation I understood
that this is not the problem of split-brain
but <b>is there any way through which can
find out the file which is not in
split-brain as well as not in sync?</b><br>
</span></font></span></font></span></div>
</blockquote>
</span><span><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New
Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2"><br>
`gluster volume heal c_glusterfs info
split-brain` should give you files that need
heal.<br>
</font></span></font></span></font></span></div>
</blockquote>
</blockquote>
</span><font size="2"><br>
Sorry I meant '</font><span><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2">gluster volume heal
c_glusterfs info' should give you the files that need
heal and '</font></span></font></span></font></span><span><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2">gluster volume heal
c_glusterfs info split-brain' the list of files in split-brain.<br>
The commands are detailed in
<a href="https://github.com/gluster/glusterfs-specs/blob/master/done/Features/heal-info-and-split-brain-resolution.md" target="_blank">https://github.com/gluster/glusterfs-specs/blob/master/done/Features/heal-info-and-split-brain-resolution.md</a><br></font></span></font></span></font></span></div></blockquote><div><br></div></span><div>Yes, I have tried this as well It is also giving Number of entries : 0 means no healing is required but the file <font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2">/opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml</font> is not in sync both of brick showing the different version of this file.<br><br></span></font></span></font></div><div><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px">You can see it in the getfattr command outcome as well.<br></span></font></span></font> </div><span class=""><div><br><br><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2"># getfattr -m . -d -e hex /opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml</font> <br>
<font face="sans-serif" size="2">getfattr: Removing leading '/' from absolute path names</font> <br>
<font face="sans-serif" size="2"># file: opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml</font> <br>
<font face="sans-serif" size="2">trusted.afr.c_glusterfs-client-0=0x000000000000000000000000</font> <br>
<font face="sans-serif" size="2">trusted.afr.c_glusterfs-client-2=0x000000000000000000000000</font> <br>
<font face="sans-serif" size="2">trusted.afr.c_glusterfs-client-4=0x000000000000000000000000</font> <br>
<font face="sans-serif" size="2">trusted.afr.c_glusterfs-client-6=0x000000000000000000000000</font> <br>
<font face="sans-serif" size="2">trusted.afr.c_glusterfs-client-8=<b>0x000000060000000000000000</b></font><b> //because client8 is the latest client in our case and starting 8 digits </b></span></font></span></font><b><br></b><div><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2"><b>00000006....are saying like there is something in changelog data.<br></b></font></span></font></span></font></span></font></span></font></div><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"></span></font></span></font><font face="sans-serif" size="2">trusted.afr.dirty=0x000000000000000000000000</font> <br>
<font face="sans-serif" size="2">trusted.bit-rot.version=0x000000000000001356d86c0c000217fd</font> <br>
<font face="sans-serif" size="2">trusted.gfid=0x9f5e354ecfda40149ddce7d5ffe760ae</font> <br>
<br>
<font face="sans-serif" size="2"># lhsh 002500 getfattr -m . -d -e hex /opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml</font> <br>
<font face="sans-serif" size="2">getfattr: Removing leading '/' from absolute path names</font> <br>
<font face="sans-serif" size="2"># file: opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml</font> <br>
<font face="sans-serif" size="2">trusted.afr.c_glusterfs-client-1=<b>0x000000000000000000000000</b></font><b> // and here we can say that there is no split brain but the file is out of sync</b><br>
<font face="sans-serif" size="2">trusted.afr.dirty=0x000000000000000000000000</font> <br>
<font face="sans-serif" size="2">trusted.bit-rot.version=0x000000000000001156d86c290005735c</font> <br>
<font face="sans-serif" size="2">trusted.gfid=0x9f5e354ecfda40149ddce7d5ffe760ae</font> </span></font></span></font><br> </div></span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF"><span><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2">
Regards,<br></font></span></font></span></font></span></div></blockquote><div> Abhishek <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF"><span><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2">
</font></span></font></span></font></span></div></blockquote></div><br></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature"><div dir="ltr"><br><br><br><br>Regards<br>
Abhishek Paliwal<br>
</div></div>
</div>