<div dir="ltr">Then how can I resolved this issue?<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Mar 14, 2016 at 1:37 PM, Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span class="">
<div>On 03/14/2016 10:36 AM, ABHISHEK
PALIWAL wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>Hi Ravishankar,<br>
<br>
</div>
I just want to inform that this file have some different
properties from other files like this is the file which
having the fixed size and when there is no space in file
the next data will start wrapping from the top of the
file. <br>
<br>
</div>
Means in this file we are doing the wrapping of the data
as well. <br>
<br>
</div>
So, I just want to know is this feature of file will effect
gluster to identify the split-brain or xattr attributes?<br>
</div>
</div>
</div>
</blockquote></span>
Hi,<br>
No it shouldn't matter at what offset the writes happen. The xattrs
only track that the write was missed (and therefore a pending
heal), irrespective of (offset, length).<br>
Ravi<div><div class="h5"><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>
<div><br>
</div>
Regards,<br>
</div>
Abhishek<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Mar 4, 2016 at 7:00 PM,
ABHISHEK PALIWAL <span dir="ltr"><<a href="mailto:abhishpaliwal@gmail.com" target="_blank"></a><a href="mailto:abhishpaliwal@gmail.com" target="_blank">abhishpaliwal@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote"><span>On Fri, Mar 4,
2016 at 6:36 PM, Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank"></a><a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span>
<div>On 03/04/2016 06:23 PM, ABHISHEK PALIWAL
wrote:<br>
</div>
<blockquote type="cite">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><font size="2"><br>
Ok, just to confirm, glusterd and
other brick processes are running
after this node rebooted? <br>
When you run the above command, you
need to check
/var/log/glusterfs/glfsheal-volname.log
logs errros. Setting client-log-level
to DEBUG would give you a more verbose
message<br>
<br>
</font></div>
</blockquote>
<div>Yes, glusterd and other brick processes
running fine. I have check the <font size="2">
/var/log/glusterfs/glfsheal-volname.log
file without the log-level= DEBUG. Here
is the logs from that file<br>
<br>
[2016-03-02 13:51:39.059440] I [MSGID:
101190]
[event-epoll.c:632:event_dispatch_epoll_worker]
0-epoll: Started thread with index 1<br>
[2016-03-02 13:51:39.072172] W [MSGID:
101012]
[common-utils.c:2776:gf_get_reserved_ports]
0-glusterfs: could not open the file
/proc/sys/net/ipv4/ip_local_reserved_ports
for getting reserved ports info [No such
file or directory]<br>
[2016-03-02 13:51:39.072228] W [MSGID:
101081]
[common-utils.c:2810:gf_process_reserved_ports]
0-glusterfs: Not able to get reserved
ports, hence there is a possibility that
glusterfs may consume reserved port<br>
[2016-03-02 13:51:39.072583] E
[socket.c:2278:socket_connect_finish]
0-gfapi: connection to <a href="http://127.0.0.1:24007" target="_blank">127.0.0.1:24007</a>
failed (Connection refused)<br>
</font></div>
</blockquote>
<br>
</span><font size="2">Not sure why ^^ occurs.
You could try flushing iptables (iptables -F),
restart glusterd and run the heal info command
again .<br>
</font></div>
</blockquote>
<div><br>
</div>
</span>
<div>No hint from the logs? I'll try your suggestion.
<br>
</div>
<span>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><font size="2"> <br>
</font><span>
<blockquote type="cite">
<div><font size="2">[2016-03-02
13:51:39.072663] E [MSGID: 104024]
[glfs-mgmt.c:738:mgmt_rpc_notify]
0-glfs-mgmt: failed to connect with
remote-host: localhost (Transport
endpoint is not connected) [Transport
endpoint is not connected]<br>
[2016-03-02 13:51:39.072700] I [MSGID:
104025]
[glfs-mgmt.c:744:mgmt_rpc_notify]
0-glfs-mgmt: Exhausted all volfile
servers [Transport endpoint is not
connected]<br>
</font> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><font size="2"> </font><span>
<blockquote type="cite">
<div dir="ltr">
<div><span><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2"># gluster
volume heal
c_glusterfs info
split-brain</font> <br>
<font face="sans-serif" size="2">c_glusterfs:
Not able to fetch
volfile from
glusterd</font> <br>
<font face="sans-serif" size="2">Volume heal
failed.</font></span></font></span></font></span></div>
</div>
</blockquote>
<font size="2"><br>
</font>
<blockquote type="cite">
<div dir="ltr">
<div><span><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px">
<br>
<br>
</span></font></span></font></span></div>
<span><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times
New Roman" size="2"><span style="font-size:16px">And
based on the your
observation I understood
that this is not the
problem of split-brain
but <b>is there any way
through which can find
out the file which is
not in split-brain as
well as not in sync?</b><br>
</span></font></span></font></span></div>
</blockquote>
</span><span><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2"><br>
`gluster volume heal
c_glusterfs info
split-brain` should give
you files that need heal.<br>
</font></span></font></span></font></span></div>
</blockquote>
</blockquote>
</span><font size="2"><br>
Sorry I meant '</font><span><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2">gluster
volume heal c_glusterfs info' should
give you the files that need heal
and '</font></span></font></span></font></span><span><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2">gluster
volume heal c_glusterfs info
split-brain' the list of files in
split-brain.<br>
The commands are detailed in
<a href="https://github.com/gluster/glusterfs-specs/blob/master/done/Features/heal-info-and-split-brain-resolution.md" target="_blank">https://github.com/gluster/glusterfs-specs/blob/master/done/Features/heal-info-and-split-brain-resolution.md</a><br>
</font></span></font></span></font></span></div>
</blockquote>
<div><br>
</div>
</span>
<div>Yes, I have tried this as well It is also giving
Number of entries : 0 means no healing is required
but the file <font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2">/opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml</font>
is not in sync both of brick showing the
different version of this file.<br>
<br>
</span></font></span></font></div>
<div><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px">You can see it in the
getfattr command outcome as well.<br>
</span></font></span></font> </div>
<span>
<div><br>
<br>
<font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2"># getfattr -m
. -d -e hex
/opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml</font>
<br>
<font face="sans-serif" size="2">getfattr:
Removing leading '/' from absolute path
names</font> <br>
<font face="sans-serif" size="2"># file:
opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml</font>
<br>
<font face="sans-serif" size="2">trusted.afr.c_glusterfs-client-0=0x000000000000000000000000</font>
<br>
<font face="sans-serif" size="2">trusted.afr.c_glusterfs-client-2=0x000000000000000000000000</font>
<br>
<font face="sans-serif" size="2">trusted.afr.c_glusterfs-client-4=0x000000000000000000000000</font>
<br>
<font face="sans-serif" size="2">trusted.afr.c_glusterfs-client-6=0x000000000000000000000000</font>
<br>
<font face="sans-serif" size="2">trusted.afr.c_glusterfs-client-8=<b>0x000000060000000000000000</b></font><b>
//because client8 is the latest client
in our case and starting 8 digits </b></span></font></span></font><b><br>
</b>
<div><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2"><b>00000006....are
saying like there is
something in changelog data.<br>
</b></font></span></font></span></font></span></font></span></font></div>
<font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"></span></font></span></font><font face="sans-serif" size="2">trusted.afr.dirty=0x000000000000000000000000</font>
<br>
<font face="sans-serif" size="2">trusted.bit-rot.version=0x000000000000001356d86c0c000217fd</font>
<br>
<font face="sans-serif" size="2">trusted.gfid=0x9f5e354ecfda40149ddce7d5ffe760ae</font>
<br>
<br>
<font face="sans-serif" size="2"># lhsh
002500 getfattr -m . -d -e hex
/opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml</font>
<br>
<font face="sans-serif" size="2">getfattr:
Removing leading '/' from absolute path
names</font> <br>
<font face="sans-serif" size="2"># file:
opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml</font>
<br>
<font face="sans-serif" size="2">trusted.afr.c_glusterfs-client-1=<b>0x000000000000000000000000</b></font><b>
// and here we can say that there is no
split brain but the file is out of sync</b><br>
<font face="sans-serif" size="2">trusted.afr.dirty=0x000000000000000000000000</font>
<br>
<font face="sans-serif" size="2">trusted.bit-rot.version=0x000000000000001156d86c290005735c</font>
<br>
<font face="sans-serif" size="2">trusted.gfid=0x9f5e354ecfda40149ddce7d5ffe760ae</font>
</span></font></span></font><br>
</div>
</span>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2"> Regards,<br>
</font></span></font></span></font></span></div>
</blockquote>
<div> Abhishek <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span><font face="Tahoma" size="2" color="black"><span style="font-size:10pt" dir="ltr"><font face="Times New Roman" size="2"><span style="font-size:16px"><font face="sans-serif" size="2"> </font></span></font></span></font></span></div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div>
<div dir="ltr"><br>
<br>
<br>
<br>
Regards<br>
Abhishek Paliwal<br>
</div>
</div>
</div>
</blockquote>
<br>
<br>
</div></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature"><div dir="ltr"><br><br><br><br>Regards<br>
Abhishek Paliwal<br>
</div></div>
</div>