<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 03/14/2016 10:36 AM, ABHISHEK
PALIWAL wrote:<br>
</div>
<blockquote
cite="mid:CA+15cFM4h=eapdR7mYPRoj3Pi4bunONa63cMxX7q0PRaaun+wg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>Hi Ravishankar,<br>
<br>
</div>
I just want to inform that this file have some different
properties from other files like this is the file which
having the fixed size and when there is no space in file
the next data will start wrapping from the top of the
file. <br>
<br>
</div>
Means in this file we are doing the wrapping of the data
as well. <br>
<br>
</div>
So, I just want to know is this feature of file will effect
gluster to identify the split-brain or xattr attributes?<br>
</div>
</div>
</div>
</blockquote>
Hi,<br>
No it shouldn't matter at what offset the writes happen. The xattrs
only track that the write was missed (and therefore a pending
heal), irrespective of (offset, length).<br>
Ravi<br>
<br>
<blockquote
cite="mid:CA+15cFM4h=eapdR7mYPRoj3Pi4bunONa63cMxX7q0PRaaun+wg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div><br>
</div>
Regards,<br>
</div>
Abhishek<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Mar 4, 2016 at 7:00 PM,
ABHISHEK PALIWAL <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:abhishpaliwal@gmail.com" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:abhishpaliwal@gmail.com">abhishpaliwal@gmail.com</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote"><span class="">On Fri, Mar 4,
2016 at 6:36 PM, Ravishankar N <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:ravishankar@redhat.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:ravishankar@redhat.com">ravishankar@redhat.com</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span>
<div>On 03/04/2016 06:23 PM, ABHISHEK PALIWAL
wrote:<br>
</div>
<blockquote type="cite">
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><font
size="2"><br>
Ok, just to confirm, glusterd and
other brick processes are running
after this node rebooted? <br>
When you run the above command, you
need to check
/var/log/glusterfs/glfsheal-volname.log
logs errros. Setting client-log-level
to DEBUG would give you a more verbose
message<br>
<br>
</font></div>
</blockquote>
<div>Yes, glusterd and other brick processes
running fine. I have check the <font
size="2">
/var/log/glusterfs/glfsheal-volname.log
file without the log-level= DEBUG. Here
is the logs from that file<br>
<br>
[2016-03-02 13:51:39.059440] I [MSGID:
101190]
[event-epoll.c:632:event_dispatch_epoll_worker]
0-epoll: Started thread with index 1<br>
[2016-03-02 13:51:39.072172] W [MSGID:
101012]
[common-utils.c:2776:gf_get_reserved_ports]
0-glusterfs: could not open the file
/proc/sys/net/ipv4/ip_local_reserved_ports
for getting reserved ports info [No such
file or directory]<br>
[2016-03-02 13:51:39.072228] W [MSGID:
101081]
[common-utils.c:2810:gf_process_reserved_ports]
0-glusterfs: Not able to get reserved
ports, hence there is a possibility that
glusterfs may consume reserved port<br>
[2016-03-02 13:51:39.072583] E
[socket.c:2278:socket_connect_finish]
0-gfapi: connection to <a
moz-do-not-send="true"
href="http://127.0.0.1:24007"
target="_blank">127.0.0.1:24007</a>
failed (Connection refused)<br>
</font></div>
</blockquote>
<br>
</span><font size="2">Not sure why ^^ occurs.
You could try flushing iptables (iptables -F),
restart glusterd and run the heal info command
again .<br>
</font></div>
</blockquote>
<div><br>
</div>
</span>
<div>No hint from the logs? I'll try your suggestion.
<br>
</div>
<span class="">
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><font
size="2"> <br>
</font><span>
<blockquote type="cite">
<div><font size="2">[2016-03-02
13:51:39.072663] E [MSGID: 104024]
[glfs-mgmt.c:738:mgmt_rpc_notify]
0-glfs-mgmt: failed to connect with
remote-host: localhost (Transport
endpoint is not connected) [Transport
endpoint is not connected]<br>
[2016-03-02 13:51:39.072700] I [MSGID:
104025]
[glfs-mgmt.c:744:mgmt_rpc_notify]
0-glfs-mgmt: Exhausted all volfile
servers [Transport endpoint is not
connected]<br>
</font> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><font
size="2"> </font><span>
<blockquote type="cite">
<div dir="ltr">
<div><span><font face="Tahoma"
size="2" color="black"><span
style="font-size:10pt"
dir="ltr"><font
face="Times New Roman"
size="2"><span
style="font-size:16px"><font
face="sans-serif"
size="2"># gluster
volume heal
c_glusterfs info
split-brain</font> <br>
<font
face="sans-serif"
size="2">c_glusterfs:
Not able to fetch
volfile from
glusterd</font> <br>
<font
face="sans-serif"
size="2">Volume heal
failed.</font></span></font></span></font></span></div>
</div>
</blockquote>
<font size="2"><br>
</font>
<blockquote type="cite">
<div dir="ltr">
<div><span><font face="Tahoma"
size="2" color="black"><span
style="font-size:10pt"
dir="ltr"><font
face="Times New Roman"
size="2"><span
style="font-size:16px">
<br>
<br>
</span></font></span></font></span></div>
<span><font face="Tahoma" size="2"
color="black"><span
style="font-size:10pt"
dir="ltr"><font face="Times
New Roman" size="2"><span
style="font-size:16px">And
based on the your
observation I understood
that this is not the
problem of split-brain
but <b>is there any way
through which can find
out the file which is
not in split-brain as
well as not in sync?</b><br>
</span></font></span></font></span></div>
</blockquote>
</span><span><font face="Tahoma"
size="2" color="black"><span
style="font-size:10pt" dir="ltr"><font
face="Times New Roman" size="2"><span
style="font-size:16px"><font
face="sans-serif" size="2"><br>
`gluster volume heal
c_glusterfs info
split-brain` should give
you files that need heal.<br>
</font></span></font></span></font></span></div>
</blockquote>
</blockquote>
</span><font size="2"><br>
Sorry I meant '</font><span><font
face="Tahoma" size="2" color="black"><span
style="font-size:10pt" dir="ltr"><font
face="Times New Roman" size="2"><span
style="font-size:16px"><font
face="sans-serif" size="2">gluster
volume heal c_glusterfs info' should
give you the files that need heal
and '</font></span></font></span></font></span><span><font
face="Tahoma" size="2" color="black"><span
style="font-size:10pt" dir="ltr"><font
face="Times New Roman" size="2"><span
style="font-size:16px"><font
face="sans-serif" size="2">gluster
volume heal c_glusterfs info
split-brain' the list of files in
split-brain.<br>
The commands are detailed in
<a moz-do-not-send="true"
href="https://github.com/gluster/glusterfs-specs/blob/master/done/Features/heal-info-and-split-brain-resolution.md"
target="_blank">https://github.com/gluster/glusterfs-specs/blob/master/done/Features/heal-info-and-split-brain-resolution.md</a><br>
</font></span></font></span></font></span></div>
</blockquote>
<div><br>
</div>
</span>
<div>Yes, I have tried this as well It is also giving
Number of entries : 0 means no healing is required
but the file <font face="Tahoma" size="2"
color="black"><span style="font-size:10pt"
dir="ltr"><font face="Times New Roman" size="2"><span
style="font-size:16px"><font
face="sans-serif" size="2">/opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml</font>
is not in sync both of brick showing the
different version of this file.<br>
<br>
</span></font></span></font></div>
<div><font face="Tahoma" size="2" color="black"><span
style="font-size:10pt" dir="ltr"><font
face="Times New Roman" size="2"><span
style="font-size:16px">You can see it in the
getfattr command outcome as well.<br>
</span></font></span></font> </div>
<span class="">
<div><br>
<br>
<font face="Tahoma" size="2" color="black"><span
style="font-size:10pt" dir="ltr"><font
face="Times New Roman" size="2"><span
style="font-size:16px"><font
face="sans-serif" size="2"># getfattr -m
. -d -e hex
/opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml</font>
<br>
<font face="sans-serif" size="2">getfattr:
Removing leading '/' from absolute path
names</font> <br>
<font face="sans-serif" size="2"># file:
opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml</font>
<br>
<font face="sans-serif" size="2">trusted.afr.c_glusterfs-client-0=0x000000000000000000000000</font>
<br>
<font face="sans-serif" size="2">trusted.afr.c_glusterfs-client-2=0x000000000000000000000000</font>
<br>
<font face="sans-serif" size="2">trusted.afr.c_glusterfs-client-4=0x000000000000000000000000</font>
<br>
<font face="sans-serif" size="2">trusted.afr.c_glusterfs-client-6=0x000000000000000000000000</font>
<br>
<font face="sans-serif" size="2">trusted.afr.c_glusterfs-client-8=<b>0x000000060000000000000000</b></font><b>
//because client8 is the latest client
in our case and starting 8 digits </b></span></font></span></font><b><br>
</b>
<div><font face="Tahoma" size="2" color="black"><span
style="font-size:10pt" dir="ltr"><font
face="Times New Roman" size="2"><span
style="font-size:16px"><font
face="Tahoma" size="2" color="black"><span
style="font-size:10pt" dir="ltr"><font
face="Times New Roman" size="2"><span
style="font-size:16px"><font
face="sans-serif" size="2"><b>00000006....are
saying like there is
something in changelog data.<br>
</b></font></span></font></span></font></span></font></span></font></div>
<font face="Tahoma" size="2" color="black"><span
style="font-size:10pt" dir="ltr"><font
face="Times New Roman" size="2"><span
style="font-size:16px"><font face="Tahoma"
size="2" color="black"><span
style="font-size:10pt" dir="ltr"><font
face="Times New Roman" size="2"><span
style="font-size:16px"></span></font></span></font><font
face="sans-serif" size="2">trusted.afr.dirty=0x000000000000000000000000</font>
<br>
<font face="sans-serif" size="2">trusted.bit-rot.version=0x000000000000001356d86c0c000217fd</font>
<br>
<font face="sans-serif" size="2">trusted.gfid=0x9f5e354ecfda40149ddce7d5ffe760ae</font>
<br>
<br>
<font face="sans-serif" size="2"># lhsh
002500 getfattr -m . -d -e hex
/opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml</font>
<br>
<font face="sans-serif" size="2">getfattr:
Removing leading '/' from absolute path
names</font> <br>
<font face="sans-serif" size="2"># file:
opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml</font>
<br>
<font face="sans-serif" size="2">trusted.afr.c_glusterfs-client-1=<b>0x000000000000000000000000</b></font><b>
// and here we can say that there is no
split brain but the file is out of sync</b><br>
<font face="sans-serif" size="2">trusted.afr.dirty=0x000000000000000000000000</font>
<br>
<font face="sans-serif" size="2">trusted.bit-rot.version=0x000000000000001156d86c290005735c</font>
<br>
<font face="sans-serif" size="2">trusted.gfid=0x9f5e354ecfda40149ddce7d5ffe760ae</font>
</span></font></span></font><br>
</div>
</span>
<blockquote class="gmail_quote" style="margin:0px 0px
0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span><font
face="Tahoma" size="2" color="black"><span
style="font-size:10pt" dir="ltr"><font
face="Times New Roman" size="2"><span
style="font-size:16px"><font
face="sans-serif" size="2"> Regards,<br>
</font></span></font></span></font></span></div>
</blockquote>
<div> Abhishek <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span><font
face="Tahoma" size="2" color="black"><span
style="font-size:10pt" dir="ltr"><font
face="Times New Roman" size="2"><span
style="font-size:16px"><font
face="sans-serif" size="2"> </font></span></font></span></font></span></div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div class="gmail_signature">
<div dir="ltr"><br>
<br>
<br>
<br>
Regards<br>
Abhishek Paliwal<br>
</div>
</div>
</div>
</blockquote>
<br>
<br>
</body>
</html>