<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
<br>
<div class="moz-cite-prefix">On 05/29/2015 03:36 PM, Alessandro De
Salvo wrote:<br>
</div>
<blockquote
cite="mid:FF136FD0-3B5E-49A0-8D0D-B24EE5FCC969@roma1.infn.it"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<div class="">Hi Pranith,</div>
<div class="">thanks to you! 2-3 days are fine, don’t worry.
However, if you can give me the details of the compilation of
glsheal you are mentioning, we could have a quick check if
everything’s fine with the fix, before you release. So just let
me know what you prefer. For me waiting 2-3 days is not a
problem though, as it is not a critical server and I could even
recreate the volumes.</div>
</blockquote>
<br>
We recently introduced code path which frees up memory in long
standing processes. Seems like this is not tested when
file-snapshots feature is on. If that option is disabled the crash
won't happen. "gluster volume heal <volname> info" Uses the
same api. But fortunately this "glfsheal" process will die as soon
as heal info output is gathered. So no need to call this freeing of
memory just before dying. For now we enabled this code path (patch:
<a class="moz-txt-link-freetext" href="http://review.gluster.org/11001">http://review.gluster.org/11001</a>) only for internal builds but not in
released versions while we stabilize that part of the code. You can
take this patch for patching glfsheal.<br>
<br>
Pranith<br>
<blockquote
cite="mid:FF136FD0-3B5E-49A0-8D0D-B24EE5FCC969@roma1.infn.it"
type="cite">
<div class="">Thanks again,</div>
<div class=""><br class="">
</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre">
</span>Alessandro</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">Il giorno 29/mag/2015, alle ore 11:54, Pranith
Kumar Karampuri <<a moz-do-not-send="true"
href="mailto:pkarampu@redhat.com" class="">pkarampu@redhat.com</a>>
ha scritto:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type" class="">
<div bgcolor="#FFFFFF" text="#000000" class=""> <br
class="">
<br class="">
<div class="moz-cite-prefix">On 05/29/2015 03:16 PM,
Alessandro De Salvo wrote:<br class="">
</div>
<blockquote
cite="mid:4355F9A3-DCCB-49D6-986A-57822B41CEFF@roma1.infn.it"
type="cite" class="">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252" class="">
<div class="">Hi Pranith,</div>
<div class="">I’m definitely sure the log is correct,
but you are also correct when you say there is no sign
of crash (even checking with grep!).</div>
<div class="">However I see core dumps (e.g. core.19430)
in /var/log/gluster) created every time I issue the
heal info command.</div>
<div class="">From gdb I see this:</div>
</blockquote>
Thanks for providing the information Alessandro. We will
fix this issue. I am wondering how we can unblock you in
the interim. There is a plan to release 3.7.1 in 2-3 days
I think. I can try to make this fix for that release. Let
me know if you can wait that long? Another possibility is
to compile just glfsheal binary with the fix which
"gluster volume heal <volname> info" internally. Let
me know.<br class="">
<br class="">
Pranith.<br class="">
<blockquote
cite="mid:4355F9A3-DCCB-49D6-986A-57822B41CEFF@roma1.infn.it"
type="cite" class="">
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">GNU gdb (GDB) Red Hat Enterprise Linux
7.6.1-64.el7</div>
<div class="">Copyright (C) 2013 Free Software
Foundation, Inc.</div>
<div class="">License GPLv3+: GNU GPL version 3 or
later <<a moz-do-not-send="true"
href="http://gnu.org/licenses/gpl.html" class="">http://gnu.org/licenses/gpl.html</a>></div>
<div class="">This is free software: you are free to
change and redistribute it.</div>
<div class="">There is NO WARRANTY, to the extent
permitted by law. Type "show copying"</div>
<div class="">and "show warranty" for details.</div>
<div class="">This GDB was configured as
"x86_64-redhat-linux-gnu".</div>
<div class="">For bug reporting instructions, please
see:</div>
<div class=""><<a moz-do-not-send="true"
href="http://www.gnu.org/software/gdb/bugs/"
class="">http://www.gnu.org/software/gdb/bugs/</a>>...</div>
<div class="">Reading symbols from
/usr/sbin/glfsheal...Reading symbols from
/usr/lib/debug/usr/sbin/glfsheal.debug...done.</div>
<div class="">done.</div>
<div class="">[New LWP 19430]</div>
<div class="">[New LWP 19431]</div>
<div class="">[New LWP 19434]</div>
<div class="">[New LWP 19436]</div>
<div class="">[New LWP 19433]</div>
<div class="">[New LWP 19437]</div>
<div class="">[New LWP 19432]</div>
<div class="">[New LWP 19435]</div>
<div class="">[Thread debugging using libthread_db
enabled]</div>
<div class="">Using host libthread_db library
"/lib64/libthread_db.so.1".</div>
<div class="">Core was generated by
`/usr/sbin/glfsheal adsnet-vm-01'.</div>
<div class="">Program terminated with signal 11,
Segmentation fault.</div>
<div class="">#0 inode_unref (inode=0x7f7a1e27806c)
at inode.c:499</div>
<div class="">499 table = inode->table;</div>
<div class="">(gdb) bt</div>
<div class="">#0 inode_unref (inode=0x7f7a1e27806c)
at inode.c:499</div>
<div class="">#1 0x00007f7a265e8a61 in fini
(this=<optimized out>) at qemu-block.c:1092</div>
<div class="">#2 0x00007f7a39a53791 in
xlator_fini_rec (xl=0x7f7a2000b9a0) at xlator.c:463</div>
<div class="">#3 0x00007f7a39a53725 in
xlator_fini_rec (xl=0x7f7a2000d450) at xlator.c:453</div>
<div class="">#4 0x00007f7a39a53725 in
xlator_fini_rec (xl=0x7f7a2000e800) at xlator.c:453</div>
<div class="">#5 0x00007f7a39a53725 in
xlator_fini_rec (xl=0x7f7a2000fbb0) at xlator.c:453</div>
<div class="">#6 0x00007f7a39a53725 in
xlator_fini_rec (xl=0x7f7a20010f80) at xlator.c:453</div>
<div class="">#7 0x00007f7a39a53725 in
xlator_fini_rec (xl=0x7f7a20012330) at xlator.c:453</div>
<div class="">#8 0x00007f7a39a53725 in
xlator_fini_rec (xl=0x7f7a200136e0) at xlator.c:453</div>
<div class="">#9 0x00007f7a39a53725 in
xlator_fini_rec (xl=0x7f7a20014b30) at xlator.c:453</div>
<div class="">#10 0x00007f7a39a53725 in
xlator_fini_rec (xl=0x7f7a20015fc0) at xlator.c:453</div>
<div class="">#11 0x00007f7a39a54eea in
xlator_tree_fini (xl=<optimized out>) at
xlator.c:545</div>
<div class="">#12 0x00007f7a39a90b25 in
glusterfs_graph_deactivate (graph=<optimized
out>) at graph.c:340</div>
<div class="">#13 0x00007f7a38d50e3c in pub_glfs_fini
(fs=fs@entry=0x7f7a3a6b6010) at glfs.c:1155</div>
<div class="">#14 0x00007f7a39f18ed4 in main
(argc=<optimized out>, argv=<optimized
out>) at glfs-heal.c:821</div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class=""><br class="">
</div>
<div class=""><span class="Apple-tab-span"
style="white-space:pre"> </span>Alessandro</div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">Il giorno 29/mag/2015, alle ore 11:12,
Pranith Kumar Karampuri <<a
moz-do-not-send="true"
href="mailto:pkarampu@redhat.com" class="">pkarampu@redhat.com</a>>
ha scritto:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type" class="">
<div bgcolor="#FFFFFF" text="#000000" class=""> <br
class="">
<br class="">
<div class="moz-cite-prefix">On 05/29/2015 02:37
PM, Alessandro De Salvo wrote:<br class="">
</div>
<blockquote
cite="mid:AB7CD500-C547-4E49-B440-14926743C0E8@roma1.infn.it"
type="cite" class="">
<meta http-equiv="Content-Type"
content="text/html; charset=windows-1252"
class="">
<div class="">Hi Pranith,</div>
<div class="">many thanks for the help!</div>
<div class="">The volume info of the
problematic volume is the following:</div>
<div class=""><br class="">
</div>
<div class="">
<div class=""># gluster volume info
adsnet-vm-01</div>
<div class=""> </div>
<div class="">Volume Name: adsnet-vm-01</div>
<div class="">Type: Replicate</div>
<div class="">Volume ID:
f8f615df-3dde-4ea6-9bdb-29a1706e864c</div>
<div class="">Status: Started</div>
<div class="">Number of Bricks: 1 x 2 = 2</div>
<div class="">Transport-type: tcp</div>
<div class="">Bricks:</div>
<div class="">Brick1: <a
moz-do-not-send="true"
href="http://gwads02.sta.adsnet.it/"
class="">gwads02.sta.adsnet.it</a>:/gluster/vm01/data</div>
<div class="">Brick2: <a
moz-do-not-send="true"
href="http://gwads03.sta.adsnet.it/"
class="">gwads03.sta.adsnet.it</a>:/gluster/vm01/data</div>
<div class="">Options Reconfigured:</div>
<div class="">nfs.disable: true</div>
<div class="">features.barrier: disable</div>
<div class="">features.file-snapshot: on</div>
<div class="">server.allow-insecure: on</div>
</div>
</blockquote>
Are you sure the attached log is correct? I do
not see any backtrace in the log file to
indicate there is a crash :-(. Could you do
"grep -i crash /var/log/glusterfs/*" to see if
there is some other file with the crash. If that
also fails, will it be possible for you to
provide the backtrace of the core by opening it
using gdb?<br class="">
<br class="">
Pranith<br class="">
<blockquote
cite="mid:AB7CD500-C547-4E49-B440-14926743C0E8@roma1.infn.it"
type="cite" class="">
<div class=""><br class="">
</div>
<div class="">The log is in attachment.</div>
<div class="">I just wanted to add that the
heal info command works fine on other
volumes hosted by the same machines, so it’s
just this volume which is causing problems.</div>
<div class="">Thanks,</div>
<div class=""><br class="">
</div>
<div class=""><span class="Apple-tab-span"
style="white-space:pre"> </span>Alessandro</div>
<div class=""><br class="">
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<br class="">
<meta http-equiv="Content-Type"
content="text/html; charset=windows-1252"
class="">
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">Il giorno 29/mag/2015, alle
ore 10:50, Pranith Kumar Karampuri <<a
moz-do-not-send="true"
href="mailto:pkarampu@redhat.com"
class="">pkarampu@redhat.com</a>>
ha scritto:</div>
<br class="Apple-interchange-newline">
<div class=""><br style="font-family:
Helvetica; font-size: 12px;
font-style: normal; font-variant:
normal; font-weight: normal;
letter-spacing: normal; line-height:
normal; orphans: auto; text-align:
start; text-indent: 0px;
text-transform: none; white-space:
normal; widows: auto; word-spacing:
0px; -webkit-text-stroke-width: 0px;"
class="">
<br style="font-family: Helvetica;
font-size: 12px; font-style: normal;
font-variant: normal; font-weight:
normal; letter-spacing: normal;
line-height: normal; orphans: auto;
text-align: start; text-indent: 0px;
text-transform: none; white-space:
normal; widows: auto; word-spacing:
0px; -webkit-text-stroke-width: 0px;"
class="">
<span style="font-family: Helvetica;
font-size: 12px; font-style: normal;
font-variant: normal; font-weight:
normal; letter-spacing: normal;
line-height: normal; orphans: auto;
text-align: start; text-indent: 0px;
text-transform: none; white-space:
normal; widows: auto; word-spacing:
0px; -webkit-text-stroke-width: 0px;
float: none; display: inline
!important;" class="">On 05/29/2015
02:18 PM, Pranith Kumar Karampuri
wrote:</span><br style="font-family:
Helvetica; font-size: 12px;
font-style: normal; font-variant:
normal; font-weight: normal;
letter-spacing: normal; line-height:
normal; orphans: auto; text-align:
start; text-indent: 0px;
text-transform: none; white-space:
normal; widows: auto; word-spacing:
0px; -webkit-text-stroke-width: 0px;"
class="">
<blockquote type="cite"
style="font-family: Helvetica;
font-size: 12px; font-style: normal;
font-variant: normal; font-weight:
normal; letter-spacing: normal;
line-height: normal; orphans: auto;
text-align: start; text-indent: 0px;
text-transform: none; white-space:
normal; widows: auto; word-spacing:
0px; -webkit-text-stroke-width: 0px;"
class=""><br class="">
<br class="">
On 05/29/2015 02:13 PM, Alessandro De
Salvo wrote:<br class="">
<blockquote type="cite" class="">Hi,<br
class="">
I'm facing a strange issue with
split brain reporting.<br class="">
I have upgraded to 3.7.0, after
stopping all gluster processes as
described in the twiki, on all
servers hosting the volumes. The
upgrade and the restart was fine,
and the volumes are accessible.<br
class="">
However I had two files in split
brain that I did not heal before
upgrading, so I tried a full heal
with 3.7.0. The heal was launched
correctly, but when I now perform an
heal info there is no output, while
the heal statistics says there are
actually 2 files in split brain. In
the logs I see something like this:<br
class="">
<br class="">
glustershd.log:<br class="">
[2015-05-29 08:28:43.008373] I
[afr-self-heal-entry.c:558:afr_selfheal_entry_do]
0-adsnet-gluster-01-replicate-0:
performing entry selfheal on
7fd1262d-949b-402e-96c2-ae487c8d4e27<br
class="">
[2015-05-29 08:28:43.012690] W
[client-rpc-fops.c:241:client3_3_mknod_cbk]
0-adsnet-gluster-01-client-1: remote
operation failed: Invalid argument.
Path: (null)<br class="">
</blockquote>
Hey could you let us know "gluster
volume info" output? Please let us
know the backtrace printed by
/var/log/glusterfs/glfsheal-<volname>.log
as well.<br class="">
</blockquote>
<span style="font-family: Helvetica;
font-size: 12px; font-style: normal;
font-variant: normal; font-weight:
normal; letter-spacing: normal;
line-height: normal; orphans: auto;
text-align: start; text-indent: 0px;
text-transform: none; white-space:
normal; widows: auto; word-spacing:
0px; -webkit-text-stroke-width: 0px;
float: none; display: inline
!important;" class="">Please attach
/var/log/glusterfs/glfsheal-<volname>.log
file to this thread so that I can take
a look.</span><br style="font-family:
Helvetica; font-size: 12px;
font-style: normal; font-variant:
normal; font-weight: normal;
letter-spacing: normal; line-height:
normal; orphans: auto; text-align:
start; text-indent: 0px;
text-transform: none; white-space:
normal; widows: auto; word-spacing:
0px; -webkit-text-stroke-width: 0px;"
class="">
<br style="font-family: Helvetica;
font-size: 12px; font-style: normal;
font-variant: normal; font-weight:
normal; letter-spacing: normal;
line-height: normal; orphans: auto;
text-align: start; text-indent: 0px;
text-transform: none; white-space:
normal; widows: auto; word-spacing:
0px; -webkit-text-stroke-width: 0px;"
class="">
<span style="font-family: Helvetica;
font-size: 12px; font-style: normal;
font-variant: normal; font-weight:
normal; letter-spacing: normal;
line-height: normal; orphans: auto;
text-align: start; text-indent: 0px;
text-transform: none; white-space:
normal; widows: auto; word-spacing:
0px; -webkit-text-stroke-width: 0px;
float: none; display: inline
!important;" class="">Pranith</span><br
style="font-family: Helvetica;
font-size: 12px; font-style: normal;
font-variant: normal; font-weight:
normal; letter-spacing: normal;
line-height: normal; orphans: auto;
text-align: start; text-indent: 0px;
text-transform: none; white-space:
normal; widows: auto; word-spacing:
0px; -webkit-text-stroke-width: 0px;"
class="">
<blockquote type="cite"
style="font-family: Helvetica;
font-size: 12px; font-style: normal;
font-variant: normal; font-weight:
normal; letter-spacing: normal;
line-height: normal; orphans: auto;
text-align: start; text-indent: 0px;
text-transform: none; white-space:
normal; widows: auto; word-spacing:
0px; -webkit-text-stroke-width: 0px;"
class=""><br class="">
Pranith<br class="">
<blockquote type="cite" class=""><br
class="">
<br class="">
So, it seems like the files to be
healed are not correctly identified,
or at least their path is null.<br
class="">
Also, every time I issue a "gluster
volume heal <volname> info" a
core dump is generated in the log
area.<br class="">
All servers are using the latest
CentOS 7.<br class="">
Any idea why this might be happening
and how to solve it?<br class="">
Thanks,<br class="">
<br class="">
Alessandro<br class="">
<br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
Gluster-users mailing list<br
class="">
<a moz-do-not-send="true"
href="mailto:Gluster-users@gluster.org"
class="">Gluster-users@gluster.org</a><br
class="">
<a moz-do-not-send="true"
href="http://www.gluster.org/mailman/listinfo/gluster-users"
class="">http://www.gluster.org/mailman/listinfo/gluster-users</a><br
class="">
</blockquote>
<br class="">
_______________________________________________<br class="">
Gluster-users mailing list<br class="">
<a moz-do-not-send="true"
href="mailto:Gluster-users@gluster.org"
class="">Gluster-users@gluster.org</a><br
class="">
<a moz-do-not-send="true"
href="http://www.gluster.org/mailman/listinfo/gluster-users"
class="">http://www.gluster.org/mailman/listinfo/gluster-users</a></blockquote>
</div>
</blockquote>
</div>
<br class="">
</blockquote>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</blockquote>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</blockquote>
<br>
</body>
</html>