<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""></div><div class=""><br class=""></div><br class=""><div><blockquote type="cite" class=""><div class="">Il giorno 29/mag/2015, alle ore 11:46, Alessandro De Salvo <<a href="mailto:Alessandro.DeSalvo@roma1.infn.it" class="">Alessandro.DeSalvo@roma1.infn.it</a>> ha scritto:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html charset=windows-1252" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Hi Pranith,</div><div class="">I’m definitely sure the log is correct, but you are also correct when you say there is no sign of crash (even checking with grep!).</div><div class="">However I see core dumps (e.g. core.19430) in /var/log/gluster) created every time I issue the heal info command.</div><div class="">From gdb I see this:</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class="">GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-64.el7</div><div class="">Copyright (C) 2013 Free Software Foundation, Inc.</div><div class="">License GPLv3+: GNU GPL version 3 or later <<a href="http://gnu.org/licenses/gpl.html" class="">http://gnu.org/licenses/gpl.html</a>></div><div class="">This is free software: you are free to change and redistribute it.</div><div class="">There is NO WARRANTY, to the extent permitted by law. Type "show copying"</div><div class="">and "show warranty" for details.</div><div class="">This GDB was configured as "x86_64-redhat-linux-gnu".</div><div class="">For bug reporting instructions, please see:</div><div class=""><<a href="http://www.gnu.org/software/gdb/bugs/" class="">http://www.gnu.org/software/gdb/bugs/</a>>...</div><div class="">Reading symbols from /usr/sbin/glfsheal...Reading symbols from /usr/lib/debug/usr/sbin/glfsheal.debug...done.</div><div class="">done.</div><div class="">[New LWP 19430]</div><div class="">[New LWP 19431]</div><div class="">[New LWP 19434]</div><div class="">[New LWP 19436]</div><div class="">[New LWP 19433]</div><div class="">[New LWP 19437]</div><div class="">[New LWP 19432]</div><div class="">[New LWP 19435]</div><div class="">[Thread debugging using libthread_db enabled]</div><div class="">Using host libthread_db library "/lib64/libthread_db.so.1".</div><div class="">Core was generated by `/usr/sbin/glfsheal adsnet-vm-01'.</div><div class="">Program terminated with signal 11, Segmentation fault.</div><div class="">#0 inode_unref (inode=0x7f7a1e27806c) at inode.c:499</div><div class="">499 table = inode->table;</div><div class="">(gdb) bt</div><div class="">#0 inode_unref (inode=0x7f7a1e27806c) at inode.c:499</div><div class="">#1 0x00007f7a265e8a61 in fini (this=<optimized out>) at qemu-block.c:1092</div><div class="">#2 0x00007f7a39a53791 in xlator_fini_rec (xl=0x7f7a2000b9a0) at xlator.c:463</div><div class="">#3 0x00007f7a39a53725 in xlator_fini_rec (xl=0x7f7a2000d450) at xlator.c:453</div><div class="">#4 0x00007f7a39a53725 in xlator_fini_rec (xl=0x7f7a2000e800) at xlator.c:453</div><div class="">#5 0x00007f7a39a53725 in xlator_fini_rec (xl=0x7f7a2000fbb0) at xlator.c:453</div><div class="">#6 0x00007f7a39a53725 in xlator_fini_rec (xl=0x7f7a20010f80) at xlator.c:453</div><div class="">#7 0x00007f7a39a53725 in xlator_fini_rec (xl=0x7f7a20012330) at xlator.c:453</div><div class="">#8 0x00007f7a39a53725 in xlator_fini_rec (xl=0x7f7a200136e0) at xlator.c:453</div><div class="">#9 0x00007f7a39a53725 in xlator_fini_rec (xl=0x7f7a20014b30) at xlator.c:453</div><div class="">#10 0x00007f7a39a53725 in xlator_fini_rec (xl=0x7f7a20015fc0) at xlator.c:453</div><div class="">#11 0x00007f7a39a54eea in xlator_tree_fini (xl=<optimized out>) at xlator.c:545</div><div class="">#12 0x00007f7a39a90b25 in glusterfs_graph_deactivate (graph=<optimized out>) at graph.c:340</div><div class="">#13 0x00007f7a38d50e3c in pub_glfs_fini (fs=fs@entry=0x7f7a3a6b6010) at glfs.c:1155</div><div class="">#14 0x00007f7a39f18ed4 in main (argc=<optimized out>, argv=<optimized out>) at glfs-heal.c:821</div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Thanks,</div><div class=""><br class=""></div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>Alessandro</div><br class=""><div class=""><blockquote type="cite" class=""><div class="">Il giorno 29/mag/2015, alle ore 11:12, Pranith Kumar Karampuri <<a href="mailto:pkarampu@redhat.com" class="">pkarampu@redhat.com</a>> ha scritto:</div><br class="Apple-interchange-newline"><div class="">
<meta content="text/html; charset=windows-1252" http-equiv="Content-Type" class="">
<div bgcolor="#FFFFFF" text="#000000" class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">On 05/29/2015 02:37 PM, Alessandro De
Salvo wrote:<br class="">
</div>
<blockquote cite="mid:AB7CD500-C547-4E49-B440-14926743C0E8@roma1.infn.it" type="cite" class="">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252" class="">
<div class="">Hi Pranith,</div>
<div class="">many thanks for the help!</div>
<div class="">The volume info of the problematic volume is the
following:</div>
<div class=""><br class="">
</div>
<div class="">
<div class=""># gluster volume info adsnet-vm-01</div>
<div class=""> </div>
<div class="">Volume Name: adsnet-vm-01</div>
<div class="">Type: Replicate</div>
<div class="">Volume ID: f8f615df-3dde-4ea6-9bdb-29a1706e864c</div>
<div class="">Status: Started</div>
<div class="">Number of Bricks: 1 x 2 = 2</div>
<div class="">Transport-type: tcp</div>
<div class="">Bricks:</div>
<div class="">Brick1: <a moz-do-not-send="true" href="http://gwads02.sta.adsnet.it/" class="">gwads02.sta.adsnet.it</a>:/gluster/vm01/data</div>
<div class="">Brick2: <a moz-do-not-send="true" href="http://gwads03.sta.adsnet.it/" class="">gwads03.sta.adsnet.it</a>:/gluster/vm01/data</div>
<div class="">Options Reconfigured:</div>
<div class="">nfs.disable: true</div>
<div class="">features.barrier: disable</div>
<div class="">features.file-snapshot: on</div>
<div class="">server.allow-insecure: on</div>
</div>
</blockquote>
Are you sure the attached log is correct? I do not see any backtrace
in the log file to indicate there is a crash :-(. Could you do "grep
-i crash /var/log/glusterfs/*" to see if there is some other file
with the crash. If that also fails, will it be possible for you to
provide the backtrace of the core by opening it using gdb?<br class="">
<br class="">
Pranith<br class="">
<blockquote cite="mid:AB7CD500-C547-4E49-B440-14926743C0E8@roma1.infn.it" type="cite" class="">
<div class=""><br class="">
</div>
<div class="">The log is in attachment.</div>
<div class="">I just wanted to add that the heal info command
works fine on other volumes hosted by the same machines, so it’s
just this volume which is causing problems.</div>
<div class="">Thanks,</div>
<div class=""><br class="">
</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre">
</span>Alessandro</div>
<div class=""><br class="">
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<br class="">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252" class="">
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">Il giorno 29/mag/2015, alle ore 10:50, Pranith
Kumar Karampuri <<a moz-do-not-send="true" href="mailto:pkarampu@redhat.com" class="">pkarampu@redhat.com</a>>
ha scritto:</div>
<br class="Apple-interchange-newline">
<div class=""><br style="font-family: Helvetica; font-size:
12px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height:
normal; orphans: auto; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows:
auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px; float:
none; display: inline !important;" class="">On 05/29/2015
02:18 PM, Pranith Kumar Karampuri wrote:</span><br style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<blockquote type="cite" style="font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height:
normal; orphans: auto; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows:
auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class="">
<br class="">
On 05/29/2015 02:13 PM, Alessandro De Salvo wrote:<br class="">
<blockquote type="cite" class="">Hi,<br class="">
I'm facing a strange issue with split brain reporting.<br class="">
I have upgraded to 3.7.0, after stopping all gluster
processes as described in the twiki, on all servers
hosting the volumes. The upgrade and the restart was
fine, and the volumes are accessible.<br class="">
However I had two files in split brain that I did not
heal before upgrading, so I tried a full heal with
3.7.0. The heal was launched correctly, but when I now
perform an heal info there is no output, while the heal
statistics says there are actually 2 files in split
brain. In the logs I see something like this:<br class="">
<br class="">
glustershd.log:<br class="">
[2015-05-29 08:28:43.008373] I
[afr-self-heal-entry.c:558:afr_selfheal_entry_do]
0-adsnet-gluster-01-replicate-0: performing entry
selfheal on 7fd1262d-949b-402e-96c2-ae487c8d4e27<br class="">
[2015-05-29 08:28:43.012690] W
[client-rpc-fops.c:241:client3_3_mknod_cbk]
0-adsnet-gluster-01-client-1: remote operation failed:
Invalid argument. Path: (null)<br class="">
</blockquote>
Hey could you let us know "gluster volume info" output?
Please let us know the backtrace printed by
/var/log/glusterfs/glfsheal-<volname>.log as well.<br class="">
</blockquote>
<span style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px; float:
none; display: inline !important;" class="">Please attach
/var/log/glusterfs/glfsheal-<volname>.log file to
this thread so that I can take a look.</span><br style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px; float:
none; display: inline !important;" class="">Pranith</span><br style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<blockquote type="cite" style="font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height:
normal; orphans: auto; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows:
auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class="">
Pranith<br class="">
<blockquote type="cite" class=""><br class="">
<br class="">
So, it seems like the files to be healed are not
correctly identified, or at least their path is null.<br class="">
Also, every time I issue a "gluster volume heal
<volname> info" a core dump is generated in the
log area.<br class="">
All servers are using the latest CentOS 7.<br class="">
Any idea why this might be happening and how to solve
it?<br class="">
Thanks,<br class="">
<br class="">
Alessandro<br class="">
<br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
Gluster-users mailing list<br class="">
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class="">
<a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users" class="">http://www.gluster.org/mailman/listinfo/gluster-users</a><br class="">
</blockquote>
<br class="">
_______________________________________________<br class="">
Gluster-users mailing list<br class="">
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class="">
<a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users" class="">http://www.gluster.org/mailman/listinfo/gluster-users</a></blockquote>
</div>
</blockquote>
</div>
<br class="">
</blockquote>
<br class="">
</div>
</div></blockquote></div><br class=""></div>_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class="">http://www.gluster.org/mailman/listinfo/gluster-users</div></blockquote></div><br class=""></body></html>