<p dir="ltr">I need to check this, few months back we faced a libc issue on RHEL. I don't have the full context of it with me right now. Does anyone recollect the issue?</p>
<p dir="ltr">-Atin<br>
Sent from one plus one</p>
<div class="gmail_quote">On Oct 7, 2015 9:29 PM, "Gene Liverman" <<a href="mailto:gliverma@westga.edu">gliverma@westga.edu</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">There are a couple of answers to that question...<div><ul><li>The core dump is from a fully patched RHEL 6 box. This is my primary box</li><li>The other two nodes are fully patched CentOS 6.</li></ul></div></div><div class="gmail_extra"><br clear="all"><div><div><div dir="ltr"><div><div dir="ltr"><div><br></div><div><br></div><div><br></div><div><br></div><div><span style="font-family:arial,helvetica,sans-serif">--</span></div><div style="color:rgb(51,102,255);font-family:tahoma,sans-serif"><font size="4"><b>Gene Liverman</b></font></div><div style="font-family:tahoma,sans-serif">Systems Integration Architect</div><div style="font-family:tahoma,sans-serif">Information Technology Services</div><div style="font-family:tahoma,sans-serif">University of West Georgia</div><div style="font-family:tahoma,sans-serif"><a href="mailto:gliverma@westga.edu" target="_blank">gliverma@westga.edu</a></div><div style="font-family:tahoma,sans-serif">678.839.5492</div><div style="font-family:tahoma,sans-serif"><br></div><div style="font-family:tahoma,sans-serif">ITS: Making Technology Work for You!</div><div style="font-family:tahoma,sans-serif"><br><img src="http://www.westga.edu/assetsDept/ucm/e-sig_wolves.png"><br></div><div style="font-family:tahoma,sans-serif"><br></div></div></div></div></div></div>
<br><div class="gmail_quote">On Wed, Oct 7, 2015 at 11:50 AM, Atin Mukherjee <span dir="ltr"><<a href="mailto:atin.mukherjee83@gmail.com" target="_blank">atin.mukherjee83@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">This looks like a glibc corruption to me. Which distribution platform are you running Gluster on?</p><span>
<p dir="ltr">-Atin<br>
Sent from one plus one</p>
</span><div><div><div class="gmail_quote">On Oct 7, 2015 9:12 PM, "Gene Liverman" <<a href="mailto:gliverma@westga.edu" target="_blank">gliverma@westga.edu</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><font face="arial, helvetica, sans-serif">Both of the requested trace commands are below:</font></div><div><br></div><div><font face="monospace, monospace">Core was generated by `/usr/sbin/glusterd --pid-file=/var/run/glusterd.pid'.</font></div><div><font face="monospace, monospace">Program terminated with signal 6, Aborted.</font></div><div><font face="monospace, monospace">#0 0x0000003b91432625 in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64</font></div><div><font face="monospace, monospace">64 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">(gdb) bt</font></div><div><font face="monospace, monospace">#0 0x0000003b91432625 in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64</font></div><div><font face="monospace, monospace">#1 0x0000003b91433e05 in abort () at abort.c:92</font></div><div><font face="monospace, monospace">#2 0x0000003b91470537 in __libc_message (do_abort=2, fmt=0x3b915588c0 "*** glibc detected *** %s: %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:198</font></div><div><font face="monospace, monospace">#3 0x0000003b91475f4e in malloc_printerr (action=3, str=0x3b9155687d "corrupted double-linked list", ptr=<value optimized out>, ar_ptr=<value optimized out>) at malloc.c:6350</font></div><div><font face="monospace, monospace">#4 0x0000003b914763d3 in malloc_consolidate (av=0x7fee90000020) at malloc.c:5216</font></div><div><font face="monospace, monospace">#5 0x0000003b91479c28 in _int_malloc (av=0x7fee90000020, bytes=<value optimized out>) at malloc.c:4415</font></div><div><font face="monospace, monospace">#6 0x0000003b9147a7ed in __libc_calloc (n=<value optimized out>, elem_size=<value optimized out>) at malloc.c:4093</font></div><div><font face="monospace, monospace">#7 0x0000003b9345c81f in __gf_calloc (nmemb=<value optimized out>, size=<value optimized out>, type=59, typestr=0x7fee9ed2d708 "gf_common_mt_rpc_trans_t") at mem-pool.c:117</font></div><div><font face="monospace, monospace">#8 0x00007fee9ed2830b in socket_server_event_handler (fd=<value optimized out>, idx=<value optimized out>, data=0xf3eca0, poll_in=1, poll_out=<value optimized out>,</font></div><div><font face="monospace, monospace"> poll_err=<value optimized out>) at socket.c:2622</font></div><div><font face="monospace, monospace">#9 0x0000003b9348b0a0 in event_dispatch_epoll_handler (data=0xf408b0) at event-epoll.c:575</font></div><div><font face="monospace, monospace">#10 event_dispatch_epoll_worker (data=0xf408b0) at event-epoll.c:678</font></div><div><font face="monospace, monospace">#11 0x0000003b91807a51 in start_thread (arg=0x7fee9db3b700) at pthread_create.c:301</font></div><div><font face="monospace, monospace">#12 0x0000003b914e893d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><br></font></div><div><div><font face="monospace, monospace">(gdb) t a a bt</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">Thread 9 (Thread 0x7fee9e53c700 (LWP 37122)):</font></div><div><font face="monospace, monospace">#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183</font></div><div><font face="monospace, monospace">#1 0x00007fee9fffcf93 in hooks_worker (args=<value optimized out>) at glusterd-hooks.c:534</font></div><div><font face="monospace, monospace">#2 0x0000003b91807a51 in start_thread (arg=0x7fee9e53c700) at pthread_create.c:301</font></div><div><font face="monospace, monospace">#3 0x0000003b914e893d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">Thread 8 (Thread 0x7feea0c99700 (LWP 36996)):</font></div><div><font face="monospace, monospace">#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239</font></div><div><font face="monospace, monospace">#1 0x0000003b9346cbdb in syncenv_task (proc=0xefa8c0) at syncop.c:607</font></div><div><font face="monospace, monospace">#2 0x0000003b93472cb0 in syncenv_processor (thdata=0xefa8c0) at syncop.c:699</font></div><div><font face="monospace, monospace">#3 0x0000003b91807a51 in start_thread (arg=0x7feea0c99700) at pthread_create.c:301</font></div><div><font face="monospace, monospace">#4 0x0000003b914e893d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">Thread 7 (Thread 0x7feea209b700 (LWP 36994)):</font></div><div><font face="monospace, monospace">#0 do_sigwait (set=<value optimized out>, sig=0x7feea209ae5c) at ../sysdeps/unix/sysv/linux/sigwait.c:65</font></div><div><font face="monospace, monospace">#1 __sigwait (set=<value optimized out>, sig=0x7feea209ae5c) at ../sysdeps/unix/sysv/linux/sigwait.c:100</font></div><div><font face="monospace, monospace">#2 0x0000000000405dfb in glusterfs_sigwaiter (arg=<value optimized out>) at glusterfsd.c:1989</font></div><div><font face="monospace, monospace">#3 0x0000003b91807a51 in start_thread (arg=0x7feea209b700) at pthread_create.c:301</font></div><div><font face="monospace, monospace">#4 0x0000003b914e893d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">Thread 6 (Thread 0x7feea2a9c700 (LWP 36993)):</font></div><div><font face="monospace, monospace">#0 0x0000003b9180efbd in nanosleep () at ../sysdeps/unix/syscall-template.S:82</font></div><div><font face="monospace, monospace">#1 0x0000003b934473ea in gf_timer_proc (ctx=0xecc010) at timer.c:205</font></div><div><font face="monospace, monospace">#2 0x0000003b91807a51 in start_thread (arg=0x7feea2a9c700) at pthread_create.c:301</font></div><div><font face="monospace, monospace">#3 0x0000003b914e893d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">Thread 5 (Thread 0x7feea9e04740 (LWP 36992)):</font></div><div><font face="monospace, monospace">#0 0x0000003b918082ad in pthread_join (threadid=140662814254848, thread_return=0x0) at pthread_join.c:89</font></div><div><font face="monospace, monospace">#1 0x0000003b9348ab4d in event_dispatch_epoll (event_pool=0xeeb5b0) at event-epoll.c:762</font></div><div><font face="monospace, monospace">#2 0x0000000000407b24 in main (argc=2, argv=0x7fff5294adc8) at glusterfsd.c:2333</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">Thread 4 (Thread 0x7feea169a700 (LWP 36995)):</font></div><div><font face="monospace, monospace">#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239</font></div><div><font face="monospace, monospace">#1 0x0000003b9346cbdb in syncenv_task (proc=0xefa500) at syncop.c:607</font></div><div><font face="monospace, monospace">#2 0x0000003b93472cb0 in syncenv_processor (thdata=0xefa500) at syncop.c:699</font></div><div><font face="monospace, monospace">#3 0x0000003b91807a51 in start_thread (arg=0x7feea169a700) at pthread_create.c:301</font></div><div><font face="monospace, monospace">#4 0x0000003b914e893d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">Thread 3 (Thread 0x7fee9d13a700 (LWP 37124)):</font></div><div><font face="monospace, monospace">#0 0x0000003b914e8f33 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82</font></div><div><font face="monospace, monospace">#1 0x0000003b9348aed1 in event_dispatch_epoll_worker (data=0xf405b0) at event-epoll.c:668</font></div><div><font face="monospace, monospace">#2 0x0000003b91807a51 in start_thread (arg=0x7fee9d13a700) at pthread_create.c:301</font></div><div><font face="monospace, monospace">#3 0x0000003b914e893d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">Thread 2 (Thread 0x7fee97fff700 (LWP 37125)):</font></div><div><font face="monospace, monospace">#0 0x0000003b914e8f33 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82</font></div><div><font face="monospace, monospace">#1 0x0000003b9348aed1 in event_dispatch_epoll_worker (data=0xf6b4d0) at event-epoll.c:668</font></div><div><font face="monospace, monospace">#2 0x0000003b91807a51 in start_thread (arg=0x7fee97fff700) at pthread_create.c:301</font></div><div><font face="monospace, monospace">#3 0x0000003b914e893d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">Thread 1 (Thread 0x7fee9db3b700 (LWP 37123)):</font></div><div><font face="monospace, monospace">#0 0x0000003b91432625 in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64</font></div><div><font face="monospace, monospace">#1 0x0000003b91433e05 in abort () at abort.c:92</font></div><div><font face="monospace, monospace">#2 0x0000003b91470537 in __libc_message (do_abort=2, fmt=0x3b915588c0 "*** glibc detected *** %s: %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:198</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">---Type <return> to continue, or q <return> to quit---</font></div><div><font face="monospace, monospace">#3 0x0000003b91475f4e in malloc_printerr (action=3, str=0x3b9155687d "corrupted double-linked list", ptr=<value optimized out>, ar_ptr=<value optimized out>) at malloc.c:6350</font></div><div><font face="monospace, monospace">#4 0x0000003b914763d3 in malloc_consolidate (av=0x7fee90000020) at malloc.c:5216</font></div><div><font face="monospace, monospace">#5 0x0000003b91479c28 in _int_malloc (av=0x7fee90000020, bytes=<value optimized out>) at malloc.c:4415</font></div><div><font face="monospace, monospace">#6 0x0000003b9147a7ed in __libc_calloc (n=<value optimized out>, elem_size=<value optimized out>) at malloc.c:4093</font></div><div><font face="monospace, monospace">#7 0x0000003b9345c81f in __gf_calloc (nmemb=<value optimized out>, size=<value optimized out>, type=59, typestr=0x7fee9ed2d708 "gf_common_mt_rpc_trans_t") at mem-pool.c:117</font></div><div><font face="monospace, monospace">#8 0x00007fee9ed2830b in socket_server_event_handler (fd=<value optimized out>, idx=<value optimized out>, data=0xf3eca0, poll_in=1, poll_out=<value optimized out>,</font></div><div><font face="monospace, monospace"> poll_err=<value optimized out>) at socket.c:2622</font></div><div><font face="monospace, monospace">#9 0x0000003b9348b0a0 in event_dispatch_epoll_handler (data=0xf408b0) at event-epoll.c:575</font></div><div><font face="monospace, monospace">#10 event_dispatch_epoll_worker (data=0xf408b0) at event-epoll.c:678</font></div><div><font face="monospace, monospace">#11 0x0000003b91807a51 in start_thread (arg=0x7fee9db3b700) at pthread_create.c:301</font></div><div><font face="monospace, monospace">#12 0x0000003b914e893d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115</font></div></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><br></font></div><div class="gmail_extra"><font face="monospace, monospace"><br clear="all"></font><div><div><div dir="ltr"><div><div dir="ltr"><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">--</font></div><div style="color:rgb(51,102,255)"><font size="4" face="monospace, monospace"><b>Gene Liverman</b></font></div><div><font face="monospace, monospace">Systems Integration Architect</font></div><div><font face="monospace, monospace">Information Technology Services</font></div><div><font face="monospace, monospace">University of West Georgia</font></div><div><a href="mailto:gliverma@westga.edu" target="_blank"><font face="monospace, monospace">gliverma@westga.edu</font></a></div><div><font face="monospace, monospace"><a href="tel:678.839.5492" value="+16788395492" target="_blank">678.839.5492</a></font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">ITS: Making Technology Work for You!</font></div><div><font face="monospace, monospace"><br><img src="http://www.westga.edu/assetsDept/ucm/e-sig_wolves.png"><br></font></div><div><font face="monospace, monospace"><br></font></div></div></div></div></div></div>
<font face="monospace, monospace"><br></font><div class="gmail_quote"><font face="monospace, monospace">On Wed, Oct 7, 2015 at 12:06 AM, Atin Mukherjee <span dir="ltr"><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span> wrote:<br></font><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><font face="monospace, monospace"><br>
<br>
On 10/07/2015 09:34 AM, Atin Mukherjee wrote:<br>
><br>
><br>
> On 10/06/2015 08:15 PM, Gene Liverman wrote:<br>
>> Sorry for the delay... they joys of multiple proverbial fires at once.<br>
>> In /var/log/messages I found this for our most recent crash:<br>
>><br>
>> Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]:<br>
>> pending frames:<br>
>> Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]:<br>
>> patchset: git://<a href="http://git.gluster.com/glusterfs.git" rel="noreferrer" target="_blank">git.gluster.com/glusterfs.git</a><br>
>> <<a href="http://git.gluster.com/glusterfs.git" rel="noreferrer" target="_blank">http://git.gluster.com/glusterfs.git</a>><br>
>> Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]:<br>
>> signal received: 6<br>
>> Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]: time<br>
>> of crash:<br>
>> Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]:<br>
>> 2015-10-03 04:26:21<br>
>> Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]:<br>
>> configuration details:<br>
>> Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]: argp 1<br>
>> Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]:<br>
>> backtrace 1<br>
>> Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]: dlfcn 1<br>
>> Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]:<br>
>> libpthread 1<br>
>> Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]:<br>
>> llistxattr 1<br>
>> Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]: setfsid 1<br>
>> Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]:<br>
>> spinlock 1<br>
>> Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]: epoll.h 1<br>
>> Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]: xattr.h 1<br>
>> Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]:<br>
>> st_atim.tv_nsec 1<br>
>> Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]:<br>
>> package-string: glusterfs 3.7.4<br>
>> Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]: ---------<br>
>><br>
>><br>
>> I have posted etc-glusterfs-glusterd.vol.log<br>
>> to <a href="http://pastebin.com/Pzq1j5J3" rel="noreferrer" target="_blank">http://pastebin.com/Pzq1j5J3</a>. I also put the core file and an<br>
>> sosreport on my web server for you but don't want to leave them there<br>
>> for long so I'd appreciate it if you'd let me know once you get them.<br>
>> They are at the following url's:<br>
>> <a href="http://www.westga.edu/~gliverma/tmp-files/core.36992" rel="noreferrer" target="_blank">http://www.westga.edu/~gliverma/tmp-files/core.36992</a><br>
> Could you get the backtrace and share with us with the following commands:<br>
><br>
> $ gdb glusterd2 <core file path><br>
> $ bt<br>
</font></div></div><font face="monospace, monospace">Also "t a a bt" output in gdb might help.<br>
</font><div><div><font face="monospace, monospace">><br>
>> <a href="http://www.westga.edu/~gliverma/tmp-files/sosreport-gliverman.gluster-crashing-20151006101239.tar.xz" rel="noreferrer" target="_blank">http://www.westga.edu/~gliverma/tmp-files/sosreport-gliverman.gluster-crashing-20151006101239.tar.xz</a><br>
>> <a href="http://www.westga.edu/~gliverma/tmp-files/sosreport-gliverman.gluster-crashing-20151006101239.tar.xz.md5" rel="noreferrer" target="_blank">http://www.westga.edu/~gliverma/tmp-files/sosreport-gliverman.gluster-crashing-20151006101239.tar.xz.md5</a><br>
>><br>
>><br>
>><br>
>><br>
>> Thanks again for the help!<br>
>> *Gene Liverman*<br>
>> Systems Integration Architect<br>
>> Information Technology Services<br>
>> University of West Georgia<br>
>> <a href="mailto:gliverma@westga.edu" target="_blank">gliverma@westga.edu</a> <mailto:<a href="mailto:gliverma@westga.edu" target="_blank">gliverma@westga.edu</a>><br>
>><br>
>> ITS: Making Technology Work for You!<br>
>><br>
>><br>
>><br>
>><br>
>> On Fri, Oct 2, 2015 at 11:18 AM, Gaurav Garg <<a href="mailto:ggarg@redhat.com" target="_blank">ggarg@redhat.com</a><br>
>> <mailto:<a href="mailto:ggarg@redhat.com" target="_blank">ggarg@redhat.com</a>>> wrote:<br>
>><br>
>> >> Pulling those logs now but how do I generate the core file you are asking<br>
>> for?<br>
>><br>
>> When there is crash then core file automatically generated based on<br>
>> your *ulimit* set option. you can find location of core file in your<br>
>> root or current working directory or where ever you have set your<br>
>> core dump file location. core file gives you information regarding<br>
>> crash, where exactly crash happened.<br>
>> you can find appropriate core file by looking at crash time in<br>
>> glusterd log's by searching "crash" keyword. you can also paste few<br>
>> line's just above latest "crash" keyword in glusterd logs.<br>
>><br>
>> Just for your curiosity if you willing to look where it crash then<br>
>> you can debug it by #gdb -c <location of core file> glusterd<br>
>><br>
>> Thank you...<br>
>><br>
>> Regards,<br>
>> Gaurav<br>
>><br>
>> ----- Original Message -----<br>
>> From: "Gene Liverman" <<a href="mailto:gliverma@westga.edu" target="_blank">gliverma@westga.edu</a> <mailto:<a href="mailto:gliverma@westga.edu" target="_blank">gliverma@westga.edu</a>>><br>
>> To: "Gaurav Garg" <<a href="mailto:ggarg@redhat.com" target="_blank">ggarg@redhat.com</a> <mailto:<a href="mailto:ggarg@redhat.com" target="_blank">ggarg@redhat.com</a>>><br>
>> Cc: "gluster-users" <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
>> <mailto:<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>>><br>
>> Sent: Friday, October 2, 2015 8:28:49 PM<br>
>> Subject: Re: [Gluster-users] glusterd crashing<br>
>><br>
>> Pulling those logs now but how do I generate the core file you are<br>
>> asking<br>
>> for?<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> --<br>
>> *Gene Liverman*<br>
>> Systems Integration Architect<br>
>> Information Technology Services<br>
>> University of West Georgia<br>
>> <a href="mailto:gliverma@westga.edu" target="_blank">gliverma@westga.edu</a> <mailto:<a href="mailto:gliverma@westga.edu" target="_blank">gliverma@westga.edu</a>><br>
>> <a href="tel:678.839.5492" value="+16788395492" target="_blank">678.839.5492</a> <tel:<a href="tel:678.839.5492" value="+16788395492" target="_blank">678.839.5492</a>><br>
>><br>
>> ITS: Making Technology Work for You!<br>
>><br>
>><br>
>><br>
>><br>
>> On Fri, Oct 2, 2015 at 2:25 AM, Gaurav Garg <<a href="mailto:ggarg@redhat.com" target="_blank">ggarg@redhat.com</a><br>
>> <mailto:<a href="mailto:ggarg@redhat.com" target="_blank">ggarg@redhat.com</a>>> wrote:<br>
>><br>
>> > Hi Gene,<br>
>> ><br>
>> > you have paste glustershd log. we asked you to paste glusterd log.<br>
>> > glusterd and glustershd both are different process. with this<br>
>> information<br>
>> > we can't find out why your glusterd crashed. could you paste<br>
>> *glusterd*<br>
>> > logs (/var/log/glusterfs/usr-local-etc-glusterfs-glusterd.vol.log*) in<br>
>> > pastebin (not in this mail thread) and give the link of pastebin<br>
>> in this<br>
>> > mail thread. Can you also attach core file or you can paste<br>
>> backtrace of<br>
>> > that core dump file.<br>
>> > It will be great if you give us sos report of the node where the crash<br>
>> > happen.<br>
>> ><br>
>> > Thanx,<br>
>> ><br>
>> > ~Gaurav<br>
>> ><br>
>> > ----- Original Message -----<br>
>> > From: "Gene Liverman" <<a href="mailto:gliverma@westga.edu" target="_blank">gliverma@westga.edu</a><br>
>> <mailto:<a href="mailto:gliverma@westga.edu" target="_blank">gliverma@westga.edu</a>>><br>
>> > To: "gluster-users" <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
>> <mailto:<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>>><br>
>> > Sent: Friday, October 2, 2015 4:47:00 AM<br>
>> > Subject: Re: [Gluster-users] glusterd crashing<br>
>> ><br>
>> > Sorry for the delay. Here is what's installed:<br>
>> > # rpm -qa | grep gluster<br>
>> > glusterfs-geo-replication-3.7.4-2.el6.x86_64<br>
>> > glusterfs-client-xlators-3.7.4-2.el6.x86_64<br>
>> > glusterfs-3.7.4-2.el6.x86_64<br>
>> > glusterfs-libs-3.7.4-2.el6.x86_64<br>
>> > glusterfs-api-3.7.4-2.el6.x86_64<br>
>> > glusterfs-fuse-3.7.4-2.el6.x86_64<br>
>> > glusterfs-server-3.7.4-2.el6.x86_64<br>
>> > glusterfs-cli-3.7.4-2.el6.x86_64<br>
>> ><br>
>> > The cmd_history.log file is attached.<br>
>> > In gluster.log I have filtered out a bunch of lines like the one<br>
>> below due<br>
>> > to make them more readable. I had a node down for multiple days due to<br>
>> > maintenance and another one went down due to a hardware failure<br>
>> during that<br>
>> > time too.<br>
>> > [2015-10-01 00:16:09.643631] W [MSGID: 114031]<br>
>> > [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-gv0-client-0: remote<br>
>> > operation failed. Path: <gfid:31f17f8c-6c96-4440-88c0-f813b3c8d364><br>
>> > (31f17f8c-6c96-4440-88c0-f813b3c8d364) [No such file or directory]<br>
>> ><br>
>> > I also filtered out a boat load of self heal lines like these two:<br>
>> > [2015-10-01 15:14:14.851015] I [MSGID: 108026]<br>
>> > [afr-self-heal-metadata.c:56:__afr_selfheal_metadata_do]<br>
>> 0-gv0-replicate-0:<br>
>> > performing metadata selfheal on f78a47db-a359-430d-a655-1d217eb848c3<br>
>> > [2015-10-01 15:14:14.856392] I [MSGID: 108026]<br>
>> > [afr-self-heal-common.c:651:afr_log_selfheal] 0-gv0-replicate-0:<br>
>> Completed<br>
>> > metadata selfheal on f78a47db-a359-430d-a655-1d217eb848c3.<br>
>> source=0 sinks=1<br>
>> ><br>
>> ><br>
>> > [root@eapps-gluster01 glusterfs]# cat glustershd.log |grep -v 'remote<br>
>> > operation failed' |grep -v 'self-heal'<br>
>> > [2015-09-27 08:46:56.893125] E [rpc-clnt.c:201:call_bail] 0-glusterfs:<br>
>> > bailing out frame type(GlusterFS Handshake) op(GETSPEC(2)) xid =<br>
>> 0x6 sent =<br>
>> > 2015-09-27 08:16:51.742731. timeout = 1800 for <a href="http://127.0.0.1:24007" rel="noreferrer" target="_blank">127.0.0.1:24007</a><br>
>> <<a href="http://127.0.0.1:24007" rel="noreferrer" target="_blank">http://127.0.0.1:24007</a>><br>
>> > [2015-09-28 12:54:17.524924] W [socket.c:588:__socket_rwv]<br>
>> 0-glusterfs:<br>
>> > readv on <a href="http://127.0.0.1:24007" rel="noreferrer" target="_blank">127.0.0.1:24007</a> <<a href="http://127.0.0.1:24007" rel="noreferrer" target="_blank">http://127.0.0.1:24007</a>> failed<br>
>> (Connection reset by peer)<br>
>> > [2015-09-28 12:54:27.844374] I<br>
>> [glusterfsd-mgmt.c:1512:mgmt_getspec_cbk]<br>
>> > 0-glusterfs: No change in volfile, continuing<br>
>> > [2015-09-28 12:57:03.485027] W [socket.c:588:__socket_rwv]<br>
>> 0-gv0-client-2:<br>
>> > readv on <a href="http://160.10.31.227:24007" rel="noreferrer" target="_blank">160.10.31.227:24007</a> <<a href="http://160.10.31.227:24007" rel="noreferrer" target="_blank">http://160.10.31.227:24007</a>> failed<br>
>> (Connection reset by peer)<br>
>> > [2015-09-28 12:57:05.872973] E [socket.c:2278:socket_connect_finish]<br>
>> > 0-gv0-client-2: connection to <a href="http://160.10.31.227:24007" rel="noreferrer" target="_blank">160.10.31.227:24007</a><br>
>> <<a href="http://160.10.31.227:24007" rel="noreferrer" target="_blank">http://160.10.31.227:24007</a>> failed (Connection<br>
>> > refused)<br>
>> > [2015-09-28 12:57:38.490578] W [socket.c:588:__socket_rwv]<br>
>> 0-glusterfs:<br>
>> > readv on <a href="http://127.0.0.1:24007" rel="noreferrer" target="_blank">127.0.0.1:24007</a> <<a href="http://127.0.0.1:24007" rel="noreferrer" target="_blank">http://127.0.0.1:24007</a>> failed (No data<br>
>> available)<br>
>> > [2015-09-28 12:57:49.054475] I<br>
>> [glusterfsd-mgmt.c:1512:mgmt_getspec_cbk]<br>
>> > 0-glusterfs: No change in volfile, continuing<br>
>> > [2015-09-28 13:01:12.062960] W [glusterfsd.c:1219:cleanup_and_exit]<br>
>> > (-->/lib64/libpthread.so.0() [0x3c65e07a51]<br>
>> > -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x405e4d]<br>
>> > -->/usr/sbin/glusterfs(cleanup_and_exit+0x65) [0x4059b5] ) 0-:<br>
>> received<br>
>> > signum (15), shutting down<br>
>> > [2015-09-28 13:01:12.981945] I [MSGID: 100030]<br>
>> [glusterfsd.c:2301:main]<br>
>> > 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version<br>
>> 3.7.4<br>
>> > (args: /usr/sbin/glusterfs -s localhost --volfile-id<br>
>> gluster/glustershd -p<br>
>> > /var/lib/glusterd/glustershd/run/glustershd.pid -l<br>
>> > /var/log/glusterfs/glustershd.log -S<br>
>> > /var/run/gluster/9a9819e90404187e84e67b01614bbe10.socket<br>
>> --xlator-option<br>
>> > *replicate*.node-uuid=416d712a-06fc-4b3c-a92f-8c82145626ff)<br>
>> > [2015-09-28 13:01:13.009171] I [MSGID: 101190]<br>
>> > [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started<br>
>> thread<br>
>> > with index 1<br>
>> > [2015-09-28 13:01:13.092483] I [graph.c:269:gf_add_cmdline_options]<br>
>> > 0-gv0-replicate-0: adding option 'node-uuid' for volume<br>
>> 'gv0-replicate-0'<br>
>> > with value '416d712a-06fc-4b3c-a92f-8c82145626ff'<br>
>> > [2015-09-28 13:01:13.100856] I [MSGID: 101190]<br>
>> > [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started<br>
>> thread<br>
>> > with index 2<br>
>> > [2015-09-28 13:01:13.103995] I [MSGID: 114020] [client.c:2118:notify]<br>
>> > 0-gv0-client-0: parent translators are ready, attempting connect on<br>
>> > transport<br>
>> > [2015-09-28 13:01:13.114745] I [MSGID: 114020] [client.c:2118:notify]<br>
>> > 0-gv0-client-1: parent translators are ready, attempting connect on<br>
>> > transport<br>
>> > [2015-09-28 13:01:13.115725] I [rpc-clnt.c:1851:rpc_clnt_reconfig]<br>
>> > 0-gv0-client-0: changing port to 49152 (from 0)<br>
>> > [2015-09-28 13:01:13.125619] I [MSGID: 114020] [client.c:2118:notify]<br>
>> > 0-gv0-client-2: parent translators are ready, attempting connect on<br>
>> > transport<br>
>> > [2015-09-28 13:01:13.132316] E [socket.c:2278:socket_connect_finish]<br>
>> > 0-gv0-client-1: connection to <a href="http://160.10.31.64:24007" rel="noreferrer" target="_blank">160.10.31.64:24007</a><br>
>> <<a href="http://160.10.31.64:24007" rel="noreferrer" target="_blank">http://160.10.31.64:24007</a>> failed (Connection<br>
>> > refused)<br>
>> > [2015-09-28 13:01:13.132650] I [MSGID: 114057]<br>
>> > [client-handshake.c:1437:select_server_supported_programs]<br>
>> 0-gv0-client-0:<br>
>> > Using Program GlusterFS 3.3, Num (1298437), Version (330)<br>
>> > [2015-09-28 13:01:13.133322] I [MSGID: 114046]<br>
>> > [client-handshake.c:1213:client_setvolume_cbk] 0-gv0-client-0:<br>
>> Connected to<br>
>> > gv0-client-0, attached to remote volume '/export/sdb1/gv0'.<br>
>> > [2015-09-28 13:01:13.133365] I [MSGID: 114047]<br>
>> > [client-handshake.c:1224:client_setvolume_cbk] 0-gv0-client-0:<br>
>> Server and<br>
>> > Client lk-version numbers are not same, reopening the fds<br>
>> > [2015-09-28 13:01:13.133782] I [MSGID: 108005]<br>
>> > [afr-common.c:3998:afr_notify] 0-gv0-replicate-0: Subvolume<br>
>> 'gv0-client-0'<br>
>> > came back up; going online.<br>
>> > [2015-09-28 13:01:13.133863] I [MSGID: 114035]<br>
>> > [client-handshake.c:193:client_set_lk_version_cbk] 0-gv0-client-0:<br>
>> Server<br>
>> > lk version = 1<br>
>> > Final graph:<br>
>> ><br>
>> ><br>
>> +------------------------------------------------------------------------------+<br>
>> > 1: volume gv0-client-0<br>
>> > 2: type protocol/client<br>
>> > 3: option clnt-lk-version 1<br>
>> > 4: option volfile-checksum 0<br>
>> > 5: option volfile-key gluster/glustershd<br>
>> > 6: option client-version 3.7.4<br>
>> > 7: option process-uuid<br>
>> > eapps-gluster01-65147-2015/09/28-13:01:12:970131-gv0-client-0-0-0<br>
>> > 8: option fops-version 1298437<br>
>> > 9: option ping-timeout 42<br>
>> > 10: option remote-host <a href="http://eapps-gluster01.uwg.westga.edu" rel="noreferrer" target="_blank">eapps-gluster01.uwg.westga.edu</a><br>
>> <<a href="http://eapps-gluster01.uwg.westga.edu" rel="noreferrer" target="_blank">http://eapps-gluster01.uwg.westga.edu</a>><br>
>> > 11: option remote-subvolume /export/sdb1/gv0<br>
>> > 12: option transport-type socket<br>
>> > 13: option username 0005f8fa-107a-4cc8-ac38-bb821c014c14<br>
>> > 14: option password 379bae9a-6529-4564-a6f5-f5a9f7424d01<br>
>> > 15: end-volume<br>
>> > 16:<br>
>> > 17: volume gv0-client-1<br>
>> > 18: type protocol/client<br>
>> > 19: option ping-timeout 42<br>
>> > 20: option remote-host <a href="http://eapps-gluster02.uwg.westga.edu" rel="noreferrer" target="_blank">eapps-gluster02.uwg.westga.edu</a><br>
>> <<a href="http://eapps-gluster02.uwg.westga.edu" rel="noreferrer" target="_blank">http://eapps-gluster02.uwg.westga.edu</a>><br>
>> > 21: option remote-subvolume /export/sdb1/gv0<br>
>> > 22: option transport-type socket<br>
>> > 23: option username 0005f8fa-107a-4cc8-ac38-bb821c014c14<br>
>> > 24: option password 379bae9a-6529-4564-a6f5-f5a9f7424d01<br>
>> > 25: end-volume<br>
>> > 26:<br>
>> > 27: volume gv0-client-2<br>
>> > 28: type protocol/client<br>
>> > 29: option ping-timeout 42<br>
>> > 30: option remote-host <a href="http://eapps-gluster03.uwg.westga.edu" rel="noreferrer" target="_blank">eapps-gluster03.uwg.westga.edu</a><br>
>> <<a href="http://eapps-gluster03.uwg.westga.edu" rel="noreferrer" target="_blank">http://eapps-gluster03.uwg.westga.edu</a>><br>
>> > 31: option remote-subvolume /export/sdb1/gv0<br>
>> > 32: option transport-type socket<br>
>> > 33: option username 0005f8fa-107a-4cc8-ac38-bb821c014c14<br>
>> > 34: option password 379bae9a-6529-4564-a6f5-f5a9f7424d01<br>
>> > 35: end-volume<br>
>> > 36:<br>
>> > 37: volume gv0-replicate-0<br>
>> > 38: type cluster/replicate<br>
>> > 39: option node-uuid 416d712a-06fc-4b3c-a92f-8c82145626ff<br>
>> > 46: subvolumes gv0-client-0 gv0-client-1 gv0-client-2<br>
>> > 47: end-volume<br>
>> > 48:<br>
>> > 49: volume glustershd<br>
>> > 50: type debug/io-stats<br>
>> > 51: subvolumes gv0-replicate-0<br>
>> > 52: end-volume<br>
>> > 53:<br>
>> ><br>
>> ><br>
>> +------------------------------------------------------------------------------+<br>
>> > [2015-09-28 13:01:13.154898] E [MSGID: 114058]<br>
>> > [client-handshake.c:1524:client_query_portmap_cbk] 0-gv0-client-2:<br>
>> failed<br>
>> > to get the port number for remote subvolume. Please run 'gluster<br>
>> volume<br>
>> > status' on server to see if brick process is running.<br>
>> > [2015-09-28 13:01:13.155031] I [MSGID: 114018]<br>
>> > [client.c:2042:client_rpc_notify] 0-gv0-client-2: disconnected from<br>
>> > gv0-client-2. Client process will keep trying to connect to<br>
>> glusterd until<br>
>> > brick's port is available<br>
>> > [2015-09-28 13:01:13.155080] W [MSGID: 108001]<br>
>> > [afr-common.c:4081:afr_notify] 0-gv0-replicate-0: Client-quorum is<br>
>> not met<br>
>> > [2015-09-29 08:11:24.728797] I [MSGID: 100011]<br>
>> > [glusterfsd.c:1291:reincarnate] 0-glusterfsd: Fetching the volume<br>
>> file from<br>
>> > server...<br>
>> > [2015-09-29 08:11:24.763338] I<br>
>> [glusterfsd-mgmt.c:1512:mgmt_getspec_cbk]<br>
>> > 0-glusterfs: No change in volfile, continuing<br>
>> > [2015-09-29 12:50:41.915254] E [rpc-clnt.c:201:call_bail]<br>
>> 0-gv0-client-2:<br>
>> > bailing out frame type(GF-DUMP) op(DUMP(1)) xid = 0xd91f sent =<br>
>> 2015-09-29<br>
>> > 12:20:36.092734. timeout = 1800 for <a href="http://160.10.31.227:24007" rel="noreferrer" target="_blank">160.10.31.227:24007</a><br>
>> <<a href="http://160.10.31.227:24007" rel="noreferrer" target="_blank">http://160.10.31.227:24007</a>><br>
>> > [2015-09-29 12:50:41.923550] W [MSGID: 114032]<br>
>> > [client-handshake.c:1623:client_dump_version_cbk] 0-gv0-client-2:<br>
>> received<br>
>> > RPC status error [Transport endpoint is not connected]<br>
>> > [2015-09-30 23:54:36.547979] W [socket.c:588:__socket_rwv]<br>
>> 0-glusterfs:<br>
>> > readv on <a href="http://127.0.0.1:24007" rel="noreferrer" target="_blank">127.0.0.1:24007</a> <<a href="http://127.0.0.1:24007" rel="noreferrer" target="_blank">http://127.0.0.1:24007</a>> failed (No data<br>
>> available)<br>
>> > [2015-09-30 23:54:46.812870] E [socket.c:2278:socket_connect_finish]<br>
>> > 0-glusterfs: connection to <a href="http://127.0.0.1:24007" rel="noreferrer" target="_blank">127.0.0.1:24007</a><br>
>> <<a href="http://127.0.0.1:24007" rel="noreferrer" target="_blank">http://127.0.0.1:24007</a>> failed (Connection refused)<br>
>> > [2015-10-01 00:14:20.997081] I<br>
>> [glusterfsd-mgmt.c:1512:mgmt_getspec_cbk]<br>
>> > 0-glusterfs: No change in volfile, continuing<br>
>> > [2015-10-01 00:15:36.770579] W [socket.c:588:__socket_rwv]<br>
>> 0-gv0-client-2:<br>
>> > readv on <a href="http://160.10.31.227:24007" rel="noreferrer" target="_blank">160.10.31.227:24007</a> <<a href="http://160.10.31.227:24007" rel="noreferrer" target="_blank">http://160.10.31.227:24007</a>> failed<br>
>> (Connection reset by peer)<br>
>> > [2015-10-01 00:15:37.906708] E [socket.c:2278:socket_connect_finish]<br>
>> > 0-gv0-client-2: connection to <a href="http://160.10.31.227:24007" rel="noreferrer" target="_blank">160.10.31.227:24007</a><br>
>> <<a href="http://160.10.31.227:24007" rel="noreferrer" target="_blank">http://160.10.31.227:24007</a>> failed (Connection<br>
>> > refused)<br>
>> > [2015-10-01 00:15:53.008130] W [glusterfsd.c:1219:cleanup_and_exit]<br>
>> > (-->/lib64/libpthread.so.0() [0x3b91807a51]<br>
>> > -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x405e4d]<br>
>> > -->/usr/sbin/glusterfs(cleanup_and_exit+0x65) [0x4059b5] ) 0-:<br>
>> received<br>
>> > signum (15), shutting down<br>
>> > [2015-10-01 00:15:53.008697] I [timer.c:48:gf_timer_call_after]<br>
>> > (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x3e2) [0x3b9480f992]<br>
>> > -->/usr/lib64/libgfrpc.so.0(__save_frame+0x76) [0x3b9480f046]<br>
>> > -->/usr/lib64/libglusterfs.so.0(gf_timer_call_after+0x1b1)<br>
>> [0x3b93447881] )<br>
>> > 0-timer: ctx cleanup started<br>
>> > [2015-10-01 00:15:53.994698] I [MSGID: 100030]<br>
>> [glusterfsd.c:2301:main]<br>
>> > 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version<br>
>> 3.7.4<br>
>> > (args: /usr/sbin/glusterfs -s localhost --volfile-id<br>
>> gluster/glustershd -p<br>
>> > /var/lib/glusterd/glustershd/run/glustershd.pid -l<br>
>> > /var/log/glusterfs/glustershd.log -S<br>
>> > /var/run/gluster/9a9819e90404187e84e67b01614bbe10.socket<br>
>> --xlator-option<br>
>> > *replicate*.node-uuid=416d712a-06fc-4b3c-a92f-8c82145626ff)<br>
>> > [2015-10-01 00:15:54.020401] I [MSGID: 101190]<br>
>> > [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started<br>
>> thread<br>
>> > with index 1<br>
>> > [2015-10-01 00:15:54.086777] I [graph.c:269:gf_add_cmdline_options]<br>
>> > 0-gv0-replicate-0: adding option 'node-uuid' for volume<br>
>> 'gv0-replicate-0'<br>
>> > with value '416d712a-06fc-4b3c-a92f-8c82145626ff'<br>
>> > [2015-10-01 00:15:54.093004] I [MSGID: 101190]<br>
>> > [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started<br>
>> thread<br>
>> > with index 2<br>
>> > [2015-10-01 00:15:54.098144] I [MSGID: 114020] [client.c:2118:notify]<br>
>> > 0-gv0-client-0: parent translators are ready, attempting connect on<br>
>> > transport<br>
>> > [2015-10-01 00:15:54.107432] I [MSGID: 114020] [client.c:2118:notify]<br>
>> > 0-gv0-client-1: parent translators are ready, attempting connect on<br>
>> > transport<br>
>> > [2015-10-01 00:15:54.115962] I [MSGID: 114020] [client.c:2118:notify]<br>
>> > 0-gv0-client-2: parent translators are ready, attempting connect on<br>
>> > transport<br>
>> > [2015-10-01 00:15:54.120474] E [socket.c:2278:socket_connect_finish]<br>
>> > 0-gv0-client-1: connection to <a href="http://160.10.31.64:24007" rel="noreferrer" target="_blank">160.10.31.64:24007</a><br>
>> <<a href="http://160.10.31.64:24007" rel="noreferrer" target="_blank">http://160.10.31.64:24007</a>> failed (Connection<br>
>> > refused)<br>
>> > [2015-10-01 00:15:54.120639] I [rpc-clnt.c:1851:rpc_clnt_reconfig]<br>
>> > 0-gv0-client-0: changing port to 49152 (from 0)<br>
>> > Final graph:<br>
>> ><br>
>> ><br>
>> +------------------------------------------------------------------------------+<br>
>> > 1: volume gv0-client-0<br>
>> > 2: type protocol/client<br>
>> > 3: option ping-timeout 42<br>
>> > 4: option remote-host <a href="http://eapps-gluster01.uwg.westga.edu" rel="noreferrer" target="_blank">eapps-gluster01.uwg.westga.edu</a><br>
>> <<a href="http://eapps-gluster01.uwg.westga.edu" rel="noreferrer" target="_blank">http://eapps-gluster01.uwg.westga.edu</a>><br>
>> > 5: option remote-subvolume /export/sdb1/gv0<br>
>> > 6: option transport-type socket<br>
>> > 7: option username 0005f8fa-107a-4cc8-ac38-bb821c014c14<br>
>> > 8: option password 379bae9a-6529-4564-a6f5-f5a9f7424d01<br>
>> > 9: end-volume<br>
>> > 10:<br>
>> > 11: volume gv0-client-1<br>
>> > 12: type protocol/client<br>
>> > 13: option ping-timeout 42<br>
>> > 14: option remote-host <a href="http://eapps-gluster02.uwg.westga.edu" rel="noreferrer" target="_blank">eapps-gluster02.uwg.westga.edu</a><br>
>> <<a href="http://eapps-gluster02.uwg.westga.edu" rel="noreferrer" target="_blank">http://eapps-gluster02.uwg.westga.edu</a>><br>
>> > 15: option remote-subvolume /export/sdb1/gv0<br>
>> > 16: option transport-type socket<br>
>> > 17: option username 0005f8fa-107a-4cc8-ac38-bb821c014c14<br>
>> > 18: option password 379bae9a-6529-4564-a6f5-f5a9f7424d01<br>
>> > 19: end-volume<br>
>> > 20:<br>
>> > 21: volume gv0-client-2<br>
>> > 22: type protocol/client<br>
>> > 23: option ping-timeout 42<br>
>> > 24: option remote-host <a href="http://eapps-gluster03.uwg.westga.edu" rel="noreferrer" target="_blank">eapps-gluster03.uwg.westga.edu</a><br>
>> <<a href="http://eapps-gluster03.uwg.westga.edu" rel="noreferrer" target="_blank">http://eapps-gluster03.uwg.westga.edu</a>><br>
>> > 25: option remote-subvolume /export/sdb1/gv0<br>
>> > 26: option transport-type socket<br>
>> > 27: option username 0005f8fa-107a-4cc8-ac38-bb821c014c14<br>
>> > 28: option password 379bae9a-6529-4564-a6f5-f5a9f7424d01<br>
>> > 29: end-volume<br>
>> > 30:<br>
>> > 31: volume gv0-replicate-0<br>
>> > 32: type cluster/replicate<br>
>> > 33: option node-uuid 416d712a-06fc-4b3c-a92f-8c82145626ff<br>
>> > 40: subvolumes gv0-client-0 gv0-client-1 gv0-client-2<br>
>> > 41: end-volume<br>
>> > 42:<br>
>> > 43: volume glustershd<br>
>> > 44: type debug/io-stats<br>
>> > 45: subvolumes gv0-replicate-0<br>
>> > 46: end-volume<br>
>> > 47:<br>
>> ><br>
>> ><br>
>> +------------------------------------------------------------------------------+<br>
>> > [2015-10-01 00:15:54.135650] I [MSGID: 114057]<br>
>> > [client-handshake.c:1437:select_server_supported_programs]<br>
>> 0-gv0-client-0:<br>
>> > Using Program GlusterFS 3.3, Num (1298437), Version (330)<br>
>> > [2015-10-01 00:15:54.136223] I [MSGID: 114046]<br>
>> > [client-handshake.c:1213:client_setvolume_cbk] 0-gv0-client-0:<br>
>> Connected to<br>
>> > gv0-client-0, attached to remote volume '/export/sdb1/gv0'.<br>
>> > [2015-10-01 00:15:54.136262] I [MSGID: 114047]<br>
>> > [client-handshake.c:1224:client_setvolume_cbk] 0-gv0-client-0:<br>
>> Server and<br>
>> > Client lk-version numbers are not same, reopening the fds<br>
>> > [2015-10-01 00:15:54.136410] I [MSGID: 108005]<br>
>> > [afr-common.c:3998:afr_notify] 0-gv0-replicate-0: Subvolume<br>
>> 'gv0-client-0'<br>
>> > came back up; going online.<br>
>> > [2015-10-01 00:15:54.136500] I [MSGID: 114035]<br>
>> > [client-handshake.c:193:client_set_lk_version_cbk] 0-gv0-client-0:<br>
>> Server<br>
>> > lk version = 1<br>
>> > [2015-10-01 00:15:54.401702] E [MSGID: 114058]<br>
>> > [client-handshake.c:1524:client_query_portmap_cbk] 0-gv0-client-2:<br>
>> failed<br>
>> > to get the port number for remote subvolume. Please run 'gluster<br>
>> volume<br>
>> > status' on server to see if brick process is running.<br>
>> > [2015-10-01 00:15:54.401834] I [MSGID: 114018]<br>
>> > [client.c:2042:client_rpc_notify] 0-gv0-client-2: disconnected from<br>
>> > gv0-client-2. Client process will keep trying to connect to<br>
>> glusterd until<br>
>> > brick's port is available<br>
>> > [2015-10-01 00:15:54.401878] W [MSGID: 108001]<br>
>> > [afr-common.c:4081:afr_notify] 0-gv0-replicate-0: Client-quorum is<br>
>> not met<br>
>> > [2015-10-01 03:57:52.755426] E [socket.c:2278:socket_connect_finish]<br>
>> > 0-gv0-client-2: connection to <a href="http://160.10.31.227:24007" rel="noreferrer" target="_blank">160.10.31.227:24007</a><br>
>> <<a href="http://160.10.31.227:24007" rel="noreferrer" target="_blank">http://160.10.31.227:24007</a>> failed (Connection<br>
>> > refused)<br>
>> > [2015-10-01 13:50:49.000708] E [socket.c:2278:socket_connect_finish]<br>
>> > 0-gv0-client-2: connection to <a href="http://160.10.31.227:24007" rel="noreferrer" target="_blank">160.10.31.227:24007</a><br>
>> <<a href="http://160.10.31.227:24007" rel="noreferrer" target="_blank">http://160.10.31.227:24007</a>> failed (Connection<br>
>> > timed out)<br>
>> > [2015-10-01 14:36:40.481673] E [MSGID: 114058]<br>
>> > [client-handshake.c:1524:client_query_portmap_cbk] 0-gv0-client-1:<br>
>> failed<br>
>> > to get the port number for remote subvolume. Please run 'gluster<br>
>> volume<br>
>> > status' on server to see if brick process is running.<br>
>> > [2015-10-01 14:36:40.481833] I [MSGID: 114018]<br>
>> > [client.c:2042:client_rpc_notify] 0-gv0-client-1: disconnected from<br>
>> > gv0-client-1. Client process will keep trying to connect to<br>
>> glusterd until<br>
>> > brick's port is available<br>
>> > [2015-10-01 14:36:41.982037] I [rpc-clnt.c:1851:rpc_clnt_reconfig]<br>
>> > 0-gv0-client-1: changing port to 49152 (from 0)<br>
>> > [2015-10-01 14:36:41.993478] I [MSGID: 114057]<br>
>> > [client-handshake.c:1437:select_server_supported_programs]<br>
>> 0-gv0-client-1:<br>
>> > Using Program GlusterFS 3.3, Num (1298437), Version (330)<br>
>> > [2015-10-01 14:36:41.994568] I [MSGID: 114046]<br>
>> > [client-handshake.c:1213:client_setvolume_cbk] 0-gv0-client-1:<br>
>> Connected to<br>
>> > gv0-client-1, attached to remote volume '/export/sdb1/gv0'.<br>
>> > [2015-10-01 14:36:41.994647] I [MSGID: 114047]<br>
>> > [client-handshake.c:1224:client_setvolume_cbk] 0-gv0-client-1:<br>
>> Server and<br>
>> > Client lk-version numbers are not same, reopening the fds<br>
>> > [2015-10-01 14:36:41.994899] I [MSGID: 108002]<br>
>> > [afr-common.c:4077:afr_notify] 0-gv0-replicate-0: Client-quorum is met<br>
>> > [2015-10-01 14:36:42.002275] I [MSGID: 114035]<br>
>> > [client-handshake.c:193:client_set_lk_version_cbk] 0-gv0-client-1:<br>
>> Server<br>
>> > lk version = 1<br>
>> ><br>
>> ><br>
>> ><br>
>> ><br>
>> > Thanks,<br>
>> > Gene Liverman<br>
>> > Systems Integration Architect<br>
>> > Information Technology Services<br>
>> > University of West Georgia<br>
>> > <a href="mailto:gliverma@westga.edu" target="_blank">gliverma@westga.edu</a> <mailto:<a href="mailto:gliverma@westga.edu" target="_blank">gliverma@westga.edu</a>><br>
>> ><br>
>> > ITS: Making Technology Work for You!<br>
>> ><br>
>> ><br>
>> ><br>
>> > On Wed, Sep 30, 2015 at 10:54 PM, Gaurav Garg < <a href="mailto:ggarg@redhat.com" target="_blank">ggarg@redhat.com</a><br>
>> <mailto:<a href="mailto:ggarg@redhat.com" target="_blank">ggarg@redhat.com</a>> > wrote:<br>
>> ><br>
>> ><br>
>> > Hi Gene,<br>
>> ><br>
>> > Could you paste or attach core file/glusterd log file/cmd history<br>
>> to find<br>
>> > out actual RCA of the crash. What steps you performed for this crash.<br>
>> ><br>
>> > >> How can I troubleshoot this?<br>
>> ><br>
>> > If you want to troubleshoot this then you can look into the<br>
>> glusterd log<br>
>> > file, core file.<br>
>> ><br>
>> > Thank you..<br>
>> ><br>
>> > Regards,<br>
>> > Gaurav<br>
>> ><br>
>> > ----- Original Message -----<br>
>> > From: "Gene Liverman" < <a href="mailto:gliverma@westga.edu" target="_blank">gliverma@westga.edu</a><br>
>> <mailto:<a href="mailto:gliverma@westga.edu" target="_blank">gliverma@westga.edu</a>> ><br>
>> > To: <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a> <mailto:<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br>
>> > Sent: Thursday, October 1, 2015 7:59:47 AM<br>
>> > Subject: [Gluster-users] glusterd crashing<br>
>> ><br>
>> > In the last few days I've started having issues with my glusterd<br>
>> service<br>
>> > crashing. When it goes down it seems to do so on all nodes in my<br>
>> replicated<br>
>> > volume. How can I troubleshoot this? I'm on a mix of CentOS 6 and<br>
>> RHEL 6.<br>
>> > Thanks!<br>
>> ><br>
>> ><br>
>> ><br>
>> > Gene Liverman<br>
>> > Systems Integration Architect<br>
>> > Information Technology Services<br>
>> > University of West Georgia<br>
>> > <a href="mailto:gliverma@westga.edu" target="_blank">gliverma@westga.edu</a> <mailto:<a href="mailto:gliverma@westga.edu" target="_blank">gliverma@westga.edu</a>><br>
>> ><br>
>> ><br>
>> > Sent from Outlook on my iPhone<br>
>> ><br>
>> ><br>
>> > _______________________________________________<br>
>> > Gluster-users mailing list<br>
>> > <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a> <mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>><br>
>> > <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
>> ><br>
>> ><br>
>> > _______________________________________________<br>
>> > Gluster-users mailing list<br>
>> > <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a> <mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>><br>
>> > <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
>> ><br>
>><br>
>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> Gluster-users mailing list<br>
>> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
>> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
>><br>
> _______________________________________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
><br>
</font></div></div></blockquote></div><br></div></div>
<br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div>
</div></div></blockquote></div><br></div>
</blockquote></div>