<div dir="ltr">Sorry for the delay... they joys of multiple proverbial fires at once. In /var/log/messages I found this for our most recent crash:<div><br></div><div><div><font face="monospace, monospace">Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]: pending frames:</font></div><div><font face="monospace, monospace">Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]: patchset: git://<a href="http://git.gluster.com/glusterfs.git">git.gluster.com/glusterfs.git</a></font></div><div><font face="monospace, monospace">Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]: signal received: 6</font></div><div><font face="monospace, monospace">Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]: time of crash:</font></div><div><font face="monospace, monospace">Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]: 2015-10-03 04:26:21</font></div><div><font face="monospace, monospace">Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]: configuration details:</font></div><div><font face="monospace, monospace">Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]: argp 1</font></div><div><font face="monospace, monospace">Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]: backtrace 1</font></div><div><font face="monospace, monospace">Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]: dlfcn 1</font></div><div><font face="monospace, monospace">Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]: libpthread 1</font></div><div><font face="monospace, monospace">Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]: llistxattr 1</font></div><div><font face="monospace, monospace">Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]: setfsid 1</font></div><div><font face="monospace, monospace">Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]: spinlock 1</font></div><div><font face="monospace, monospace">Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]: epoll.h 1</font></div><div><font face="monospace, monospace">Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]: xattr.h 1</font></div><div><font face="monospace, monospace">Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]: st_atim.tv_nsec 1</font></div><div><font face="monospace, monospace">Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]: package-string: glusterfs 3.7.4</font></div><div><font face="monospace, monospace">Oct 3 00:26:21 eapps-gluster01 etc-glusterfs-glusterd.vol[36992]: ---------</font></div></div><div><br></div><div><br></div><div>I have posted etc-glusterfs-glusterd.vol.log to <a href="http://pastebin.com/Pzq1j5J3">http://pastebin.com/Pzq1j5J3</a>. I also put the core file and an sosreport on my web server for you but don't want to leave them there for long so I'd appreciate it if you'd let me know once you get them. They are at the following url's:</div><div><a href="http://www.westga.edu/~gliverma/tmp-files/core.36992">http://www.westga.edu/~gliverma/tmp-files/core.36992</a></div><a href="http://www.westga.edu/~gliverma/tmp-files/sosreport-gliverman.gluster-crashing-20151006101239.tar.xz">http://www.westga.edu/~gliverma/tmp-files/sosreport-gliverman.gluster-crashing-20151006101239.tar.xz</a><div><a href="http://www.westga.edu/~gliverma/tmp-files/sosreport-gliverman.gluster-crashing-20151006101239.tar.xz.md5">http://www.westga.edu/~gliverma/tmp-files/sosreport-gliverman.gluster-crashing-20151006101239.tar.xz.md5</a><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div>Thanks again for the help!</div><div class="gmail_extra"><div><div class="gmail_signature"><div dir="ltr"><div dir="ltr"><div style="color:rgb(51,102,255);font-family:tahoma,sans-serif"><font size="4"><b>Gene Liverman</b></font></div><div style="font-family:tahoma,sans-serif">Systems Integration Architect</div><div style="font-family:tahoma,sans-serif">Information Technology Services</div><div style="font-family:tahoma,sans-serif">University of West Georgia</div><div style="font-family:tahoma,sans-serif"><a href="mailto:gliverma@westga.edu" target="_blank">gliverma@westga.edu</a></div><div style="font-family:tahoma,sans-serif"><br></div><div style="font-family:tahoma,sans-serif">ITS: Making Technology Work for You!</div><div style="font-family:tahoma,sans-serif"><br><img src="http://www.westga.edu/assetsDept/ucm/e-sig_wolves.png"><br></div><div style="font-family:tahoma,sans-serif"><br></div></div></div></div></div>
<br><div class="gmail_quote">On Fri, Oct 2, 2015 at 11:18 AM, Gaurav Garg <span dir="ltr"><<a href="mailto:ggarg@redhat.com" target="_blank">ggarg@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">>> Pulling those logs now but how do I generate the core file you are asking<br>
for?<br>
<br>
</span>When there is crash then core file automatically generated based on your *ulimit* set option. you can find location of core file in your root or current working directory or where ever you have set your core dump file location. core file gives you information regarding crash, where exactly crash happened.<br>
you can find appropriate core file by looking at crash time in glusterd log's by searching "crash" keyword. you can also paste few line's just above latest "crash" keyword in glusterd logs.<br>
<br>
Just for your curiosity if you willing to look where it crash then you can debug it by #gdb -c <location of core file> glusterd<br>
<br>
Thank you...<br>
<span class=""><br>
Regards,<br>
Gaurav<br>
<br>
----- Original Message -----<br>
From: "Gene Liverman" <<a href="mailto:gliverma@westga.edu">gliverma@westga.edu</a>><br>
</span><span class="">To: "Gaurav Garg" <<a href="mailto:ggarg@redhat.com">ggarg@redhat.com</a>><br>
Cc: "gluster-users" <<a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br>
Sent: Friday, October 2, 2015 8:28:49 PM<br>
Subject: Re: [Gluster-users] glusterd crashing<br>
<br>
</span><span class="">Pulling those logs now but how do I generate the core file you are asking<br>
for?<br>
<br>
<br>
<br>
<br>
<br>
--<br>
</span>*Gene Liverman*<br>
<span class="im HOEnZb">Systems Integration Architect<br>
Information Technology Services<br>
University of West Georgia<br>
<a href="mailto:gliverma@westga.edu">gliverma@westga.edu</a><br>
<a href="tel:678.839.5492" value="+16788395492">678.839.5492</a><br>
<br>
ITS: Making Technology Work for You!<br>
<br>
<br>
<br>
<br>
</span><div class="HOEnZb"><div class="h5">On Fri, Oct 2, 2015 at 2:25 AM, Gaurav Garg <<a href="mailto:ggarg@redhat.com">ggarg@redhat.com</a>> wrote:<br>
<br>
> Hi Gene,<br>
><br>
> you have paste glustershd log. we asked you to paste glusterd log.<br>
> glusterd and glustershd both are different process. with this information<br>
> we can't find out why your glusterd crashed. could you paste *glusterd*<br>
> logs (/var/log/glusterfs/usr-local-etc-glusterfs-glusterd.vol.log*) in<br>
> pastebin (not in this mail thread) and give the link of pastebin in this<br>
> mail thread. Can you also attach core file or you can paste backtrace of<br>
> that core dump file.<br>
> It will be great if you give us sos report of the node where the crash<br>
> happen.<br>
><br>
> Thanx,<br>
><br>
> ~Gaurav<br>
><br>
> ----- Original Message -----<br>
> From: "Gene Liverman" <<a href="mailto:gliverma@westga.edu">gliverma@westga.edu</a>><br>
> To: "gluster-users" <<a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br>
> Sent: Friday, October 2, 2015 4:47:00 AM<br>
> Subject: Re: [Gluster-users] glusterd crashing<br>
><br>
> Sorry for the delay. Here is what's installed:<br>
> # rpm -qa | grep gluster<br>
> glusterfs-geo-replication-3.7.4-2.el6.x86_64<br>
> glusterfs-client-xlators-3.7.4-2.el6.x86_64<br>
> glusterfs-3.7.4-2.el6.x86_64<br>
> glusterfs-libs-3.7.4-2.el6.x86_64<br>
> glusterfs-api-3.7.4-2.el6.x86_64<br>
> glusterfs-fuse-3.7.4-2.el6.x86_64<br>
> glusterfs-server-3.7.4-2.el6.x86_64<br>
> glusterfs-cli-3.7.4-2.el6.x86_64<br>
><br>
> The cmd_history.log file is attached.<br>
> In gluster.log I have filtered out a bunch of lines like the one below due<br>
> to make them more readable. I had a node down for multiple days due to<br>
> maintenance and another one went down due to a hardware failure during that<br>
> time too.<br>
> [2015-10-01 00:16:09.643631] W [MSGID: 114031]<br>
> [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-gv0-client-0: remote<br>
> operation failed. Path: <gfid:31f17f8c-6c96-4440-88c0-f813b3c8d364><br>
> (31f17f8c-6c96-4440-88c0-f813b3c8d364) [No such file or directory]<br>
><br>
> I also filtered out a boat load of self heal lines like these two:<br>
> [2015-10-01 15:14:14.851015] I [MSGID: 108026]<br>
> [afr-self-heal-metadata.c:56:__afr_selfheal_metadata_do] 0-gv0-replicate-0:<br>
> performing metadata selfheal on f78a47db-a359-430d-a655-1d217eb848c3<br>
> [2015-10-01 15:14:14.856392] I [MSGID: 108026]<br>
> [afr-self-heal-common.c:651:afr_log_selfheal] 0-gv0-replicate-0: Completed<br>
> metadata selfheal on f78a47db-a359-430d-a655-1d217eb848c3. source=0 sinks=1<br>
><br>
><br>
> [root@eapps-gluster01 glusterfs]# cat glustershd.log |grep -v 'remote<br>
> operation failed' |grep -v 'self-heal'<br>
> [2015-09-27 08:46:56.893125] E [rpc-clnt.c:201:call_bail] 0-glusterfs:<br>
> bailing out frame type(GlusterFS Handshake) op(GETSPEC(2)) xid = 0x6 sent =<br>
> 2015-09-27 08:16:51.742731. timeout = 1800 for <a href="http://127.0.0.1:24007" rel="noreferrer" target="_blank">127.0.0.1:24007</a><br>
> [2015-09-28 12:54:17.524924] W [socket.c:588:__socket_rwv] 0-glusterfs:<br>
> readv on <a href="http://127.0.0.1:24007" rel="noreferrer" target="_blank">127.0.0.1:24007</a> failed (Connection reset by peer)<br>
> [2015-09-28 12:54:27.844374] I [glusterfsd-mgmt.c:1512:mgmt_getspec_cbk]<br>
> 0-glusterfs: No change in volfile, continuing<br>
> [2015-09-28 12:57:03.485027] W [socket.c:588:__socket_rwv] 0-gv0-client-2:<br>
> readv on <a href="http://160.10.31.227:24007" rel="noreferrer" target="_blank">160.10.31.227:24007</a> failed (Connection reset by peer)<br>
> [2015-09-28 12:57:05.872973] E [socket.c:2278:socket_connect_finish]<br>
> 0-gv0-client-2: connection to <a href="http://160.10.31.227:24007" rel="noreferrer" target="_blank">160.10.31.227:24007</a> failed (Connection<br>
> refused)<br>
> [2015-09-28 12:57:38.490578] W [socket.c:588:__socket_rwv] 0-glusterfs:<br>
> readv on <a href="http://127.0.0.1:24007" rel="noreferrer" target="_blank">127.0.0.1:24007</a> failed (No data available)<br>
> [2015-09-28 12:57:49.054475] I [glusterfsd-mgmt.c:1512:mgmt_getspec_cbk]<br>
> 0-glusterfs: No change in volfile, continuing<br>
> [2015-09-28 13:01:12.062960] W [glusterfsd.c:1219:cleanup_and_exit]<br>
> (-->/lib64/libpthread.so.0() [0x3c65e07a51]<br>
> -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x405e4d]<br>
> -->/usr/sbin/glusterfs(cleanup_and_exit+0x65) [0x4059b5] ) 0-: received<br>
> signum (15), shutting down<br>
> [2015-09-28 13:01:12.981945] I [MSGID: 100030] [glusterfsd.c:2301:main]<br>
> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.4<br>
> (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p<br>
> /var/lib/glusterd/glustershd/run/glustershd.pid -l<br>
> /var/log/glusterfs/glustershd.log -S<br>
> /var/run/gluster/9a9819e90404187e84e67b01614bbe10.socket --xlator-option<br>
> *replicate*.node-uuid=416d712a-06fc-4b3c-a92f-8c82145626ff)<br>
> [2015-09-28 13:01:13.009171] I [MSGID: 101190]<br>
> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread<br>
> with index 1<br>
> [2015-09-28 13:01:13.092483] I [graph.c:269:gf_add_cmdline_options]<br>
> 0-gv0-replicate-0: adding option 'node-uuid' for volume 'gv0-replicate-0'<br>
> with value '416d712a-06fc-4b3c-a92f-8c82145626ff'<br>
> [2015-09-28 13:01:13.100856] I [MSGID: 101190]<br>
> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread<br>
> with index 2<br>
> [2015-09-28 13:01:13.103995] I [MSGID: 114020] [client.c:2118:notify]<br>
> 0-gv0-client-0: parent translators are ready, attempting connect on<br>
> transport<br>
> [2015-09-28 13:01:13.114745] I [MSGID: 114020] [client.c:2118:notify]<br>
> 0-gv0-client-1: parent translators are ready, attempting connect on<br>
> transport<br>
> [2015-09-28 13:01:13.115725] I [rpc-clnt.c:1851:rpc_clnt_reconfig]<br>
> 0-gv0-client-0: changing port to 49152 (from 0)<br>
> [2015-09-28 13:01:13.125619] I [MSGID: 114020] [client.c:2118:notify]<br>
> 0-gv0-client-2: parent translators are ready, attempting connect on<br>
> transport<br>
> [2015-09-28 13:01:13.132316] E [socket.c:2278:socket_connect_finish]<br>
> 0-gv0-client-1: connection to <a href="http://160.10.31.64:24007" rel="noreferrer" target="_blank">160.10.31.64:24007</a> failed (Connection<br>
> refused)<br>
> [2015-09-28 13:01:13.132650] I [MSGID: 114057]<br>
> [client-handshake.c:1437:select_server_supported_programs] 0-gv0-client-0:<br>
> Using Program GlusterFS 3.3, Num (1298437), Version (330)<br>
> [2015-09-28 13:01:13.133322] I [MSGID: 114046]<br>
> [client-handshake.c:1213:client_setvolume_cbk] 0-gv0-client-0: Connected to<br>
> gv0-client-0, attached to remote volume '/export/sdb1/gv0'.<br>
> [2015-09-28 13:01:13.133365] I [MSGID: 114047]<br>
> [client-handshake.c:1224:client_setvolume_cbk] 0-gv0-client-0: Server and<br>
> Client lk-version numbers are not same, reopening the fds<br>
> [2015-09-28 13:01:13.133782] I [MSGID: 108005]<br>
> [afr-common.c:3998:afr_notify] 0-gv0-replicate-0: Subvolume 'gv0-client-0'<br>
> came back up; going online.<br>
> [2015-09-28 13:01:13.133863] I [MSGID: 114035]<br>
> [client-handshake.c:193:client_set_lk_version_cbk] 0-gv0-client-0: Server<br>
> lk version = 1<br>
> Final graph:<br>
><br>
> +------------------------------------------------------------------------------+<br>
> 1: volume gv0-client-0<br>
> 2: type protocol/client<br>
> 3: option clnt-lk-version 1<br>
> 4: option volfile-checksum 0<br>
> 5: option volfile-key gluster/glustershd<br>
> 6: option client-version 3.7.4<br>
> 7: option process-uuid<br>
> eapps-gluster01-65147-2015/09/28-13:01:12:970131-gv0-client-0-0-0<br>
> 8: option fops-version 1298437<br>
> 9: option ping-timeout 42<br>
> 10: option remote-host <a href="http://eapps-gluster01.uwg.westga.edu" rel="noreferrer" target="_blank">eapps-gluster01.uwg.westga.edu</a><br>
> 11: option remote-subvolume /export/sdb1/gv0<br>
> 12: option transport-type socket<br>
> 13: option username 0005f8fa-107a-4cc8-ac38-bb821c014c14<br>
> 14: option password 379bae9a-6529-4564-a6f5-f5a9f7424d01<br>
> 15: end-volume<br>
> 16:<br>
> 17: volume gv0-client-1<br>
> 18: type protocol/client<br>
> 19: option ping-timeout 42<br>
> 20: option remote-host <a href="http://eapps-gluster02.uwg.westga.edu" rel="noreferrer" target="_blank">eapps-gluster02.uwg.westga.edu</a><br>
> 21: option remote-subvolume /export/sdb1/gv0<br>
> 22: option transport-type socket<br>
> 23: option username 0005f8fa-107a-4cc8-ac38-bb821c014c14<br>
> 24: option password 379bae9a-6529-4564-a6f5-f5a9f7424d01<br>
> 25: end-volume<br>
> 26:<br>
> 27: volume gv0-client-2<br>
> 28: type protocol/client<br>
> 29: option ping-timeout 42<br>
> 30: option remote-host <a href="http://eapps-gluster03.uwg.westga.edu" rel="noreferrer" target="_blank">eapps-gluster03.uwg.westga.edu</a><br>
> 31: option remote-subvolume /export/sdb1/gv0<br>
> 32: option transport-type socket<br>
> 33: option username 0005f8fa-107a-4cc8-ac38-bb821c014c14<br>
> 34: option password 379bae9a-6529-4564-a6f5-f5a9f7424d01<br>
> 35: end-volume<br>
> 36:<br>
> 37: volume gv0-replicate-0<br>
> 38: type cluster/replicate<br>
> 39: option node-uuid 416d712a-06fc-4b3c-a92f-8c82145626ff<br>
> 46: subvolumes gv0-client-0 gv0-client-1 gv0-client-2<br>
> 47: end-volume<br>
> 48:<br>
> 49: volume glustershd<br>
> 50: type debug/io-stats<br>
> 51: subvolumes gv0-replicate-0<br>
> 52: end-volume<br>
> 53:<br>
><br>
> +------------------------------------------------------------------------------+<br>
> [2015-09-28 13:01:13.154898] E [MSGID: 114058]<br>
> [client-handshake.c:1524:client_query_portmap_cbk] 0-gv0-client-2: failed<br>
> to get the port number for remote subvolume. Please run 'gluster volume<br>
> status' on server to see if brick process is running.<br>
> [2015-09-28 13:01:13.155031] I [MSGID: 114018]<br>
> [client.c:2042:client_rpc_notify] 0-gv0-client-2: disconnected from<br>
> gv0-client-2. Client process will keep trying to connect to glusterd until<br>
> brick's port is available<br>
> [2015-09-28 13:01:13.155080] W [MSGID: 108001]<br>
> [afr-common.c:4081:afr_notify] 0-gv0-replicate-0: Client-quorum is not met<br>
> [2015-09-29 08:11:24.728797] I [MSGID: 100011]<br>
> [glusterfsd.c:1291:reincarnate] 0-glusterfsd: Fetching the volume file from<br>
> server...<br>
> [2015-09-29 08:11:24.763338] I [glusterfsd-mgmt.c:1512:mgmt_getspec_cbk]<br>
> 0-glusterfs: No change in volfile, continuing<br>
> [2015-09-29 12:50:41.915254] E [rpc-clnt.c:201:call_bail] 0-gv0-client-2:<br>
> bailing out frame type(GF-DUMP) op(DUMP(1)) xid = 0xd91f sent = 2015-09-29<br>
> 12:20:36.092734. timeout = 1800 for <a href="http://160.10.31.227:24007" rel="noreferrer" target="_blank">160.10.31.227:24007</a><br>
> [2015-09-29 12:50:41.923550] W [MSGID: 114032]<br>
> [client-handshake.c:1623:client_dump_version_cbk] 0-gv0-client-2: received<br>
> RPC status error [Transport endpoint is not connected]<br>
> [2015-09-30 23:54:36.547979] W [socket.c:588:__socket_rwv] 0-glusterfs:<br>
> readv on <a href="http://127.0.0.1:24007" rel="noreferrer" target="_blank">127.0.0.1:24007</a> failed (No data available)<br>
> [2015-09-30 23:54:46.812870] E [socket.c:2278:socket_connect_finish]<br>
> 0-glusterfs: connection to <a href="http://127.0.0.1:24007" rel="noreferrer" target="_blank">127.0.0.1:24007</a> failed (Connection refused)<br>
> [2015-10-01 00:14:20.997081] I [glusterfsd-mgmt.c:1512:mgmt_getspec_cbk]<br>
> 0-glusterfs: No change in volfile, continuing<br>
> [2015-10-01 00:15:36.770579] W [socket.c:588:__socket_rwv] 0-gv0-client-2:<br>
> readv on <a href="http://160.10.31.227:24007" rel="noreferrer" target="_blank">160.10.31.227:24007</a> failed (Connection reset by peer)<br>
> [2015-10-01 00:15:37.906708] E [socket.c:2278:socket_connect_finish]<br>
> 0-gv0-client-2: connection to <a href="http://160.10.31.227:24007" rel="noreferrer" target="_blank">160.10.31.227:24007</a> failed (Connection<br>
> refused)<br>
> [2015-10-01 00:15:53.008130] W [glusterfsd.c:1219:cleanup_and_exit]<br>
> (-->/lib64/libpthread.so.0() [0x3b91807a51]<br>
> -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x405e4d]<br>
> -->/usr/sbin/glusterfs(cleanup_and_exit+0x65) [0x4059b5] ) 0-: received<br>
> signum (15), shutting down<br>
> [2015-10-01 00:15:53.008697] I [timer.c:48:gf_timer_call_after]<br>
> (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x3e2) [0x3b9480f992]<br>
> -->/usr/lib64/libgfrpc.so.0(__save_frame+0x76) [0x3b9480f046]<br>
> -->/usr/lib64/libglusterfs.so.0(gf_timer_call_after+0x1b1) [0x3b93447881] )<br>
> 0-timer: ctx cleanup started<br>
> [2015-10-01 00:15:53.994698] I [MSGID: 100030] [glusterfsd.c:2301:main]<br>
> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.4<br>
> (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p<br>
> /var/lib/glusterd/glustershd/run/glustershd.pid -l<br>
> /var/log/glusterfs/glustershd.log -S<br>
> /var/run/gluster/9a9819e90404187e84e67b01614bbe10.socket --xlator-option<br>
> *replicate*.node-uuid=416d712a-06fc-4b3c-a92f-8c82145626ff)<br>
> [2015-10-01 00:15:54.020401] I [MSGID: 101190]<br>
> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread<br>
> with index 1<br>
> [2015-10-01 00:15:54.086777] I [graph.c:269:gf_add_cmdline_options]<br>
> 0-gv0-replicate-0: adding option 'node-uuid' for volume 'gv0-replicate-0'<br>
> with value '416d712a-06fc-4b3c-a92f-8c82145626ff'<br>
> [2015-10-01 00:15:54.093004] I [MSGID: 101190]<br>
> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread<br>
> with index 2<br>
> [2015-10-01 00:15:54.098144] I [MSGID: 114020] [client.c:2118:notify]<br>
> 0-gv0-client-0: parent translators are ready, attempting connect on<br>
> transport<br>
> [2015-10-01 00:15:54.107432] I [MSGID: 114020] [client.c:2118:notify]<br>
> 0-gv0-client-1: parent translators are ready, attempting connect on<br>
> transport<br>
> [2015-10-01 00:15:54.115962] I [MSGID: 114020] [client.c:2118:notify]<br>
> 0-gv0-client-2: parent translators are ready, attempting connect on<br>
> transport<br>
> [2015-10-01 00:15:54.120474] E [socket.c:2278:socket_connect_finish]<br>
> 0-gv0-client-1: connection to <a href="http://160.10.31.64:24007" rel="noreferrer" target="_blank">160.10.31.64:24007</a> failed (Connection<br>
> refused)<br>
> [2015-10-01 00:15:54.120639] I [rpc-clnt.c:1851:rpc_clnt_reconfig]<br>
> 0-gv0-client-0: changing port to 49152 (from 0)<br>
> Final graph:<br>
><br>
> +------------------------------------------------------------------------------+<br>
> 1: volume gv0-client-0<br>
> 2: type protocol/client<br>
> 3: option ping-timeout 42<br>
> 4: option remote-host <a href="http://eapps-gluster01.uwg.westga.edu" rel="noreferrer" target="_blank">eapps-gluster01.uwg.westga.edu</a><br>
> 5: option remote-subvolume /export/sdb1/gv0<br>
> 6: option transport-type socket<br>
> 7: option username 0005f8fa-107a-4cc8-ac38-bb821c014c14<br>
> 8: option password 379bae9a-6529-4564-a6f5-f5a9f7424d01<br>
> 9: end-volume<br>
> 10:<br>
> 11: volume gv0-client-1<br>
> 12: type protocol/client<br>
> 13: option ping-timeout 42<br>
> 14: option remote-host <a href="http://eapps-gluster02.uwg.westga.edu" rel="noreferrer" target="_blank">eapps-gluster02.uwg.westga.edu</a><br>
> 15: option remote-subvolume /export/sdb1/gv0<br>
> 16: option transport-type socket<br>
> 17: option username 0005f8fa-107a-4cc8-ac38-bb821c014c14<br>
> 18: option password 379bae9a-6529-4564-a6f5-f5a9f7424d01<br>
> 19: end-volume<br>
> 20:<br>
> 21: volume gv0-client-2<br>
> 22: type protocol/client<br>
> 23: option ping-timeout 42<br>
> 24: option remote-host <a href="http://eapps-gluster03.uwg.westga.edu" rel="noreferrer" target="_blank">eapps-gluster03.uwg.westga.edu</a><br>
> 25: option remote-subvolume /export/sdb1/gv0<br>
> 26: option transport-type socket<br>
> 27: option username 0005f8fa-107a-4cc8-ac38-bb821c014c14<br>
> 28: option password 379bae9a-6529-4564-a6f5-f5a9f7424d01<br>
> 29: end-volume<br>
> 30:<br>
> 31: volume gv0-replicate-0<br>
> 32: type cluster/replicate<br>
> 33: option node-uuid 416d712a-06fc-4b3c-a92f-8c82145626ff<br>
> 40: subvolumes gv0-client-0 gv0-client-1 gv0-client-2<br>
> 41: end-volume<br>
> 42:<br>
> 43: volume glustershd<br>
> 44: type debug/io-stats<br>
> 45: subvolumes gv0-replicate-0<br>
> 46: end-volume<br>
> 47:<br>
><br>
> +------------------------------------------------------------------------------+<br>
> [2015-10-01 00:15:54.135650] I [MSGID: 114057]<br>
> [client-handshake.c:1437:select_server_supported_programs] 0-gv0-client-0:<br>
> Using Program GlusterFS 3.3, Num (1298437), Version (330)<br>
> [2015-10-01 00:15:54.136223] I [MSGID: 114046]<br>
> [client-handshake.c:1213:client_setvolume_cbk] 0-gv0-client-0: Connected to<br>
> gv0-client-0, attached to remote volume '/export/sdb1/gv0'.<br>
> [2015-10-01 00:15:54.136262] I [MSGID: 114047]<br>
> [client-handshake.c:1224:client_setvolume_cbk] 0-gv0-client-0: Server and<br>
> Client lk-version numbers are not same, reopening the fds<br>
> [2015-10-01 00:15:54.136410] I [MSGID: 108005]<br>
> [afr-common.c:3998:afr_notify] 0-gv0-replicate-0: Subvolume 'gv0-client-0'<br>
> came back up; going online.<br>
> [2015-10-01 00:15:54.136500] I [MSGID: 114035]<br>
> [client-handshake.c:193:client_set_lk_version_cbk] 0-gv0-client-0: Server<br>
> lk version = 1<br>
> [2015-10-01 00:15:54.401702] E [MSGID: 114058]<br>
> [client-handshake.c:1524:client_query_portmap_cbk] 0-gv0-client-2: failed<br>
> to get the port number for remote subvolume. Please run 'gluster volume<br>
> status' on server to see if brick process is running.<br>
> [2015-10-01 00:15:54.401834] I [MSGID: 114018]<br>
> [client.c:2042:client_rpc_notify] 0-gv0-client-2: disconnected from<br>
> gv0-client-2. Client process will keep trying to connect to glusterd until<br>
> brick's port is available<br>
> [2015-10-01 00:15:54.401878] W [MSGID: 108001]<br>
> [afr-common.c:4081:afr_notify] 0-gv0-replicate-0: Client-quorum is not met<br>
> [2015-10-01 03:57:52.755426] E [socket.c:2278:socket_connect_finish]<br>
> 0-gv0-client-2: connection to <a href="http://160.10.31.227:24007" rel="noreferrer" target="_blank">160.10.31.227:24007</a> failed (Connection<br>
> refused)<br>
> [2015-10-01 13:50:49.000708] E [socket.c:2278:socket_connect_finish]<br>
> 0-gv0-client-2: connection to <a href="http://160.10.31.227:24007" rel="noreferrer" target="_blank">160.10.31.227:24007</a> failed (Connection<br>
> timed out)<br>
> [2015-10-01 14:36:40.481673] E [MSGID: 114058]<br>
> [client-handshake.c:1524:client_query_portmap_cbk] 0-gv0-client-1: failed<br>
> to get the port number for remote subvolume. Please run 'gluster volume<br>
> status' on server to see if brick process is running.<br>
> [2015-10-01 14:36:40.481833] I [MSGID: 114018]<br>
> [client.c:2042:client_rpc_notify] 0-gv0-client-1: disconnected from<br>
> gv0-client-1. Client process will keep trying to connect to glusterd until<br>
> brick's port is available<br>
> [2015-10-01 14:36:41.982037] I [rpc-clnt.c:1851:rpc_clnt_reconfig]<br>
> 0-gv0-client-1: changing port to 49152 (from 0)<br>
> [2015-10-01 14:36:41.993478] I [MSGID: 114057]<br>
> [client-handshake.c:1437:select_server_supported_programs] 0-gv0-client-1:<br>
> Using Program GlusterFS 3.3, Num (1298437), Version (330)<br>
> [2015-10-01 14:36:41.994568] I [MSGID: 114046]<br>
> [client-handshake.c:1213:client_setvolume_cbk] 0-gv0-client-1: Connected to<br>
> gv0-client-1, attached to remote volume '/export/sdb1/gv0'.<br>
> [2015-10-01 14:36:41.994647] I [MSGID: 114047]<br>
> [client-handshake.c:1224:client_setvolume_cbk] 0-gv0-client-1: Server and<br>
> Client lk-version numbers are not same, reopening the fds<br>
> [2015-10-01 14:36:41.994899] I [MSGID: 108002]<br>
> [afr-common.c:4077:afr_notify] 0-gv0-replicate-0: Client-quorum is met<br>
> [2015-10-01 14:36:42.002275] I [MSGID: 114035]<br>
> [client-handshake.c:193:client_set_lk_version_cbk] 0-gv0-client-1: Server<br>
> lk version = 1<br>
><br>
><br>
><br>
><br>
> Thanks,<br>
> Gene Liverman<br>
> Systems Integration Architect<br>
> Information Technology Services<br>
> University of West Georgia<br>
> <a href="mailto:gliverma@westga.edu">gliverma@westga.edu</a><br>
><br>
> ITS: Making Technology Work for You!<br>
><br>
><br>
><br>
> On Wed, Sep 30, 2015 at 10:54 PM, Gaurav Garg < <a href="mailto:ggarg@redhat.com">ggarg@redhat.com</a> > wrote:<br>
><br>
><br>
> Hi Gene,<br>
><br>
> Could you paste or attach core file/glusterd log file/cmd history to find<br>
> out actual RCA of the crash. What steps you performed for this crash.<br>
><br>
> >> How can I troubleshoot this?<br>
><br>
> If you want to troubleshoot this then you can look into the glusterd log<br>
> file, core file.<br>
><br>
> Thank you..<br>
><br>
> Regards,<br>
> Gaurav<br>
><br>
> ----- Original Message -----<br>
> From: "Gene Liverman" < <a href="mailto:gliverma@westga.edu">gliverma@westga.edu</a> ><br>
> To: <a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br>
> Sent: Thursday, October 1, 2015 7:59:47 AM<br>
> Subject: [Gluster-users] glusterd crashing<br>
><br>
> In the last few days I've started having issues with my glusterd service<br>
> crashing. When it goes down it seems to do so on all nodes in my replicated<br>
> volume. How can I troubleshoot this? I'm on a mix of CentOS 6 and RHEL 6.<br>
> Thanks!<br>
><br>
><br>
><br>
> Gene Liverman<br>
> Systems Integration Architect<br>
> Information Technology Services<br>
> University of West Georgia<br>
> <a href="mailto:gliverma@westga.edu">gliverma@westga.edu</a><br>
><br>
><br>
> Sent from Outlook on my iPhone<br>
><br>
><br>
> _______________________________________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
><br>
><br>
> _______________________________________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
><br>
</div></div></blockquote></div><br></div></div>