<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-AU" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hello,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We are trying to run a pair of ActiveMQ nodes on top of glusterfs, using the approach described in
<a href="http://activemq.apache.org/shared-file-system-master-slave.html">http://activemq.apache.org/shared-file-system-master-slave.html</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">This seemed to work at first, but if I start rebooting machines while under load I seem to quickly get into this problem:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> [2015-08-05 08:54:40.475351] I [afr-self-heal-common.c:705:afr_mark_sources] 0-gv0-replicate-0: split-brain possible, no source detected<o:p></o:p></p>
<p class="MsoNormal"> [2015-08-05 08:54:40.475373] W [fuse-bridge.c:184:fuse_entry_cbk] 0-glusterfs-fuse: 61819: LOOKUP() /kahadb/db.data => -1 (Input/output error)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">(from /var/log/glusterfs/srv-amq.log , more of the log below)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Afterwards the whole cluster ceases to function, since the affected file is crucial to ActiveMQ’s storage backend.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have gotten into this situation three times by now, recovering in between by rebuilding the glusterfs configuration from scratch (stop volume, delete, empty bricks, create, start). The trigger is always a “sudo reboot” on one of the nodes.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Am I wrong to expect this to work or is this an issue with my configuration or glusterfs itself?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Cheers,<o:p></o:p></p>
<p class="MsoNormal"> Peter<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">More detail:<o:p></o:p></p>
<p class="MsoNormal">-----<o:p></o:p></p>
<p class="MsoNormal">qmaster@srvamqpy01:~$ cat /etc/issue<o:p></o:p></p>
<p class="MsoNormal">Ubuntu 12.04.5 LTS \n \l<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">qmaster@srvamqpy01:~$ uname -a<o:p></o:p></p>
<p class="MsoNormal">Linux srvamqpy01 3.13.0-61-generic #100~precise1-Ubuntu SMP Wed Jul 29 12:06:40 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux<o:p></o:p></p>
<p class="MsoNormal">qmaster@srvamqpy01:~$ gluster --version<o:p></o:p></p>
<p class="MsoNormal">glusterfs 3.2.5 built on Jan 31 2012 07:39:59<o:p></o:p></p>
<p class="MsoNormal">[…]<o:p></o:p></p>
<p class="MsoNormal">qmaster@srvamqpy01:~$ cat /etc/fstab<o:p></o:p></p>
<p class="MsoNormal">[…]<o:p></o:p></p>
<p class="MsoNormal">/dev/sdb1 /data/brick1 ext4 acl,user_xattr 0 2<o:p></o:p></p>
<p class="MsoNormal">srvamqpy01:/gv0 /srv/amq glusterfs defaults,nobootwait,_netdev,direct-io-mode=disable 0 0<o:p></o:p></p>
<p class="MsoNormal">-----<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Command used to create the volume:<o:p></o:p></p>
<p class="MsoNormal">-----<o:p></o:p></p>
<p class="MsoNormal">gluster volume create gv0 replica 2 srvamqpy01:/data/brick1/gv0 srvamqpy02:/data/brick1/gv0<o:p></o:p></p>
<p class="MsoNormal">-----<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">And more of the log:<o:p></o:p></p>
<p class="MsoNormal">-----<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:51:54.50969] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-gv0-client-0: changing port to 24011 (from 0)<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:51:54.51313] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-gv0-client-1: changing port to 24011 (from 0)<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:51:58.32060] I [client-handshake.c:1090:select_server_supported_programs] 0-gv0-client-0: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:51:58.32239] I [client-handshake.c:913:client_setvolume_cbk] 0-gv0-client-0: Connected to 10.254.2.137:24011, attached to remote volume '/data/brick1/gv0'.<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:51:58.32257] I [afr-common.c:3141:afr_notify] 0-gv0-replicate-0: Subvolume 'gv0-client-0' came back up; going online.<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:51:58.32359] I [client-handshake.c:1090:select_server_supported_programs] 0-gv0-client-1: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:51:58.33070] I [client-handshake.c:913:client_setvolume_cbk] 0-gv0-client-1: Connected to 10.254.2.164:24011, attached to remote volume '/data/brick1/gv0'.<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:51:58.35521] I [fuse-bridge.c:3339:fuse_graph_setup] 0-fuse: switched to graph 0<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:51:58.35642] I [fuse-bridge.c:2927:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.22<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:51:58.36851] I [afr-common.c:1520:afr_set_root_inode_on_first_lookup] 0-gv0-replicate-0: added root inode<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:06.24620] I [afr-common.c:1038:afr_launch_self_heal] 0-gv0-replicate-0: background meta-data data self-heal triggered. path: /kahadb/lock<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:06.28557] I [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] 0-gv0-replicate-0: background meta-data data self-heal completed on /kahadb/lock<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:16.64428] I [afr-common.c:1038:afr_launch_self_heal] 0-gv0-replicate-0: background meta-data self-heal triggered. path: /kahadb/lock<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:16.65701] I [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] 0-gv0-replicate-0: background meta-data self-heal completed on /kahadb/lock<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:21.692657] W [socket.c:1494:__socket_proto_state_machine] 0-gv0-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.254.2.164:24011)<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:21.693353] I [client.c:1883:client_rpc_notify] 0-gv0-client-1: disconnected<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:26.71942] W [client3_1-fops.c:4699:client3_1_lk] 0-gv0-client-1: (-1909467425): failed to get fd ctx. EBADFD<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:26.71988] W [client3_1-fops.c:4751:client3_1_lk] 0-gv0-client-1: failed to send the fop: File descriptor in bad state<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:32.35552] E [socket.c:1685:socket_connect_finish] 0-gv0-client-1: connection to 10.254.2.164:24011 failed (Connection refused)<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:35.36179] I [client-handshake.c:1090:select_server_supported_programs] 0-gv0-client-1: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:35.37641] I [client-handshake.c:913:client_setvolume_cbk] 0-gv0-client-1: Connected to 10.254.2.164:24011, attached to remote volume '/data/brick1/gv0'.<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:36.538807] I [afr-open.c:432:afr_openfd_sh] 0-gv0-replicate-0: data missing-entry gfid self-heal triggered. path: /kahadb/db-4.log, reason: Replicate up down flush, data lock is held<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:36.539349] I [afr-self-heal-common.c:1203:sh_missing_entries_create] 0-gv0-replicate-0: no missing files - /kahadb/db-4.log. proceeding to metadata check<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:36.540105] W [dict.c:418:dict_unref] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x7fea25a93ec5] (-->/usr/lib/glusterfs/3.2.5/xlator/protocol/client.so(client3_1_fstat_cbk+0x312) [0x7fea228f8902] (-->/usr/lib/glusterfs/3.2.5/xlator/cluster/replicate.so(afr_sh_data_fstat_cbk+0x1d5)
[0x7fea226a0405]))) 0-dict: dict is NULL<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:36.772749] I [afr-self-heal-algorithm.c:520:sh_diff_loop_driver_done] 0-gv0-replicate-0: diff self-heal on /kahadb/db-4.log: completed. (1 blocks of 252 were different (0.40%))<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:36.775638] I [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] 0-gv0-replicate-0: background data missing-entry gfid self-heal completed on /kahadb/db-4.log<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:36.785113] I [afr-open.c:432:afr_openfd_sh] 0-gv0-replicate-0: data missing-entry gfid self-heal triggered. path: /kahadb/db.redo, reason: Replicate up down flush, data lock is held<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:36.785214] I [afr-open.c:432:afr_openfd_sh] 0-gv0-replicate-0: data missing-entry gfid self-heal triggered. path: /kahadb/db.data, reason: Replicate up down flush, data lock is held<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:36.785458] I [afr-self-heal-common.c:1858:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-gv0-replicate-0: Non blocking entrylks failed.<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:36.785480] I [afr-self-heal-common.c:963:afr_sh_missing_entries_done] 0-gv0-replicate-0: split brain found, aborting selfheal of /kahadb/db.data<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:36.785496] E [afr-self-heal-common.c:2074:afr_self_heal_completion_cbk] 0-gv0-replicate-0: background data missing-entry gfid self-heal failed on /kahadb/db.data<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:36.786139] I [afr-self-heal-common.c:1203:sh_missing_entries_create] 0-gv0-replicate-0: no missing files - /kahadb/db.redo. proceeding to metadata check<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:36.787147] I [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] 0-gv0-replicate-0: background data missing-entry gfid self-heal completed on /kahadb/db.redo<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:56.948495] I [afr-common.c:1038:afr_launch_self_heal] 0-gv0-replicate-0: background entry self-heal triggered. path: /kahadb<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:56.949790] I [afr-self-heal-entry.c:644:afr_sh_entry_expunge_entry_cbk] 0-gv0-replicate-0: missing entry /kahadb/db.free on gv0-client-0<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:56.952400] E [afr-self-heal-common.c:1054:afr_sh_common_lookup_resp_handler] 0-gv0-replicate-0: path /kahadb/lock on subvolume gv0-client-1 => -1 (No such file or directory)<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:52:56.953281] I [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] 0-gv0-replicate-0: background entry self-heal completed on /kahadb<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:53:37.196481] I [client3_1-fops.c:1025:client3_1_removexattr_cbk] 0-gv0-client-0: remote operation failed: No data available<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:53:37.196735] I [client3_1-fops.c:1025:client3_1_removexattr_cbk] 0-gv0-client-1: remote operation failed: No data available<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:53:37.196917] W [fuse-bridge.c:850:fuse_err_cbk] 0-glusterfs-fuse: 54284: REMOVEXATTR() /kahadb/db-4.log => -1 (No data available)<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:53:37.200487] I [client3_1-fops.c:1025:client3_1_removexattr_cbk] 0-gv0-client-0: remote operation failed: No data available<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:53:37.200746] I [client3_1-fops.c:1025:client3_1_removexattr_cbk] 0-gv0-client-1: remote operation failed: No data available<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:53:37.200936] W [fuse-bridge.c:850:fuse_err_cbk] 0-glusterfs-fuse: 54291: REMOVEXATTR() /kahadb/db-5.log => -1 (No data available)<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:53:48.674314] W [client3_1-fops.c:3655:client3_1_flush] 0-gv0-client-1: (-2161116166): failed to get fd ctx. EBADFD<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:53:48.674350] W [client3_1-fops.c:3692:client3_1_flush] 0-gv0-client-1: failed to send the fop: File descriptor in bad state<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:53:48.676375] W [client3_1-fops.c:3655:client3_1_flush] 0-gv0-client-1: (-1443019630): failed to get fd ctx. EBADFD<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:53:48.676396] W [client3_1-fops.c:3692:client3_1_flush] 0-gv0-client-1: failed to send the fop: File descriptor in bad state<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:53:48.762598] W [client3_1-fops.c:4699:client3_1_lk] 0-gv0-client-1: (-1909467425): failed to get fd ctx. EBADFD<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:53:48.762662] W [client3_1-fops.c:4751:client3_1_lk] 0-gv0-client-1: failed to send the fop: File descriptor in bad state<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:53:48.764122] W [client3_1-fops.c:3655:client3_1_flush] 0-gv0-client-1: (-1909467425): failed to get fd ctx. EBADFD<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:53:48.764142] W [client3_1-fops.c:3692:client3_1_flush] 0-gv0-client-1: failed to send the fop: File descriptor in bad state<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:54:40.467613] I [afr-self-heal-common.c:705:afr_mark_sources] 0-gv0-replicate-0: split-brain possible, no source detected<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:54:40.467839] I [afr-self-heal-common.c:705:afr_mark_sources] 0-gv0-replicate-0: split-brain possible, no source detected<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:54:40.467861] W [fuse-bridge.c:184:fuse_entry_cbk] 0-glusterfs-fuse: 61809: LOOKUP() /kahadb/db.data => -1 (Input/output error)<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:54:40.468151] I [afr-self-heal-common.c:705:afr_mark_sources] 0-gv0-replicate-0: split-brain possible, no source detected<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:54:40.468171] W [fuse-bridge.c:184:fuse_entry_cbk] 0-glusterfs-fuse: 61811: LOOKUP() /kahadb/db.data => -1 (Input/output error)<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:54:40.473764] I [afr-self-heal-common.c:705:afr_mark_sources] 0-gv0-replicate-0: split-brain possible, no source detected<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:54:40.473797] W [fuse-bridge.c:184:fuse_entry_cbk] 0-glusterfs-fuse: 61812: LOOKUP() /kahadb/db.data => -1 (Input/output error)<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:54:40.475351] I [afr-self-heal-common.c:705:afr_mark_sources] 0-gv0-replicate-0: split-brain possible, no source detected<o:p></o:p></p>
<p class="MsoNormal">[2015-08-05 08:54:40.475373] W [fuse-bridge.c:184:fuse_entry_cbk] 0-glusterfs-fuse: 61819: LOOKUP() /kahadb/db.data => -1 (Input/output error)<o:p></o:p></p>
<p class="MsoNormal">-----<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<br> ­­ </body>
</html>