<div dir="ltr">Hi guys,<div><br></div><div>My glusterfs setup is across 3 servers (gfs1 ,gfs2, hpc1) with 2 bricks (on gfs1 and gfs2).</div><div><br></div><div>The reason i have 3 servers is that i wanted the gluster to stay online when one server goes down as i frequently lose my gfs2 server. I have no idea what causes this and im at a dead end.</div><div><br></div><div>I reinstalled glusterfs-server on my gfs2 server and i found that my performance dropped to the point my R/W speed is around 1.5 kB/S on my 10G LAN . Pressing tab when i ls into a folder takes around 20-30 seconds to autocomplete, sometimes longer. Previously i had blazing fast speeds. When i take gluster2 off the network, the speed is back up.</div><div><br></div><div>Both my servers are now on the fritz and the bricks keep going offline and i have no idea what to do. Can anyone help shed some light on this? I have no idea what i need to provide so ill provide the logs and config. </div><div><br></div><div><br></div><div>::EDIT::</div><div>As i was typing, gluster went down again and when i brought it back up, the speed went back to normal. This didnt happen the last 3 or four times i restarted the servers.. I would still like to know what is going on...<br></div><div>::::::::<br></div><div><br></div><div><br></div><div><br></div><div><div> gluster volume info</div><div><br></div><div> Volume Name: gfsvolume</div><div>Type: Replicate</div><div>Volume ID: a29bd2fb-b1ef-4481-be10-c2f4faf4059b</div><div>Status: Started</div><div>Number of Bricks: 1 x 2 = 2</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: gfs1:/export/sda/brick</div><div>Brick2: gfs2:/export/sda/brick</div><div>Options Reconfigured:</div><div>performance.quick-read: off</div><div>network.ping-timeout: 30</div><div>network.frame-timeout: 90</div><div>performance.cache-max-file-size: 2MB</div><div>cluster.server-quorum-type: none</div><div>nfs.addr-namelookup: off</div><div>nfs.trusted-write: off</div><div>performance.write-behind-window-size: 4MB</div><div>cluster.data-self-heal-algorithm: diff</div><div>performance.cache-refresh-timeout: 60</div><div>performance.cache-size: 6442450944</div><div>cluster.quorum-type: fixed</div><div>auth.allow: 172.*</div><div>cluster.quorum-count: 1</div><div>cluster.server-quorum-ratio: 50%</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">/var/log//glusterfs/</blockquote><blockquote> <br><blockquote>[2015-02-26 00:13:36.207408] I [glusterd-volume-ops.c:481:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume gfsvolume<br><blockquote>[2015-02-26 00:13:36.211810] E [glusterd-syncop.c:101:gd_collate_errors] 0-: Commit failed on gfs2. Please check log file for details.<br>[2015-02-26 00:14:17.939391] W [glusterfsd.c:1095:cleanup_and_exit] (--&gt;/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f13408f8e6d] (--&gt;/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f1340fa6b50] (--&gt;/us<br>r/sbin/glusterd(glusterfs_sigwaiter+0xd5) [0x7f1342832d55]))) 0-: received signum (15), shutting down<br>[2015-02-26 00:14:20.127606] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.0 (/usr/sbin/glusterd -p /var/run/glusterd.pid)<br>[2015-02-26 00:14:20.136366] I [glusterd.c:1148:init] 0-management: Using /var/lib/glusterd as working directory<br>[2015-02-26 00:14:20.137768] I [socket.c:3561:socket_init] 0-socket.management: SSL support is NOT enabled<br>[2015-02-26 00:14:20.137797] I [socket.c:3576:socket_init] 0-socket.management: using system polling thread<br>[2015-02-26 00:14:20.139452] W [rdma.c:4194:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device)<br>[2015-02-26 00:14:20.139483] E [rdma.c:4482:init] 0-rdma.management: Failed to initialize IB Device<br>[2015-02-26 00:14:20.139497] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport: &#39;rdma&#39; initialization failed<br>[2015-02-26 00:14:20.139568] W [rpcsvc.c:1483:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed<br>[2015-02-26 00:14:20.139719] I [socket.c:3561:socket_init] 0-socket.management: SSL support is NOT enabled<br>[2015-02-26 00:14:20.139741] I [socket.c:3576:socket_init] 0-socket.management: using system polling thread<br>[2015-02-26 00:14:23.351872] I [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2<br>[2015-02-26 00:14:23.355502] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0<br>[2015-02-26 00:14:23.355552] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1<br>[2015-02-26 00:14:23.584714] I [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect returned 0<br>[2015-02-26 00:14:23.588106] I [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect returned 0<br>[2015-02-26 00:14:23.588223] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br>[2015-02-26 00:14:23.588328] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled<br>[2015-02-26 00:14:23.588347] I [socket.c:3576:socket_init] 0-management: using system polling thread<br>[2015-02-26 00:14:23.589290] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br>[2015-02-26 00:14:23.589378] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled<br>[2015-02-26 00:14:23.589397] I [socket.c:3576:socket_init] 0-management: using system polling thread<br>[2015-02-26 00:14:23.589885] I [glusterd.c:138:glusterd_uuid_init] 0-management: retrieved UUID: 49acc9c2-4809-4da5-a6f0-6a3d48314070<br>Final graph:<br>+------------------------------------------------------------------------------+<br>  1: volume management<br>  2:     type mgmt/glusterd<br>  3:     option rpc-auth.auth-glusterfs on<br>  4:     option rpc-auth.auth-unix on<br>  5:     option rpc-auth.auth-null on<br>  6:     option transport.socket.listen-backlog 128<br>  7:     option transport.socket.read-fail-log off<br>  8:     option transport.socket.keepalive-interval 2<br>  9:     option transport.socket.keepalive-time 10<br> 10:     option transport-type rdma<br> 11:     option working-directory /var/lib/glusterd<br> 12: end-volume<br> 13: <br>+------------------------------------------------------------------------------+<br>[2015-02-26 00:14:23.780644] I [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2, port: 0<br>[2015-02-26 00:14:23.798102] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br>[2015-02-26 00:14:23.798242] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled<br>[2015-02-26 00:14:23.798276] I [socket.c:3576:socket_init] 0-management: using system polling thread<br>[2015-02-26 00:14:24.800202] E [glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/ec396f4c7328565d7dafb2c74d51d072.socket error: Permission denied<br>[2015-02-26 00:14:24.800858] I [glusterd-utils.c:4155:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV3 successfully<br>[2015-02-26 00:14:24.801261] I [glusterd-utils.c:4160:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV1 successfully<br>[2015-02-26 00:14:24.801649] I [glusterd-utils.c:4165:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 successfully<br>[2015-02-26 00:14:24.802028] I [glusterd-utils.c:4170:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v4 successfully<br>[2015-02-26 00:14:24.802410] I [glusterd-utils.c:4175:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v1 successfully<br>[2015-02-26 00:14:24.802793] I [glusterd-utils.c:4180:glusterd_nfs_pmap_deregister] 0-: De-registered ACL v3 successfully<br>[2015-02-26 00:14:24.806276] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br>[2015-02-26 00:14:24.806387] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled<br>[2015-02-26 00:14:24.806406] I [socket.c:3576:socket_init] 0-management: using system polling thread<br>[2015-02-26 00:14:25.807341] E [glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/35e2381254b245fb5c6f66b8a82d585e.socket error: No such file or directory<br>[2015-02-26 00:14:25.810987] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br>[2015-02-26 00:14:25.811100] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled[2015-02-26 00:14:25.811119] I [socket.c:3576:socket_init] 0-management: using system polling thread<br>[2015-02-26 00:14:25.811329] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: adbb7505-3342-4c6d-be3d-75938633612c<br>[2015-02-26 00:14:25.811370] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1<br>[2015-02-26 00:14:25.811391] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend<br>[2015-02-26 00:14:25.811407] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, hostname:hpc1<br>[2015-02-26 00:14:25.811510] I [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 2<br>[2015-02-26 00:14:25.982101] I [socket.c:2238:socket_event_handler] 0-transport: disconnecting now<br>[2015-02-26 00:14:25.982172] I [socket.c:2238:socket_event_handler] 0-transport: disconnecting now<br>[2015-02-26 00:14:25.982265] I [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1, port: 0<br>[2015-02-26 00:14:26.003332] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /export/sda/brick on port 49153<br>[2015-02-26 00:14:26.159051] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae<br>[2015-02-26 00:14:26.159112] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1<br>[2015-02-26 00:14:26.159130] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend<br>[2015-02-26 00:14:26.178634] I [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: adbb7505-3342-4c6d-be3d-75938633612c<br>[2015-02-26 00:14:26.178794] I [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to gfs2 (0), ret: 0<br>[2015-02-26 00:14:26.194289] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1<br>[2015-02-26 00:14:26.194348] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2<br>[2015-02-26 00:14:26.208631] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: adbb7505-3342-4c6d-be3d-75938633612c<br>[2015-02-26 00:14:26.373031] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae<br>[2015-02-26 00:14:26.373101] I [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 2<br>[2015-02-26 00:14:26.385510] I [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae<br>[2015-02-26 00:14:26.385645] I [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to hpc1 (0), ret: 0<br>[2015-02-26 00:14:26.399963] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1<br>[2015-02-26 00:14:26.400010] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2<br>[2015-02-26 00:14:26.414448] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: adbb7505-3342-4c6d-be3d-75938633612c<br>[2015-02-26 00:14:26.668753] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae<br>[2015-02-26 00:14:32.167298] W [socket.c:522:__socket_rwv] 0-management: readv on <a href="http://172.20.20.22:24007">172.20.20.22:24007</a> failed (No data available)<br>[2015-02-26 00:14:33.590941] E [socket.c:2161:socket_connect_finish] 0-management: connection to <a href="http://172.20.20.22:24007">172.20.20.22:24007</a> failed (Connection refused)<br>[2015-02-26 00:14:39.060572] W [glusterfsd.c:1095:cleanup_and_exit] (--&gt;/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f727130ce6d] (--&gt;/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f72719bab50] (--&gt;/usr/sbin/glusterd(glusterfs_sigwaiter+0xd5) [0x7f7273246d55]))) 0-: received signum (15), shutting down<br>[2015-02-26 00:14:42.990079] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.0 (/usr/sbin/glusterd -p /var/run/glusterd.pid)<br>[2015-02-26 00:14:42.994671] I [glusterd.c:1148:init] 0-management: Using /var/lib/glusterd as working directory<br>[2015-02-26 00:14:42.996068] I [socket.c:3561:socket_init] 0-socket.management: SSL support is NOT enabled<br>[2015-02-26 00:14:42.996098] I [socket.c:3576:socket_init] 0-socket.management: using system polling thread<br>[2015-02-26 00:14:42.996892] W [rdma.c:4194:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device)<br>[2015-02-26 00:14:42.996919] E [rdma.c:4482:init] 0-rdma.management: Failed to initialize IB Device<br>[2015-02-26 00:14:42.996932] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport: &#39;rdma&#39; initialization failed<br> [2015-02-26 00:14:42.997004] W [rpcsvc.c:1483:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed<br>[2015-02-26 00:14:42.997138] I [socket.c:3561:socket_init] 0-socket.management: SSL support is NOT enabled<br>[2015-02-26 00:14:42.997156] I [socket.c:3576:socket_init] 0-socket.management: using system polling thread<br>[2015-02-26 00:14:46.319737] I [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2<br>[2015-02-26 00:14:46.323519] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0<br>[2015-02-26 00:14:46.323567] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1<br>[2015-02-26 00:14:46.561399] I [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect returned 0<br>[2015-02-26 00:14:46.564795] I [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect returned 0<br>[2015-02-26 00:14:46.564919] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br>[2015-02-26 00:14:46.565035] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled<br>[2015-02-26 00:14:46.565054] I [socket.c:3576:socket_init] 0-management: using system polling thread<br>[2015-02-26 00:14:46.566216] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br>[2015-02-26 00:14:46.566329] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled<br>[2015-02-26 00:14:46.566348] I [socket.c:3576:socket_init] 0-management: using system polling thread<br>[2015-02-26 00:14:46.566832] I [glusterd.c:138:glusterd_uuid_init] 0-management: retrieved UUID: 49acc9c2-4809-4da5-a6f0-6a3d48314070<br>Final graph:<br>+------------------------------------------------------------------------------+<br>  1: volume management<br>  2:     type mgmt/glusterd<br>  3:     option rpc-auth.auth-glusterfs on<br>  4:     option rpc-auth.auth-unix on<br>  5:     option rpc-auth.auth-null on<br>  6:     option transport.socket.listen-backlog 128<br>  7:     option transport.socket.read-fail-log off<br>  8:     option transport.socket.keepalive-interval 2<br>  9:     option transport.socket.keepalive-time 10<br> 10:     option transport-type rdma<br> 11:     option working-directory /var/lib/glusterd<br> 12: end-volume<br> 13: <br>+------------------------------------------------------------------------------+<br>[2015-02-26 00:14:46.570634] E [socket.c:2161:socket_connect_finish] 0-management: connection to <a href="http://172.20.20.22:24007">172.20.20.22:24007</a> failed (Connection refused)<br>[2015-02-26 00:14:46.571244] I [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 2<br>[2015-02-26 00:14:47.001039] I [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae<br>[2015-02-26 00:14:47.001183] I [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to hpc1 (0), ret: 0<br>[2015-02-26 00:14:47.015772] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1<br>[2015-02-26 00:14:47.015820] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2<br>[2015-02-26 00:14:47.030090] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br>[2015-02-26 00:14:47.030178] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled<br>[2015-02-26 00:14:47.030196] I [socket.c:3576:socket_init] 0-management: using system polling thread<br>[2015-02-26 00:14:48.031850] E [glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/ec396f4c7328565d7dafb2c74d51d072.socket error: Permission denied<br>[2015-02-26 00:14:48.032392] I [glusterd-utils.c:4155:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV3 successfully<br>[2015-02-26 00:14:48.032794] I [glusterd-utils.c:4160:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV1 successfully<br>[2015-02-26 00:14:48.033182] I [glusterd-utils.c:4165:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 successfully<br>[2015-02-26 00:14:48.033571] I [glusterd-utils.c:4170:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v4 successfully<br>[2015-02-26 00:14:48.033964] I [glusterd-utils.c:4175:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v1 successfully<br>[2015-02-26 00:14:48.034356] I [glusterd-utils.c:4180:glusterd_nfs_pmap_deregister] 0-: De-registered ACL v3 successfully<br>[2015-02-26 00:14:48.037872] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br>[2015-02-26 00:14:48.037982] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled<br>[2015-02-26 00:14:48.038001] I [socket.c:3576:socket_init] 0-management: using system polling thread<br>[2015-02-26 00:14:49.039002] E [glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/35e2381254b245fb5c6f66b8a82d585e.socket error: No such file or directory<br>[2015-02-26 00:14:49.042564] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br>[2015-02-26 00:14:49.042670] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled<br>[2015-02-26 00:14:49.042690] I [socket.c:3576:socket_init] 0-management: using system polling thread<br>[2015-02-26 00:14:49.042958] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae<br>[2015-02-26 00:14:49.042998] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1<br>[2015-02-26 00:14:49.043013] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend<br>[2015-02-26 00:14:49.043319] I [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1, port: 0<br>[2015-02-26 00:14:49.204554] I [socket.c:2238:socket_event_handler] 0-transport: disconnecting now<br>[2015-02-26 00:14:49.204657] I [socket.c:2238:socket_event_handler] 0-transport: disconnecting now<br>[2015-02-26 00:14:49.204717] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae<br>[2015-02-26 00:14:49.814028] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /export/sda/brick on port 49153<br>[2015-02-26 00:14:53.678347] I [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 2<br>[2015-02-26 00:14:53.848276] I [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: adbb7505-3342-4c6d-be3d-75938633612c<br>[2015-02-26 00:14:53.848407] I [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to gfs2 (0), ret: 0<br>[2015-02-26 00:14:53.862892] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1<br>[2015-02-26 00:14:53.862952] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2<br>[2015-02-26 00:14:53.877685] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae<br>[2015-02-26 00:14:55.217486] I [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2, port: 0<br>[2015-02-26 00:14:55.234370] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: adbb7505-3342-4c6d-be3d-75938633612c<br>[2015-02-26 00:14:55.234433] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1<br>[2015-02-26 00:14:55.234451] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend<br>[2015-02-26 00:14:55.234468] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, hostname:hpc1<br>[2015-02-26 00:15:14.637613] E [glusterd-volume-ops.c:1047:glusterd_op_stage_start_volume] 0-management: Volume gfsvolume already started<br>[2015-02-26 00:15:14.637661] E [glusterd-syncop.c:912:gd_stage_op_phase] 0-management: Staging of operation &#39;Volume Start&#39; failed on localhost : Volume gfsvolume already started<br>[2015-02-26 00:24:41.265078] W [socket.c:522:__socket_rwv] 0-management: readv on <a href="http://172.20.20.22:24007">172.20.20.22:24007</a> failed (No data available)<br>[2015-02-26 00:24:43.631739] E [socket.c:2161:socket_connect_finish] 0-management: connection to <a href="http://172.20.20.22:24007">172.20.20.22:24007</a> failed (Connection refused)<br>[2015-02-26 00:24:56.174790] W [socket.c:522:__socket_rwv] 0-management: readv on /var/run/91bfef953906a83fc14aa435a886ac4d.socket failed (No data available)<br>[2015-02-26 00:24:56.175299] I [glusterd-handler.c:3713:__glusterd_brick_rpc_notify] 0-management: Disconnected from gfs1:/export/sda/brick<br>[2015-02-26 00:25:24.076560] W [glusterd-op-sm.c:3404:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<br>[2015-02-26 00:25:24.078109] I [glusterd-handler.c:3530:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gfsvolume<br>[2015-02-26 00:25:31.041343] E [glusterd-volume-ops.c:1047:glusterd_op_stage_start_volume] 0-management: Volume gfsvolume already started<br>[2015-02-26 00:25:31.041404] E [glusterd-syncop.c:912:gd_stage_op_phase] 0-management: Staging of operation &#39;Volume Start&#39; failed on localhost : Volume gfsvolume already started<br>[2015-02-26 00:25:43.389260] W [glusterfsd.c:1095:cleanup_and_exit] (--&gt;/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f8f8cfa7e6d] (--&gt;/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f8f8d655b50] (--&gt;/usr/sbin/glusterd(glusterfs_sigwaiter+0xd5) [0x7f8f8eee1d55]))) 0-: received signum (15), shutting down<br>[2015-02-26 00:25:45.437325] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.0 (/usr/sbin/glusterd -p /var/run/glusterd.pid)<br>[2015-02-26 00:25:45.487307] I [glusterd.c:1148:init] 0-management: Using /var/lib/glusterd as working directory<br>[2015-02-26 00:25:45.488734] I [socket.c:3561:socket_init] 0-socket.management: SSL support is NOT enabled<br>[2015-02-26 00:25:45.488765] I [socket.c:3576:socket_init] 0-socket.management: using system polling thread<br>[2015-02-26 00:25:45.489557] W [rdma.c:4194:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device)<br>[2015-02-26 00:25:45.489584] E [rdma.c:4482:init] 0-rdma.management: Failed to initialize IB Device<br>[2015-02-26 00:25:45.489597] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport: &#39;rdma&#39; initialization failed<br>[2015-02-26 00:25:45.489668] W [rpcsvc.c:1483:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed<br>[2015-02-26 00:25:45.489799] I [socket.c:3561:socket_init] 0-socket.management: SSL support is NOT enabled<br>[2015-02-26 00:25:45.489818] I [socket.c:3576:socket_init] 0-socket.management: using system polling thread<br>[2015-02-26 00:25:48.706235] I [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2</blockquote></blockquote></blockquote><div><br></div><div><br></div></div><div><br></div><div>Please Do let me know if there is anything else i can provide to help.  Apologies if this is a simple fix but i have never found an answer after a week of searching. </div><div><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><div>Thank You Kindly,</div><div>Kaamesh</div></div></div></div>
</div></div>