[2015-11-29 22:40:26.791167] C [rpc-clnt-ping.c:109:rpc_clnt_ping_timer_expired] 0-ph-fs-0-client-0: server 10.11.100.1:49152 has not responded in the last 2 seconds, disconnecting. [2015-11-29 22:40:26.801987] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7fcd149c3580] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7fcd14798787] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fcd1479889e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x91)[0x7fcd14798951] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7fcd14798f1f] ))))) 0-ph-fs-0-client-0: forced unwinding frame type(GlusterFS 3.3) op(INODELK(29)) called at 2015-11-29 22:40:21.995126 (xid=0x2bfd33) [2015-11-29 22:40:26.802041] E [client-rpc-fops.c:1523:client3_3_inodelk_cbk] 0-ph-fs-0-client-0: remote operation failed: Koncový komunikační bod není připojen [2015-11-29 22:40:26.802267] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7fcd149c3580] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7fcd14798787] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fcd1479889e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x91)[0x7fcd14798951] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7fcd14798f1f] ))))) 0-ph-fs-0-client-0: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2015-11-29 22:40:24.791015 (xid=0x2bfd34) [2015-11-29 22:40:26.810744] I [socket.c:3303:socket_submit_request] 0-ph-fs-0-client-0: not connected (priv->connected = 0) [2015-11-29 22:40:26.823947] W [rpc-clnt.c:1562:rpc_clnt_submit] 0-ph-fs-0-client-0: failed to submit rpc-request (XID: 0x2bfd35 Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (ph-fs-0-client-0) [2015-11-29 22:40:26.823977] E [client-rpc-fops.c:1523:client3_3_inodelk_cbk] 0-ph-fs-0-client-0: remote operation failed: Koncový komunikační bod není připojen [2015-11-29 22:40:26.823978] W [rpc-clnt-ping.c:154:rpc_clnt_ping_cbk] 0-ph-fs-0-client-0: socket disconnected [2015-11-29 22:40:26.824044] I [client.c:2215:client_rpc_notify] 0-ph-fs-0-client-0: disconnected from ph-fs-0-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2015-11-29 22:40:26.824364] W [client-rpc-fops.c:5611:client3_3_rchecksum] 0-ph-fs-0-client-0: (a5d467b8-3bf6-424e-8123-c81ff8f0a3fc) remote_fd is -1. EBADFD [2015-11-29 22:40:26.824427] I [rpc-clnt.c:1761:rpc_clnt_reconfig] 0-ph-fs-0-client-0: changing port to 49152 (from 0) [2015-11-29 22:40:28.184866] I [client-handshake.c:1413:select_server_supported_programs] 0-ph-fs-0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-11-29 22:40:28.204008] W [client-rpc-fops.c:2845:client3_3_readv_cbk] 0-ph-fs-0-client-0: remote operation failed: Koncový komunikační bod není připojen [2015-11-29 22:40:28.204134] I [client-handshake.c:1200:client_setvolume_cbk] 0-ph-fs-0-client-0: Connected to ph-fs-0-client-0, attached to remote volume '/gfs/s3-sata-10k/fs'. [2015-11-29 22:40:28.204161] I [client-handshake.c:1210:client_setvolume_cbk] 0-ph-fs-0-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-11-29 22:40:28.204189] I [client-handshake.c:1052:client_post_handshake] 0-ph-fs-0-client-0: 2 fds open - Delaying child_up until they are re-opened [2015-11-29 22:40:28.204319] I [client-handshake.c:811:client3_3_reopendir_cbk] 0-ph-fs-0-client-0: reopendir on succeeded (fd = 0) [2015-11-29 22:40:28.204469] I [client-handshake.c:674:client_child_up_reopen_done] 0-ph-fs-0-client-0: last fd open'd/lock-self-heal'd - notifying CHILD-UP [2015-11-29 22:40:28.204564] I [client-handshake.c:188:client_set_lk_version_cbk] 0-ph-fs-0-client-0: Server lk version = 1 [2015-11-29 22:40:31.754136] E [client-rpc-fops.c:1523:client3_3_inodelk_cbk] 0-ph-fs-0-client-0: remote operation failed: Nepřípustný argument [2015-11-29 22:40:31.754659] E [client-rpc-fops.c:1523:client3_3_inodelk_cbk] 0-ph-fs-0-client-0: remote operation failed: Nepřípustný argument [2015-11-29 22:47:11.584673] I [afr-self-heal-common.c:476:afr_log_selfheal] 0-ph-fs-0-replicate-0: Completed data selfheal on d0ff0185-f41d-4858-b80b-2624f42899df. source=1 sinks=0 [2015-11-29 22:50:08.599778] I [afr-self-heal-common.c:476:afr_log_selfheal] 0-ph-fs-0-replicate-0: Completed data selfheal on a5d467b8-3bf6-424e-8123-c81ff8f0a3fc. source=0 sinks=1 [2015-11-29 22:58:22.002161] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-ph-fs-0-client-1: remote operation failed: Adresář nebo soubor neexistuje. Path: (ca290e22-71a9-46ca-9d39-e0a013c63ce0) [2015-11-29 22:58:22.004212] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-ph-fs-0-client-1: remote operation failed: Adresář nebo soubor neexistuje. Path: (ca290e22-71a9-46ca-9d39-e0a013c63ce0) [2015-11-29 22:58:22.006015] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-ph-fs-0-client-1: remote operation failed: Adresář nebo soubor neexistuje. Path: /f1607f25aa52f4fb6f98f20ef0f3f9d7 (84269273-6a1e-4c2e-9ded-0ada5a38007d) [2015-11-29 22:58:22.126221] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-ph-fs-0-client-1: remote operation failed: Adresář nebo soubor neexistuje. Path: /f1607f25aa52f4fb6f98f20ef0f3f9d7 (84269273-6a1e-4c2e-9ded-0ada5a38007d) [2015-11-29 22:58:22.127435] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-ph-fs-0-client-1: remote operation failed: Adresář nebo soubor neexistuje. Path: (e70135c8-ede5-4987-942c-2796a649dc88) [2015-11-29 22:58:22.129519] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-ph-fs-0-client-1: remote operation failed: Adresář nebo soubor neexistuje. Path: (e70135c8-ede5-4987-942c-2796a649dc88) [2015-11-29 22:59:23.003061] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-ph-fs-0-client-1: remote operation failed: Adresář nebo soubor neexistuje. Path: (ca290e22-71a9-46ca-9d39-e0a013c63ce0) [2015-11-29 22:59:23.005249] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-ph-fs-0-client-1: remote operation failed: Adresář nebo soubor neexistuje. Path: (ca290e22-71a9-46ca-9d39-e0a013c63ce0) [2015-11-29 22:59:23.007082] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-ph-fs-0-client-1: remote operation failed: Adresář nebo soubor neexistuje. Path: /f1607f25aa52f4fb6f98f20ef0f3f9d7 (84269273-6a1e-4c2e-9ded-0ada5a38007d) [2015-11-29 22:59:23.008779] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-ph-fs-0-client-1: remote operation failed: Adresář nebo soubor neexistuje. Path: /f1607f25aa52f4fb6f98f20ef0f3f9d7 (84269273-6a1e-4c2e-9ded-0ada5a38007d) [2015-11-29 22:59:23.010273] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-ph-fs-0-client-1: remote operation failed: Adresář nebo soubor neexistuje. Path: (e70135c8-ede5-4987-942c-2796a649dc88) [2015-11-29 22:59:23.011912] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-ph-fs-0-client-1: remote operation failed: Adresář nebo soubor neexistuje. Path: (e70135c8-ede5-4987-942c-2796a649dc88) [2015-11-29 23:00:24.009741] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-ph-fs-0-client-1: remote operation failed: Adresář nebo soubor neexistuje. Path: (ca290e22-71a9-46ca-9d39-e0a013c63ce0) [2015-11-29 23:00:24.011080] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-ph-fs-0-client-1: remote operation failed: Adresář nebo soubor neexistuje. Path: (ca290e22-71a9-46ca-9d39-e0a013c63ce0) [2015-11-29 23:00:24.013069] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-ph-fs-0-client-1: remote operation failed: Adresář nebo soubor neexistuje. Path: /f1607f25aa52f4fb6f98f20ef0f3f9d7 (84269273-6a1e-4c2e-9ded-0ada5a38007d) [2015-11-29 23:00:24.014876] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-ph-fs-0-client-1: remote operation failed: Adresář nebo soubor neexistuje. Path: /f1607f25aa52f4fb6f98f20ef0f3f9d7 (84269273-6a1e-4c2e-9ded-0ada5a38007d) [2015-11-29 23:00:24.016334] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-ph-fs-0-client-1: remote operation failed: Adresář nebo soubor neexistuje. Path: (e70135c8-ede5-4987-942c-2796a649dc88) [2015-11-29 23:00:24.017862] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-ph-fs-0-client-1: remote operation failed: Adresář nebo soubor neexistuje. Path: (e70135c8-ede5-4987-942c-2796a649dc88) [2015-11-29 23:03:38.556419] I [afr-self-heal-common.c:476:afr_log_selfheal] 0-ph-fs-0-replicate-0: Completed data selfheal on e70135c8-ede5-4987-942c-2796a649dc88. source=0 sinks=1