<div dir="ltr">Hi Sung Sik Park,<div><br></div><div>Please find my comments inline.</div><div><br></div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Dec 3, 2015 at 7:19 AM, 박성식 <span dir="ltr"><<a href="mailto:mulgo79@gmail.com" target="_blank">mulgo79@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>Hi all</div><div><br></div><div>Gluster-3.7.6 in 'Peer Rejected' problem occurs in the following test case.(it doesn't occur if don't enable the volume quota)</div><div><br></div><div>The problem is caused from the commit point '3d3176958b7da48dbacb1a5a0fedf26322a38297'.</div><div><br></div><div>Can you help with the following problems?</div><div><br></div><div># Test enviroment</div><div><br></div><div>1. Ubuntu 14.04 Server</div><div> (1) c00002-dn00001: 192.16.63.21</div><div> (2) c00002-dn00002: 192.16.63.22</div><div> (3) c00002-dn00003: 192.16.63.23</div><div>2. Glusterfs-3.7.6</div><div><br></div><div># Test procedure</div><div><br></div><div>1. Gluster installation</div><div><br></div><div>$ git clone git://<a href="http://git.gluster.com/glusterfs.git" target="_blank">git.gluster.com/glusterfs.git</a> <a href="http://git.gluster.com" target="_blank">git.gluster.com</a></div><div>$ cd <a href="http://git.gluster.com/" target="_blank">git.gluster.com/</a></div><div>$ git checkout v3.7.6</div><div>$ ./autogen.sh ; ./configure ; make ; make install ; ldconfig</div><div><br></div><div>2. Glusterd startup</div><div><br></div><div>$ /etc/init.d/glusterd start</div><div><br></div><div>3. Gluster peer: 2 Server peer</div><div><br></div><div>[c00002-dn00001] $ gluster peer probe 192.16.63.22</div><div>[c00002-dn00001] $ gluster volume create tvol0001 c00002-dn00001:/brick/tvol0001 c00002-dn00002:/brick/tvol0001 force</div><div>[c00002-dn00001] $ gluster volume start tvol0001</div><div>[c00002-dn00001] $ gluster volume quota tvol0001 enable</div><div><br></div><div>4. Another server peer</div><div><br></div><div>[c00002-dn00001] $ gluster peer probe 192.16.63.23</div><div>peer probe: success.</div><div><br></div><div>[c00002-dn00001] $ gluster peer status</div><div>Uuid: dd749b9b-d4e1-4b94-8f36-c03064bc5a57</div><div>State: Peer Rejected (Connected)</div><div><br></div><div># c00002-dn00001 log</div><div>[2015-12-03 00:37:52.464600] I [MSGID: 106487] [glusterd-handler.c:1422:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</div><div>[2015-12-03 00:38:51.035018] I [MSGID: 106487] [glusterd-handler.c:1301:__glusterd_handle_cli_deprobe] 0-glusterd: Received CLI deprobe req</div><div>[2015-12-03 00:38:51.036338] I [MSGID: 106493] [glusterd-rpc-ops.c:599:__glusterd_friend_remove_cbk] 0-glusterd: Received ACC from uuid: dd749b9b-d4e1-4b94-8f36-c03064bc5a57, host: c00002-dn00001, port: 0</div><div>[2015-12-03 00:38:51.041859] I [MSGID: 106493] [glusterd-rpc-ops.c:695:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: cb22d272-d563-4bda-802b-c93798130108</div><div>[2015-12-03 00:38:56.441189] I [MSGID: 106487] [glusterd-handler.c:1189:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req 192.16.63.23 24007</div><div>[2015-12-03 00:38:56.441828] I [MSGID: 106129] [glusterd-handler.c:3611:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: 192.16.63.23 (24007)</div><div>[2015-12-03 00:38:56.445471] I [rpc-clnt.c:984:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600</div><div>[2015-12-03 00:38:56.445954] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 23, Invalid argument</div><div>[2015-12-03 00:38:56.445977] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument</div><div>[2015-12-03 00:38:56.446232] I [MSGID: 106498] [glusterd-handler.c:3539:glusterd_friend_add] 0-management: connect returned 0</div><div>[2015-12-03 00:38:56.457933] I [MSGID: 106511] [glusterd-rpc-ops.c:257:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: dd749b9b-d4e1-4b94-8f36-c03064bc5a57, host: 192.16.63.23</div><div>[2015-12-03 00:38:56.457992] I [MSGID: 106511] [glusterd-rpc-ops.c:417:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req</div><div>[2015-12-03 00:38:56.519048] I [MSGID: 106493] [glusterd-rpc-ops.c:481:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: dd749b9b-d4e1-4b94-8f36-c03064bc5a57, host: 192.16.63.23, port: 0</div><div>[2015-12-03 00:38:56.524556] I [MSGID: 106163] [glusterd-handshake.c:1193:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30707</div><div>[2015-12-03 00:38:56.527701] I [MSGID: 106490] [glusterd-handler.c:2895:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: dd749b9b-d4e1-4b94-8f36-c03064bc5a57</div><div>[2015-12-03 00:38:56.528598] I [MSGID: 106493] [glusterd-handler.c:2958:__glusterd_handle_probe_query] 0-glusterd: Responded to c00002-dn00003, op_ret: 0, op_errno: 0, ret: 0</div><div>[2015-12-03 00:38:56.529154] I [MSGID: 106490] [glusterd-handler.c:2550:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: dd749b9b-d4e1-4b94-8f36-c03064bc5a57</div><div>[2015-12-03 00:38:56.529743] E [MSGID: 106010] [glusterd-utils.c:2727:glusterd_compare_friend_volume] 0-management: Version of Cksums tvol0001 differ. local cksum = 870323923, remote cksum = 870586066 on peer 192.16.63.23</div><div>[2015-12-03 00:38:56.529905] I [MSGID: 106493] [glusterd-handler.c:3791:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.16.63.23 (0), ret: 0</div><div>[2015-12-03 00:38:59.243087] I [MSGID: 106487] [glusterd-handler.c:1422:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</div><div><br></div><div># c00002-dn00003 log</div><div>[2015-12-03 00:38:45.556047] I [MSGID: 106491] [glusterd-handler.c:2600:__glusterd_handle_incoming_unfriend_req] 0-glusterd: Received unfriend from uuid: 090e2d18-7a6a-4d11-89cd-7c583180cba3</div><div>[2015-12-03 00:38:45.556835] I [MSGID: 106493] [glusterd-handler.c:3758:glusterd_xfer_friend_remove_resp] 0-glusterd: Responded to c00002-dn00001 (0), ret: 0</div><div>[2015-12-03 00:38:45.556863] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: already stopped</div><div>[2015-12-03 00:38:45.556873] I [MSGID: 106510] [glusterd-sm.c:637:glusterd_peer_detach_cleanup] 0-management: Deleting stale volume tvol0001</div><div>[2015-12-03 00:38:45.571032] W [socket.c:588:__socket_rwv] 0-nfs: readv on /var/run/gluster/ce041cf16b3a2a739b2381df8b2eb141.socket failed (No data available)</div><div>[2015-12-03 00:38:46.557936] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped</div><div>[2015-12-03 00:38:46.563409] W [socket.c:588:__socket_rwv] 0-quotad: readv on /var/run/gluster/a3c5c6ab5ef0542c80d4dcee7b6f6b00.socket failed (No data available)</div><div>[2015-12-03 00:38:47.559114] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped</div><div>[2015-12-03 00:38:47.559217] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped</div><div>[2015-12-03 00:38:47.565412] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-management: size=588 max=3 total=6</div><div>[2015-12-03 00:38:47.565589] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-management: size=124 max=3 total=6</div><div>[2015-12-03 00:38:50.969101] I [MSGID: 106163] [glusterd-handshake.c:1193:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30707</div><div>[2015-12-03 00:38:50.972274] I [MSGID: 106490] [glusterd-handler.c:2895:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: 090e2d18-7a6a-4d11-89cd-7c583180cba3</div><div>[2015-12-03 00:38:50.973833] I [MSGID: 106129] [glusterd-handler.c:2930:__glusterd_handle_probe_query] 0-glusterd: Unable to find peerinfo for host: c00002-dn00001 (24007)</div><div>[2015-12-03 00:38:50.976661] I [rpc-clnt.c:984:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600</div><div>[2015-12-03 00:38:50.977655] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 12, Invalid argument</div><div>[2015-12-03 00:38:50.977679] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument</div><div>[2015-12-03 00:38:50.978170] I [MSGID: 106498] [glusterd-handler.c:3539:glusterd_friend_add] 0-management: connect returned 0</div><div>[2015-12-03 00:38:50.978682] I [MSGID: 106493] [glusterd-handler.c:2958:__glusterd_handle_probe_query] 0-glusterd: Responded to c00002-dn00001, op_ret: 0, op_errno: 0, ret: 0</div><div>[2015-12-03 00:38:50.979553] I [MSGID: 106490] [glusterd-handler.c:2550:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 090e2d18-7a6a-4d11-89cd-7c583180cba3</div><div>[2015-12-03 00:38:50.987658] E [MSGID: 106187] [glusterd-utils.c:4541:glusterd_brick_start] 0-management: Could not find peer on which brick c00002-dn00002:/brick/tvol0001 resides</div><div>[2015-12-03 00:38:50.987678] E [MSGID: 106005] [glusterd-op-sm.c:2184:glusterd_start_bricks] 0-management: Failed to start c00002-dn00002:/brick/tvol0001 for tvol0001</div><div>[2015-12-03 00:38:51.017178] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped</div><div>[2015-12-03 00:38:51.026769] W [socket.c:3009:socket_connect] 0-nfs: Ignore failed connection attempt on /var/run/gluster/ce041cf16b3a2a739b2381df8b2eb141.socket, (No such file or directory)</div><div>[2015-12-03 00:38:51.026992] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped</div><div>[2015-12-03 00:38:51.032213] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped</div><div>[2015-12-03 00:38:51.036689] W [socket.c:3009:socket_connect] 0-quotad: Ignore failed connection attempt on /var/run/gluster/a3c5c6ab5ef0542c80d4dcee7b6f6b00.socket, (No such file or directory)</div><div>[2015-12-03 00:38:51.036778] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped</div><div>[2015-12-03 00:38:51.036828] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped</div><div>[2015-12-03 00:38:51.039228] I [MSGID: 106493] [glusterd-handler.c:3791:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to c00002-dn00001 (0), ret: 0</div><div>[2015-12-03 00:38:51.044296] W [socket.c:588:__socket_rwv] 0-quotad: readv on /var/run/gluster/a3c5c6ab5ef0542c80d4dcee7b6f6b00.socket failed (Invalid argument)</div><div>[2015-12-03 00:38:51.044652] I [MSGID: 106006] [glusterd-svc-mgmt.c:323:glusterd_svc_common_rpc_notify] 0-management: quotad has disconnected from glusterd.</div><div>[2015-12-03 00:38:51.044868] W [socket.c:588:__socket_rwv] 0-nfs: readv on /var/run/gluster/ce041cf16b3a2a739b2381df8b2eb141.socket failed (Invalid argument)</div><div>[2015-12-03 00:38:51.044902] I [MSGID: 106006] [glusterd-svc-mgmt.c:323:glusterd_svc_common_rpc_notify] 0-management: nfs has disconnected from glusterd.</div><div>[2015-12-03 00:38:51.049332] I [MSGID: 106511] [glusterd-rpc-ops.c:257:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 090e2d18-7a6a-4d11-89cd-7c583180cba3, host: c00002-dn00001</div><div>[2015-12-03 00:38:51.049365] I [MSGID: 106511] [glusterd-rpc-ops.c:417:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req</div><div>[2015-12-03 00:38:51.050941] I [MSGID: 106493] [glusterd-rpc-ops.c:481:__glusterd_friend_add_cbk] 0-glusterd: Received RJT from uuid: 090e2d18-7a6a-4d11-89cd-7c583180cba3, host: c00002-dn00001, port: 0</div><div><br></div><div># c00002-dn00001 /var/lib/glusterd/vols/tvol0001/info</div><div>type=0</div><div>count=2</div><div>status=1</div><div>sub_count=0</div><div>stripe_count=1</div><div>replica_count=1</div><div>disperse_count=0</div><div>redundancy_count=0</div><div>version=3</div><div>transport-type=0</div><div>volume-id=cb9864b7-0bb4-439d-b60b-595d2e1e0cc3</div><div>username=eb29fba6-a51a-4e65-abbd-8ef03d7bb22d</div><div>password=c987bfd7-8da3-4379-abe1-7dcbcda7d49d</div><div>op-version=3</div><div>client-op-version=3</div><div>quota-version=1</div><div>parent_volname=N/A</div><div>restored_from_snap=00000000-0000-0000-0000-000000000000</div><div>snap-max-hard-limit=256</div><div>features.quota-deem-statfs=on</div><div>features.inode-quota=on</div><div>features.quota=on</div><div>performance.readdir-ahead=on</div><div>brick-0=c00002-dn00001:-brick-tvol0001</div><div>brick-1=c00002-dn00002:-brick-tvol0001</div><div><br></div><div># c00002-dn00003 /var/lib/glusterd/vols/tvol0001/info</div><div>type=0</div><div>count=2</div><div>status=1</div><div>sub_count=0</div><div>stripe_count=1</div><div>replica_count=1</div><div>disperse_count=0</div><div>redundancy_count=0</div><div>version=3</div><div>transport-type=0</div><div>volume-id=cb9864b7-0bb4-439d-b60b-595d2e1e0cc3</div><div>username=eb29fba6-a51a-4e65-abbd-8ef03d7bb22d</div><div>password=c987bfd7-8da3-4379-abe1-7dcbcda7d49d</div><div>op-version=3</div><div>client-op-version=3</div><div>quota-version=0</div></div></blockquote><div> </div><div>Somehow, parameter 'quota-version' is not updated here. I will try to re-create this problem in-house and fix this issue ASAP.</div><div>For a workaround, you can try below steps</div><div>1) stop glusterd on 'c00002-dn00003'</div><div>2) copy volume 'info' and 'cksum' file from 'c00002-dn00001' to ''c00002-dn00003'</div><div>3) start glusterd</div><div>4) try peer probe</div><div> </div><div>Thanks,</div><div>Vijay</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>parent_volname=N/A</div><div>restored_from_snap=00000000-0000-0000-0000-000000000000</div><div>snap-max-hard-limit=256</div><div>features.quota-deem-statfs=on</div><div>features.inode-quota=on</div><div>features.quota=on</div><div>performance.readdir-ahead=on</div><div>brick-0=c00002-dn00001:-brick-tvol0001</div><div>brick-1=c00002-dn00002:-brick-tvol0001</div><div><br></div><div><div dir="ltr"><p style="margin:0cm 0cm 0.0001pt;text-align:justify;font-family:맑은고딕"><span lang="EN-US" style="font-family:돋움">------------------------------------------------------------------------------------------<u></u><u></u></span></p><p style="margin:0cm 0cm 0.0001pt;text-align:justify;font-family:맑은고딕"><span lang="EN-US" style="font-family:돋움">Sung Sik, Park [</span><span style="font-family:돋움">박성식<span lang="EN-US">, </span>朴宬植<span lang="EN-US">]</span></span></p><p style="margin:0cm 0cm 0.0001pt;text-align:justify;font-family:맑은고딕"><span lang="EN-US" style="font-family:돋움">Email: <span style="color:blue"><a href="mailto:sypark@wisetodd.com" style="color:rgb(17,85,204)" target="_blank">mulgo79@gmail.com</a></span></span></p><p style="margin:0cm 0cm 0.0001pt;text-align:justify;font-family:맑은고딕"><span lang="EN-US" style="font-family:돋움">------------------------------------------------------------------------------------------</span></p></div></div>
</div>
<br>_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br></blockquote></div><br></div></div>