<div dir="ltr"><div>VM pauses are happening because quorum is not met. i.e. at least two of the brick in the setup are not connected to the client. So it is becoming Read-only filesystem.<br><br></div>Pranith<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 11, 2016 at 2:53 PM, <a href="mailto:maokx@sina.com">maokx@sina.com</a> <span dir="ltr"><<a href="mailto:maokx@sina.com" target="_blank">maokx@sina.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>
<div><span></span>Hi all:</div><div><br></div><div>can glusterfs replicate run in <span style="background-color:rgba(0,0,0,0);font-size:10.5pt;line-height:1.5">100M ethernet?</span></div><div><span style="background-color:rgba(0,0,0,0);font-size:10.5pt;line-height:1.5">gluster:</span></div><div><span style="background-color:rgba(0,0,0,0)">[root@server187 qemu]# rpm -aq|grep gluster
<br>glusterfs-3.6.2-1.el7.x86_64
<br>glusterfs-extra-xlators-3.6.2-1.el7.x86_64
<br>svserver-gluster-4.16.14-0.el7.centos.noarch
<br>glusterfs-api-3.6.2-1.el7.x86_64
<br>glusterfs-fuse-3.6.2-1.el7.x86_64
<br>glusterfs-cli-3.6.2-1.el7.x86_64
<br>glusterfs-server-3.6.2-1.el7.x86_64
<br>glusterfs-rdma-3.6.2-1.el7.x86_64
<br>glusterfs-libs-3.6.2-1.el7.x86_64
<br>glusterfs-devel-3.6.2-1.el7.x86_64
<br>[root@server187 qemu]# <br></span></div><div><span style="background-color:rgba(0,0,0,0);font-size:10.5pt;line-height:1.5"><b>1、why vm on running server has paused?</b></span><b style="font-size:10.5pt;line-height:1.5;background-color:window"><span style="font-size:10.5pt;line-height:1.5;background-color:rgba(0,0,0,0)">Is </span><span style="font-size:10.5pt;line-height:1.5;background-color:window">100M ethernet</span><span style="font-size:10.5pt;line-height:1.5;background-color:rgba(0,0,0,0)"> lead to?</span></b></div><div><div><span style="background-color:rgba(0,0,0,0);font-size:10.5pt;line-height:1.5">I use replicate 3 glusterfs,one server down,two server still up.</span></div><div>In this case,one vm on <span style="background-color:rgba(0,0,0,0);font-size:10.5pt;line-height:1.5">server running is become pause </span><span style="font-size:10.5pt;line-height:1.5;background-color:window">.</span></div></div><div><span style="font-size:10.5pt;line-height:1.5;background-color:window">detail:</span></div><div><img src="cid:_Foxmail.1@f080b889-4492-d13e-bd74-ee5629d64124"></div><div><img src="cid:_Foxmail.1@2dda1714-cf00-6514-4da7-e44da79eb292" border="0"></div><div><span style="background-color:rgba(0,0,0,0)">[2016-05-05 08:51:22.942682] I [rpc-clnt.c:1761:rpc_clnt_reconfig] 0-pool01-client-2: changing port to 49152 (from 0)<br>[2016-05-05 08:51:22.947960] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier<br>[2016-05-05 08:51:22.948196] I [client-handshake.c:1413:select_server_supported_programs] 0-pool01-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)<br>[2016-05-05 08:51:22.948346] I [client-handshake.c:1200:client_setvolume_cbk] 0-pool01-client-2: Connected to pool01-client-2, attached to remote volume '/data/data01'.<br>[2016-05-05 08:51:22.948358] I [client-handshake.c:1210:client_setvolume_cbk] 0-pool01-client-2: Server and Client lk-version numbers are not same, reopening the fds<br>[2016-05-05 08:51:22.948365] I [client-handshake.c:1052:client_post_handshake] 0-pool01-client-2: 3 fds open - Delaying child_up until they are re-opened<br>[2016-05-05 08:51:22.948612] I [client-handshake.c:674:client_child_up_reopen_done] 0-pool01-client-2: last fd open'd/lock-self-heal'd - notifying CHILD-UP<br>[2016-05-05 08:51:22.948646] I [MSGID: 108002] [afr-common.c:3631:afr_notify] 0-pool01-replicate-0: Client-quorum is met<br>[2016-05-05 08:51:22.948677] I [client-handshake.c:188:client_set_lk_version_cbk] 0-pool01-client-2: Server lk version = 1<br>[2016-05-05 08:51:23.945232] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier<br>[2016-05-05 08:51:26.948173] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier<br>[2016-05-05 08:51:29.397014] I [afr-common.c:1484:afr_local_discovery_cbk] 0-pool01-replicate-0: selecting local read_child pool01-client-2<br>[2016-05-05 08:51:29.951076] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier<br>[2016-05-05 08:51:32.954015] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier<br>[2016-05-05 08:51:35.956908] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier<br><font color="#0000ff">[2016-05-05 08:51:36.238878] W [fuse-bridge.c:1261:fuse_err_cbk] 0-glusterfs-fuse: 407688: FSYNC() ERR => -1 (Read-only file system)<br>[2016-05-05 08:51:36.280494] W [fuse-bridge.c:1261:fuse_err_cbk] 0-glusterfs-fuse: 407689: FSYNC() ERR => -1 (Read-only file system)</font><br>[2016-05-05 08:51:38.959928] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier<br>[2016-05-05 08:51:41.962890] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier<br>[2016-05-05 08:51:44.965974] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier<br>[2016-05-05 08:51:47.968896] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier<br>[2016-05-05 08:51:50.971802] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier<br>[2016-05-05 08:51:53.974781] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier</span></div><div><br></div><div><br></div><div><br></div><div><span style="background-color:rgba(0,0,0,0);font-size:10.5pt;line-height:1.5"><br></span></div><div><b><span style="background-color:rgba(0,0,0,0);font-size:10.5pt;line-height:1.5">2、</span><span style="font-size:10.5pt;line-height:1.5;background-color:rgba(0,0,0,0)">why </span><span style="font-size:10.5pt;line-height:1.5;background-color:rgba(0,0,0,0)">Response so slowly?</span><span style="font-size:10.5pt;line-height:1.5;background-color:rgba(0,0,0,0)">Is </span><span style="font-size:10.5pt;line-height:1.5;background-color:window">100M ethernet</span><span style="font-size:10.5pt;line-height:1.5;background-color:rgba(0,0,0,0)"> lead to?</span></b></div><div><span style="background-color:rgba(0,0,0,0);font-size:10.5pt;line-height:1.5">I use replicate 3 glusterfs,one server down,two server still up.</span></div><div>In this case,one brick on <span style="background-color:rgba(0,0,0,0);font-size:10.5pt;line-height:1.5">server running is become readonly because of </span><span style="font-size:10.5pt;line-height:1.5;background-color:window">Client-quorum is not met.</span></div><div>But a<span style="background-color:rgba(0,0,0,0);font-size:10.5pt;line-height:1.5">fter about three hours,that brick </span><span style="font-size:10.5pt;line-height:1.5;background-color:window">become better </span><span style="font-size:10.5pt;line-height:1.5;background-color:window">automatic</span><span style="background-color:window;font-size:10.5pt;line-height:1.5">.</span></div><div><span style="background-color:rgba(0,0,0,0);font-size:10.5pt;line-height:1.5">why </span><span style="background-color:rgba(0,0,0,0);font-size:10.5pt;line-height:1.5">Response so slowly?</span><span style="background-color:rgba(0,0,0,0);font-size:10.5pt;line-height:1.5">Is </span>100M ethernet<span style="background-color:rgba(0,0,0,0);font-size:10.5pt;line-height:1.5"> lead to?</span></div><div><span style="background-color:rgba(0,0,0,0);font-size:10.5pt;line-height:1.5">Detail:</span></div><div><img src="cid:_Foxmail.1@f080b889-4492-d13e-bd74-ee5629d64124" border="0"></div><div><span style="background-color:rgba(0,0,0,0)">[2016-05-04 06:49:47.924382] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier<br>[2016-05-04 06:50:47.262468] W [socket.c:611:__socket_rwv] 0-pool01-client-0: readv on <a href="http://192.168.101.185:49152" target="_blank">192.168.101.185:49152</a> failed (No data available)<br>[2016-05-04 06:50:47.262530] I [client.c:2215:client_rpc_notify] 0-pool01-client-0: disconnected from pool01-client-0. Client process will keep trying to connect to glusterd until brick's port is available<br>[2016-05-04 06:50:47.262546] W [MSGID: 108001] [afr-common.c:3635:afr_notify] 0-pool01-replicate-0: Client-quorum is not met<br>[2016-05-04 06:50:54.310210] I [afr-common.c:1484:afr_local_discovery_cbk] 0-pool01-replicate-0: selecting local read_child pool01-client-2<br>[2016-05-04 06:50:57.932864] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier<br>[2016-05-04 06:50:57.933201] E [socket.c:2267:socket_connect_finish] 0-pool01-client-0: connection to <a href="http://192.168.101.185:24007" target="_blank">192.168.101.185:24007</a> failed (Connection refused)<br>[2016-05-04 06:51:00.935714] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier<br>[2016-05-04 06:51:03.349865] W [fuse-bridge.c:2270:fuse_writev_cbk] 0-glusterfs-fuse: 37106: WRITE => -1 (Read-only file system)<br>[2016-05-04 06:51:03.851940] W [fuse-bridge.c:2270:fuse_writev_cbk] 0-glusterfs-fuse: 37117: WRITE => -1 (Read-only file system)<br>[2016-05-04 06:51:03.938817] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier<br><font color="#0000ff">[2016-05-04 06:51:04.354396] W [fuse-bridge.c:2270:fuse_writev_cbk] 0-glusterfs-fuse: 37142: WRITE => -1 (Read-only file system)</font></span></div><div><span style="background-color:rgba(0,0,0,0)">...........</span></div><div><span style="background-color:rgba(0,0,0,0)">[2016-05-04 09:43:20.018833] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:43:21.775009] W [fuse-bridge.c:1079:fuse_setattr_cbk] 0-glusterfs-fuse: 51883: SETATTR() /__DIRECT_IO_TEST__ => -1 (Read-only file system)
<br>[2016-05-04 09:43:23.021665] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:43:26.024536] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:43:29.027486] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:43:31.791565] W [fuse-bridge.c:1079:fuse_setattr_cbk] 0-glusterfs-fuse: 51903: SETATTR() /__DIRECT_IO_TEST__ => -1 (Read-only file system)
<br>[2016-05-04 09:43:32.030394] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:43:35.033372] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:43:38.036299] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:43:41.039224] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:43:41.807885] W [fuse-bridge.c:1079:fuse_setattr_cbk] 0-glusterfs-fuse: 51915: SETATTR() /__DIRECT_IO_TEST__ => -1 (Read-only file system)
<br>[2016-05-04 09:43:44.042165] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:43:47.045042] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:43:50.047917] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:43:51.824073] W [fuse-bridge.c:1079:fuse_setattr_cbk] 0-glusterfs-fuse: 51927: SETATTR() /__DIRECT_IO_TEST__ => -1 (Read-only file system)
<br>[2016-05-04 09:43:53.050793] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:43:56.053775] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:43:59.056653] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:44:01.839749] W [fuse-bridge.c:1079:fuse_setattr_cbk] 0-glusterfs-fuse: 51939: SETATTR() /__DIRECT_IO_TEST__ => -1 (Read-only file system)
<br>[2016-05-04 09:44:02.059552] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:44:07.062665] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:44:07.063593] I [rpc-clnt.c:1761:rpc_clnt_reconfig] 0-pool01-client-1: changing port to 49152 (from 0)
<br><font color="#0000ff">[2016-05-04 09:44:07.066088] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:44:07.066641] I [client-handshake.c:1413:select_server_supported_programs] 0-pool01-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
<br>[2016-05-04 09:44:07.067234] I [client-handshake.c:1200:client_setvolume_cbk] 0-pool01-client-1: Connected to pool01-client-1, attached to remote volume '/data/data01'.
<br>[2016-05-04 09:44:07.067247] I [client-handshake.c:1210:client_setvolume_cbk] 0-pool01-client-1: Server and Client lk-version numbers are not same, reopening the fds
<br>[2016-05-04 09:44:07.067253] I [client-handshake.c:1052:client_post_handshake] 0-pool01-client-1: 2 fds open - Delaying child_up until they are re-opened
<br>[2016-05-04 09:44:07.067778] I [client-handshake.c:674:client_child_up_reopen_done] 0-pool01-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP
</font><br>[2016-05-04 09:44:07.067804] I [MSGID: 108002] [afr-common.c:3631:afr_notify] 0-pool01-replicate-0: Client-quorum is met
<br>[2016-05-04 09:44:07.068272] I [client-handshake.c:188:client_set_lk_version_cbk] 0-pool01-client-1: Server lk version = 1
<br>[2016-05-04 09:44:08.065350] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:44:11.068215] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:44:11.842992] I [afr-common.c:1484:afr_local_discovery_cbk] 0-pool01-replicate-0: selecting local read_child pool01-client-2
<br>[2016-05-04 09:44:11.865405] W [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-pool01-client-2: remote operation failed: No data available
<br>[2016-05-04 09:44:11.865633] W [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-pool01-client-1: remote operation failed: No data available
<br>[2016-05-04 09:44:11.866102] W [fuse-bridge.c:1261:fuse_err_cbk] 0-glusterfs-fuse: 51953: REMOVEXATTR() /__DIRECT_IO_TEST__ => -1 (No data available)
<br>[2016-05-04 09:44:14.071042] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:44:17.073986] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:44:20.076992] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:44:23.079942] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:44:26.082841] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:44:29.085667] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:44:32.088532] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:44:35.091452] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:44:38.094352] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:44:41.097332] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:44:44.100262] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:44:47.103083] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:44:50.105942] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:44:53.108953] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:44:56.111865] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:44:59.114713] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:45:02.117608] W [common-utils.c:2621:gf_ports_reserved] 0-glusterfs-socket: 54321 is not a valid port identifier
<br>[2016-05-04 09:45:04.595414] W [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-pool01-client-2: remote operation failed: No data available
<br>[2016-05-04 09:45:04.595634] W [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-pool01-client-1: remote operation failed: No data available
<br>[2016-05-04 09:45:04.595966] W [fuse-bridge.c:1261:fuse_err_cbk] 0-glusterfs-fuse: 52091: REMOVEXATTR() /__DIRECT_IO_TEST__ => -1 (No data available)</span></div>
<div><br></div><hr style="width:210px;min-height:1px" size="1" color="#b5c4df" align="left"><span class="HOEnZb"><font color="#888888">
<div><span><div style="MARGIN:10px;FONT-FAMILY:verdana;FONT-SIZE:10pt"><div><a href="mailto:maokx@sina.com" target="_blank">maokx@sina.com</a></div></div></span></div>
</font></span></div><br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>