<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>
<div>setting server.manage-gids to on fixed remote i/o error messages for me, accessing over nfs, gluster 3.6.2-2 on Debian Wheezy. worth a shot.</div>
<div>
<div id="MAC_OUTLOOK_SIGNATURE"></div>
</div>
</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:12pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span><<a href="mailto:gluster-users-bounces@gluster.org">gluster-users-bounces@gluster.org</a>> on behalf of Mark Selby <<a href="mailto:mselby@unseelie.name">mselby@unseelie.name</a>><br>
<span style="font-weight:bold">Date: </span>Monday, March 7, 2016 at 8:11 AM<br>
<span style="font-weight:bold">To: </span>Jiffin Tony Thottan <<a href="mailto:jthottan@redhat.com">jthottan@redhat.com</a>>, "<a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>" <<a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [Gluster-users] NFS Client issues with Gluster Server 3.6.9<br>
</div>
<div><br>
</div>
<div>
<div>
<div>Here are the logs that you requested</div>
<div><br>
</div>
<div>Please let me know if I can send you anything else.</div>
<div><br>
</div>
<div>I really appreciate you taking a look at this - thanks!</div>
<div><br>
</div>
<div><br>
</div>
<div>root@dc1strg001x /root 547# gluster vol info backups</div>
<div><br>
</div>
<div>Volume Name: backups</div>
<div>Type: Replicate</div>
<div>Volume ID: 71a26ea6-632d-4a1d-8610-e782ce2a5100</div>
<div>Status: Started</div>
<div>Number of Bricks: 1 x 2 = 2</div>
<div>Transport-type: tcp</div>
<div>Bricks:</div>
<div>Brick1: dc1strg001x:/zfspool/glusterfs/backups/data</div>
<div>Brick2: dc1strg002x:/zfspool/glusterfs/backups/data</div>
<div>Options Reconfigured:</div>
<div>nfs.disable: off</div>
<div><br>
</div>
<div><br>
</div>
<div>root@dc1strg001x /var/log/glusterfs 551# cat nfs.log</div>
<div>[2016-03-07 16:03:14.257919] I [MSGID: 100030] [glusterfsd.c:2318:main] </div>
<div>0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.6 </div>
<div>(args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p </div>
<div>/var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S </div>
<div>/var/run/gluster/ad38be3bd1baece29e1b672e6659ae60.socket)</div>
<div>[2016-03-07 16:03:14.267862] I [MSGID: 101190] </div>
<div>[event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread </div>
<div>with index 1</div>
<div>[2016-03-07 16:03:14.273283] I </div>
<div>[rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: </div>
<div>Configured rpc.outstanding-rpc-limit with value 16</div>
<div>[2016-03-07 16:03:14.284154] W [MSGID: 112153] </div>
<div>[mount3.c:3929:mnt3svc_init] 0-nfs-mount: Exports auth has been disabled!</div>
<div>[2016-03-07 16:03:14.306163] I [rpc-drc.c:694:rpcsvc_drc_init] </div>
<div>0-rpc-service: DRC is turned OFF</div>
<div>[2016-03-07 16:03:14.306216] I [MSGID: 112110] [nfs.c:1494:init] 0-nfs: </div>
<div>NFS service started</div>
<div>[2016-03-07 16:03:14.312901] I [MSGID: 101190] </div>
<div>[event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread </div>
<div>with index 2</div>
<div>[2016-03-07 16:03:14.314078] W [graph.c:357:_log_if_unknown_option] </div>
<div>0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized</div>
<div>[2016-03-07 16:03:14.314137] W [graph.c:357:_log_if_unknown_option] </div>
<div>0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized</div>
<div>[2016-03-07 16:03:14.314185] W [graph.c:357:_log_if_unknown_option] </div>
<div>0-nfs-server: option 'transport-type' is not recognized</div>
<div>[2016-03-07 16:03:14.314270] I [MSGID: 114020] [client.c:2118:notify] </div>
<div>0-backups-client-0: parent translators are ready, attempting connect on </div>
<div>transport</div>
<div>[2016-03-07 16:03:14.315341] I [MSGID: 114020] [client.c:2118:notify] </div>
<div>0-backups-client-1: parent translators are ready, attempting connect on </div>
<div>transport</div>
<div>[2016-03-07 16:03:14.315923] I [rpc-clnt.c:1847:rpc_clnt_reconfig] </div>
<div>0-backups-client-0: changing port to 49152 (from 0)</div>
<div>Final graph:</div>
<div>+------------------------------------------------------------------------------+</div>
<div> 1: volume backups-client-0</div>
<div> 2: type protocol/client</div>
<div> 3: option ping-timeout 42</div>
<div> 4: option remote-host dc1strg001x</div>
<div> 5: option remote-subvolume /zfspool/glusterfs/backups/data</div>
<div> 6: option transport-type socket</div>
<div> 7: option username 42fa7a62-1420-4169-ad00-53c3481dbe5b</div>
<div> 8: option password b71b3c88-51e0-464c-8b09-14b661fdb4d3</div>
<div> 9: option send-gids true</div>
<div> 10: end-volume</div>
<div> 11:</div>
<div> 12: volume backups-client-1</div>
<div> 13: type protocol/client</div>
<div> 14: option ping-timeout 42</div>
<div> 15: option remote-host dc1strg002x</div>
<div>[2016-03-07 16:03:14.317412] I [MSGID: 114057] </div>
<div>[client-handshake.c:1437:select_server_supported_programs] </div>
<div>0-backups-client-0: Using Program GlusterFS 3.3, Num (1298437), Version </div>
<div>(330)</div>
<div> 16: option remote-subvolume /zfspool/glusterfs/backups/data</div>
<div> 17: option transport-type socket</div>
<div> 18: option username 42fa7a62-1420-4169-ad00-53c3481dbe5b</div>
<div> 19: option password b71b3c88-51e0-464c-8b09-14b661fdb4d3</div>
<div> 20: option send-gids true</div>
<div> 21: end-volume</div>
<div> 22:</div>
<div> 23: volume backups-replicate-0</div>
<div> 24: type cluster/replicate</div>
<div> 25: subvolumes backups-client-0 backups-client-1</div>
<div> 26: end-volume</div>
<div> 27:</div>
<div> 28: volume backups-dht</div>
<div> 29: type cluster/distribute</div>
<div> 30: subvolumes backups-replicate-0</div>
<div> 31: end-volume</div>
<div> 32:</div>
<div> 33: volume backups-write-behind</div>
<div> 34: type performance/write-behind</div>
<div> 35: subvolumes backups-dht</div>
<div> 36: end-volume</div>
<div> 37:</div>
<div> 38: volume backups</div>
<div> 39: type debug/io-stats</div>
<div> 40: option latency-measurement off</div>
<div> 41: option count-fop-hits off</div>
<div> 42: subvolumes backups-write-behind</div>
<div> 43: end-volume</div>
<div> 44:</div>
<div> 45: volume nfs-server</div>
<div> 46: type nfs/server</div>
<div> 47: option rpc-auth.auth-glusterfs on</div>
<div> 48: option rpc-auth.auth-unix on</div>
<div> 49: option rpc-auth.auth-null on</div>
<div> 50: option rpc-auth.ports.insecure on</div>
<div> 51: option rpc-auth-allow-insecure on</div>
<div> 52: option transport-type socket</div>
<div> 53: option transport.socket.listen-port 2049</div>
<div> 54: option nfs.dynamic-volumes on</div>
<div> 55: option nfs.nlm on</div>
<div> 56: option nfs.drc off</div>
<div> 57: option rpc-auth.addr.backups.allow *</div>
<div> 58: option nfs3.backups.volume-id 71a26ea6-632d-4a1d-8610-e782ce2a5100</div>
<div> 59: option nfs.backups.disable off</div>
<div> 60: option nfs.logs.disable off</div>
<div> 61: option nfs.users.disable off</div>
<div> 62: subvolumes backups</div>
<div> 63: end-volume</div>
<div> 64:</div>
<div>+------------------------------------------------------------------------------+</div>
<div>[2016-03-07 16:03:14.318157] I [MSGID: 114046] </div>
<div>[client-handshake.c:1213:client_setvolume_cbk] 0-backups-client-0: </div>
<div>Connected to backups-client-0, attached to remote volume </div>
<div>'/zfspool/glusterfs/backups/data'.</div>
<div>[2016-03-07 16:03:14.318276] I [MSGID: 114047] </div>
<div>[client-handshake.c:1224:client_setvolume_cbk] 0-backups-client-0: </div>
<div>Server and Client lk-version numbers are not same, reopening the fds</div>
<div>[2016-03-07 16:03:14.318400] I [MSGID: 108005] </div>
<div>[afr-common.c:3841:afr_notify] 0-backups-replicate-0: Subvolume </div>
<div>'backups-client-0' came back up; going online.</div>
<div>[2016-03-07 16:03:14.318470] I [MSGID: 114035] </div>
<div>[client-handshake.c:193:client_set_lk_version_cbk] 0-backups-client-0: </div>
<div>Server lk version = 1</div>
<div>[2016-03-07 16:03:14.496642] I [rpc-clnt.c:1847:rpc_clnt_reconfig] </div>
<div>0-backups-client-1: changing port to 49152 (from 0)</div>
<div>[2016-03-07 16:03:14.498394] I [MSGID: 114057] </div>
<div>[client-handshake.c:1437:select_server_supported_programs] </div>
<div>0-backups-client-1: Using Program GlusterFS 3.3, Num (1298437), Version </div>
<div>(330)</div>
<div>[2016-03-07 16:03:14.505580] I [MSGID: 114046] </div>
<div>[client-handshake.c:1213:client_setvolume_cbk] 0-backups-client-1: </div>
<div>Connected to backups-client-1, attached to remote volume </div>
<div>'/zfspool/glusterfs/backups/data'.</div>
<div>[2016-03-07 16:03:14.505627] I [MSGID: 114047] </div>
<div>[client-handshake.c:1224:client_setvolume_cbk] 0-backups-client-1: </div>
<div>Server and Client lk-version numbers are not same, reopening the fds</div>
<div>[2016-03-07 16:03:14.506210] I [MSGID: 114035] </div>
<div>[client-handshake.c:193:client_set_lk_version_cbk] 0-backups-client-1: </div>
<div>Server lk version = 1</div>
<div>[2016-03-07 16:03:14.507836] I [MSGID: 108031] </div>
<div>[afr-common.c:1782:afr_local_discovery_cbk] 0-backups-replicate-0: </div>
<div>selecting local read_child backups-client-0</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>On 3/6/16 9:13 PM, Jiffin Tony Thottan wrote:</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div><br>
</div>
<div><br>
</div>
<div>On 05/03/16 07:12, Mark Selby wrote:</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>I am trying to use GlusterFS as a general purpose NFS file server. I </div>
<div>have tried using the FUSE client but the performance fall off vs NFS </div>
<div>is quite large</div>
<div><br>
</div>
<div>Both the client and the server are Ubuntu 14.04.</div>
<div><br>
</div>
<div>I am using Gluster 3.6.9 because of the FUSE performance issues that </div>
<div>have been reported with 3.7.8 (see </div>
<div><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1309462">https://bugzilla.redhat.com/show_bug.cgi?id=1309462</a>)</div>
<div><br>
</div>
<div>I am having serious issues with a generic NFS client as shown by the </div>
<div>issues below. Basically most FOPs are giving me a Remote I/O error.</div>
<div><br>
</div>
<div>I would not think I was 1st person to see these issues - but my </div>
<div>Google Fu is not working.</div>
<div><br>
</div>
<div>Any and all help would be much appreciated</div>
<div><br>
</div>
<div>BTW - These operation against a plain Linux NFS server work fine.</div>
<div><br>
</div>
<div><br>
</div>
<div>root@dc1strg001x /var/log 448# gluster volume status</div>
<div>Status of volume: backups</div>
<div>Gluster process Port Online Pid</div>
<div>------------------------------------------------------------------------------
</div>
<div><br>
</div>
<div>Brick dc1strg001x:/zfspool/glusterfs/backups/data 49152 Y 6462</div>
<div>Brick dc1strg002x:/zfspool/glusterfs/backups/data 49152 Y 6382</div>
<div>NFS Server on localhost 2049 Y 6619</div>
<div>Self-heal Daemon on localhost N/A Y 6626</div>
<div>NFS Server on dc1strg002x 2049 Y 6502</div>
<div>Self-heal Daemon on dc1strg002x N/A Y 6509</div>
<div><br>
</div>
<div><br>
</div>
<div>root@vc1test001 /root 735# mount -o vers=3 -t nfs </div>
<div>dc1strg001x:/backups /mnt/backups_nfs</div>
<div><br>
</div>
<div>root@vc1test001 /mnt/backups_nfs 737# dd if=/dev/zero of=testfile </div>
<div>bs=16k count=16384</div>
<div>16384+0 records in</div>
<div>16384+0 records out</div>
<div>268435456 bytes (268 MB) copied, 2.46237 s, 109 MB/s</div>
<div><br>
</div>
<div>root@vc1test001 /mnt/backups_nfs 738# rm testfile</div>
<div><br>
</div>
<div>root@vc1test001 /mnt/backups_nfs 739# dd if=/dev/zero of=testfile </div>
<div>bs=16k count=16384</div>
<div>dd: failed to open ~testfile~: Remote I/O error</div>
<div><br>
</div>
<div>root@vc1test001 /var/tmp 743# rsync -av testfile /mnt/backups_nfs/</div>
<div>sending incremental file list</div>
<div>testfile</div>
<div>rsync: mkstemp "/mnt/backups_nfs/.testfile.bzg47C" failed: Remote I/O </div>
<div>error (121)</div>
<div><br>
</div>
<div>sent 1,074,004,056 bytes received 121 bytes 165,231,411.85 bytes/sec</div>
<div>total size is 1,073,741,824 speedup is 1.00</div>
<div>rsync error: some files/attrs were not transferred (see previous </div>
<div>errors) (code 23) at main.c(1183) [sender=3.1.0]</div>
<div><br>
</div>
</blockquote>
<div><br>
</div>
<div>Can you please provide the volume configuration(gluster vol info )</div>
<div>and log file for nfs server which u mounted (/var/log/glusterfs)</div>
<div><br>
</div>
<div>-- </div>
<div>Jiffin</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>_______________________________________________</div>
<div>Gluster-users mailing list</div>
<div><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a></div>
<div><a href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></div>
</blockquote>
<div><br>
</div>
</blockquote>
<div><br>
</div>
<div>_______________________________________________</div>
<div>Gluster-users mailing list</div>
<div><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a></div>
<div><a href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></div>
<div><br>
</div>
</div>
</div>
</span>
</body>
</html>