[Gluster-users] Gluster trouble

Wed Jul 3 14:55:59 UTC 2013

Back in March I posted about some gluster problems:

http://gluster.org/pipermail/gluster-users/2013-March/035737.html
http://gluster.org/pipermail/gluster-users/2013-March/035655.html

I'm still in the same situation - a straightforward 2-node, 2-way AFR setup with each server mounting the single shared volume via NFS using gluster 3.3.0 (can't use 3.3.1 dues to its NFS issues) on 64-bit linux (ubuntu lucid). Gluster appears to be working, but won't mount on boot by any means I've tried, and it's still logging prodigious amounts of incomprehensible rubbish (to me!).

gluster says everything is ok:

gluster volume status
Status of volume: shared
Gluster process                                         Port    Online  Pid
------------------------------------------------------------------------------
Brick 192.168.1.10:/var/shared                          24009   Y       3097
Brick 192.168.1.11:/var/shared                          24009   Y       3020
NFS Server on localhost                                 38467   Y       3103
Self-heal Daemon on localhost                           N/A     Y       3109
NFS Server on 192.168.1.11                              38467   Y       3057
Self-heal Daemon on 192.168.1.11                        N/A     Y       3096

(other node says the same thing with IPs the other way around)

Yet the logs tell a different story.

In syslog, this happens every second:

Jul  3 00:17:29 web1 init: glusterd main process (14958) terminated with status 255
Jul  3 00:17:29 web1 init: glusterd main process ended, respawning

In /var/log/glusterfs/etc-glusterfs-glusterd.vol.log I have lots of this:

[2013-07-03 14:24:08.350429] I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.3.0
[2013-07-03 14:24:08.350592] E [glusterfsd.c:1296:glusterfs_pidfile_setup] 0-glusterfsd: pidfile /var/run/glusterd.pid lock error (Resource temporarily unavailable)

In /var/log/glusterfs/glustershd.log, every minute I get hundreds of these:

2013-07-03 14:24:00.792751] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-shared-replicate-0: background  meta-data self-heal completed on <gfid:16adce4d-1933-485f-8359-66c47c757cd3>
[2013-07-03 14:24:00.794251] I [afr-common.c:1340:afr_launch_self_heal] 0-shared-replicate-0: background  meta-data self-heal triggered. path: <gfid:52bd33a0-0df8-408a-bf6f-c5b6a48c4bd3>, reason: lookup detected pending operations
[2013-07-03 14:24:00.796411] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-shared-replicate-0: background  meta-data self-heal completed on <gfid:52bd33a0-0df8-408a-bf6f-c5b6a48c4bd3>

'gluster volume heal shared info' says:

Heal operation on volume shared has been successful

Brick 192.168.1.10:/var/shared
Number of entries: 335
...

I'm not clear whether this means it has 335 files still to fix, or whether it's done so already.

Both servers are logging the same kind of stuff. I'm sure all these are related since they happen at about the same rate.

The lock error looks the most interesting, but I've no idea why that should happen. As before, I've tried deleting all traces of gluster, reinstalling and reconfiguring and putting all the data back on, but nothing changes.

Here's the command I used to create the volume:

gluster volume create shared replica 2 transport tcp 192.168.1.10:/var/shared 192.168.1.11:/var/shared

Here's the volume file it created:

+------------------------------------------------------------------------------+
  1: volume shared-posix
  2:     type storage/posix
  3:     option directory /var/shared
  4:     option volume-id 2600e26c-b6c4-448f-a6f6-ad27c14745a0
  5: end-volume
  6:
  7: volume shared-access-control
  8:     type features/access-control
  9:     subvolumes shared-posix
 10: end-volume
 11:
 12: volume shared-locks
 13:     type features/locks
 14:     subvolumes shared-access-control
 15: end-volume
 16:
 17: volume shared-io-threads
 18:     type performance/io-threads
 19:     subvolumes shared-locks
 20: end-volume
 21:
 22: volume shared-index
 23:     type features/index
 24:     option index-base /var/shared/.glusterfs/indices
 25:     subvolumes shared-io-threads
 26: end-volume
 27:
 28: volume shared-marker
 29:     type features/marker
 30:     option volume-uuid 2600e26c-b6c4-448f-a6f6-ad27c14745a0
 31:     option timestamp-file /var/lib/glusterd/vols/shared/marker.tstamp
 32:     option xtime off
 33:     option quota off
 34:     subvolumes shared-index
 35: end-volume
 36:
 37: volume /var/shared
 38:     type debug/io-stats
 39:     option latency-measurement off
 40:     option count-fop-hits off
 41:     subvolumes shared-marker
 42: end-volume
 43:
 44: volume shared-server
 45:     type protocol/server
 46:     option transport-type tcp
 47:     option auth.login./var/shared.allow 94017411-d986-48e4-a7ac-47c1db14fba0
 48:     option auth.login.94017411-d986-48e4-a7ac-47c1db14fba0.password 3929acf9-fcf1-4684-b271-07927d375c9b
 49:     option auth.addr./var/shared.allow *
 50:     subvolumes /var/shared
 51: end-volume

Despite all this, I've not seen gluster do anything visibly wrong - if I create a file on the shared volume it appears on the other node, checksums match, clients can read etc, but I don't want to be running on luck! It's all very troubling, and it's making a right mess of my new distributed logging system...

Any ideas?

Marcus