[Gluster-users] Problem rebalancing a distributed volume

[Kyle Johnson]

Hello everyone,

I am having trouble with a distributed volume.  In short, the rebalance 
command does not seem to work for me: Existing files are not migrated, 
and new files are not created on the new brick.

I am running glusterfs 3.7.6 on two servers:

  1) FreeBSD 10.3-RELEASE (colossus2 - 192.168.110.1)
  2) CentOS 6.7 (colossus - 192.168.110.2)

The bricks are zfs-backed on both servers, and the network consists of 
two direct-connected cat6 cables on 10gig NICs.  The NICs are bonded 
(lagg'd) together with mode 4 (LACP).

Here is what I am seeing:

root at colossus ~]# gluster volume create fubar 
192.168.110.2:/ftp/bricks/fubar
volume create: fubar: success: please start the volume to access data
[root at colossus ~]# gluster volume start fubar
volume start: fubar: success
[root at colossus ~]# mount -t glusterfs 192.168.110.2:/fubar /mnt/test
[root at colossus ~]# touch /mnt/test/file{1..100}
[root at colossus ~]# ls /mnt/test/| wc -l
100
[root at colossus ~]# ls /ftp/bricks/fubar | wc -l
100

# So far, so good.

[root at colossus ~]# gluster volume add-brick fubar 
192.168.110.1:/tank/bricks/fubar
volume add-brick: success

# For good measure, I'll do an explicit fix-layout first.

[root at colossus ~]# gluster volume rebalance fubar fix-layout start
volume rebalance: fubar: success: Rebalance on fubar has been started 
successfully. Use rebalance status command to check status of the 
rebalance process.
ID: 2da23238-dbe4-4759-97b2-08879db271e7

[root at colossus ~]# gluster volume rebalance fubar status
                                     Node Rebalanced-files size       
scanned      failures       skipped status   run time in secs
                                ---------      ----------- -----------   
-----------   -----------   ----------- ------------     --------------
                                localhost                0 
0Bytes             0             0             0 fix-layout 
completed               0.00
                            192.168.110.1                0 
0Bytes             0             0             0 fix-layout 
completed               0.00
volume rebalance: fubar: success

# Now to do the actual rebalance.

[root at colossus ~]# gluster volume rebalance fubar start
volume rebalance: fubar: success: Rebalance on fubar has been started 
successfully. Use rebalance status command to check status of the 
rebalance process.
ID: 67160a67-01b2-4a51-9a11-114aa6269ee9

[root at colossus ~]# gluster volume rebalance fubar status
                                     Node Rebalanced-files size       
scanned      failures       skipped status   run time in secs
                                ---------      ----------- -----------   
-----------   -----------   ----------- ------------     --------------
                                localhost                0 
0Bytes           100             0             0 completed               
0.00
                            192.168.110.1                0 
0Bytes             0             0             0 completed               
0.00
volume rebalance: fubar: success
[root at colossus ~]# ls /mnt/test/ | wc -l
101
[root at colossus ~]# ls /ftp/bricks/fubar/ | wc -l
100

# As the output shows, 100 files were scanned, but none were moved.

# And for another test, I'll create 100 new post-fix-layout files

[root at colossus ~]# touch /mnt/test/file{101..200}
[root at colossus ~]# ls /ftp/bricks/fubar/ | wc -l
199


# And as you can see here, they were all created on the first server.  
The second server isn't touched at all.


Not sure if this is relevant, but if I create the volume with both 
bricks to begin with, files are properly distributed.

Thanks!
Kyle

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160705/3729535c/attachment.html>