<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>
</p>
<div class="moz-text-flowed" style="font-family: -moz-fixed;
font-size: 12px;" lang="x-unicode">Hello everyone,
<br>
<br>
I am having trouble with a distributed volume. In short, the
rebalance command does not seem to work for me: Existing files are
not migrated, and new files are not created on the new brick.
<br>
<br>
I am running glusterfs 3.7.6 on two servers:
<br>
<br>
1) FreeBSD 10.3-RELEASE (colossus2 - 192.168.110.1)
<br>
2) CentOS 6.7 (colossus - 192.168.110.2)
<br>
<br>
The bricks are zfs-backed on both servers, and the network
consists of two direct-connected cat6 cables on 10gig NICs. The
NICs are bonded (lagg'd) together with mode 4 (LACP).
<br>
<br>
Here is what I am seeing:
<br>
<br>
root@colossus ~]# gluster volume create fubar
192.168.110.2:/ftp/bricks/fubar
<br>
volume create: fubar: success: please start the volume to access
data
<br>
[root@colossus ~]# gluster volume start fubar
<br>
volume start: fubar: success
<br>
[root@colossus ~]# mount -t glusterfs 192.168.110.2:/fubar
/mnt/test
<br>
[root@colossus ~]# touch /mnt/test/file{1..100}
<br>
[root@colossus ~]# ls <i class="moz-txt-slash"><span
class="moz-txt-tag">/</span>mnt/test<span class="moz-txt-tag">/</span></i>|
wc -l
<br>
100
<br>
[root@colossus ~]# ls /ftp/bricks/fubar | wc -l
<br>
100
<br>
<br>
# So far, so good.
<br>
<br>
[root@colossus ~]# gluster volume add-brick fubar
192.168.110.1:/tank/bricks/fubar
<br>
volume add-brick: success
<br>
<br>
# For good measure, I'll do an explicit fix-layout first.
<br>
<br>
[root@colossus ~]# gluster volume rebalance fubar fix-layout start
<br>
volume rebalance: fubar: success: Rebalance on fubar has been
started successfully. Use rebalance status command to check status
of the rebalance process.
<br>
ID: 2da23238-dbe4-4759-97b2-08879db271e7
<br>
<br>
[root@colossus ~]# gluster volume rebalance fubar status
<br>
Node Rebalanced-files
size scanned failures skipped
status run time in secs
<br>
--------- -----------
----------- ----------- ----------- -----------
------------ --------------
<br>
localhost 0
0Bytes 0 0 0 fix-layout
completed 0.00
<br>
192.168.110.1 0
0Bytes 0 0 0 fix-layout
completed 0.00
<br>
volume rebalance: fubar: success
<br>
<br>
# Now to do the actual rebalance.
<br>
<br>
[root@colossus ~]# gluster volume rebalance fubar start
<br>
volume rebalance: fubar: success: Rebalance on fubar has been
started successfully. Use rebalance status command to check status
of the rebalance process.
<br>
ID: 67160a67-01b2-4a51-9a11-114aa6269ee9
<br>
<br>
[root@colossus ~]# gluster volume rebalance fubar status
<br>
Node Rebalanced-files
size scanned failures skipped
status run time in secs
<br>
--------- -----------
----------- ----------- ----------- -----------
------------ --------------
<br>
localhost 0
0Bytes 100 0 0
completed 0.00
<br>
192.168.110.1 0
0Bytes 0 0 0
completed 0.00
<br>
volume rebalance: fubar: success
<br>
[root@colossus ~]# ls <i class="moz-txt-slash"><span
class="moz-txt-tag">/</span>mnt/test<span class="moz-txt-tag">/</span></i>
| wc -l
<br>
101
<br>
[root@colossus ~]# ls <i class="moz-txt-slash"><span
class="moz-txt-tag">/</span>ftp/bricks/fubar<span
class="moz-txt-tag">/</span></i> | wc -l
<br>
100
<br>
<br>
# As the output shows, 100 files were scanned, but none were
moved.
<br>
<br>
# And for another test, I'll create 100 new post-fix-layout files
<br>
<br>
[root@colossus ~]# touch /mnt/test/file{101..200}
<br>
[root@colossus ~]# ls <i class="moz-txt-slash"><span
class="moz-txt-tag">/</span>ftp/bricks/fubar<span
class="moz-txt-tag">/</span></i> | wc -l
<br>
199
<br>
<br>
<br>
# And as you can see here, they were all created on the first
server. The second server isn't touched at all.
<br>
<br>
<br>
Not sure if this is relevant, but if I create the volume with both
bricks to begin with, files are properly distributed.
<br>
<br>
Thanks!
<br>
Kyle
<br>
<br>
</div>
</body>
</html>