<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
I started a remove-brick operation on a distributed volume roughly eight days ago and it has never finished. The counters (besides runtime in secs) have been the same for several days now:<br>
<br>
Node Rebalanced-files size scanned failures skipped status run time in secs<br>
localhost 5039 169.1GB 851242 0 0 in progress 689421.00<br>
<br>
The brick has ~3.5T of data in it, so clearly more data needs to be evacuated, but nothing is moving.
<br>
<br>
The only errors I've found in the volume log are the following:<br>
<br>
[2015-08-12 14:59:16.657643] E [glusterd-hooks.c:328:glusterd_hooks_run_hooks] 0-management: Failed to open dir /var/lib/glusterd/hooks/1/remove-brick/pre, due to No such file or directory<br>
[2015-08-12 14:59:21.920355] E [glusterd-hooks.c:328:glusterd_hooks_run_hooks] 0-management: Failed to open dir /var/lib/glusterd/hooks/1/remove-brick/post, due to No such file or directory<br>
<br>
Which is indeed strange, because the rpm reports it should have placed those hooks:<br>
rpm -q --filesbypkg glusterfs-server | grep /var/lib/glusterd/hooks/1/remove-brick<br>
glusterfs-server /var/lib/glusterd/hooks/1/remove-brick<br>
glusterfs-server /var/lib/glusterd/hooks/1/remove-brick/post<br>
glusterfs-server /var/lib/glusterd/hooks/1/remove-brick/pre<br>
<br>
What's even stranger is that rpm --verify works even though the files are not there:<br>
rpm --verify glusterfs-server # returns nothing<br>
ls -alh /var/lib/glusterd/hooks/1/remove-brick<br>
ls: cannot access /var/lib/glusterd/hooks/1/remove-brick: No such file or directory<br>
<br>
Any help would be greatly appreciated. Thanks!<br>
<br>
<br>
<br>
</div>
</body>
</html>