<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi all
<div class=""><br class="">
</div>
<div class="">We have currently a production cluster with 2 nodes in a distributed replicated setup with glusterfs version 3.6.4 which was updated from gluster version 3.5.x. I just expanded the cluster with 2 extra nodes with glusterfs version 3.6.4 installed
but when running the rebalancing command some errors show up:</div>
<div class=""><br class="">
</div>
<div class="">
<div class=""><i class="">gluster volume rebalance uploadtmp start</i></div>
<div class="">volume rebalance: uploadtmp: success: Initiated rebalance on volume uploadtmp.</div>
<div class="">Execute "gluster volume rebalance <volume-name> status" to check status.</div>
<div class="">ID: 49264fc3-6cb2-424b-8522-425989b125d9</div>
</div>
<div class=""><br class="">
</div>
<div class="">
<div class=""><i class="">gluster volume rebalance uploadtmp status</i></div>
<div class=""> Node Rebalanced-files size scanned failures skipped status run time in secs</div>
<div class=""> --------- ----------- ----------- ----------- ----------- ----------- ------------ --------------</div>
<div class=""> localhost 0 0Bytes 5 2 0 completed 2.00</div>
<div class=""> <a href="http://gfs01b-dcg.intnet.be" class="">gfs01b-dcg.intnet.be</a> 0 0Bytes 5 2 0 completed 1.00</div>
<div class=""> <a href="http://gfs02a-dcg.intnet.be" class="">gfs02a-dcg.intnet.be</a> 0 0Bytes 5 2 0 completed 2.00</div>
<div class=""> <a href="http://gfs02b-dcg.intnet.be" class="">gfs02b-dcg.intnet.be</a> 0 0Bytes 5 2 0 completed 2.00</div>
<div class="">volume rebalance: uploadtmp: success:</div>
</div>
<div class=""><br class="">
</div>
<div class="">
<div class=""><i class="">tail -200 uploadtmp-rebalance.log</i></div>
<div class="">[2015-09-14 11:41:24.881814] I [MSGID: 100030] [glusterfsd.c:2018:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.4 (args: /usr/sbin/glusterfs -s localhost --volfile-id rebalance/uploadtmp --xlator-option *dht.use-readdirp=yes
--xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off
--xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=052e85f3-2693-43d1-a586-3ebe11e86055 --socket-file /var/run/gluster/gluster-rebalance-69232195-1b1c-47c4-8ef9-a1145cb7fc7a.sock --pid-file /var/lib/glusterd/vols/uploadtmp/rebalance/052e85f3-2693-43d1-a586-3ebe11e86055.pid
-l /var/log/glusterfs/uploadtmp-rebalance.log)</div>
<div class="">[2015-09-14 11:41:29.901106] I [graph.c:269:gf_add_cmdline_options] 0-uploadtmp-dht: adding option 'node-uuid' for volume 'uploadtmp-dht' with value '052e85f3-2693-43d1-a586-3ebe11e86055'</div>
<div class="">[2015-09-14 11:41:29.901142] I [graph.c:269:gf_add_cmdline_options] 0-uploadtmp-dht: adding option 'rebalance-cmd' for volume 'uploadtmp-dht' with value '1'</div>
<div class="">[2015-09-14 11:41:29.901152] I [graph.c:269:gf_add_cmdline_options] 0-uploadtmp-dht: adding option 'readdir-optimize' for volume 'uploadtmp-dht' with value 'on'</div>
<div class="">[2015-09-14 11:41:29.901159] I [graph.c:269:gf_add_cmdline_options] 0-uploadtmp-dht: adding option 'assert-no-child-down' for volume 'uploadtmp-dht' with value 'yes'</div>
<div class="">[2015-09-14 11:41:29.901166] I [graph.c:269:gf_add_cmdline_options] 0-uploadtmp-dht: adding option 'lookup-unhashed' for volume 'uploadtmp-dht' with value 'yes'</div>
<div class="">[2015-09-14 11:41:29.901173] I [graph.c:269:gf_add_cmdline_options] 0-uploadtmp-dht: adding option 'use-readdirp' for volume 'uploadtmp-dht' with value 'yes'</div>
<div class="">[2015-09-14 11:41:29.901180] I [graph.c:269:gf_add_cmdline_options] 0-uploadtmp-replicate-1: adding option 'readdir-failover' for volume 'uploadtmp-replicate-1' with value 'off'</div>
<div class="">[2015-09-14 11:41:29.901188] I [graph.c:269:gf_add_cmdline_options] 0-uploadtmp-replicate-1: adding option 'entry-self-heal' for volume 'uploadtmp-replicate-1' with value 'off'</div>
<div class="">[2015-09-14 11:41:29.901195] I [graph.c:269:gf_add_cmdline_options] 0-uploadtmp-replicate-1: adding option 'metadata-self-heal' for volume 'uploadtmp-replicate-1' with value 'off'</div>
<div class="">[2015-09-14 11:41:29.901208] I [graph.c:269:gf_add_cmdline_options] 0-uploadtmp-replicate-1: adding option 'data-self-heal' for volume 'uploadtmp-replicate-1' with value 'off'</div>
<div class="">[2015-09-14 11:41:29.901215] I [graph.c:269:gf_add_cmdline_options] 0-uploadtmp-replicate-0: adding option 'readdir-failover' for volume 'uploadtmp-replicate-0' with value 'off'</div>
<div class="">[2015-09-14 11:41:29.901222] I [graph.c:269:gf_add_cmdline_options] 0-uploadtmp-replicate-0: adding option 'entry-self-heal' for volume 'uploadtmp-replicate-0' with value 'off'</div>
<div class="">[2015-09-14 11:41:29.901228] I [graph.c:269:gf_add_cmdline_options] 0-uploadtmp-replicate-0: adding option 'metadata-self-heal' for volume 'uploadtmp-replicate-0' with value 'off'</div>
<div class="">[2015-09-14 11:41:29.901234] I [graph.c:269:gf_add_cmdline_options] 0-uploadtmp-replicate-0: adding option 'data-self-heal' for volume 'uploadtmp-replicate-0' with value 'off'</div>
<div class="">[2015-09-14 11:41:29.901726] I [dht-shared.c:337:dht_init_regex] 0-uploadtmp-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$</div>
<div class="">[2015-09-14 11:41:29.909507] W [graph.c:344:_log_if_unknown_option] 0-uploadtmp-replicate-1: option 'readdir-failover' is not recognized</div>
<div class="">[2015-09-14 11:41:29.909611] W [graph.c:344:_log_if_unknown_option] 0-uploadtmp-replicate-0: option 'readdir-failover' is not recognized</div>
<div class="">[2015-09-14 11:41:29.909644] I [client.c:2280:notify] 0-uploadtmp-client-0: parent translators are ready, attempting connect on transport</div>
<div class="">[2015-09-14 11:41:29.910192] I [client.c:2280:notify] 0-uploadtmp-client-1: parent translators are ready, attempting connect on transport</div>
<div class="">[2015-09-14 11:41:29.911618] I [client.c:2280:notify] 0-uploadtmp-client-2: parent translators are ready, attempting connect on transport</div>
<div class="">[2015-09-14 11:41:29.912884] I [client.c:2280:notify] 0-uploadtmp-client-3: parent translators are ready, attempting connect on transport</div>
<div class="">Final graph:</div>
<div class="">+------------------------------------------------------------------------------+</div>
<div class=""> 1: volume uploadtmp-client-0</div>
<div class=""> 2: type protocol/client</div>
<div class=""> 3: option ping-timeout 42</div>
<div class=""> 4: option remote-host <a href="http://gfs01a-dcg.intnet.be" class="">
gfs01a-dcg.intnet.be</a></div>
<div class=""> 5: option remote-subvolume /mnt/uploadtmp/brick1</div>
<div class=""> 6: option transport-type socket</div>
<div class=""> 7: option send-gids true</div>
<div class=""> 8: end-volume</div>
<div class=""> 9:</div>
<div class=""> 10: volume uploadtmp-client-1</div>
<div class=""> 11: type protocol/client</div>
<div class=""> 12: option ping-timeout 42</div>
<div class=""> 13: option remote-host <a href="http://gfs01b-dcg.intnet.be" class="">
gfs01b-dcg.intnet.be</a></div>
<div class=""> 14: option remote-subvolume /mnt/uploadtmp/brick1</div>
<div class=""> 15: option transport-type socket</div>
<div class=""> 16: option send-gids true</div>
<div class=""> 17: end-volume</div>
<div class=""> 18:</div>
<div class=""> 19: volume uploadtmp-replicate-0</div>
<div class=""> 20: type cluster/replicate</div>
<div class=""> 21: option data-self-heal off</div>
<div class=""> 22: option metadata-self-heal off</div>
<div class=""> 23: option entry-self-heal off</div>
<div class=""> 24: option readdir-failover off</div>
<div class=""> 25: subvolumes uploadtmp-client-0 uploadtmp-client-1</div>
<div class=""> 26: end-volume</div>
<div class=""> 27:</div>
<div class=""> 28: volume uploadtmp-client-2</div>
<div class=""> 29: type protocol/client</div>
<div class=""> 30: option ping-timeout 42</div>
<div class=""> 31: option remote-host <a href="http://gfs02a-dcg.intnet.be" class="">
gfs02a-dcg.intnet.be</a></div>
<div class=""> 32: option remote-subvolume /mnt/uploadtmp/brick1</div>
<div class=""> 33: option transport-type socket</div>
<div class=""> 34: option send-gids true</div>
<div class=""> 35: end-volume</div>
<div class=""> 36:</div>
<div class=""> 37: volume uploadtmp-client-3</div>
<div class=""> 38: type protocol/client</div>
<div class=""> 39: option ping-timeout 42</div>
<div class=""> 40: option remote-host <a href="http://gfs02b-dcg.intnet.be" class="">
gfs02b-dcg.intnet.be</a></div>
<div class=""> 41: option remote-subvolume /mnt/uploadtmp/brick1</div>
<div class=""> 42: option transport-type socket</div>
<div class=""> 43: option send-gids true</div>
<div class=""> 44: end-volume</div>
<div class=""> 45:</div>
<div class=""> 46: volume uploadtmp-replicate-1</div>
<div class=""> 47: type cluster/replicate</div>
<div class=""> 48: option data-self-heal off</div>
<div class=""> 49: option metadata-self-heal off</div>
<div class=""> 50: option entry-self-heal off</div>
<div class=""> 51: option readdir-failover off</div>
<div class=""> 52: subvolumes uploadtmp-client-2 uploadtmp-client-3</div>
<div class=""> 53: end-volume</div>
<div class=""> 54:</div>
<div class=""> 55: volume uploadtmp-dht</div>
<div class=""> 56: type cluster/distribute</div>
<div class=""> 57: option use-readdirp yes</div>
<div class=""> 58: option lookup-unhashed yes</div>
<div class=""> 59: option assert-no-child-down yes</div>
<div class=""> 60: option readdir-optimize on</div>
<div class=""> 61: option rebalance-cmd 1</div>
<div class=""> 62: option node-uuid 052e85f3-2693-43d1-a586-3ebe11e86055</div>
<div class=""> 63: subvolumes uploadtmp-replicate-0 uploadtmp-replicate-1</div>
<div class=""> 64: end-volume</div>
<div class=""> 65:</div>
<div class=""> 66: volume uploadtmp-write-behind</div>
<div class=""> 67: type performance/write-behind</div>
<div class=""> 68: subvolumes uploadtmp-dht</div>
<div class=""> 69: end-volume</div>
<div class=""> 70:</div>
<div class=""> 71: volume uploadtmp-read-ahead</div>
<div class=""> 72: type performance/read-ahead</div>
<div class=""> 73: subvolumes uploadtmp-write-behind</div>
<div class=""> 74: end-volume</div>
<div class=""> 75:</div>
<div class=""> 76: volume uploadtmp-io-cache</div>
<div class=""> 77: type performance/io-cache</div>
<div class=""> 78: subvolumes uploadtmp-read-ahead</div>
<div class=""> 79: end-volume</div>
<div class=""> 80:</div>
<div class=""> 81: volume uploadtmp-quick-read</div>
<div class=""> 82: type performance/quick-read</div>
<div class=""> 83: subvolumes uploadtmp-io-cache</div>
<div class=""> 84: end-volume</div>
<div class=""> 85:</div>
<div class=""> 86: volume uploadtmp-open-behind</div>
<div class=""> 87: type performance/open-behind</div>
<div class=""> 88: option read-after-open yes</div>
<div class=""> 89: subvolumes uploadtmp-quick-read</div>
<div class=""> 90: end-volume</div>
<div class=""> 91:</div>
<div class=""> 92: volume uploadtmp-md-cache</div>
<div class=""> 93: type performance/md-cache</div>
<div class=""> 94: subvolumes uploadtmp-open-behind</div>
<div class=""> 95: end-volume</div>
<div class=""> 96:</div>
<div class=""> 97: volume uploadtmp</div>
<div class=""> 98: type debug/io-stats</div>
<div class=""> 99: option latency-measurement off</div>
<div class="">100: option count-fop-hits off</div>
<div class="">101: subvolumes uploadtmp-md-cache</div>
<div class="">102: end-volume</div>
<div class="">103:</div>
<div class="">+------------------------------------------------------------------------------+</div>
<div class="">[2015-09-14 11:41:29.915765] I [rpc-clnt.c:1761:rpc_clnt_reconfig] 0-uploadtmp-client-0: changing port to 49152 (from 0)</div>
<div class="">[2015-09-14 11:41:29.916607] I [client-handshake.c:1413:select_server_supported_programs] 0-uploadtmp-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)</div>
<div class="">[2015-09-14 11:41:29.916922] I [client-handshake.c:1200:client_setvolume_cbk] 0-uploadtmp-client-0: Connected to uploadtmp-client-0, attached to remote volume '/mnt/uploadtmp/brick1'.</div>
<div class="">[2015-09-14 11:41:29.916937] I [client-handshake.c:1210:client_setvolume_cbk] 0-uploadtmp-client-0: Server and Client lk-version numbers are not same, reopening the fds</div>
<div class="">[2015-09-14 11:41:29.916994] I [MSGID: 108005] [afr-common.c:3672:afr_notify] 0-uploadtmp-replicate-0: Subvolume 'uploadtmp-client-0' came back up; going online.</div>
<div class="">[2015-09-14 11:41:29.917033] I [client-handshake.c:188:client_set_lk_version_cbk] 0-uploadtmp-client-0: Server lk version = 1</div>
<div class="">[2015-09-14 11:41:34.884359] I [rpc-clnt.c:1761:rpc_clnt_reconfig] 0-uploadtmp-client-1: changing port to 49152 (from 0)</div>
<div class="">[2015-09-14 11:41:34.892294] I [client-handshake.c:1413:select_server_supported_programs] 0-uploadtmp-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)</div>
<div class="">[2015-09-14 11:41:34.893999] I [client-handshake.c:1200:client_setvolume_cbk] 0-uploadtmp-client-1: Connected to uploadtmp-client-1, attached to remote volume '/mnt/uploadtmp/brick1'.</div>
<div class="">[2015-09-14 11:41:34.894020] I [client-handshake.c:1210:client_setvolume_cbk] 0-uploadtmp-client-1: Server and Client lk-version numbers are not same, reopening the fds</div>
<div class="">[2015-09-14 11:41:34.894446] I [client-handshake.c:188:client_set_lk_version_cbk] 0-uploadtmp-client-1: Server lk version = 1</div>
<div class="">[2015-09-14 11:41:34.894864] I [rpc-clnt.c:1761:rpc_clnt_reconfig] 0-uploadtmp-client-3: changing port to 49153 (from 0)</div>
<div class="">[2015-09-14 11:41:34.895947] I [rpc-clnt.c:1761:rpc_clnt_reconfig] 0-uploadtmp-client-2: changing port to 49153 (from 0)</div>
<div class="">[2015-09-14 11:41:34.898011] I [client-handshake.c:1413:select_server_supported_programs] 0-uploadtmp-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)</div>
<div class="">[2015-09-14 11:41:34.898861] I [client-handshake.c:1413:select_server_supported_programs] 0-uploadtmp-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)</div>
<div class="">[2015-09-14 11:41:34.899259] I [client-handshake.c:1200:client_setvolume_cbk] 0-uploadtmp-client-3: Connected to uploadtmp-client-3, attached to remote volume '/mnt/uploadtmp/brick1'.</div>
<div class="">[2015-09-14 11:41:34.899279] I [client-handshake.c:1210:client_setvolume_cbk] 0-uploadtmp-client-3: Server and Client lk-version numbers are not same, reopening the fds</div>
<div class="">[2015-09-14 11:41:34.899325] I [MSGID: 108005] [afr-common.c:3672:afr_notify] 0-uploadtmp-replicate-1: Subvolume 'uploadtmp-client-3' came back up; going online.</div>
<div class="">[2015-09-14 11:41:34.899923] I [client-handshake.c:1200:client_setvolume_cbk] 0-uploadtmp-client-2: Connected to uploadtmp-client-2, attached to remote volume '/mnt/uploadtmp/brick1'.</div>
<div class="">[2015-09-14 11:41:34.899951] I [client-handshake.c:1210:client_setvolume_cbk] 0-uploadtmp-client-2: Server and Client lk-version numbers are not same, reopening the fds</div>
<div class="">[2015-09-14 11:41:34.900191] I [client-handshake.c:188:client_set_lk_version_cbk] 0-uploadtmp-client-3: Server lk version = 1</div>
<div class="">[2015-09-14 11:41:34.902578] I [client-handshake.c:188:client_set_lk_version_cbk] 0-uploadtmp-client-2: Server lk version = 1</div>
<div class="">[2015-09-14 11:41:34.910808] I [afr-common.c:1477:afr_local_discovery_cbk] 0-uploadtmp-replicate-0: selecting local read_child uploadtmp-client-0</div>
<div class="">[2015-09-14 11:41:34.917249] I [dht-common.c:3309:dht_setxattr] 0-uploadtmp-dht: fixing the layout of /</div>
<div class="">[2015-09-14 11:41:34.917299] I [dht-selfheal.c:960:dht_fix_layout_of_directory] 0-uploadtmp-dht: subvolume 0 (uploadtmp-replicate-0): 251980 chunks</div>
<div class="">[2015-09-14 11:41:34.917314] I [dht-selfheal.c:960:dht_fix_layout_of_directory] 0-uploadtmp-dht: subvolume 1 (uploadtmp-replicate-1): 251980 chunks</div>
<div class="">[2015-09-14 11:41:34.917327] I [dht-selfheal.c:1065:dht_selfheal_layout_new_directory] 0-uploadtmp-dht: chunk size = 0xffffffff / 503960 = 0x214a</div>
<div class="">[2015-09-14 11:41:34.917361] I [dht-selfheal.c:1103:dht_selfheal_layout_new_directory] 0-uploadtmp-dht: assigning range size 0x7ffe51f8 to uploadtmp-replicate-0</div>
<div class="">[2015-09-14 11:41:34.917376] I [dht-selfheal.c:1103:dht_selfheal_layout_new_directory] 0-uploadtmp-dht: assigning range size 0x7ffe51f8 to uploadtmp-replicate-1</div>
<div class="">[2015-09-14 11:41:34.917410] I [MSGID: 109036] [dht-common.c:6296:dht_log_new_layout_for_dir_selfheal] 0-uploadtmp-dht: Setting layout of / with [Subvol_name: uploadtmp-replicate-0, Err: -1 , Start: 2147373560 , Stop: 4294967295 ], [Subvol_name:
uploadtmp-replicate-1, Err: -1 , Start: 0 , Stop: 2147373559 ],</div>
<div class=""><b class="">[2015-09-14 11:41:34.922000] W [client-rpc-fops.c:1023:client3_3_setxattr_cbk] 0-uploadtmp-client-3: remote operation failed: Permission denied</b></div>
<div class=""><b class="">[2015-09-14 11:41:34.922071] W [client-rpc-fops.c:1023:client3_3_setxattr_cbk] 0-uploadtmp-client-2: remote operation failed: Permission denied</b></div>
<div class="">[2015-09-14 11:41:34.926973] I [dht-rebalance.c:1405:gf_defrag_migrate_data] 0-uploadtmp-dht: migrate data called on /</div>
<div class=""><b class="">[2015-09-14 11:41:34.929176] W [client-rpc-fops.c:2145:client3_3_setattr_cbk] 0-uploadtmp-client-3: remote operation failed: Operation not permitted</b></div>
<div class=""><b class="">[2015-09-14 11:41:34.929231] W [client-rpc-fops.c:2145:client3_3_setattr_cbk] 0-uploadtmp-client-2: remote operation failed: Operation not permitted</b></div>
<div class=""><b class="">[2015-09-14 11:41:34.929660] E [MSGID: 109004] [dht-selfheal.c:1382:dht_dir_attr_heal] 0-dht: Directory attr heal failed. Failed to set uid/gid on path / on subvol uploadtmp-replicate-1, gfid = 00000000-0000-0000-0000-000000000001
[Operation not permitted]</b></div>
<div class="">[2015-09-14 11:41:35.453870] I [dht-rebalance.c:1649:gf_defrag_migrate_data] 0-uploadtmp-dht: Migration operation on dir / took 0.53 secs</div>
<div class=""><b class="">[2015-09-14 11:41:35.794673] W [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-uploadtmp-client-2: remote operation failed: Permission denied. Path: /atheneumwillebroek-rvl</b></div>
<div class=""><b class="">[2015-09-14 11:41:35.794874] W [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-uploadtmp-client-3: remote operation failed: Permission denied. Path: /atheneumwillebroek-rvl</b></div>
<div class=""><b class="">[2015-09-14 11:41:35.795868] W [MSGID: 109005] [dht-selfheal.c:582:dht_selfheal_dir_mkdir_cbk] 0-uploadtmp-dht: Directory selfheal failed: path = /atheneumwillebroek-rvl, gfid = 120476ff-55bb-43ab-9155-740c7db69fef [Permission denied]</b></div>
<div class="">[2015-09-14 11:41:35.795953] I [MSGID: 109036] [dht-common.c:6296:dht_log_new_layout_for_dir_selfheal] 0-uploadtmp-dht: Setting layout of /atheneumwillebroek-rvl with [Subvol_name: uploadtmp-replicate-0, Err: 0 , Start: 0 , Stop: 4294967295 ],
[Subvol_name: uploadtmp-replicate-1, Err: 2 , Start: 0 , Stop: 0 ],</div>
<div class="">[2015-09-14 11:41:35.798301] I [afr-lk-common.c:1078:afr_lock_blocking] 0-uploadtmp-replicate-1: unable to lock on even one child</div>
<div class="">[2015-09-14 11:41:35.798669] I [afr-transaction.c:1096:afr_post_blocking_inodelk_cbk] 0-uploadtmp-replicate-1: Blocking inodelks failed.</div>
<div class="">[2015-09-14 11:41:35.798829] I [dht-common.c:3309:dht_setxattr] 0-uploadtmp-dht: fixing the layout of /atheneumwillebroek-rvl</div>
<div class="">[2015-09-14 11:41:35.798853] I [dht-selfheal.c:960:dht_fix_layout_of_directory] 0-uploadtmp-dht: subvolume 0 (uploadtmp-replicate-0): 251980 chunks</div>
<div class="">[2015-09-14 11:41:35.798863] I [dht-selfheal.c:960:dht_fix_layout_of_directory] 0-uploadtmp-dht: subvolume 1 (uploadtmp-replicate-1): 251980 chunks</div>
<div class="">[2015-09-14 11:41:35.798873] I [dht-selfheal.c:1065:dht_selfheal_layout_new_directory] 0-uploadtmp-dht: chunk size = 0xffffffff / 251980 = 0x4294</div>
<div class="">[2015-09-14 11:41:35.798936] I [dht-selfheal.c:1103:dht_selfheal_layout_new_directory] 0-uploadtmp-dht: assigning range size 0xfffca3f0 to uploadtmp-replicate-0</div>
<div class="">[2015-09-14 11:41:35.798964] I [MSGID: 109036] [dht-common.c:6296:dht_log_new_layout_for_dir_selfheal] 0-uploadtmp-dht: Setting layout of /atheneumwillebroek-rvl with [Subvol_name: uploadtmp-replicate-0, Err: -1 , Start: 0 , Stop: 4294967295 ],
[Subvol_name: uploadtmp-replicate-1, Err: 116 , Start: 0 , Stop: 0 ],</div>
<div class="">[2015-09-14 11:41:35.801072] I [afr-lk-common.c:1078:afr_lock_blocking] 0-uploadtmp-replicate-1: unable to lock on even one child</div>
<div class="">[2015-09-14 11:41:35.801148] I [afr-transaction.c:1096:afr_post_blocking_inodelk_cbk] 0-uploadtmp-replicate-1: Blocking inodelks failed.</div>
<div class="">[2015-09-14 11:41:35.999884] I [dht-rebalance.c:1405:gf_defrag_migrate_data] 0-uploadtmp-dht: migrate data called on /atheneumwillebroek-rvl</div>
<div class="">[2015-09-14 11:41:36.541735] I [dht-rebalance.c:1649:gf_defrag_migrate_data] 0-uploadtmp-dht: Migration operation on dir /atheneumwillebroek-rvl took 0.54 secs</div>
<div class="">[2015-09-14 11:41:36.557300] I [dht-common.c:3309:dht_setxattr] 0-uploadtmp-dht: fixing the layout of /atheneumwillebroek-rvl/G387KnvoHHBMrKjrPHg6NFHhk144221575722193683</div>
<div class="">[2015-09-14 11:41:36.557347] I [dht-selfheal.c:960:dht_fix_layout_of_directory] 0-uploadtmp-dht: subvolume 0 (uploadtmp-replicate-0): 251980 chunks</div>
<div class="">[2015-09-14 11:41:36.557361] I [dht-selfheal.c:960:dht_fix_layout_of_directory] 0-uploadtmp-dht: subvolume 1 (uploadtmp-replicate-1): 251980 chunks</div>
<div class="">[2015-09-14 11:41:36.557370] I [dht-selfheal.c:1065:dht_selfheal_layout_new_directory] 0-uploadtmp-dht: chunk size = 0xffffffff / 251980 = 0x4294</div>
<div class="">[2015-09-14 11:41:36.557389] I [dht-selfheal.c:1103:dht_selfheal_layout_new_directory] 0-uploadtmp-dht: assigning range size 0xfffca3f0 to uploadtmp-replicate-0</div>
<div class="">[2015-09-14 11:41:36.557407] I [MSGID: 109036] [dht-common.c:6296:dht_log_new_layout_for_dir_selfheal] 0-uploadtmp-dht: Setting layout of /atheneumwillebroek-rvl/G387KnvoHHBMrKjrPHg6NFHhk144221575722193683 with [Subvol_name: uploadtmp-replicate-0,
Err: -1 , Start: 0 , Stop: 4294967295 ], [Subvol_name: uploadtmp-replicate-1, Err: 116 , Start: 0 , Stop: 0 ],</div>
<div class="">[2015-09-14 11:41:36.559315] I [afr-lk-common.c:1078:afr_lock_blocking] 0-uploadtmp-replicate-1: unable to lock on even one child</div>
<div class="">[2015-09-14 11:41:36.559339] I [afr-transaction.c:1096:afr_post_blocking_inodelk_cbk] 0-uploadtmp-replicate-1: Blocking inodelks failed.</div>
<div class=""><b class="">[2015-09-14 11:41:36.562725] E [dht-rebalance.c:1685:gf_defrag_fix_layout] 0-uploadtmp-dht: Lookup failed on /atheneumwillebroek-rvl/G387KnvoHHBMrKjrPHg6NFHhk144221575722193683</b></div>
<div class=""><b class="">[2015-09-14 11:41:36.562744] E [MSGID: 109016] [dht-rebalance.c:1815:gf_defrag_fix_layout] 0-uploadtmp-dht: Fix layout failed for /atheneumwillebroek-rvl/G387KnvoHHBMrKjrPHg6NFHhk144221575722193683</b></div>
<div class=""><b class="">[2015-09-14 11:41:36.562826] E [MSGID: 109016] [dht-rebalance.c:1815:gf_defrag_fix_layout] 0-uploadtmp-dht: Fix layout failed for /atheneumwillebroek-rvl</b></div>
<div class="">[2015-09-14 11:41:36.563249] I [MSGID: 109028] [dht-rebalance.c:2111:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 2.00 secs</div>
<div class="">[2015-09-14 11:41:36.563273] I [MSGID: 109028] [dht-rebalance.c:2115:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 5, failures: 2, skipped: 0</div>
<div class=""><b class="">[2015-09-14 11:41:36.563589] W [glusterfsd.c:1194:cleanup_and_exit] (--> 0-: received signum (15), shutting down</b></div>
</div>
<div class=""><b class=""><br class="">
</b></div>
<div class="">As far as I can see all gluster processes run under the user root. Anybody any ideas on how to fix this?</div>
<div class=""><br class="">
</div>
<div class="">Thanks in advance.</div>
<div class=""><br class="">
</div>
<div class="">Kind regards.</div>
<div class="">Davy</div>
</body>
</html>