<div dir="ltr">Dietmar,<div>I just learnt that in 3.6.7, the ignore_deletes option gets ignored in the code and in fact the DELETE operation does get propagated to the slave and enforced i.e. you should see entries getting deleted on the slave irrespective of the state of the ignore_deletes option as dumped by the geo-replication config command.</div><div><br></div><div>I too see the "Reserved option" being output at the shell command-line when attempting to change the ignore_deletes config option. Since the option gets ignored, changing the option value is not needed.</div><div><br></div><div>-----</div><div><br></div><div>On a different note, if its not too much to ask, please consider upgrading to the latest 3.7.x builds. As usual, there's much that's been fixed since 3.6.7 and features that have been added as well. The fixes contain enforcing the ignore_deletes option and it can actually be toggled and is effective as well. The fix sets the default value of ignore_deletes to false which helps propagating the DELETE operations to slave clusters.</div><div><br></div><div>--</div><div>Milind</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jan 21, 2016 at 2:28 AM, Dietmar Putz <span dir="ltr"><<a href="mailto:putz@3qmedien.net" target="_blank">putz@3qmedien.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Hi Milind,<br>
<br>
thanks again for your answer...<br>
damn...i found an old mail from Venky Shankar and obviously i had a
wrong view on ignore_deletes....<br>
but<br>
<tt>[ 20:42:53 ] - root@gluster-ger-ber-07 ~ $gluster volume
geo-replication ger-ber-01 gluster-wien-07::aut-wien-vol-01 config
ignore_deletes false</tt><tt><br>
</tt><tt>Reserved option</tt><tt><br>
</tt><tt>geo-replication command failed</tt><tt><br>
</tt><tt>[ 20:43:06 ] - root@gluster-ger-ber-07 ~ $gluster volume
geo-replication ger-ber-01 gluster-wien-07::aut-wien-vol-01 config
ignore_deletes no</tt><tt><br>
</tt><tt>Reserved option</tt><tt><br>
</tt><tt>geo-replication command failed</tt><tt><br>
</tt><tt>[ 20:43:11 ] - root@gluster-ger-ber-07 ~ </tt><tt><br>
</tt>i stopped the geo-replication and tried it again...same result.<br>
possibly i should start a new replication from scratch and set
ignore_deletes to false before starting the replication.<br>
<br>
meanwhile the reason for the new 'operation not permitted' messages
were found...<br>
a directory on the master and a file on the slave in the same
directory-level with the same name...<br>
<br>
best regards<span class="HOEnZb"><font color="#888888"><br>
dietmar</font></span><div><div class="h5"><br>
<br>
<br>
<br>
<br>
<br>
<div>Am 20.01.2016 um 19:28 schrieb Milind
Changire:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Dietmar,
<div>I just looked at your very first post describing the
problem and I found<br>
</div>
<div><br>
</div>
<div><span style="font-size:12.8px">ignore_deletes: true</span><br>
</div>
<div><span style="font-size:12.8px"><br>
</span></div>
<div><span style="font-size:12.8px">in the geo-replication
config command output.</span></div>
<div><span style="font-size:12.8px"><br>
</span></div>
<div><span style="font-size:12.8px">If you'd like the slave
volume to replicate file deletion as well, then the
"ignore_deletes" should be set to "false"</span></div>
<div><span style="font-size:12.8px">That should help resolve the
CREATE + DELETE + CREATE issue.</span></div>
<div><span style="font-size:12.8px"><br>
</span></div>
<div><span style="font-size:12.8px">If this doesn't help, then
strace output for gsyncd could be the savior.</span></div>
<div><span style="font-size:12.8px"><br>
</span></div>
<div><span style="font-size:12.8px">-----</span></div>
<div><span style="font-size:12.8px">To add further ...</span></div>
<div><span style="font-size:12.8px">Lately we've stumbled across
another issue with the CREATE + RENAME + CREATE sequence on
geo-replication restart. Two fixes have been posted and are
available for review upstream.</span></div>
<div><span style="font-size:12.8px"><br>
</span></div>
<div>geo-rep: avoid creating multiple entries with same gfid -- <a href="http://review.gluster.org/13186" target="_blank"></a><a href="http://review.gluster.org/13186" target="_blank">http://review.gluster.org/13186</a></div>
<div>geo-rep: hard-link rename issues on changelog replay -- <a href="http://review.gluster.org/13189" target="_blank"></a><a href="http://review.gluster.org/13189" target="_blank">http://review.gluster.org/13189</a><br>
</div>
<div><br>
</div>
<div>I'll post info about the fix propagation plan for the 3.6.x
series later.</div>
<div><br>
</div>
<div>--</div>
<div>Milind</div>
<div><br>
</div>
<div><span style="font-size:12.8px"><br>
</span></div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, Jan 20, 2016 at 11:23 PM,
Dietmar Putz <span dir="ltr"><<a href="mailto:putz@3qmedien.net" target="_blank">putz@3qmedien.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Milind,<br>
<br>
thank you for your reply...<br>
meanwhile i realized that the setfattr command don't work so
i decided to delete the affected files and directories but
without stopping the geo-replication...i did it before i red
you mail.<br>
the affected folders are already replicated with the same
gfid like on the master...so this is solved for the moment.<br>
afterwards i did not receive the 'errcode: 23' messages on
the masters and the 'Operation not permitted' messages on
the slaves for 2 1/2 days but the geo-replication restarted
all the time about every 2 - 4 hours on each active master /
slave with below shown "OSError: [Errno 16] Device or
resource busy" message on master and slave.<br>
anytime the geo-replication restarts i saw following line
embedded in the event of restart (as shown far below) :<span><br>
<br>
I [dht-layout.c:663:dht_layout_normalize]
0-aut-wien-vol-01-dht: Found anomalies in
/.gfid/cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2/.dstXXb70G3x
(gfid = 00000000-0000-0000-0000-000000000000). Holes=1
overlaps=0<br>
<br>
</span>
i have found 24 of these hidden .dst* folders. All of them
are stored in the same 2 different subfolders on the master
and slave but the shown .dstXXb70G3x is the only one that
just exists on the slave volume. I checked that folder and
deleted it on the slave since it was empty. i believe that
such folders somehow belongs to the geo-replication process
but i have no details.<br>
however, about one day after deletion this folder was
recreated on the slave again but since deletion there are no
more 'Found anomalies' messages.<br>
currently the geo-replication still restarts frequently with
the far below shown message and unfortunately some
'Operation not permitted' messages appears again but for
different files than before.<br>
I already checked all folders on master/slave for different
gfid's but there are no more different gfid's. i guess there
is no way around to compare all gfid's of all files on
master and slave...since it is a dist-repl. volume there are
several million lines.<br>
if geo-replication then still not works i will start the
suggested strace of the gsyncd. in regard to the strace i
have two questions...<br>
<br>
do i need to start the strace on all active masters / slaves
or is it sufficient to trace one active master and the
corresponding active slave ?<br>
should i try to capture a geo-rep restart by the strace or
is it sufficient to let it run one minute randomly ?<br>
<br>
<br>
maybe this is of interest to solve the problem...on the
active masters there are lines like :<br>
...<br>
[2016-01-19 18:34:58.441606] W
[master(/gluster-export):1015:process] _GMaster: incomplete
sync, retrying changelogs: XSYNC-CHANGELOG.1453225971<br>
[2016-01-19 18:36:27.313515] W
[master(/gluster-export):1015:process] _GMaster: incomplete
sync, retrying changelogs: XSYNC-CHANGELOG.1453225971<br>
[2016-01-19 18:37:56.337660] W
[master(/gluster-export):996:process] _GMaster: changelogs
XSYNC-CHANGELOG.1453225971 could not be processed - moving
on...<br>
[2016-01-19 18:37:56.339124] W
[master(/gluster-export):1000:process] _GMaster: SKIPPED
GFID = 5a47cc07-f32f-4685-ae8e-4969995f3f1c,<huge list
with gfid's><br>
<br>
they end up in a huge list of comma separated gfid's. is
there any hint to get benefit out of this xsync-changelogs,
a way to find out what was incomplete ?<br>
<br>
one more thing that concerns me...i'm trying to understand
the distributed geo-replication.<br>
the master volume is a living object, accessed by some
clients who are upload, delete and recreate files and
folders. not very frequent but i observed the mentioned two
folders with different gfid's on master / slave and now they
are deleted by any client on the master volume. The
geo-replication is still in hybrid crawl and afaik can not
delete or rename files and folders on the slave volume until
changelog mode is reached. When now a client recreates the
same folders on the master again they get a new gfid
assigned which are different from the still existing gfid's
on the slave i believe...so geo-replication should get in
conflict again because of existing folders on the slave with
the same path but a different gfid than on the master...like
for any other files which are deleted and later recreated
while geo-rep is in hybrid crawl. is that right ?<br>
if so it will be difficult to reach the changelog modus on
large gluster volumes because in our case the initial hybrid
crawl took some days for about 45 TB...or r/w access needs
to be stopped for that time ?<br>
<br>
thanks in advance and<br>
best regards<span><font color="#888888"><br>
dietmar</font></span><span><br>
<br>
<br>
<br>
<br>
Am 19.01.2016 um 10:53 schrieb Milind Changire:<br>
</span>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>
Hi Dietmar,<br>
After discussion with Aravinda we realized that
unfortunately the suggestion to:<br>
</span><span>
setfattr -n glusterfs.geo-rep.trigger-sync -v "1"
<DIR><br>
setfattr -n glusterfs.geo-rep.trigger-sync -v "1"
<file-path><br>
<br>
</span><span>
won't work with 3.6.7, since provision for that
workaround was added after 3.6.7.<br>
<br>
There's an alternative way to achieve the
geo-replication:<br>
1. stop geo-replication<br>
2. delete files and directories with conflicting gfid on
SLAVE<br>
3. use the "touch" command to touch files and
directories with conflicting gfid<br>
on MASTER<br>
4. start geo-replication<br>
<br>
This *should* get things correctly replicated to SLAVE.<br>
Geo-replication should start with hybrid-crawl and
trigger the replication to SLAVE.<br>
<br>
If not, then there's more to look at.<br>
You could then send us output of the strace command for
the gsyncd process, while<br>
geo-replication is running:<br>
# strace -ff -p <gsyncd-pid> -o gsyncd-strace<br>
<br>
You could terminate strace after about one minute and
send us all the gsyncd-strace.<pid><br>
files which will help us debug the issue if its not
resolved by the alternative<br>
mechanism mentioned above.<br>
<br>
Also, crawl status Hybrid Crawl is not an entirely bad
thing. It just could mean<br>
that there's a lot of entries that are being processed.
However, if things don't<br>
return back to the normal state after trying out the
alternative suggestion, we<br>
could take a look at the strace output and get some
clues.<br>
<br>
--<br>
Milind<br>
<br>
</span>
<div>
<div>
----- Original Message -----<br>
From: "Dietmar Putz" <<a href="mailto:putz@3qmedien.net" target="_blank">putz@3qmedien.net</a>><br>
To: "Aravinda" <<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>>,
<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>,
"Milind Changire" <<a href="mailto:mchangir@redhat.com" target="_blank">mchangir@redhat.com</a>><br>
Sent: Thursday, January 14, 2016 11:07:55 PM<br>
Subject: Re: [Gluster-users] geo-replication 3.6.7 -
no transition from hybrid to changelog crawl<br>
<br>
Hello all,<br>
<br>
after some days of inactivity i started another
attempt to solve this<br>
geo-replication issue...step by step.<br>
it looks like that some of the directories on the
slave volume does not<br>
have the same gfid like the corresponding directory on
the master volume.<br>
<br>
for example :<br>
<br>
on a master-node i can see a lot of 'errcode: 23'
lines like :<br>
[2016-01-14 09:58:36.96585] W
[master(/gluster-export):301:regjob]<br>
_GMaster: Rsync:
.gfid/a8d0387d-c5ad-4eeb-9fc6-637fb8299a50 [errcode:
23]<br>
<br>
on the corresponding slave the corresponding message :<br>
[2016-01-14 09:57:06.070452] W
[fuse-bridge.c:1967:fuse_create_cbk]<br>
0-glusterfs-fuse: 1185648:
/.gfid/a8d0387d-c5ad-4eeb-9fc6-637fb8299a50<br>
=> -1 (Operation not permitted)<br>
<br>
This is the file on the master, the file is still not
replicated to the<br>
slave.<br>
<br>
120533444364 97332 -rw-r--r-- 2 2001 2001 99662854
Jan 8 13:40<br>
/gluster-export/3912/uploads/BSZ-2015/Z_002895D0-C832-4698-84E6-89F34CDEC2AE_20144555_ST_1.mp4<br>
120533444364 97332 -rw-r--r-- 2 2001 2001 99662854
Jan 8 13:40<br>
/gluster-export/.glusterfs/a8/d0/a8d0387d-c5ad-4eeb-9fc6-637fb8299a50<br>
<br>
The directory on the slave already contain some files,
all of them are<br>
not available on the master anymore, obviously deleted
in meantime on<br>
the master by a client.<br>
i have deleted and recreated this file on the master
and observed the<br>
logs for recurrence of the newly created gfid of this
file...same as before.<br>
<br>
in <a href="http://comments.gmane.org/gmane.comp.file-systems.gluster.user/20703" rel="noreferrer" target="_blank">http://comments.gmane.org/gmane.comp.file-systems.gluster.user/20703</a><br>
a user reports a geo-replication problem which is
possibly caused by<br>
different gfid's of underlying directories.<br>
and yes, the directory of this file-example above
shows that the gfid of<br>
the underlying directory differs from the gfid on the
master while the<br>
most other directories have the same gfid.<br>
<br>
master :<br>
...<br>
# file: gluster-export/3912/uploads/BSP-2012<br>
trusted.gfid=0x8f1d480351bb455b9adde190f2c2b350<br>
--------------<br>
# file: gluster-export/3912/uploads/BSZ-2003<br>
trusted.gfid=0xe80adc088e604234b778997d8e8c2018<br>
--------------<br>
# file: gluster-export/3912/uploads/BSZ-2004<br>
trusted.gfid=0xfe417dd16bbe4ae4a6a1936cfee7aced<br>
--------------<br>
# file: gluster-export/3912/uploads/BSZ-2010<br>
trusted.gfid=0x8044e436407d4ed3a67c81df8a7ad47f
###<br>
--------------<br>
# file: gluster-export/3912/uploads/BSZ-2015<br>
trusted.gfid=0x0c30f50480204e02b65d4716a048b029
###<br>
<br>
slave :<br>
...<br>
# file: gluster-export/3912/uploads/BSP-2012<br>
trusted.gfid=0x8f1d480351bb455b9adde190f2c2b350<br>
--------------<br>
# file: gluster-export/3912/uploads/BSZ-2003<br>
trusted.gfid=0xe80adc088e604234b778997d8e8c2018<br>
--------------<br>
# file: gluster-export/3912/uploads/BSZ-2004<br>
trusted.gfid=0xfe417dd16bbe4ae4a6a1936cfee7aced<br>
--------------<br>
# file: gluster-export/3912/uploads/BSZ-2010<br>
trusted.gfid=0xd83e8fb568c74e33a2091c547512a6ce
###<br>
--------------<br>
# file: gluster-export/3912/uploads/BSZ-2015<br>
trusted.gfid=0xa406e1bec7f3454d8f2ce9c5f9c70eb3
###<br>
<br>
<br>
now the question...how to fix this..?<br>
in the thread above Aravinda wrote :<br>
<br>
...<br>
To fix the issue,<br>
-----------------<br>
Find the parent directory of "main.mdb",<br>
Get the GFID of that directory, using getfattr<br>
Check the GFID of the same directory in Slave(To
confirm GFIDs are different)<br>
To fix the issue, Delete that directory in Slave.<br>
Set virtual xattr for that directory and all the files
inside that directory.<br>
setfattr -n glusterfs.geo-rep.trigger-sync -v
"1" <DIR><br>
setfattr -n glusterfs.geo-rep.trigger-sync -v
"1" <file-path><br>
<br>
Geo-rep will recreate the directory with Proper GFID
and starts sync.<br>
<br>
deletion of the affected slave directory might be
helpful...<br>
but do i have to execute above shown setfattr commands
on the master or<br>
do they just speed up synchronization ?<br>
usually sync should start automatically or could there
be a problem<br>
because crawl status is still in 'hybrid crawl'...?<br>
<br>
thanks in advance...<br>
best regards<br>
dietmar<br>
<br>
<br>
<br>
<br>
On 04.01.2016 12:08, Dietmar Putz wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello Aravinda,<br>
<br>
thank you for your reply.<br>
i just made a 'find /gluster-export -type f -exec ls
-lisa {} \; ><br>
ls-lisa-gluster-export-`hostname`.out' on each brick
and checked the<br>
output for files with less than 2 link counts.<br>
i found nothing...all files on each brick have exact
2 links.<br>
<br>
the entire output for all bricks contain more than 7
million lines<br>
including .glusterfs but without non relevant
directories and files..<br>
tron@dp-server:~/geo_rep_3$ cat
ls-lisa-gluster-wien-0* | egrep -v<br>
'indices|landfill|changelogs|health_check' | wc -l<br>
7007316<br>
<br>
link count is on $4 :<br>
tron@dp-server:~/geo_rep_3$ cat
ls-lisa-gluster-wien-0* | egrep -v<br>
'indices|landfill|changelogs|health_check' | awk
'{if($4=="2")print}'<br>
| tail -1<br>
62648153697 4 -rw-rw-rw- 2 root root 1713 Jan 4
01:44<br>
/gluster-export/3500/files/16/01/387233/3500-6dqMmBcVby97PQtR.ism<br>
<br>
tron@dp-server:~/geo_rep_3$ cat
ls-lisa-gluster-wien-0* | egrep -v<br>
'indices|landfill|changelogs|health_check' | awk
'{if($4=="1")print}'<br>
tron@dp-server:~/geo_rep_3$<br>
tron@dp-server:~/geo_rep_3$ cat
ls-lisa-gluster-wien-0* | egrep -v<br>
'indices|landfill|changelogs|health_check' | awk
'{if($4!="2")print}'<br>
tron@dp-server:~/geo_rep_3$<br>
tron@dp-server:~/geo_rep_3$ cat
ls-lisa-gluster-wien-0* | egrep -v<br>
'indices|landfill|changelogs|health_check' | awk
'{print $4}' | sort |<br>
uniq -c<br>
7007316 2<br>
tron@dp-server:~/geo_rep_3$<br>
<br>
If i understood you right this can not be the reason
for the problem.<br>
is there any other hint which i can check on the
master or slave to<br>
analyse the problem....?<br>
<br>
Any help would be very appreciated<br>
best regards<br>
dietmar<br>
<br>
<br>
<br>
Am 04.01.2016 um 07:14 schrieb Aravinda:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<br>
Looks like issue with Geo-rep due to race between
Create and Rename.<br>
Geo-replication uses gfid-access (Mount Volume
with aux-gfid-mount)<br>
to create and rename files. If Create and Rename
replayed more than<br>
once then Geo-rep creates a two files with same
GFID(not hardlink).<br>
This causes one file without backend GFID link.<br>
<br>
Milind is working on the patch to disallow the
creation of second<br>
file with same GFID.<br>
@Milind, Please provide more update about your
patch.<br>
<br>
As a workaround, identify all the files in Slave
volume which do not<br>
have backend links and delete those files(Only in
Slaves, keep backup<br>
if required)<br>
<br>
In Brick backend, Crawl and look for files with
link count less than<br>
2. (Exclude .glusterfs and .trashcan directory)<br>
<br>
regards<br>
Aravinda<br>
<br>
On 01/02/2016 09:56 PM, Dietmar Putz wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello all,<br>
<br>
one more time i need some help with a
geo-replication problem.<br>
recently i started a new geo-replication. the
master volume contains<br>
about 45 TB data and the slave volume was new
created before<br>
geo-replication setup was done.<br>
master and slave is a 6 node distributed
replicated volume running<br>
glusterfs-server 3.6.7-ubuntu1~trusty1.<br>
geo-rep was starting without problems. since few
days the slave<br>
volume contains about 200 GB more data than the
master volume and i<br>
expected that the crawl status changes from
'hybrid crawl' to<br>
'changelog crawl' but it remains in 'hybrid
crawl'.<br>
the 'status detail' output far below shows more
than 10 million<br>
synced files while the entire master volume
contains just about 2<br>
million files. some tests show that files are
not deleted on the<br>
slave volume.<br>
as far as i know the hybrid crawl has the
limitation of not<br>
replicating deletes and renames to the slave
thus the geo-rep needs<br>
to achieve the 'changelog crawl' status after
initial sync...<br>
usually this should happen more or less
automatically, is this right ?<br>
<br>
the geo-rep frequently fails with below shown
"OSError: [Errno 16]<br>
Device or resource busy", this error appears
about every 3-4 hours<br>
on each active master node.<br>
i guess the frequent appearance of this error
prevent geo-rep from<br>
changing to 'changelog crawl', does somebody
experienced such<br>
problem, is this the cause of the problem ?<br>
<br>
i found some similar reports on <a href="http://gluster.org" rel="noreferrer" target="_blank">gluster.org</a> for gfs 3.5,
3.6 and 3.7<br>
but none of them point me to a solution...<br>
does anybody know a solution or is there a
workaround to achieve the<br>
changelog crawl status...?<br>
<br>
Any help would be very appreciated<br>
best regards<br>
dietmar<br>
<br>
<br>
<br>
<br>
Master gluster-ger-ber-07:<br>
-----------------------------<br>
<br>
[2016-01-02 11:39:48.122546] I
[master(/gluster-export):1343:crawl]<br>
_GMaster: processing xsync changelog<br>
/var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01/9d7139ecf10a6fc33a<br>
6e41d8d6e56984/xsync/XSYNC-CHANGELOG.1451724692<br>
[2016-01-02 11:42:55.182342] I
[master(/gluster-export):1343:crawl]<br>
_GMaster: processing xsync changelog<br>
/var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01/9d7139ecf10a6fc33a<br>
6e41d8d6e56984/xsync/XSYNC-CHANGELOG.1451724751<br>
[2016-01-02 11:44:11.168962] I
[master(/gluster-export):1340:crawl]<br>
_GMaster: finished hybrid crawl syncing, stime:
(-1, 0)<br>
[2016-01-02 11:44:11.246845] I<br>
[master(/gluster-export):490:crawlwrap]
_GMaster: primary master<br>
with volume id
6a071cfa-b150-4f0b-b1ed-96ab5d4bd671 ...<br>
[2016-01-02 11:44:11.265209] I<br>
[master(/gluster-export):501:crawlwrap]
_GMaster: crawl interval: 3<br>
seconds<br>
[2016-01-02 11:44:11.896940] I
[master(/gluster-export):1192:crawl]<br>
_GMaster: slave's time: (-1, 0)<br>
[2016-01-02 11:44:12.171761] E
[repce(/gluster-export):207:__call__]<br>
RepceClient: call
18897:139899553576768:1451735052.09 (entry_ops)<br>
failed on peer with OSError<br>
[2016-01-02 11:44:12.172101] E<br>
[syncdutils(/gluster-export):270:log_raise_exception]
<top>: FAIL:<br>
Traceback (most recent call last):<br>
File<br>
"/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/gsyncd.py",<br>
line 164, in main<br>
main_i()<br>
File<br>
"/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/gsyncd.py",<br>
line 643, in main_i<br>
local.service_loop(*[r for r in [remote] if
r])<br>
File<br>
"/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py",<br>
line 1344, in service_loop<br>
g2.crawlwrap()<br>
File<br>
"/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py",<br>
line 539, in crawlwrap<br>
self.crawl(no_stime_update=no_stime_update)<br>
File<br>
"/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py",<br>
line 1204, in crawl<br>
self.process(changes)<br>
File<br>
"/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py",<br>
line 956, in process<br>
self.process_change(change, done, retry)<br>
File<br>
"/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py",<br>
line 920, in process_change<br>
self.slave.server.entry_ops(entries)<br>
File<br>
"/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/repce.py",<br>
line 226, in __call__<br>
return self.ins(self.meth, *a)<br>
File<br>
"/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/repce.py",<br>
line 208, in __call__<br>
raise res<br>
OSError: [Errno 16] Device or resource busy<br>
[2016-01-02 11:44:12.258982] I<br>
[syncdutils(/gluster-export):214:finalize]
<top>: exiting.<br>
[2016-01-02 11:44:12.321808] I
[repce(agent):92:service_loop]<br>
RepceServer: terminating on reaching EOF.<br>
[2016-01-02 11:44:12.349766] I
[syncdutils(agent):214:finalize]<br>
<top>: exiting.<br>
[2016-01-02 11:44:12.435992] I
[monitor(monitor):141:set_state]<br>
Monitor: new state: faulty<br>
[2016-01-02 11:44:23.164284] I
[monitor(monitor):215:monitor]<br>
Monitor:
------------------------------------------------------------<br>
[2016-01-02 11:44:23.169981] I
[monitor(monitor):216:monitor]<br>
Monitor: starting gsyncd worker<br>
[2016-01-02 11:44:23.216662] I
[changelogagent(agent):72:__init__]<br>
ChangelogAgent: Agent listining...<br>
[2016-01-02 11:44:23.239778] I
[gsyncd(/gluster-export):633:main_i]<br>
<top>: syncing:
gluster://localhost:ger-ber-01 -><br>
<a>ssh://root@gluster-wien-07-int:gluster://localhost:aut-wien-vol-01</a><br>
[2016-01-02 11:44:26.358613] I<br>
[master(/gluster-export):75:gmaster_builder]
<top>: setting up xsync<br>
change detection mode<br>
[2016-01-02 11:44:26.358983] I<br>
[master(/gluster-export):413:__init__] _GMaster:
using 'rsync' as<br>
the sync engine<br>
[2016-01-02 11:44:26.359985] I<br>
[master(/gluster-export):75:gmaster_builder]
<top>: setting up<br>
changelog change detection mode<br>
[2016-01-02 11:44:26.360243] I<br>
[master(/gluster-export):413:__init__] _GMaster:
using 'rsync' as<br>
the sync engine<br>
[2016-01-02 11:44:26.361159] I<br>
[master(/gluster-export):75:gmaster_builder]
<top>: setting up<br>
changeloghistory change detection mode<br>
[2016-01-02 11:44:26.361377] I<br>
[master(/gluster-export):413:__init__] _GMaster:
using 'rsync' as<br>
the sync engine<br>
[2016-01-02 11:44:26.402601] I<br>
[master(/gluster-export):1311:register]
_GMaster: xsync temp<br>
directory:<br>
/var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01/9d7139ecf10a6fc33a6e<br>
41d8d6e56984/xsync<br>
[2016-01-02 11:44:26.402848] I<br>
[resource(/gluster-export):1318:service_loop]
GLUSTER: Register<br>
time: 1451735066<br>
[2016-01-02 11:44:27.26012] I<br>
[master(/gluster-export):490:crawlwrap]
_GMaster: primary master<br>
with volume id
6a071cfa-b150-4f0b-b1ed-96ab5d4bd671 ...<br>
[2016-01-02 11:44:27.31605] I<br>
[master(/gluster-export):501:crawlwrap]
_GMaster: crawl interval: 1<br>
seconds<br>
[2016-01-02 11:44:27.66868] I
[master(/gluster-export):1226:crawl]<br>
_GMaster: starting history crawl... turns: 1,
stime: (-1, 0)<br>
[2016-01-02 11:44:27.67043] I
[master(/gluster-export):1229:crawl]<br>
_GMaster: stime not available, abandoning
history crawl<br>
[2016-01-02 11:44:27.112426] I<br>
[master(/gluster-export):490:crawlwrap]
_GMaster: primary master<br>
with volume id
6a071cfa-b150-4f0b-b1ed-96ab5d4bd671 ...<br>
[2016-01-02 11:44:27.117506] I<br>
[master(/gluster-export):501:crawlwrap]
_GMaster: crawl interval: 60<br>
seconds<br>
[2016-01-02 11:44:27.140610] I
[master(/gluster-export):1333:crawl]<br>
_GMaster: starting hybrid crawl..., stime: (-1,
0)<br>
[2016-01-02 11:45:23.417233] I
[monitor(monitor):141:set_state]<br>
Monitor: new state: Stable<br>
[2016-01-02 11:45:48.225915] I
[master(/gluster-export):1343:crawl]<br>
_GMaster: processing xsync changelog<br>
/var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01/9d7139ecf10a6fc33a<br>
6e41d8d6e56984/xsync/XSYNC-CHANGELOG.1451735067<br>
[2016-01-02 11:47:08.65231] I
[master(/gluster-export):1343:crawl]<br>
_GMaster: processing xsync changelog<br>
/var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01/9d7139ecf10a6fc33a6<br>
e41d8d6e56984/xsync/XSYNC-CHANGELOG.1451735148<br>
...<br>
<br>
<br>
slave gluster-wien-07 :<br>
------------------------<br>
<br>
[2016-01-02 11:44:12.007744] W
[fuse-bridge.c:1261:fuse_err_cbk]<br>
0-glusterfs-fuse: 1959820: SETXATTR()<br>
/.gfid/5e436e5b-086b-4720-9e70-0e49c8e09698
=> -1 (File exists)<br>
[2016-01-02 11:44:12.010970] W<br>
[client-rpc-fops.c:240:client3_3_mknod_cbk]<br>
0-aut-wien-vol-01-client-5: remote operation
failed: File exists.<br>
Path:<br>
<gfid:666bceac-7c14-4efd-81fe-8185458fcf1f>/11-kxyrM3NgdtBWPFv4.webm<br>
[2016-01-02 11:44:12.011327] W<br>
[client-rpc-fops.c:240:client3_3_mknod_cbk]<br>
0-aut-wien-vol-01-client-4: remote operation
failed: File exists.<br>
Path:<br>
<gfid:666bceac-7c14-4efd-81fe-8185458fcf1f>/11-kxyrM3NgdtBWPFv4.webm<br>
[2016-01-02 11:44:12.012054] W
[fuse-bridge.c:1261:fuse_err_cbk]<br>
0-glusterfs-fuse: 1959822: SETXATTR()<br>
/.gfid/666bceac-7c14-4efd-81fe-8185458fcf1f
=> -1 (File exists)<br>
[2016-01-02 11:44:12.024743] W<br>
[client-rpc-fops.c:240:client3_3_mknod_cbk]<br>
0-aut-wien-vol-01-client-5: remote operation
failed: File exists.<br>
Path:
<gfid:5bfd6f99-07e8-4b2f-844b-aa0b6535c055>/Gf4FYbpDTC7yK2mv.png<br>
[2016-01-02 11:44:12.024970] W<br>
[client-rpc-fops.c:240:client3_3_mknod_cbk]<br>
0-aut-wien-vol-01-client-4: remote operation
failed: File exists.<br>
Path:
<gfid:5bfd6f99-07e8-4b2f-844b-aa0b6535c055>/Gf4FYbpDTC7yK2mv.png<br>
[2016-01-02 11:44:12.025601] W
[fuse-bridge.c:1261:fuse_err_cbk]<br>
0-glusterfs-fuse: 1959823: SETXATTR()<br>
/.gfid/5bfd6f99-07e8-4b2f-844b-aa0b6535c055
=> -1 (File exists)<br>
[2016-01-02 11:44:12.100688] I<br>
[dht-selfheal.c:1065:dht_selfheal_layout_new_directory]<br>
0-aut-wien-vol-01-dht: chunk size = 0xffffffff /
57217563 = 0x4b<br>
[2016-01-02 11:44:12.100765] I<br>
[dht-selfheal.c:1103:dht_selfheal_layout_new_directory]<br>
0-aut-wien-vol-01-dht: assigning range size
0x5542c4a3 to<br>
aut-wien-vol-01-replicate-0<br>
[2016-01-02 11:44:12.100785] I<br>
[dht-selfheal.c:1103:dht_selfheal_layout_new_directory]<br>
0-aut-wien-vol-01-dht: assigning range size
0x5542c4a3 to<br>
aut-wien-vol-01-replicate-1<br>
[2016-01-02 11:44:12.100800] I<br>
[dht-selfheal.c:1103:dht_selfheal_layout_new_directory]<br>
0-aut-wien-vol-01-dht: assigning range size
0x5542c4a3 to<br>
aut-wien-vol-01-replicate-2<br>
[2016-01-02 11:44:12.100839] I [MSGID: 109036]<br>
[dht-common.c:6296:dht_log_new_layout_for_dir_selfheal]<br>
0-aut-wien-vol-01-dht: Setting layout of<br>
<gfid:d4815ee4-3348-4105-9136-d0219d956ed8>/.dstXXX0HUpRD
with<br>
[Subvol_name: aut-wien-vol-01-re<br>
plicate-0, Err: -1 , Start: 0 , Stop: 1430439074
], [Subvol_name:<br>
aut-wien-vol-01-replicate-1, Err: -1 , Start:
1430439075 , Stop:<br>
2860878149 ], [Subvol_name:
aut-wien-vol-01-replicate-2, Err: -1 ,<br>
Start: 2860878150 , Stop: 4294967295 ],<br>
[2016-01-02 11:44:12.114192] W<br>
[client-rpc-fops.c:306:client3_3_mkdir_cbk]<br>
0-aut-wien-vol-01-client-2: remote operation
failed: File exists.<br>
Path:
<gfid:cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2>/.dstXXb70G3x<br>
[2016-01-02 11:44:12.114275] W<br>
[client-rpc-fops.c:306:client3_3_mkdir_cbk]<br>
0-aut-wien-vol-01-client-3: remote operation
failed: File exists.<br>
Path:
<gfid:cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2>/.dstXXb70G3x<br>
[2016-01-02 11:44:12.114879] W
[fuse-bridge.c:1261:fuse_err_cbk]<br>
0-glusterfs-fuse: 1959831: SETXATTR()<br>
/.gfid/cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2
=> -1 (File exists)<br>
[2016-01-02 11:44:12.118473] I<br>
[dht-layout.c:663:dht_layout_normalize]
0-aut-wien-vol-01-dht: Found<br>
anomalies in<br>
/.gfid/cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2/.dstXXb70G3x
(gfid =<br>
00000000-0000-0000-0000-000000000000). Holes=1
overlaps=0<br>
[2016-01-02 11:44:12.118537] I<br>
[dht-selfheal.c:1065:dht_selfheal_layout_new_directory]<br>
0-aut-wien-vol-01-dht: chunk size = 0xffffffff /
57217563 = 0x4b<br>
[2016-01-02 11:44:12.118562] I<br>
[dht-selfheal.c:1103:dht_selfheal_layout_new_directory]<br>
0-aut-wien-vol-01-dht: assigning range size
0x5542c4a3 to<br>
aut-wien-vol-01-replicate-2<br>
[2016-01-02 11:44:12.118579] I<br>
[dht-selfheal.c:1103:dht_selfheal_layout_new_directory]<br>
0-aut-wien-vol-01-dht: assigning range size
0x5542c4a3 to<br>
aut-wien-vol-01-replicate-0<br>
[2016-01-02 11:44:12.118613] I<br>
[dht-selfheal.c:1103:dht_selfheal_layout_new_directory]<br>
0-aut-wien-vol-01-dht: assigning range size
0x5542c4a3 to<br>
aut-wien-vol-01-replicate-1<br>
[2016-01-02 11:44:12.120352] I [MSGID: 109036]<br>
[dht-common.c:6296:dht_log_new_layout_for_dir_selfheal]<br>
0-aut-wien-vol-01-dht: Setting layout of<br>
/.gfid/cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2/.dstXXb70G3x
with<br>
[Subvol_name: aut-wien-vol-01-rep<br>
licate-0, Err: -1 , Start: 1430439075 , Stop:
2860878149 ],<br>
[Subvol_name: aut-wien-vol-01-replicate-1, Err:
-1 , Start:<br>
2860878150 , Stop: 4294967295 ], [Subvol_name:<br>
aut-wien-vol-01-replicate-2, Err: -1 , Start: 0
, Stop: 1430439074 ],<br>
[2016-01-02 11:44:12.630949] I
[fuse-bridge.c:4927:fuse_thread_proc]<br>
0-fuse: unmounting /tmp/gsyncd-aux-mount-tOUOsz<br>
[2016-01-02 11:44:12.633952] W
[glusterfsd.c:1211:cleanup_and_exit]<br>
(--> 0-: received signum (15), shutting down<br>
[2016-01-02 11:44:12.633964] I
[fuse-bridge.c:5607:fini] 0-fuse:<br>
Unmounting '/tmp/gsyncd-aux-mount-tOUOsz'.<br>
[2016-01-02 11:44:23.946702] I [MSGID: 100030]<br>
[glusterfsd.c:2035:main] 0-/usr/sbin/glusterfs:
Started running<br>
/usr/sbin/glusterfs version 3.6.7 (args:
/usr/sbin/glusterfs<br>
--aux-gfid-mount
--log-file=/var/log/glusterfs/geo-replication-slav<br>
es/6a071cfa-b150-4f0b-b1ed-96ab5d4bd671:gluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.gluster.log<br>
--volfile-server=localhost
--volfile-id=aut-wien-vol-01<br>
--client-pid=-1 /tmp/gsyncd-aux-mount-otU3wS)<br>
[2016-01-02 11:44:24.042128] I
[dht-shared.c:337:dht_init_regex]<br>
0-aut-wien-vol-01-dht: using regex
rsync-hash-regex = ^\.(.+)\.[^.]+$<br>
[2016-01-02 11:44:24.046315] I
[client.c:2268:notify]<br>
0-aut-wien-vol-01-client-0: parent translators
are ready, attempting<br>
connect on transport<br>
[2016-01-02 11:44:24.046532] I
[client.c:2268:notify]<br>
0-aut-wien-vol-01-client-1: parent translators
are ready, attempting<br>
connect on transport<br>
[2016-01-02 11:44:24.046664] I
[client.c:2268:notify]<br>
0-aut-wien-vol-01-client-2: parent translators
are ready, attempting<br>
connect on transport<br>
[2016-01-02 11:44:24.046806] I
[client.c:2268:notify]<br>
0-aut-wien-vol-01-client-3: parent translators
are ready, attempting<br>
connect on transport<br>
[2016-01-02 11:44:24.046940] I
[client.c:2268:notify]<br>
0-aut-wien-vol-01-client-4: parent translators
are ready, attempting<br>
connect on transport<br>
[2016-01-02 11:44:24.047070] I
[client.c:2268:notify]<br>
0-aut-wien-vol-01-client-5: parent translators
are ready, attempting<br>
connect on transport<br>
Final graph:<br>
+------------------------------------------------------------------------------+<br>
<br>
1: volume aut-wien-vol-01-client-0<br>
2: type protocol/client<br>
3: option ping-timeout 10<br>
4: option remote-host gluster-wien-02-int<br>
5: option remote-subvolume
/gluster-export<br>
6: option transport-type socket<br>
7: option username
6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929<br>
8: option password
8777e154-476c-449a-89b2-3199872e4a1f<br>
9: option send-gids true<br>
10: end-volume<br>
11:<br>
12: volume aut-wien-vol-01-client-1<br>
13: type protocol/client<br>
14: option ping-timeout 10<br>
15: option remote-host gluster-wien-03-int<br>
16: option remote-subvolume
/gluster-export<br>
17: option transport-type socket<br>
18: option username
6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929<br>
19: option password
8777e154-476c-449a-89b2-3199872e4a1f<br>
20: option send-gids true<br>
21: end-volume<br>
22:<br>
23: volume aut-wien-vol-01-replicate-0<br>
24: type cluster/replicate<br>
25: subvolumes aut-wien-vol-01-client-0
aut-wien-vol-01-client-1<br>
26: end-volume<br>
27:<br>
28: volume aut-wien-vol-01-client-2<br>
29: type protocol/client<br>
30: option ping-timeout 10<br>
31: option remote-host gluster-wien-04-int<br>
32: option remote-subvolume
/gluster-export<br>
33: option transport-type socket<br>
34: option username
6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929<br>
35: option password
8777e154-476c-449a-89b2-3199872e4a1f<br>
36: option send-gids true<br>
37: end-volume<br>
38:<br>
39: volume aut-wien-vol-01-client-3<br>
40: type protocol/client<br>
41: option ping-timeout 10<br>
42: option remote-host gluster-wien-05-int<br>
43: option remote-subvolume
/gluster-export<br>
44: option transport-type socket<br>
45: option username
6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929<br>
46: option password
8777e154-476c-449a-89b2-3199872e4a1f<br>
47: option send-gids true<br>
48: end-volume<br>
49:<br>
50: volume aut-wien-vol-01-replicate-1<br>
51: type cluster/replicate<br>
52: subvolumes aut-wien-vol-01-client-2
aut-wien-vol-01-client-3<br>
53: end-volume<br>
54:<br>
55: volume aut-wien-vol-01-client-4<br>
56: type protocol/client<br>
57: option ping-timeout 10<br>
58: option remote-host gluster-wien-06-int<br>
59: option remote-subvolume
/gluster-export<br>
60: option transport-type socket<br>
61: option username
6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929<br>
62: option password
8777e154-476c-449a-89b2-3199872e4a1f<br>
63: option send-gids true<br>
64: end-volume<br>
65:<br>
66: volume aut-wien-vol-01-client-5<br>
67: type protocol/client<br>
68: option ping-timeout 10<br>
69: option remote-host gluster-wien-07-int<br>
70: option remote-subvolume
/gluster-export<br>
71: option transport-type socket<br>
72: option username
6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929<br>
73: option password
8777e154-476c-449a-89b2-3199872e4a1f<br>
74: option send-gids true<br>
75: end-volume<br>
76:<br>
77: volume aut-wien-vol-01-replicate-2<br>
78: type cluster/replicate<br>
79: subvolumes aut-wien-vol-01-client-4
aut-wien-vol-01-client-5<br>
80: end-volume<br>
81:<br>
82: volume aut-wien-vol-01-dht<br>
83: type cluster/distribute<br>
84: subvolumes aut-wien-vol-01-replicate-0<br>
aut-wien-vol-01-replicate-1
aut-wien-vol-01-replicate-2<br>
85: end-volume<br>
86:<br>
87: volume aut-wien-vol-01-write-behind<br>
88: type performance/write-behind<br>
89: subvolumes aut-wien-vol-01-dht<br>
90: end-volume<br>
91:<br>
92: volume aut-wien-vol-01-read-ahead<br>
93: type performance/read-ahead<br>
94: subvolumes
aut-wien-vol-01-write-behind<br>
95: end-volume<br>
96:<br>
97: volume aut-wien-vol-01-io-cache<br>
98: type performance/io-cache<br>
99: option min-file-size 0<br>
100: option cache-timeout 2<br>
101: option cache-size 1024MB<br>
102: subvolumes aut-wien-vol-01-read-ahead<br>
103: end-volume<br>
104:<br>
105: volume aut-wien-vol-01-quick-read<br>
106: type performance/quick-read<br>
107: option cache-size 1024MB<br>
108: subvolumes aut-wien-vol-01-io-cache<br>
109: end-volume<br>
110:<br>
111: volume aut-wien-vol-01-open-behind<br>
112: type performance/open-behind<br>
113: subvolumes aut-wien-vol-01-quick-read<br>
114: end-volume<br>
115:<br>
116: volume aut-wien-vol-01-md-cache<br>
117: type performance/md-cache<br>
118: subvolumes aut-wien-vol-01-open-behind<br>
119: end-volume<br>
120:<br>
121: volume aut-wien-vol-01<br>
122: type debug/io-stats<br>
123: option latency-measurement off<br>
124: option count-fop-hits off<br>
125: subvolumes aut-wien-vol-01-md-cache<br>
126: end-volume<br>
127:<br>
128: volume gfid-access-autoload<br>
129: type features/gfid-access<br>
130: subvolumes aut-wien-vol-01<br>
131: end-volume<br>
132:<br>
133: volume meta-autoload<br>
134: type meta<br>
135: subvolumes gfid-access-autoload<br>
136: end-volume<br>
137:<br>
+------------------------------------------------------------------------------+<br>
<br>
[2016-01-02 11:44:24.047642] I
[rpc-clnt.c:1761:rpc_clnt_reconfig]<br>
0-aut-wien-vol-01-client-5: changing port to
49153 (from 0)<br>
[2016-01-02 11:44:24.047927] I<br>
[client-handshake.c:1413:select_server_supported_programs]<br>
0-aut-wien-vol-01-client-5: Using Program
GlusterFS 3.3, Num<br>
(1298437), Version (330)<br>
[2016-01-02 11:44:24.048044] I<br>
[client-handshake.c:1200:client_setvolume_cbk]<br>
0-aut-wien-vol-01-client-5: Connected to
aut-wien-vol-01-client-5,<br>
attached to remote volume '/gluster-export'.<br>
[2016-01-02 11:44:24.048050] I<br>
[client-handshake.c:1210:client_setvolume_cbk]<br>
0-aut-wien-vol-01-client-5: Server and Client
lk-version numbers are<br>
not same, reopening the fds<br>
[2016-01-02 11:44:24.048088] I [MSGID: 108005]<br>
[afr-common.c:3684:afr_notify]
0-aut-wien-vol-01-replicate-2:<br>
Subvolume 'aut-wien-vol-01-client-5' came back
up; going online.<br>
[2016-01-02 11:44:24.048114] I<br>
[client-handshake.c:188:client_set_lk_version_cbk]<br>
0-aut-wien-vol-01-client-5: Server lk version =
1<br>
[2016-01-02 11:44:24.048124] I
[rpc-clnt.c:1761:rpc_clnt_reconfig]<br>
0-aut-wien-vol-01-client-0: changing port to
49153 (from 0)<br>
[2016-01-02 11:44:24.048132] I
[rpc-clnt.c:1761:rpc_clnt_reconfig]<br>
0-aut-wien-vol-01-client-1: changing port to
49153 (from 0)<br>
[2016-01-02 11:44:24.048138] I
[rpc-clnt.c:1761:rpc_clnt_reconfig]<br>
0-aut-wien-vol-01-client-2: changing port to
49153 (from 0)<br>
[2016-01-02 11:44:24.048146] I
[rpc-clnt.c:1761:rpc_clnt_reconfig]<br>
0-aut-wien-vol-01-client-3: changing port to
49153 (from 0)<br>
[2016-01-02 11:44:24.048153] I
[rpc-clnt.c:1761:rpc_clnt_reconfig]<br>
0-aut-wien-vol-01-client-4: changing port to
49153 (from 0)<br>
[2016-01-02 11:44:24.049070] I<br>
[client-handshake.c:1413:select_server_supported_programs]<br>
0-aut-wien-vol-01-client-0: Using Program
GlusterFS 3.3, Num<br>
(1298437), Version (330)<br>
[2016-01-02 11:44:24.049094] I<br>
[client-handshake.c:1413:select_server_supported_programs]<br>
0-aut-wien-vol-01-client-3: Using Program
GlusterFS 3.3, Num<br>
(1298437), Version (330)<br>
[2016-01-02 11:44:24.049113] I<br>
[client-handshake.c:1413:select_server_supported_programs]<br>
0-aut-wien-vol-01-client-2: Using Program
GlusterFS 3.3, Num<br>
(1298437), Version (330)<br>
[2016-01-02 11:44:24.049131] I<br>
[client-handshake.c:1413:select_server_supported_programs]<br>
0-aut-wien-vol-01-client-1: Using Program
GlusterFS 3.3, Num<br>
(1298437), Version (330)<br>
[2016-01-02 11:44:24.049224] I<br>
[client-handshake.c:1413:select_server_supported_programs]<br>
0-aut-wien-vol-01-client-4: Using Program
GlusterFS 3.3, Num<br>
(1298437), Version (330)<br>
[2016-01-02 11:44:24.049307] I<br>
[client-handshake.c:1200:client_setvolume_cbk]<br>
0-aut-wien-vol-01-client-0: Connected to
aut-wien-vol-01-client-0,<br>
attached to remote volume '/gluster-export'.<br>
[2016-01-02 11:44:24.049312] I<br>
[client-handshake.c:1210:client_setvolume_cbk]<br>
0-aut-wien-vol-01-client-0: Server and Client
lk-version numbers are<br>
not same, reopening the fds<br>
[2016-01-02 11:44:24.049324] I [MSGID: 108005]<br>
[afr-common.c:3684:afr_notify]
0-aut-wien-vol-01-replicate-0:<br>
Subvolume 'aut-wien-vol-01-client-0' came back
up; going online.<br>
[2016-01-02 11:44:24.049384] I<br>
[client-handshake.c:1200:client_setvolume_cbk]<br>
0-aut-wien-vol-01-client-3: Connected to
aut-wien-vol-01-client-3,<br>
attached to remote volume '/gluster-export'.<br>
[2016-01-02 11:44:24.049389] I<br>
[client-handshake.c:1210:client_setvolume_cbk]<br>
0-aut-wien-vol-01-client-3: Server and Client
lk-version numbers are<br>
not same, reopening the fds<br>
[2016-01-02 11:44:24.049400] I [MSGID: 108005]<br>
[afr-common.c:3684:afr_notify]
0-aut-wien-vol-01-replicate-1:<br>
Subvolume 'aut-wien-vol-01-client-3' came back
up; going online.<br>
[2016-01-02 11:44:24.049418] I<br>
[client-handshake.c:1200:client_setvolume_cbk]<br>
0-aut-wien-vol-01-client-2: Connected to
aut-wien-vol-01-client-2,<br>
attached to remote volume '/gluster-export'.<br>
[2016-01-02 11:44:24.049422] I<br>
[client-handshake.c:1210:client_setvolume_cbk]<br>
0-aut-wien-vol-01-client-2: Server and Client
lk-version numbers are<br>
not same, reopening the fds<br>
[2016-01-02 11:44:24.049460] I<br>
[client-handshake.c:1200:client_setvolume_cbk]<br>
0-aut-wien-vol-01-client-1: Connected to
aut-wien-vol-01-client-1,<br>
attached to remote volume '/gluster-export'.<br>
[2016-01-02 11:44:24.049465] I<br>
[client-handshake.c:1210:client_setvolume_cbk]<br>
0-aut-wien-vol-01-client-1: Server and Client
lk-version numbers are<br>
not same, reopening the fds<br>
[2016-01-02 11:44:24.049493] I<br>
[client-handshake.c:188:client_set_lk_version_cbk]<br>
0-aut-wien-vol-01-client-0: Server lk version =
1<br>
[2016-01-02 11:44:24.049567] I<br>
[client-handshake.c:188:client_set_lk_version_cbk]<br>
0-aut-wien-vol-01-client-3: Server lk version =
1<br>
[2016-01-02 11:44:24.049632] I<br>
[client-handshake.c:1200:client_setvolume_cbk]<br>
0-aut-wien-vol-01-client-4: Connected to
aut-wien-vol-01-client-4,<br>
attached to remote volume '/gluster-export'.<br>
[2016-01-02 11:44:24.049638] I<br>
[client-handshake.c:1210:client_setvolume_cbk]<br>
0-aut-wien-vol-01-client-4: Server and Client
lk-version numbers are<br>
not same, reopening the fds<br>
[2016-01-02 11:44:24.052103] I
[fuse-bridge.c:5086:fuse_graph_setup]<br>
0-fuse: switched to graph 0<br>
[2016-01-02 11:44:24.052150] I<br>
[client-handshake.c:188:client_set_lk_version_cbk]<br>
0-aut-wien-vol-01-client-2: Server lk version =
1<br>
[2016-01-02 11:44:24.052163] I<br>
[client-handshake.c:188:client_set_lk_version_cbk]<br>
0-aut-wien-vol-01-client-4: Server lk version =
1<br>
[2016-01-02 11:44:24.052192] I<br>
[client-handshake.c:188:client_set_lk_version_cbk]<br>
0-aut-wien-vol-01-client-1: Server lk version =
1<br>
[2016-01-02 11:44:24.052204] I
[fuse-bridge.c:4015:fuse_init]<br>
0-glusterfs-fuse: FUSE inited with protocol
versions: glusterfs 7.22<br>
kernel 7.20<br>
[2016-01-02 11:44:24.053991] I<br>
[afr-common.c:1491:afr_local_discovery_cbk]<br>
0-aut-wien-vol-01-replicate-2: selecting local
read_child<br>
aut-wien-vol-01-client-5<br>
[2016-01-02 11:45:48.613563] W<br>
[client-rpc-fops.c:306:client3_3_mkdir_cbk]<br>
0-aut-wien-vol-01-client-5: remote operation
failed: File exists.<br>
Path: /keys<br>
[2016-01-02 11:45:48.614131] W<br>
[client-rpc-fops.c:306:client3_3_mkdir_cbk]<br>
0-aut-wien-vol-01-client-4: remote operation
failed: File exists.<br>
Path: /keys<br>
[2016-01-02 11:45:48.614436] W
[fuse-bridge.c:1261:fuse_err_cbk]<br>
0-glusterfs-fuse: 12: SETXATTR()<br>
/.gfid/00000000-0000-0000-0000-000000000001
=> -1 (File exists)<br>
...<br>
<br>
<br>
[ 13:41:40 ] - root@gluster-ger-ber-07<br>
/var/log/glusterfs/geo-replication/ger-ber-01
$gluster volume<br>
geo-replication ger-ber-01
gluster-wien-07::aut-wien-vol-01 status<br>
detail<br>
<br>
MASTER NODE MASTER VOL MASTER BRICK<br>
SLAVE STATUS
CHECKPOINT<br>
STATUS CRAWL STATUS FILES SYNCD FILES
PENDING BYTES<br>
PENDING DELETES PENDING FILES SKIPPED<br>
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
<br>
gluster-ger-ber-07 ger-ber-01
/gluster-export<br>
gluster-wien-07-int::aut-wien-vol-01 Active
N/A<br>
Hybrid Crawl 10743644 8192 0 0
0<br>
gluster-ger-ber-11 ger-ber-01
/gluster-export<br>
gluster-wien-03-int::aut-wien-vol-01 Active
N/A<br>
Hybrid Crawl 16037091 8192 0 0
0<br>
gluster-ger-ber-10 ger-ber-01
/gluster-export<br>
gluster-wien-02-int::aut-wien-vol-01 Passive
N/A<br>
N/A 0 0 0 0 0<br>
gluster-ger-ber-12 ger-ber-01
/gluster-export<br>
gluster-wien-06-int::aut-wien-vol-01 Passive
N/A<br>
N/A 0 0 0 0 0<br>
gluster-ger-ber-09 ger-ber-01
/gluster-export<br>
gluster-wien-05-int::aut-wien-vol-01 Active
N/A<br>
Hybrid Crawl 16180514 8192 0 0
0<br>
gluster-ger-ber-08 ger-ber-01
/gluster-export<br>
gluster-wien-04-int::aut-wien-vol-01 Passive
N/A<br>
N/A 0 0 0 0 0<br>
<br>
<br>
[ 13:41:55 ] - root@gluster-ger-ber-07<br>
/var/log/glusterfs/geo-replication/ger-ber-01
$gluster volume<br>
geo-replication ger-ber-01
gluster-wien-07::aut-wien-vol-01 config<br>
special_sync_mode: partial<br>
state_socket_unencoded:<br>
/var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-07_aut-wien-vol-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.socket<br>
gluster_log_file:<br>
/var/log/glusterfs/geo-replication/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.gluster.log<br>
ssh_command: ssh -p 2503
-oPasswordAuthentication=no<br>
-oStrictHostKeyChecking=no -i<br>
/var/lib/glusterd/geo-replication/secret.pem<br>
ignore_deletes: true<br>
change_detector: changelog<br>
ssh_command_tar: ssh -p 2503
-oPasswordAuthentication=no<br>
-oStrictHostKeyChecking=no -i<br>
/var/lib/glusterd/geo-replication/tar_ssh.pem<br>
state_file:<br>
/var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-07_aut-wien-vol-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.status<br>
remote_gsyncd: /nonexistent/gsyncd<br>
log_file:<br>
/var/log/glusterfs/geo-replication/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.log<br>
changelog_log_file:<br>
/var/log/glusterfs/geo-replication/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01-changes.log<br>
socketdir: /var/run<br>
working_dir:<br>
/var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01<br>
state_detail_file:<br>
/var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-07_aut-wien-vol-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01-detail.status<br>
session_owner:
6a071cfa-b150-4f0b-b1ed-96ab5d4bd671<br>
gluster_command_dir: /usr/sbin/<br>
pid_file:<br>
/var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-07_aut-wien-vol-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.pid<br>
georep_session_working_dir:<br>
/var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-07_aut-wien-vol-01/<br>
gluster_params: aux-gfid-mount<br>
volume_id: 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671<br>
[ 13:42:11 ] - root@gluster-ger-ber-07<br>
/var/log/glusterfs/geo-replication/ger-ber-01 $<br>
<br>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote>
</blockquote>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote>
</div>
</div>
</blockquote>
<div>
<div>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br></div>