<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Sure,
<div class=""><br class="">
</div>
<div class=""><a href="https://dl.dropboxusercontent.com/u/2663552/logs.tgz" class="">https://dl.dropboxusercontent.com/u/2663552/logs.tgz</a></div>
<div class=""><br class="">
</div>
<div class="">Yesterday I restart the geo-rep (and reset the changelog.changelog option). Today it looks converged and changelog keeps doing his job.</div>
<div class=""><br class="">
</div>
<div class="">BUT</div>
<div class=""><br class="">
</div>
<div class="">hybridcrawl doesn’t seem to update symlink links if they changed on master:</div>
<div class=""><br class="">
</div>
<div class="">From master:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">ll -n /usr/global/images/3.2/latest</div>
<div class="">lrwxrwxrwx 1 499 499 3 Jun 1 21:40 /usr/global/images/3.2/latest -> S22</div>
</div>
<div class=""><br class="">
</div>
<div class="">On slave:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">ls /usr/global/images/3.2/latest</div>
<div class="">lrwxrwxrwx 1 root root 2 May 9 07:01 /usr/global/images/3.2/latest -> S3</div>
</div>
<div class=""><br class="">
</div>
<div class="">The point is I can’t get the gfid from the symlink because it resolve the target folder.</div>
<div class=""><br class="">
</div>
<div class="">And by the way all data synced in hybrid crawl are root.root on the slave (they should keep the owner from the master as it also exist on the slave).</div>
<div class=""><br class="">
</div>
<div class="">So.</div>
<div class=""><br class="">
</div>
<div class="">1/I will need to remove symlinks from the salve and retrigger an hybrid crawl (again)</div>
<div class="">2/I will need to update permissions of the salve according to permissions on master (will be long and difficult)</div>
<div class="">3/Or I missed something here.</div>
<div class=""><br class="">
</div>
<div class="">Thanks!</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">-- <br class="">
Cyril Peponnet </div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Jun 1, 2015, at 10:20 PM, Kotresh Hiremath Ravishankar <<a href="mailto:khiremat@redhat.com" class="">khiremat@redhat.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Hi
Cyril,</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Could
you please attach the geo-replication logs?</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Thanks
and Regards,</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Kotresh
H R</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">-----
Original Message -----</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
From: "Cyril N PEPONNET (Cyril)" <<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">cyril.peponnet@alcatel-lucent.com</a>><br class="">
To: "Kotresh Hiremath Ravishankar" <<a href="mailto:khiremat@redhat.com" class="">khiremat@redhat.com</a>><br class="">
Cc: "gluster-users" <<a href="mailto:gluster-users@gluster.org" class="">gluster-users@gluster.org</a>><br class="">
Sent: Monday, June 1, 2015 10:34:42 PM<br class="">
Subject: Re: [Gluster-users] Geo-Replication - Changelog socket is not present - Falling back to xsync<br class="">
<br class="">
Some news,<br class="">
<br class="">
Looks like changelog is not working anymore. When I touch a file in master it<br class="">
doesnt propagate to slave…<br class="">
<br class="">
.processing folder contain a thousand of changelog not processed.<br class="">
<br class="">
I had to stop the geo-rep, reset changelog.changelog to the volume and<br class="">
restart the geo-rep. It’s now sending missing files using hybrid crawl.<br class="">
<br class="">
So geo-repo is not working as expected.<br class="">
<br class="">
Another thing, we use symlink to point to latest release build, and it seems<br class="">
that symlinks are not synced when they change from master to slave.<br class="">
<br class="">
Any idea on how I can debug this ?<br class="">
<br class="">
--<br class="">
Cyril Peponnet<br class="">
<br class="">
On May 29, 2015, at 3:01 AM, Kotresh Hiremath Ravishankar<br class="">
<<a href="mailto:khiremat@redhat.com" class="">khiremat@redhat.com</a><<a href="mailto:khiremat@redhat.com" class="">mailto:khiremat@redhat.com</a>>> wrote:<br class="">
<br class="">
Yes, geo-rep internally uses fuse mount.<br class="">
I will explore further and get back to you<br class="">
if there is a way.<br class="">
<br class="">
Thanks and Regards,<br class="">
Kotresh H R<br class="">
<br class="">
----- Original Message -----<br class="">
From: "Cyril N PEPONNET (Cyril)"<br class="">
<<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">cyril.peponnet@alcatel-lucent.com</a><<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">mailto:cyril.peponnet@alcatel-lucent.com</a>>><br class="">
To: "Kotresh Hiremath Ravishankar"<br class="">
<<a href="mailto:khiremat@redhat.com" class="">khiremat@redhat.com</a><<a href="mailto:khiremat@redhat.com" class="">mailto:khiremat@redhat.com</a>>><br class="">
Cc: "gluster-users"<br class="">
<<a href="mailto:gluster-users@gluster.org" class="">gluster-users@gluster.org</a><<a href="mailto:gluster-users@gluster.org" class="">mailto:gluster-users@gluster.org</a>>><br class="">
Sent: Thursday, May 28, 2015 10:12:57 PM<br class="">
Subject: Re: [Gluster-users] Geo-Replication - Changelog socket is not<br class="">
present - Falling back to xsync<br class="">
<br class="">
One more thing:<br class="">
<br class="">
nfs.volume-access read-only works only for nfs clients, glusterfs client have<br class="">
still write access<br class="">
<br class="">
features.read-only on need a vol restart and set RO for everyone but in this<br class="">
case, geo-rep goes faulty.<br class="">
<br class="">
[2015-05-28 09:42:27.917897] E [repce(/export/raid/usr_global):188:__call__]<br class="">
RepceClient: call 8739:139858642609920:1432831347.73 (keep_alive) failed on<br class="">
peer with OSError<br class="">
[2015-05-28 09:42:27.918102] E<br class="">
[syncdutils(/export/raid/usr_global):240:log_raise_exception] <top>: FAIL:<br class="">
Traceback (most recent call last):<br class="">
File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 266, in<br class="">
twrap<br class="">
tf(*aa)<br class="">
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 391, in<br class="">
keep_alive<br class="">
cls.slave.server.keep_alive(vi)<br class="">
File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 204, in<br class="">
__call__<br class="">
return self.ins(self.meth, *a)<br class="">
File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 189, in<br class="">
__call__<br class="">
raise res<br class="">
OSError: [Errno 30] Read-<br class="">
<br class="">
So there is no proper way to protect the salve against write.<br class="">
<br class="">
--<br class="">
Cyril Peponnet<br class="">
<br class="">
On May 28, 2015, at 8:54 AM, Cyril Peponnet<br class="">
<<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">cyril.peponnet@alcatel-lucent.com</a><<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">mailto:cyril.peponnet@alcatel-lucent.com</a>><<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">mailto:cyril.peponnet@alcatel-lucent.com</a>>><br class="">
wrote:<br class="">
<br class="">
Hi Kotresh,<br class="">
<br class="">
Inline.<br class="">
<br class="">
Again, thank for you time.<br class="">
<br class="">
--<br class="">
Cyril Peponnet<br class="">
<br class="">
On May 27, 2015, at 10:47 PM, Kotresh Hiremath Ravishankar<br class="">
<<a href="mailto:khiremat@redhat.com" class="">khiremat@redhat.com</a><<a href="mailto:khiremat@redhat.com" class="">mailto:khiremat@redhat.com</a>><<a href="mailto:khiremat@redhat.com" class="">mailto:khiremat@redhat.com</a>>><br class="">
wrote:<br class="">
<br class="">
Hi Cyril,<br class="">
<br class="">
Replies inline.<br class="">
<br class="">
Thanks and Regards,<br class="">
Kotresh H R<br class="">
<br class="">
----- Original Message -----<br class="">
From: "Cyril N PEPONNET (Cyril)"<br class="">
<<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">cyril.peponnet@alcatel-lucent.com</a><<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">mailto:cyril.peponnet@alcatel-lucent.com</a>><<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">mailto:cyril.peponnet@alcatel-lucent.com</a>>><br class="">
To: "Kotresh Hiremath Ravishankar"<br class="">
<<a href="mailto:khiremat@redhat.com" class="">khiremat@redhat.com</a><<a href="mailto:khiremat@redhat.com" class="">mailto:khiremat@redhat.com</a>><<a href="mailto:khiremat@redhat.com" class="">mailto:khiremat@redhat.com</a>>><br class="">
Cc: "gluster-users"<br class="">
<<a href="mailto:gluster-users@gluster.org" class="">gluster-users@gluster.org</a><<a href="mailto:gluster-users@gluster.org" class="">mailto:gluster-users@gluster.org</a>><<a href="mailto:gluster-users@gluster.org" class="">mailto:gluster-users@gluster.org</a>>><br class="">
Sent: Wednesday, May 27, 2015 9:28:00 PM<br class="">
Subject: Re: [Gluster-users] Geo-Replication - Changelog socket is not<br class="">
present - Falling back to xsync<br class="">
<br class="">
Hi and thanks again for those explanation.<br class="">
<br class="">
Due to lot of missing files and not up to date (with gfid mismatch some<br class="">
time), I reset the index (or I think I do) by:<br class="">
<br class="">
deleting the geo-reop, reset geo-replication.indexing (set it to off does not<br class="">
work for me), and recreate it again.<br class="">
<br class="">
Resetting index does not initiate geo-replication from the version changelog<br class="">
is<br class="">
introduced. It works only for the versions prior to it.<br class="">
<br class="">
NOTE 1: Recreation of geo-rep session will work only if slave doesn't contain<br class="">
file with mismatch gfids. If there are, slave should be cleaned up<br class="">
before recreating.<br class="">
<br class="">
I started it again to transfert missing files Ill take of gfid missmatch<br class="">
afterward. Our vol is almost 5TB and it took almost 2 month to crawl to the<br class="">
slave I did’nt want to start over :/<br class="">
<br class="">
<br class="">
NOTE 2: Another method exists now to initiate a full sync. It also expects<br class="">
slave<br class="">
files should not be in gfid mismatch state (meaning, slave volume<br class="">
should not<br class="">
written by any other means other than geo-replication). The method is<br class="">
to<br class="">
reset stime on all the bricks of master.<br class="">
<br class="">
<br class="">
Following are the steps to trigger full sync!!!. Let me know if any<br class="">
comments/doubts.<br class="">
================================================<br class="">
1. Stop geo-replication<br class="">
2. Remove stime extended attribute all the master brick root using<br class="">
following command.<br class="">
setfattr -x<br class="">
trusted.glusterfs.<MASTER_VOL_UUID>.<SLAVE_VOL_UUID>.stime<br class="">
<brick-root><br class="">
NOTE: 1. If AFR is setup, do this for all replicated set<br class="">
<br class="">
2. Above mentioned stime key can be got as follows:<br class="">
Using 'gluster volume info <mastervol>', get all brick<br class="">
paths and dump all the<br class="">
extended attributes, using 'getfattr -d -m . -e hex<br class="">
<brick-path>', which will<br class="">
dump stime key which should be removed.<br class="">
<br class="">
3. The technique, re-triggers complete sync. It involves<br class="">
complete xsync crawl.<br class="">
If there are rename issues, it might hit the rsync error<br class="">
on complete re-sync as well.<br class="">
So it is recommended, if the problematic files on slaves<br class="">
are known, remove them and initiate<br class="">
complete sync.<br class="">
<br class="">
Is complete sync will send again the data if present of not ? How to track<br class="">
down rename issue ? master is a living volume with lot of creation / rename<br class="">
/ deletion.<br class="">
<br class="">
<br class="">
3. Start geo-replicatoin.<br class="">
<br class="">
The above technique can also be used to trigger data sync only on one<br class="">
particular brick.<br class="">
Just removing stime extended attribute only on brick root of master<br class="">
to be synced will<br class="">
do. If AFR is setup, remove stime on all replicated set of bricks.<br class="">
<br class="">
================================<br class="">
<br class="">
<br class="">
So for now it’s still in hybrid crawl process.<br class="">
<br class="">
I end up with that because some entire folder where not synced up by the<br class="">
first hybrid crawl (and touch does nothing afterward in changelog). In fact<br class="">
touch anyfile doesnt trigger any resync, only delete/rename/change do.<br class="">
<br class="">
<br class="">
In newer geo-replication, from the version history crawl is introduced,<br class="">
xsync<br class="">
crawl is minimized. Once it reaches the timestamp where it gets the<br class="">
historical changelogs,<br class="">
it starts using history changelogs. Touch will be recorded as SETATTR in<br class="">
Changelog so<br class="">
Geo-rep will not sync the data. So the new virtual setattr interface is<br class="">
introduced<br class="">
which is mentioned in previous mail.<br class="">
<br class="">
1/<br class="">
1. Directories:<br class="">
#setfattr -n glusterfs.geo-rep.trigger-sync -v "1" <DIR><br class="">
2. Files:<br class="">
#setfattr -n glusterfs.geo-rep.trigger-sync -v “1" <file-path><br class="">
<br class="">
Is is recursive ? (for directories) or I have to do that on each mismatching<br class="">
files ? Should I do that on master or slave ?<br class="">
<br class="">
<br class="">
No, it is not recursive, it should be done for every missing files and<br class="">
directories.<br class="">
And directories should be done before the files inside it.<br class="">
It should be done on master.<br class="">
<br class="">
<br class="">
I don’t understand the difference between setfattr -n<br class="">
glusterfs.geo-rep.trigger-sync -v “1” <DIR> (vol level) and setfattr -x<br class="">
trusted.glusterfs.<MASTER_VOL_UUID>.<SLAVE_VOL_UUID>.stime <brick-root><br class="">
(brick level)<br class="">
<br class="">
<br class="">
2/ For the RO I can pass the Option: nfs.volume-access to read-only, this<br class="">
will pass the vol in RO for nfs mount and glusterfs mount. Correct ?<br class="">
<br class="">
Yes, that should do.<br class="">
<br class="">
Cool ! Thanks!<br class="">
<br class="">
<br class="">
Thank you so much for your help.<br class="">
--<br class="">
Cyril Peponnet<br class="">
<br class="">
On May 26, 2015, at 11:29 PM, Kotresh Hiremath Ravishankar<br class="">
<<a href="mailto:khiremat@redhat.com" class="">khiremat@redhat.com</a><<a href="mailto:khiremat@redhat.com" class="">mailto:khiremat@redhat.com</a>><<a href="mailto:khiremat@redhat.com" class="">mailto:khiremat@redhat.com</a>>><br class="">
wrote:<br class="">
<br class="">
Hi Cyril,<br class="">
<br class="">
Need some clarifications. Comments inline.<br class="">
<br class="">
Thanks and Regards,<br class="">
Kotresh H R<br class="">
<br class="">
----- Original Message -----<br class="">
From: "Cyril N PEPONNET (Cyril)"<br class="">
<<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">cyril.peponnet@alcatel-lucent.com</a><<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">mailto:cyril.peponnet@alcatel-lucent.com</a>><<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">mailto:cyril.peponnet@alcatel-lucent.com</a>>><br class="">
To: "Kotresh Hiremath Ravishankar"<br class="">
<<a href="mailto:khiremat@redhat.com" class="">khiremat@redhat.com</a><<a href="mailto:khiremat@redhat.com" class="">mailto:khiremat@redhat.com</a>><<a href="mailto:khiremat@redhat.com" class="">mailto:khiremat@redhat.com</a>>><br class="">
Cc: "gluster-users"<br class="">
<<a href="mailto:gluster-users@gluster.org" class="">gluster-users@gluster.org</a><<a href="mailto:gluster-users@gluster.org" class="">mailto:gluster-users@gluster.org</a>><<a href="mailto:gluster-users@gluster.org" class="">mailto:gluster-users@gluster.org</a>>><br class="">
Sent: Tuesday, May 26, 2015 11:43:44 PM<br class="">
Subject: Re: [Gluster-users] Geo-Replication - Changelog socket is not<br class="">
present - Falling back to xsync<br class="">
<br class="">
So, changelog is still active but I notice that some file were missing.<br class="">
<br class="">
So I ‘m running a rsync -avn between the two vol (master and slave) to<br class="">
sync<br class="">
then again by touching the missing files (hopping geo-rep will do the<br class="">
rest).<br class="">
<br class="">
Are you running rsync -avn for missed files between master and slave<br class="">
volumes ?<br class="">
If yes, that is dangerous and it should not be done. Geo-replication<br class="">
demands gfid<br class="">
of files between master and slave to be intact (meaning the gfid of<br class="">
'file1' in<br class="">
master vol should be same as 'file1' in slave). It is required because,<br class="">
the data sync<br class="">
happens using 'gfid' not the 'pathname' of the file. So if manual rsync is<br class="">
used<br class="">
to sync files between master and slave using pathname, gfids will change<br class="">
and<br class="">
further syncing on those files fails through geo-rep.<br class="">
<br class="">
A virtual setxattr interface is provided to sync missing files through<br class="">
geo-replication.<br class="">
It makes sure gfids are intact.<br class="">
<br class="">
NOTE: Directories have to be synced to slave before trying setxattr for<br class="">
files inside it.<br class="">
<br class="">
1. Directories:<br class="">
#setfattr -n glusterfs.geo-rep.trigger-sync -v "1" <DIR><br class="">
2. Files:<br class="">
#setfattr -n glusterfs.geo-rep.trigger-sync -v "1" <file-path><br class="">
<br class="">
One question, can I pass the slave vol a RO ? Because if somebody change a<br class="">
file in the slave it’s no longer synced (changes and delete but rename<br class="">
keep<br class="">
synced between master and slave).<br class="">
<br class="">
Will it have an impact on geo-replication process if I pass the slave vol<br class="">
a<br class="">
RO ?<br class="">
<br class="">
Again if slave volume is modified by something else other than geo-rep, we<br class="">
might<br class="">
end up in mismatch of gfids. So exposing the slave volume to consumers as<br class="">
RO is always<br class="">
a good idea. It doesn't affect geo-rep as it internally mounts in RW.<br class="">
<br class="">
Hope this helps. Let us know if anything else. We are happy to help you.<br class="">
<br class="">
Thanks again.<br class="">
<br class="">
<br class="">
--<br class="">
Cyril Peponnet<br class="">
<br class="">
On May 25, 2015, at 12:43 AM, Kotresh Hiremath Ravishankar<br class="">
<<a href="mailto:khiremat@redhat.com" class="">khiremat@redhat.com</a><<a href="mailto:khiremat@redhat.com" class="">mailto:khiremat@redhat.com</a>><<a href="mailto:khiremat@redhat.com" class="">mailto:khiremat@redhat.com</a>><<a href="mailto:khiremat@redhat.com" class="">mailto:khiremat@redhat.com</a>>><br class="">
wrote:<br class="">
<br class="">
Hi Cyril,<br class="">
<br class="">
Answers inline<br class="">
<br class="">
Thanks and Regards,<br class="">
Kotresh H R<br class="">
<br class="">
----- Original Message -----<br class="">
From: "Cyril N PEPONNET (Cyril)"<br class="">
<<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">cyril.peponnet@alcatel-lucent.com</a><<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">mailto:cyril.peponnet@alcatel-lucent.com</a>><<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">mailto:cyril.peponnet@alcatel-lucent.com</a>><<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">mailto:cyril.peponnet@alcatel-lucent.com</a>>><br class="">
To: "Kotresh Hiremath Ravishankar"<br class="">
<<a href="mailto:khiremat@redhat.com" class="">khiremat@redhat.com</a><<a href="mailto:khiremat@redhat.com" class="">mailto:khiremat@redhat.com</a>><<a href="mailto:khiremat@redhat.com" class="">mailto:khiremat@redhat.com</a>><<a href="mailto:khiremat@redhat.com" class="">mailto:khiremat@redhat.com</a>>><br class="">
Cc: "gluster-users"<br class="">
<<a href="mailto:gluster-users@gluster.org" class="">gluster-users@gluster.org</a><<a href="mailto:gluster-users@gluster.org" class="">mailto:gluster-users@gluster.org</a>><<a href="mailto:gluster-users@gluster.org" class="">mailto:gluster-users@gluster.org</a>><<a href="mailto:gluster-users@gluster.org" class="">mailto:gluster-users@gluster.org</a>>><br class="">
Sent: Friday, May 22, 2015 9:34:47 PM<br class="">
Subject: Re: [Gluster-users] Geo-Replication - Changelog socket is not<br class="">
present - Falling back to xsync<br class="">
<br class="">
One last question, correct me if I’m wrong.<br class="">
<br class="">
When you start a geo-rep process it starts with xsync aka hybrid crawling<br class="">
(sending files every 60s, with files windows set as 8192 files per sent).<br class="">
<br class="">
When the crawl is done it should use changelog detector and dynamically<br class="">
change things to slaves.<br class="">
<br class="">
1/ During the hybride crawl, if we delete files from master (and they were<br class="">
already transfered to the slave), xsync process will not delete them from<br class="">
the slave (and we can’t change as the option as is hardcoded).<br class="">
When it will pass to changelog, will it remove the non existent folders<br class="">
and<br class="">
files on the slave that are no longer on the master ?<br class="">
<br class="">
<br class="">
You are right, xsync does not sync delete files, once it is already<br class="">
synced.<br class="">
After xsync, when it switches to changelog, it doesn't delete all the non<br class="">
existing<br class="">
entries on slave that are no longer on the master. Changelog is capable of<br class="">
deleting<br class="">
files from the time it got switched to changelog.<br class="">
<br class="">
2/ With changelog, if I add a file of 10GB and after a file of 1KB, will<br class="">
the<br class="">
changelog process with queue (waiting for the 10GB file to be sent) or are<br class="">
the sent done in thread ?<br class="">
(ex I add a 10GB file and I delete it after 1min, what will happen ?)<br class="">
<br class="">
Changelog records the operations happened in master and is replayed by<br class="">
geo-replication<br class="">
on to slave volume. Geo-replication syncs files in two phases.<br class="">
<br class="">
1. Phase-1: Create entries through RPC( 0 byte files on slave keeping<br class="">
gfid<br class="">
intact as in master)<br class="">
2. Phase-2: Sync data, through rsync/tar_over_ssh (Multi threaded)<br class="">
<br class="">
Ok, now keeping that in mind, Phase-1 happens serially, and the phase two<br class="">
happens parallely.<br class="">
Zero byte files of 10GB and 1KB gets created on slave serially and data<br class="">
for<br class="">
the same syncs<br class="">
parallely. Another thing to remember, geo-rep makes sure that, syncing<br class="">
data<br class="">
to file is tried<br class="">
only after zero byte file for the same is created already.<br class="">
<br class="">
<br class="">
In latest release 3.7, xsync crawl is minimized by the feature called<br class="">
history<br class="">
crawl introduced in 3.6.<br class="">
So the chances of missing deletes/renames are less.<br class="">
<br class="">
Thanks.<br class="">
<br class="">
--<br class="">
Cyril Peponnet<br class="">
<br class="">
On May 21, 2015, at 10:22 PM, Kotresh Hiremath Ravishankar<br class="">
<<a href="mailto:khiremat@redhat.com" class="">khiremat@redhat.com</a><<a href="mailto:khiremat@redhat.com" class="">mailto:khiremat@redhat.com</a>><<a href="mailto:khiremat@redhat.com" class="">mailto:khiremat@redhat.com</a>><<a href="mailto:khiremat@redhat.com" class="">mailto:khiremat@redhat.com</a>>><br class="">
wrote:<br class="">
<br class="">
Great, hope that should work. Let's see<br class="">
<br class="">
Thanks and Regards,<br class="">
Kotresh H R<br class="">
<br class="">
----- Original Message -----<br class="">
From: "Cyril N PEPONNET (Cyril)"<br class="">
<<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">cyril.peponnet@alcatel-lucent.com</a><<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">mailto:cyril.peponnet@alcatel-lucent.com</a>><<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">mailto:cyril.peponnet@alcatel-lucent.com</a>><<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">mailto:cyril.peponnet@alcatel-lucent.com</a>>><br class="">
To: "Kotresh Hiremath Ravishankar"<br class="">
<<a href="mailto:khiremat@redhat.com" class="">khiremat@redhat.com</a><<a href="mailto:khiremat@redhat.com" class="">mailto:khiremat@redhat.com</a>><<a href="mailto:khiremat@redhat.com" class="">mailto:khiremat@redhat.com</a>><<a href="mailto:khiremat@redhat.com" class="">mailto:khiremat@redhat.com</a>>><br class="">
Cc: "gluster-users"<br class="">
<<a href="mailto:gluster-users@gluster.org" class="">gluster-users@gluster.org</a><<a href="mailto:gluster-users@gluster.org" class="">mailto:gluster-users@gluster.org</a>><<a href="mailto:gluster-users@gluster.org" class="">mailto:gluster-users@gluster.org</a>><<a href="mailto:gluster-users@gluster.org" class="">mailto:gluster-users@gluster.org</a>>><br class="">
Sent: Friday, May 22, 2015 5:31:13 AM<br class="">
Subject: Re: [Gluster-users] Geo-Replication - Changelog socket is not<br class="">
present - Falling back to xsync<br class="">
<br class="">
Thanks to JoeJulian / Kaushal I managed to re-enable the changelog option<br class="">
and<br class="">
the socket is now present.<br class="">
<br class="">
For the record I had some clients running rhs gluster-fuse and our nodes<br class="">
are<br class="">
running glusterfs release and op-version are not “compatible”.<br class="">
<br class="">
Now I have to wait for the init crawl see if it switches to changelog<br class="">
detector mode.<br class="">
<br class="">
Thanks Kotresh<br class="">
--<br class="">
Cyril Peponnet<br class="">
<br class="">
On May 21, 2015, at 8:39 AM, Cyril Peponnet<br class="">
<<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">cyril.peponnet@alcatel-lucent.com</a><<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">mailto:cyril.peponnet@alcatel-lucent.com</a>><<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">mailto:cyril.peponnet@alcatel-lucent.com</a>><<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">mailto:cyril.peponnet@alcatel-lucent.com</a>>><br class="">
wrote:<br class="">
<br class="">
Hi,<br class="">
<br class="">
Unfortunately,<br class="">
<br class="">
# gluster vol set usr_global changelog.changelog off<br class="">
volume set: failed: Staging failed on<br class="">
<a href="http://mvdcgluster01.us.alcatel-lucent.com/" class="">mvdcgluster01.us.alcatel-lucent.com</a><<a href="http://mvdcgluster01.us.alcatel-lucent.com/" class="">http://mvdcgluster01.us.alcatel-lucent.com/</a>><<a href="http://mvdcgluster01.us.alcatel-lucent.com<http://mvdcgluster01.us.alcatel-lucent.com/>><http://mvdcgluster01.us.alcatel-lucent.com<http://mvdcgluster01.us.alcatel-lucent.com/>" class="">http://mvdcgluster01.us.alcatel-lucent.com<http://mvdcgluster01.us.alcatel-lucent.com/>><http://mvdcgluster01.us.alcatel-lucent.com<http://mvdcgluster01.us.alcatel-lucent.com/></a>>.<br class="">
Error: One or more connected clients cannot support the feature being<br class="">
set.<br class="">
These clients need to be upgraded or disconnected before running this<br class="">
command again<br class="">
<br class="">
<br class="">
I don’t know really why, I have some clients using 3.6 as fuse client<br class="">
others are running on 3.5.2.<br class="">
<br class="">
Any advice ?<br class="">
<br class="">
--<br class="">
Cyril Peponnet<br class="">
<br class="">
On May 20, 2015, at 5:17 AM, Kotresh Hiremath Ravishankar<br class="">
<<a href="mailto:khiremat@redhat.com" class="">khiremat@redhat.com</a><<a href="mailto:khiremat@redhat.com" class="">mailto:khiremat@redhat.com</a>><<a href="mailto:khiremat@redhat.com" class="">mailto:khiremat@redhat.com</a>><<a href="mailto:khiremat@redhat.com" class="">mailto:khiremat@redhat.com</a>>><br class="">
wrote:<br class="">
<br class="">
Hi Cyril,<br class="">
<br class="">
From the brick logs, it seems the changelog-notifier thread has got<br class="">
killed<br class="">
for some reason,<br class="">
as notify is failing with EPIPE.<br class="">
<br class="">
Try the following. It should probably help:<br class="">
1. Stop geo-replication.<br class="">
2. Disable changelog: gluster vol set <master-vol-name><br class="">
changelog.changelog off<br class="">
3. Enable changelog: glluster vol set <master-vol-name><br class="">
changelog.changelog on<br class="">
4. Start geo-replication.<br class="">
<br class="">
Let me know if it works.<br class="">
<br class="">
Thanks and Regards,<br class="">
Kotresh H R<br class="">
<br class="">
----- Original Message -----<br class="">
From: "Cyril N PEPONNET (Cyril)"<br class="">
<<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">cyril.peponnet@alcatel-lucent.com</a><<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">mailto:cyril.peponnet@alcatel-lucent.com</a>><<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">mailto:cyril.peponnet@alcatel-lucent.com</a>><<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">mailto:cyril.peponnet@alcatel-lucent.com</a>>><br class="">
To: "gluster-users"<br class="">
<<a href="mailto:gluster-users@gluster.org" class="">gluster-users@gluster.org</a><<a href="mailto:gluster-users@gluster.org" class="">mailto:gluster-users@gluster.org</a>><<a href="mailto:gluster-users@gluster.org" class="">mailto:gluster-users@gluster.org</a>><<a href="mailto:gluster-users@gluster.org" class="">mailto:gluster-users@gluster.org</a>>><br class="">
Sent: Tuesday, May 19, 2015 3:16:22 AM<br class="">
Subject: [Gluster-users] Geo-Replication - Changelog socket is not<br class="">
present - Falling back to xsync<br class="">
<br class="">
Hi Gluster Community,<br class="">
<br class="">
I have a 3 nodes setup at location A and a two node setup at location<br class="">
B.<br class="">
<br class="">
All running 3.5.2 under Centos-7.<br class="">
<br class="">
I have one volume I sync through georeplication process.<br class="">
<br class="">
So far so good, the first step of geo-replication is done<br class="">
(hybrid-crawl).<br class="">
<br class="">
Now I’d like to use the change log detector in order to delete files on<br class="">
the<br class="">
slave when they are gone on master.<br class="">
<br class="">
But it always fallback to xsync mecanism (even when I force it using<br class="">
config<br class="">
changelog_detector changelog):<br class="">
<br class="">
[2015-05-18 12:29:49.543922] I [monitor(monitor):129:monitor] Monitor:<br class="">
------------------------------------------------------------<br class="">
[2015-05-18 12:29:49.544018] I [monitor(monitor):130:monitor] Monitor:<br class="">
starting gsyncd worker<br class="">
[2015-05-18 12:29:49.614002] I [gsyncd(/export/raid/vol):532:main_i]<br class="">
<top>:<br class="">
syncing: <a href="gluster://localhost:vol" class="">gluster://localhost:vol</a> -><br class="">
<a href="ssh://root@x.x.x.x:gluster://localhost:vol" class="">ssh://root@x.x.x.x:gluster://localhost:vol</a><br class="">
[2015-05-18 12:29:54.696532] I<br class="">
[master(/export/raid/vol):58:gmaster_builder]<br class="">
<top>: setting up xsync change detection mode<br class="">
[2015-05-18 12:29:54.696888] I [master(/export/raid/vol):357:__init__]<br class="">
_GMaster: using 'rsync' as the sync engine<br class="">
[2015-05-18 12:29:54.697930] I<br class="">
[master(/export/raid/vol):58:gmaster_builder]<br class="">
<top>: setting up changelog change detection mode<br class="">
[2015-05-18 12:29:54.698160] I [master(/export/raid/vol):357:__init__]<br class="">
_GMaster: using 'rsync' as the sync engine<br class="">
[2015-05-18 12:29:54.699239] I [master(/export/raid/vol):1104:register]<br class="">
_GMaster: xsync temp directory:<br class="">
/var/run/gluster/vol/ssh%3A%2F%2Froot%40x.x.x.x%3Agluster%3A%2F%2F127.0.0.1%3Avol/ce749a38ba30d4171cd674ec00ab24f9/xsync<br class="">
[2015-05-18 12:30:04.707216] I<br class="">
[master(/export/raid/vol):682:fallback_xsync]<br class="">
_GMaster: falling back to xsync mode<br class="">
[2015-05-18 12:30:04.742422] I<br class="">
[syncdutils(/export/raid/vol):192:finalize]<br class="">
<top>: exiting.<br class="">
[2015-05-18 12:30:05.708123] I [monitor(monitor):157:monitor] Monitor:<br class="">
worker(/export/raid/vol) died in startup phase<br class="">
[2015-05-18 12:30:05.708369] I [monitor(monitor):81:set_state] Monitor:<br class="">
new<br class="">
state: faulty<br class="">
[201<br class="">
<br class="">
After some python debugging and stack strace printing I figure out<br class="">
that:<br class="">
<br class="">
/var/run/gluster/vol/ssh%3A%2F%2Froot%40x.x.x.x%3Agluster%3A%2F%2F127.0.0.1%3Avol/ce749a38ba30d4171cd674ec00ab24f9/changes.log<br class="">
<br class="">
[2015-05-18 19:41:24.511423] I<br class="">
[gf-changelog.c:179:gf_changelog_notification_init] 0-glusterfs:<br class="">
connecting<br class="">
to changelog socket:<br class="">
/var/run/gluster/changelog-ce749a38ba30d4171cd674ec00ab24f9.sock<br class="">
(brick:<br class="">
/export/raid/vol)<br class="">
[2015-05-18 19:41:24.511445] W<br class="">
[gf-changelog.c:189:gf_changelog_notification_init] 0-glusterfs:<br class="">
connection<br class="">
attempt 1/5...<br class="">
[2015-05-18 19:41:26.511556] W<br class="">
[gf-changelog.c:189:gf_changelog_notification_init] 0-glusterfs:<br class="">
connection<br class="">
attempt 2/5...<br class="">
[2015-05-18 19:41:28.511670] W<br class="">
[gf-changelog.c:189:gf_changelog_notification_init] 0-glusterfs:<br class="">
connection<br class="">
attempt 3/5...<br class="">
[2015-05-18 19:41:30.511790] W<br class="">
[gf-changelog.c:189:gf_changelog_notification_init] 0-glusterfs:<br class="">
connection<br class="">
attempt 4/5...<br class="">
[2015-05-18 19:41:32.511890] W<br class="">
[gf-changelog.c:189:gf_changelog_notification_init] 0-glusterfs:<br class="">
connection<br class="">
attempt 5/5...<br class="">
[2015-05-18 19:41:34.512016] E<br class="">
[gf-changelog.c:204:gf_changelog_notification_init] 0-glusterfs: could<br class="">
not<br class="">
connect to changelog socket! bailing out...<br class="">
<br class="">
<br class="">
/var/run/gluster/changelog-ce749a38ba30d4171cd674ec00ab24f9.sock<br class="">
doesn’t<br class="">
exist. So the<br class="">
https://github.com/gluster/glusterfs/blob/release-3.5/xlators/features/changelog/lib/src/gf-changelog.c#L431<br class="">
is failing because<br class="">
https://github.com/gluster/glusterfs/blob/release-3.5/xlators/features/changelog/lib/src/gf-changelog.c#L153<br class="">
cannot open the socket file.<br class="">
<br class="">
And I don’t find any error related to changelog in log files, except on<br class="">
brick<br class="">
logs node 2 (site A)<br class="">
<br class="">
bricks/export-raid-vol.log-20150517:[2015-05-14 17:06:52.636908] E<br class="">
[changelog-helpers.c:168:changelog_rollover_changelog] 0-vol-changelog:<br class="">
Failed to send file name to notify thread (reason: Broken pipe)<br class="">
bricks/export-raid-vol.log-20150517:[2015-05-14 17:06:52.636949] E<br class="">
[changelog-helpers.c:280:changelog_handle_change] 0-vol-changelog:<br class="">
Problem<br class="">
rolling over changelog(s)<br class="">
<br class="">
gluster vol status is all fine, and change-log options are enabled in<br class="">
vol<br class="">
file<br class="">
<br class="">
volume vol-changelog<br class="">
type features/changelog<br class="">
option changelog on<br class="">
option changelog-dir /export/raid/vol/.glusterfs/changelogs<br class="">
option changelog-brick /export/raid/vol<br class="">
subvolumes vol-posix<br class="">
end-volume<br class="">
<br class="">
Any help will be appreciated :)<br class="">
<br class="">
Oh Btw, hard to stop / restart the volume as I have around 4k clients<br class="">
connected.<br class="">
<br class="">
Thanks !<br class="">
<br class="">
--<br class="">
Cyril Peponnet<br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
Gluster-users mailing list<br class="">
Gluster-users@gluster.org<br class="">
http://www.gluster.org/mailman/listinfo/gluster-users</blockquote>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>