<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class="">Hi Gluster Community,</div>
<div class=""><br class="">
</div>
<div class="">I have a 3 nodes setup at location A and a two node setup at location B.</div>
<div class=""><br class="">
</div>
<div class="">All running 3.5.2 under Centos-7.</div>
<div class=""><br class="">
</div>
<div class="">I have one volume I sync through georeplication process.</div>
<div class=""><br class="">
</div>
<div class="">So far so good, the first step of geo-replication is done (hybrid-crawl).</div>
<div class=""><br class="">
</div>
<div class="">Now I’d like to use the change log detector in order to delete files on the slave when they are gone on master.</div>
<div class=""><br class="">
</div>
<div class="">But it always fallback to xsync mecanism (even when I force it using config changelog_detector changelog):</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">[2015-05-18 12:29:49.543922] I [monitor(monitor):129:monitor] Monitor: ------------------------------------------------------------</div>
<div class="">[2015-05-18 12:29:49.544018] I [monitor(monitor):130:monitor] Monitor: starting gsyncd worker</div>
<div class="">[2015-05-18 12:29:49.614002] I [gsyncd(/export/raid/vol):532:main_i] <top>: syncing:
<a href="gluster://localhost:vol" class="">gluster://localhost:vol</a> -> <a href="ssh://root@x.x.x.x:gluster://localhost:vol" class="">
ssh://root@x.x.x.x:gluster://localhost:vol</a></div>
<div class="">[2015-05-18 12:29:54.696532] I [master(/export/raid/vol):58:gmaster_builder] <top>: setting up xsync change detection mode</div>
<div class="">[2015-05-18 12:29:54.696888] I [master(/export/raid/vol):357:__init__] _GMaster: using 'rsync' as the sync engine</div>
<div class="">[2015-05-18 12:29:54.697930] I [master(/export/raid/vol):58:gmaster_builder] <top>: setting up changelog change detection mode</div>
<div class="">[2015-05-18 12:29:54.698160] I [master(/export/raid/vol):357:__init__] _GMaster: using 'rsync' as the sync engine</div>
<div class="">[2015-05-18 12:29:54.699239] I [master(/export/raid/vol):1104:register] _GMaster: xsync temp directory: /var/run/gluster/vol/ssh%3A%2F%2Froot%40x.x.x.x%3Agluster%3A%2F%2F127.0.0.1%3Avol/ce749a38ba30d4171cd674ec00ab24f9/xsync</div>
<div class="">[2015-05-18 12:30:04.707216] I [master(/export/raid/vol):682:fallback_xsync] _GMaster: falling back to xsync mode</div>
<div class="">[2015-05-18 12:30:04.742422] I [syncdutils(/export/raid/vol):192:finalize] <top>: exiting.</div>
<div class="">[2015-05-18 12:30:05.708123] I [monitor(monitor):157:monitor] Monitor: worker(/export/raid/vol) died in startup phase</div>
<div class="">[2015-05-18 12:30:05.708369] I [monitor(monitor):81:set_state] Monitor: new state: faulty</div>
<div class="">[201</div>
</div>
<div class=""><br class="">
</div>
<div class="">After some python debugging and stack strace printing I figure out that:</div>
<div class=""><br class="">
</div>
<div class="">/var/run/gluster/vol/ssh%3A%2F%2Froot%40x.x.x.x%3Agluster%3A%2F%2F127.0.0.1%3Avol/ce749a38ba30d4171cd674ec00ab24f9/changes.log</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">[2015-05-18 19:41:24.511423] I [gf-changelog.c:179:gf_changelog_notification_init] 0-glusterfs: connecting to changelog socket: /var/run/gluster/changelog-ce749a38ba30d4171cd674ec00ab24f9.sock (brick: /export/raid/vol)</div>
<div class="">
<div class="">[2015-05-18 19:41:24.511445] W [gf-changelog.c:189:gf_changelog_notification_init] 0-glusterfs: connection attempt 1/5...</div>
<div class="">[2015-05-18 19:41:26.511556] W [gf-changelog.c:189:gf_changelog_notification_init] 0-glusterfs: connection attempt 2/5...</div>
<div class="">[2015-05-18 19:41:28.511670] W [gf-changelog.c:189:gf_changelog_notification_init] 0-glusterfs: connection attempt 3/5...</div>
<div class="">[2015-05-18 19:41:30.511790] W [gf-changelog.c:189:gf_changelog_notification_init] 0-glusterfs: connection attempt 4/5...</div>
<div class="">[2015-05-18 19:41:32.511890] W [gf-changelog.c:189:gf_changelog_notification_init] 0-glusterfs: connection attempt 5/5...</div>
<div class="">[2015-05-18 19:41:34.512016] E [gf-changelog.c:204:gf_changelog_notification_init] 0-glusterfs: could not connect to changelog socket! bailing out...</div>
</div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
/var/run/gluster/changelog-ce749a38ba30d4171cd674ec00ab24f9.sock doesn’t exist. So the <a href="https://github.com/gluster/glusterfs/blob/release-3.5/xlators/features/changelog/lib/src/gf-changelog.c#L431" class="">https://github.com/gluster/glusterfs/blob/release-3.5/xlators/features/changelog/lib/src/gf-changelog.c#L431</a> is
failing because <a href="https://github.com/gluster/glusterfs/blob/release-3.5/xlators/features/changelog/lib/src/gf-changelog.c#L153" class="">https://github.com/gluster/glusterfs/blob/release-3.5/xlators/features/changelog/lib/src/gf-changelog.c#L153</a> cannot
open the socket file.
<div class=""><br class="">
</div>
<div class="">And I don’t find any error related to changelog in log files, except on brick logs node 2 (site A)</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">bricks/export-raid-vol.log-20150517:[2015-05-14 17:06:52.636908] E [changelog-helpers.c:168:changelog_rollover_changelog] 0-vol-changelog: Failed to send file name to notify thread (reason: Broken pipe)</div>
<div class="">bricks/export-raid-vol.log-20150517:[2015-05-14 17:06:52.636949] E [changelog-helpers.c:280:changelog_handle_change] 0-vol-changelog: Problem rolling over changelog(s)</div>
</div>
<div class=""><br class="">
</div>
<div class="">gluster vol status is all fine, and change-log options are enabled in vol file</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">volume vol-changelog</div>
<div class=""> type features/changelog</div>
<div class=""> option changelog on</div>
<div class=""> option changelog-dir /export/raid/vol/.glusterfs/changelogs</div>
<div class=""> option changelog-brick /export/raid/vol</div>
<div class=""> subvolumes vol-posix</div>
<div class="">end-volume</div>
</div>
<div class=""><br class="">
</div>
<div class="">Any help will be appreciated :)</div>
<div class=""><br class="">
</div>
<div class="">Oh Btw, hard to stop / restart the volume as I have around 4k clients connected.</div>
<div class=""><br class="">
</div>
<div class="">Thanks !</div>
<div class=""><br class="">
<div class="">
<div class="">-- <br class="">
Cyril Peponnet </div>
<br class="">
</div>
</div>
</body>
</html>