<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hi Kris,<br>
<br>
These failures are mostly while transferring data from Master to
Slave. Geo-replication is not intelligent to find exact number and
reason for failures. I sent patch to effectively handle the
rsync/tar failures. This patch is expected to be available with
3.7.8 release.<br>
<a class="moz-txt-link-freetext" href="http://review.gluster.org/#/c/12856/">http://review.gluster.org/#/c/12856/</a><br>
<br>
Above patch identifies the reason for failure and logs in
/var/log/glusterfs/geo-replication/<MASTERVOL>/*.log. <br>
<br>
Answers inline.<br>
<pre class="moz-signature" cols="72">regards
Aravinda</pre>
<div class="moz-cite-prefix">On 02/03/2016 07:16 AM, Kris Laib
wrote:<br>
</div>
<blockquote
cite="mid:BLUPR06MB1796DF463BC5974E5CEB14C398D00@BLUPR06MB1796.namprd06.prod.outlook.com"
type="cite">
<pre wrap="">I recently setup geo-replication between two sites, and everything appears to be working, however, if I do a "gluster volume geo-replication SourceVol DestVol status detail", I see that the "failures" column seems to be incrementing for each file it transfers. I've looked at the logs in /var/log/glusterfs/geo-replication/SourceVol/* , but nothing stood out to me.
I've verified that the MD5 sum matches on the files at the destination after it's been replicated, so I'm unsure what is triggering the failure. Aside from the incrementing failure column, it all looks good to me.</pre>
</blockquote>
Looks like these issues can be ignored. Errors may be due to xattrs
syncing. With the above patch, logging is improved to log the
failure reasons.<br>
<blockquote
cite="mid:BLUPR06MB1796DF463BC5974E5CEB14C398D00@BLUPR06MB1796.namprd06.prod.outlook.com"
type="cite">
<pre wrap="">
All boxes in both clusters are CentOS 6.7, running glusterfs 3.7.5
My questions:
1) What's causing the failures to be logged (please let me know what additional logs to provide)?</pre>
</blockquote>
Possible reasons for Sync failures,<br>
- May be failure to sync xattrs/acls to Slave.<br>
- I/O error on Slave<br>
- Files unlinked in Slave<br>
- GFID Conflict<br>
<blockquote
cite="mid:BLUPR06MB1796DF463BC5974E5CEB14C398D00@BLUPR06MB1796.namprd06.prod.outlook.com"
type="cite">
<pre wrap="">
2) How do I reset the failures counter? It stays at the current value even once the files are deleted from the source</pre>
</blockquote>
Will fix this issue in upcoming releases.<br>
<blockquote
cite="mid:BLUPR06MB1796DF463BC5974E5CEB14C398D00@BLUPR06MB1796.namprd06.prod.outlook.com"
type="cite">
<pre wrap="">
3) Is there documentation somewhere I missed that explains what Entry, Data, Meta, and Failures are in the status detail view for geo-replication?</pre>
</blockquote>
Entry - Pending Entry operations - Entry operations like CREATE,
RENAME, UNLINK etc, which are executed as RPC calls.<br>
Data - Pending Data operations - Rsync/Tar is used to transfer data
from Master to Slave.<br>
Meta - Pending Meta operations - Changing ownership, permissions are
executed as RPC calls.<br>
<br>
Failures - Going to be obsolete since Errors will be logged in log
files.<br>
<blockquote
cite="mid:BLUPR06MB1796DF463BC5974E5CEB14C398D00@BLUPR06MB1796.namprd06.prod.outlook.com"
type="cite">
<pre wrap="">
4) Should "gluster volume geo-replication status" show me anything on the destination? Mine simply outputs "No active geo-replication sessions", though the command on the source works fine.</pre>
</blockquote>
Geo-replication status only available from Master.<br>
<br>
<blockquote
cite="mid:BLUPR06MB1796DF463BC5974E5CEB14C398D00@BLUPR06MB1796.namprd06.prod.outlook.com"
type="cite">
<pre wrap="">
Thanks,
Kris
-----------
Here's an example of what I'm seeing:
[<a class="moz-txt-link-abbreviated" href="mailto:root@gfs-prd02.mgmt">root@gfs-prd02.mgmt</a> temp]$ mount | grep "gfs-prd.mgmt:/prd_items"
gfs-prd.mgmt:/prd_items on /mnt/temp type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)
[<a class="moz-txt-link-abbreviated" href="mailto:root@gfs-prd02.mgmt">root@gfs-prd02.mgmt</a> temp]$ pwd
/mnt/temp
[<a class="moz-txt-link-abbreviated" href="mailto:root@gfs-prd02.mgmt">root@gfs-prd02.mgmt</a> temp]$ gluster volume geo-replication prd_items <a class="moz-txt-link-abbreviated" href="mailto:georep@gfs-slc01.mgmt::dr_items">georep@gfs-slc01.mgmt::dr_items</a> status detail
MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ENTRY DATA META FAILURES CHECKPOINT TIME CHECKPOINT COMPLETED CHECKPOINT COMPLETION TIME
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
gfs-prd02.mgmt prd_items /data/glusterfs/prd_items_brick1/brick1 georep <a class="moz-txt-link-abbreviated" href="mailto:georep@gfs-slc01.mgmt::dr_items">georep@gfs-slc01.mgmt::dr_items</a> gfs-slc03.mgmt Active Changelog Crawl 2016-02-03 01:28:26 0 0 0 2 N/A N/A N/A
gfs-prd03.mgmt prd_items /data/glusterfs/prd_items_brick1/brick1 georep <a class="moz-txt-link-abbreviated" href="mailto:georep@gfs-slc01.mgmt::dr_items">georep@gfs-slc01.mgmt::dr_items</a> gfs-slc02.mgmt Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A
[<a class="moz-txt-link-abbreviated" href="mailto:root@gfs-prd02.mgmt">root@gfs-prd02.mgmt</a> temp]$ dd if=/dev/urandom of=test01 bs=1M count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.179957 s, 5.8 MB/s
[<a class="moz-txt-link-abbreviated" href="mailto:root@gfs-prd02.mgmt">root@gfs-prd02.mgmt</a> temp]$ dd if=/dev/urandom of=test02 bs=1M count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.181704 s, 5.8 MB/s
[<a class="moz-txt-link-abbreviated" href="mailto:root@gfs-prd02.mgmt">root@gfs-prd02.mgmt</a> temp]$ dd if=/dev/urandom of=test03 bs=1M count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.181575 s, 5.8 MB/s
[<a class="moz-txt-link-abbreviated" href="mailto:root@gfs-prd02.mgmt">root@gfs-prd02.mgmt</a> temp]$ gluster volume geo-replication prd_items <a class="moz-txt-link-abbreviated" href="mailto:georep@gfs-slc01.mgmt::dr_items">georep@gfs-slc01.mgmt::dr_items</a> status detail
MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ENTRY DATA META FAILURES CHECKPOINT TIME CHECKPOINT COMPLETED CHECKPOINT COMPLETION TIME
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
gfs-prd02.mgmt prd_items /data/glusterfs/prd_items_brick1/brick1 georep <a class="moz-txt-link-abbreviated" href="mailto:georep@gfs-slc01.mgmt::dr_items">georep@gfs-slc01.mgmt::dr_items</a> gfs-slc03.mgmt Active Changelog Crawl 2016-02-03 01:29:11 0 0 0 5 N/A N/A N/A
gfs-prd03.mgmt prd_items /data/glusterfs/prd_items_brick1/brick1 georep <a class="moz-txt-link-abbreviated" href="mailto:georep@gfs-slc01.mgmt::dr_items">georep@gfs-slc01.mgmt::dr_items</a> gfs-slc02.mgmt Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A
Snippet from /var/log/glusterfs/geo-replication/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items.log
==============
[2016-02-03 01:26:14.702147] I [master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 20 crawls, 1 turns
[2016-02-03 01:26:14.718751] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462711, 0)
[2016-02-03 01:27:15.371404] I [master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 20 crawls, 1 turns
[2016-02-03 01:27:15.379936] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462771, 0)
[2016-02-03 01:27:58.15177] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462831, 0)
[2016-02-03 01:27:59.54204] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877
[2016-02-03 01:28:00.101433] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877
[2016-02-03 01:28:01.104391] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877
[2016-02-03 01:28:02.312911] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877
[2016-02-03 01:28:03.356967] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877
[2016-02-03 01:28:04.364518] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877
[2016-02-03 01:28:05.565407] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877
[2016-02-03 01:28:06.610732] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877
[2016-02-03 01:28:07.620142] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877
[2016-02-03 01:28:08.818785] W [master(/data/glusterfs/prd_items_brick1/brick1):1010:process] _GMaster: changelogs CHANGELOG.1454462877 could not be processed - moving on...
[2016-02-03 01:28:08.826875] W [master(/data/glusterfs/prd_items_brick1/brick1):1014:process] _GMaster: SKIPPED GFID = f5a44b23-cdd8-4278-92d1-1b2a16e8c9d5
[2016-02-03 01:28:15.94074] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462876, 0)
[2016-02-03 01:28:18.669955] I [master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 17 crawls, 3 turns
[2016-02-03 01:28:27.720731] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462891, 0)
[2016-02-03 01:28:58.387859] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462906, 0)
[2016-02-03 01:28:59.360420] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937
[2016-02-03 01:29:00.430566] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937
[2016-02-03 01:29:01.610398] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937
[2016-02-03 01:29:02.667408] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937
[2016-02-03 01:29:03.692369] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937
[2016-02-03 01:29:04.947613] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937
[2016-02-03 01:29:05.932200] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937
[2016-02-03 01:29:06.947702] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937
[2016-02-03 01:29:08.186101] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937
[2016-02-03 01:29:09.203042] W [master(/data/glusterfs/prd_items_brick1/brick1):1010:process] _GMaster: changelogs CHANGELOG.1454462937 could not be processed - moving on...
[2016-02-03 01:29:09.205854] W [master(/data/glusterfs/prd_items_brick1/brick1):1014:process] _GMaster: SKIPPED GFID = c99a05c9-9e7d-41cf-b148-9bec91f527c8
[2016-02-03 01:29:15.432744] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462936, 0)
[2016-02-03 01:29:16.743381] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952
[2016-02-03 01:29:17.944102] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952
[2016-02-03 01:29:18.970493] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952
[2016-02-03 01:29:20.77699] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952
[2016-02-03 01:29:21.197900] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952
[2016-02-03 01:29:22.259825] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952
[2016-02-03 01:29:23.349780] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952
[2016-02-03 01:29:24.469357] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952
[2016-02-03 01:29:25.526793] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952
[2016-02-03 01:29:26.612977] W [master(/data/glusterfs/prd_items_brick1/brick1):1010:process] _GMaster: changelogs CHANGELOG.1454462952 could not be processed - moving on...
[2016-02-03 01:29:26.615757] W [master(/data/glusterfs/prd_items_brick1/brick1):1014:process] _GMaster: SKIPPED GFID = a7dc6c28-9239-44af-909f-a2938bb387b6,550ffdd3-4636-45b8-b278-fbfe120b9d85
[2016-02-03 01:29:29.828999] I [master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 16 crawls, 3 turns
[2016-02-03 01:30:15.98556] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462951, 0)
[2016-02-03 01:30:30.709733] I [master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 20 crawls, 1 turns
[2016-02-03 01:31:12.918898] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454463011, 0)
[2016-02-03 01:31:31.423204] I [master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 20 crawls, 1 turns
Snippet from /var/log/glusterfs/geo-replication/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items.%2Fdata%2Fglusterfs%2Fprd_items_brick1%2Fbrick1-changes.log
===============
[2016-02-03 01:26:12.472345] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2
[2016-02-03 01:26:12.472396] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1721, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog)
[2016-02-03 01:26:12.472415] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000
[2016-02-03 01:26:12.472426] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for
[2016-02-03 01:26:12.472440] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2478 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462772.472206), (vec: 1, len: 4104)
[2016-02-03 01:26:12.472467] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8
[2016-02-03 01:26:12.472481] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24
[2016-02-03 01:26:12.472502] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6b9, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog)
[2016-02-03 01:26:14.952267] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462772 to processed directory
[2016-02-03 01:27:12.531138] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2
[2016-02-03 01:27:12.531177] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1722, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog)
[2016-02-03 01:27:12.531195] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000
[2016-02-03 01:27:12.531207] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for
[2016-02-03 01:27:12.531220] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2479 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462832.531035), (vec: 1, len: 4104)
[2016-02-03 01:27:12.531241] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8
[2016-02-03 01:27:12.531256] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24
[2016-02-03 01:27:12.531274] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6ba, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog)
[2016-02-03 01:27:15.660249] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462832 to processed directory
[2016-02-03 01:27:57.576147] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2
[2016-02-03 01:27:57.576198] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1723, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog)
[2016-02-03 01:27:57.576217] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000
[2016-02-03 01:27:57.576230] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for
[2016-02-03 01:27:57.576246] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2480 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462877.576035), (vec: 1, len: 4104)
[2016-02-03 01:27:57.576282] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8
[2016-02-03 01:27:57.576297] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24
[2016-02-03 01:27:57.576327] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6bb, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog)
[2016-02-03 01:28:08.833040] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462877 to processed directory
[2016-02-03 01:28:12.589377] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2
[2016-02-03 01:28:12.589420] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1724, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog)
[2016-02-03 01:28:12.589439] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000
[2016-02-03 01:28:12.589451] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for
[2016-02-03 01:28:12.589466] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2481 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462892.589270), (vec: 1, len: 4104)
[2016-02-03 01:28:12.589487] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8
[2016-02-03 01:28:12.589500] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24
[2016-02-03 01:28:12.589521] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6bc, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog)
[2016-02-03 01:28:15.398144] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462892 to processed directory
[2016-02-03 01:28:27.604453] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2
[2016-02-03 01:28:27.604510] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1725, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog)
[2016-02-03 01:28:27.604530] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000
[2016-02-03 01:28:27.604542] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for
[2016-02-03 01:28:27.604557] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2482 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462907.604293), (vec: 1, len: 4104)
[2016-02-03 01:28:27.604580] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8
[2016-02-03 01:28:27.604594] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24
[2016-02-03 01:28:27.604615] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6bd, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog)
[2016-02-03 01:28:28.066455] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462907 to processed directory
[2016-02-03 01:28:57.634864] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2
[2016-02-03 01:28:57.634938] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1726, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog)
[2016-02-03 01:28:57.635006] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000
[2016-02-03 01:28:57.635028] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for
[2016-02-03 01:28:57.635049] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2483 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462937.634652), (vec: 1, len: 4104)
[2016-02-03 01:28:57.635079] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8
[2016-02-03 01:28:57.635098] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24
[2016-02-03 01:28:57.635127] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6be, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog)
[2016-02-03 01:29:09.211888] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462937 to processed directory
[2016-02-03 01:29:12.650651] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2
[2016-02-03 01:29:12.650705] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1727, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog)
[2016-02-03 01:29:12.650726] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000
[2016-02-03 01:29:12.650739] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for
[2016-02-03 01:29:12.650755] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2484 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462952.650534), (vec: 1, len: 4104)
[2016-02-03 01:29:12.650787] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8
[2016-02-03 01:29:12.650802] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24
[2016-02-03 01:29:12.650824] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6bf, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog)
[2016-02-03 01:29:26.622432] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462952 to processed directory
[2016-02-03 01:30:12.693458] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2
[2016-02-03 01:30:12.693499] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1728, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog)
[2016-02-03 01:30:12.693528] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000
[2016-02-03 01:30:12.693544] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for
[2016-02-03 01:30:12.693562] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2485 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454463012.693353), (vec: 1, len: 4104)
[2016-02-03 01:30:12.693587] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8
[2016-02-03 01:30:12.693610] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24
[2016-02-03 01:30:12.693638] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6c0, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog)
[2016-02-03 01:30:15.432599] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454463012 to processed directory
[2016-02-03 01:31:12.754211] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2
[2016-02-03 01:31:12.754243] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1729, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog)
[2016-02-03 01:31:12.754260] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000
[2016-02-03 01:31:12.754273] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for
[2016-02-03 01:31:12.754294] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2486 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454463072.754125), (vec: 1, len: 4104)
[2016-02-03 01:31:12.754323] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8
[2016-02-03 01:31:12.754338] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24
[2016-02-03 01:31:12.754357] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6c1, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog)
[2016-02-03 01:31:13.156812] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454463072 to processed directory
[2016-02-03 01:32:12.816173] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2
[2016-02-03 01:32:12.816219] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1730, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog)
[2016-02-03 01:32:12.816236] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000
[2016-02-03 01:32:12.816248] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for
[2016-02-03 01:32:12.816261] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2487 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454463132.816061), (vec: 1, len: 4104)
[2016-02-03 01:32:12.816298] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8
[2016-02-03 01:32:12.816319] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24
[2016-02-03 01:32:12.816361] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6c2, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog)
[2016-02-03 01:32:13.864741] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454463132 to processed directory
[2016-02-03 01:33:12.877209] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2
[2016-02-03 01:33:12.877247] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1731, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog)
[2016-02-03 01:33:12.877276] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000
[2016-02-03 01:33:12.877308] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for
[2016-02-03 01:33:12.877332] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2488 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454463192.877131), (vec: 1, len: 4104)
[2016-02-03 01:33:12.877360] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8
[2016-02-03 01:33:12.877380] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24
[2016-02-03 01:33:12.877408] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6c3, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog)
[2016-02-03 01:33:14.577963] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454463192 to processed directory
Snippet from /var/log/glusterfs/geo-replication/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items.%2Fdata%2Fglusterfs%2Fprd_items_brick1%2Fbrick1.gluster.log
=============
[2016-02-01 20:45:23.591390] I [MSGID: 108031] [afr-common.c:1783:afr_local_discovery_cbk] 0-prd_items-replicate-0: selecting local read_child prd_items-client-0
[2016-02-03 01:27:58.464278] E [fuse-bridge.c:3336:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2016-02-03 01:29:03.558047] E [fuse-bridge.c:3336:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2016-02-03 01:29:25.386117] E [fuse-bridge.c:3336:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</body>
</html>