<div dir="ltr"><div>I also noted that the second master gfs2 alternates between passive/faulty. Not sure if this matters but, I have changed the /etc/hosts file to change 127.0.0.1 to gfs1 and so on because my node would not be in peer cluster state. </div><div><br></div>Gluster version : 3.7.6-1<div>OS: RHEL 7</div><div><br></div><div><br></div><div><div>[root@gfs1 ~]# cat /var/log/glusterfs/geo-replication/gvol/ssh%3A%2F%2Froot%4054.172.172.245%3Agluster%3A%2F%2F127.0.0.1%3Axvol.log</div><div>[2015-11-17 10:30:30.244277] I [monitor(monitor):362:distribute] <top>: slave bricks: [{'host': 'xfs1', 'dir': '/data/brick/xvol'}, {'host': 'xfs2', 'dir': '/data/brick/xvol'}]</div><div>[2015-11-17 10:30:30.245239] I [monitor(monitor):383:distribute] <top>: worker specs: [('/data/brick/gvol', 'ssh://root@xfs2:gluster://localhost:xvol', 1)]</div><div>[2015-11-17 10:30:30.433696] I [monitor(monitor):221:monitor] Monitor: ------------------------------------------------------------</div><div>[2015-11-17 10:30:30.433882] I [monitor(monitor):222:monitor] Monitor: starting gsyncd worker</div><div>[2015-11-17 10:30:30.561599] I [gsyncd(/data/brick/gvol):650:main_i] <top>: syncing: gluster://localhost:gvol -> ssh://root@xfs2:gluster://localhost:xvol</div><div>[2015-11-17 10:30:30.573781] I [changelogagent(agent):75:__init__] ChangelogAgent: Agent listining...</div><div>[2015-11-17 10:30:34.26421] I [master(/data/brick/gvol):83:gmaster_builder] <top>: setting up xsync change detection mode</div><div>[2015-11-17 10:30:34.26695] I [master(/data/brick/gvol):404:__init__] _GMaster: using 'rsync' as the sync engine</div><div>[2015-11-17 10:30:34.27324] I [master(/data/brick/gvol):83:gmaster_builder] <top>: setting up changelog change detection mode</div><div>[2015-11-17 10:30:34.27477] I [master(/data/brick/gvol):404:__init__] _GMaster: using 'rsync' as the sync engine</div><div>[2015-11-17 10:30:34.27873] I [master(/data/brick/gvol):83:gmaster_builder] <top>: setting up changeloghistory change detection mode</div><div>[2015-11-17 10:30:34.28048] I [master(/data/brick/gvol):404:__init__] _GMaster: using 'rsync' as the sync engine</div><div>[2015-11-17 10:30:36.40117] I [master(/data/brick/gvol):1229:register] _GMaster: xsync temp directory: /var/lib/misc/glusterfsd/gvol/ssh%3A%2F%2Froot%4054.172.172.245%3Agluster%3A%2F%2F127.0.0.1%3Axvol/0c4166e49b1b516d061ed475806364b9/xsync</div><div>[2015-11-17 10:30:36.40409] I [resource(/data/brick/gvol):1432:service_loop] GLUSTER: Register time: 1447774236</div><div>[2015-11-17 10:30:36.65299] I [master(/data/brick/gvol):530:crawlwrap] _GMaster: primary master with volume id f77a024e-a865-493e-9ce2-d7dbe99ee6d5 ...</div><div>[2015-11-17 10:30:36.67856] I [master(/data/brick/gvol):539:crawlwrap] _GMaster: crawl interval: 1 seconds</div><div>[2015-11-17 10:31:36.185137] I [master(/data/brick/gvol):552:crawlwrap] _GMaster: 0 crawls, 0 turns</div><div>[2015-11-17 10:32:36.315582] I [master(/data/brick/gvol):552:crawlwrap] _GMaster: 0 crawls, 0 turns</div><div>[2015-11-17 10:33:36.438072] I [master(/data/brick/gvol):552:crawlwrap] _GMaster: 0 crawls, 0 turns</div></div><div><br></div><div><br></div><div><div>[root@gfs2 ~]#cat /var/log/glusterfs/geo-replication/gvol/ssh%3A%2F%2Froot%4054.172.172.245%3Agluster%3A%2F%2F127.0.0.1%3Axvol.log | less</div><div>[2015-11-17 10:30:30.498424] I [monitor(monitor):362:distribute] <top>: slave bricks: [{'host': 'xfs1', 'dir': '/data/brick/xvol'}, {'host': 'xfs2', 'dir': '/data/brick/xvol'}]</div><div>[2015-11-17 10:30:30.499473] I [monitor(monitor):383:distribute] <top>: worker specs: [('/data/brick/gvol', 'ssh://root@xfs1:gluster://localhost:xvol', 1)]</div><div>[2015-11-17 10:30:30.679028] I [monitor(monitor):221:monitor] Monitor: ------------------------------------------------------------</div><div>[2015-11-17 10:30:30.679259] I [monitor(monitor):222:monitor] Monitor: starting gsyncd worker</div><div>[2015-11-17 10:30:30.807980] I [gsyncd(/data/brick/gvol):650:main_i] <top>: syncing: gluster://localhost:gvol -> ssh://root@xfs1:gluster://localhost:xvol</div><div>[2015-11-17 10:30:30.820440] I [changelogagent(agent):75:__init__] ChangelogAgent: Agent listining...</div><div>[2015-11-17 10:30:34.358032] I [master(/data/brick/gvol):83:gmaster_builder] <top>: setting up xsync change detection mode</div><div>[2015-11-17 10:30:34.358304] I [master(/data/brick/gvol):404:__init__] _GMaster: using 'rsync' as the sync engine</div><div>[2015-11-17 10:30:34.359335] I [master(/data/brick/gvol):83:gmaster_builder] <top>: setting up changelog change detection mode</div><div>[2015-11-17 10:30:34.359496] I [master(/data/brick/gvol):404:__init__] _GMaster: using 'rsync' as the sync engine</div><div>[2015-11-17 10:30:34.359890] I [master(/data/brick/gvol):83:gmaster_builder] <top>: setting up changeloghistory change detection mode</div><div>[2015-11-17 10:30:34.360044] I [master(/data/brick/gvol):404:__init__] _GMaster: using 'rsync' as the sync engine</div><div>[2015-11-17 10:30:36.371203] I [master(/data/brick/gvol):1229:register] _GMaster: xsync temp directory: /var/lib/misc/glusterfsd/gvol/ssh%3A%2F%2Froot%4054.172.172.245%3Agluster%3A%2F%2F127.0.0.1%3Axvol/0c4166e49b1b516d061ed475806364b9/xsync</div><div>[2015-11-17 10:30:36.371514] I [resource(/data/brick/gvol):1432:service_loop] GLUSTER: Register time: 1447774236</div><div>[2015-11-17 10:30:36.383291] I [master(/data/brick/gvol):530:crawlwrap] _GMaster: primary master with volume id f77a024e-a865-493e-9ce2-d7dbe99ee6d5 ...</div><div>[2015-11-17 10:30:36.386276] I [master(/data/brick/gvol):539:crawlwrap] _GMaster: crawl interval: 1 seconds</div><div>[2015-11-17 10:30:46.558255] E [repce(/data/brick/gvol):207:__call__] RepceClient: call 29036:140624661567232:1447774246.47 (keep_alive) failed on peer with OSError</div><div>[2015-11-17 10:30:46.558463] E [syncdutils(/data/brick/gvol):276:log_raise_exception] <top>: FAIL:</div><div>Traceback (most recent call last):</div><div> File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 306, in twrap</div><div> tf(*aa)</div><div> File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 438, in keep_alive</div><div> cls.slave.server.keep_alive(vi)</div><div> File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 226, in __call__</div><div> return self.ins(self.meth, *a)</div><div> File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 208, in __call__</div><div> raise res</div><div>OSError: [Errno 5] Input/output error</div></div><div><br></div><div><br></div><div><div><br></div></div><div><br></div><div>-----------</div><div><br></div><div><div>[root@gfs1 ~]# ps aux | grep gsyncd</div><div>root 15837 0.0 1.0 368584 11148 ? Ssl 11:08 0:00 /usr/bin/python /usr/libexec/glusterfs/python/syncdaemon/gsyncd.py --path=/data/brick/gvol --monitor -c /var/lib/glusterd/geo-replication/gvol_xfs1_xvol/gsyncd.conf --iprefix=/var :gvol --glusterd-uuid=c6e8cdef-bc46-4684-9c75-fc348fefb95e xfs1::xvol</div><div>root 15867 0.0 1.7 884044 18064 ? Ssl 11:08 0:00 python /usr/libexec/glusterfs/python/syncdaemon/gsyncd.py --path=/data/brick/gvol -c /var/lib/glusterd/geo-replication/gvol_xfs1_xvol/gsyncd.conf --iprefix=/var :gvol --glusterd-uuid=c6e8cdef-bc46-4684-9c75-fc348fefb95e xfs1::xvol -N -p --slave-id ff6d57c8-cfb5-40b3-843f-bcd79cdd6164 --local-path /data/brick/gvol --agent --rpc-fd 7,10,9,8</div><div>root 15868 0.0 1.7 847644 17292 ? Sl 11:08 0:00 python /usr/libexec/glusterfs/python/syncdaemon/gsyncd.py --path=/data/brick/gvol -c /var/lib/glusterd/geo-replication/gvol_xfs1_xvol/gsyncd.conf --iprefix=/var :gvol --glusterd-uuid=c6e8cdef-bc46-4684-9c75-fc348fefb95e xfs1::xvol -N -p --slave-id ff6d57c8-cfb5-40b3-843f-bcd79cdd6164 --feedback-fd 12 --local-path /data/brick/gvol --local-id .%2Fdata%2Fbrick%2Fgvol --rpc-fd 9,8,7,10 --subvol-num 1 --resource-remote ssh://root@xfs2:gluster://localhost:xvol</div><div>root 15879 0.0 0.4 80384 4244 ? S 11:08 0:00 ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-5bwc6n/21cd0d364db39da791c9bc6dcf62c55b.sock root@xfs2 /nonexistent/gsyncd --session-owner f77a024e-a865-493e-9ce2-d7dbe99ee6d5 -N --listen --timeout 120 gluster://localhost:xvol</div><div>root 15887 0.1 3.9 630404 40476 ? Ssl 11:08 0:02 /usr/sbin/glusterfs --aux-gfid-mount --acl --log-file=/var/log/glusterfs/geo-replication/gvol/ssh%3A%2F%2Froot%4054.172.172.245%3Agluster%3A%2F%2F127.0.0.1%3Axvol.%2Fdata%2Fbrick%2Fgvol.gluster.log --volfile-server=localhost --volfile-id=gvol --client-pid=-1 /tmp/gsyncd-aux-mount-IOxY7_</div><div>root 16540 0.0 0.0 112640 956 pts/0 R+ 11:26 0:00 grep --color=auto gsyncd</div></div><div>--------------</div><div><div>[root@gfs2 ec2-user]# ps aux | grep gsyncd</div><div>root 3099 0.0 1.3 368488 13568 ? Ssl 11:08 0:00 /usr/bin/python /usr/libexec/glusterfs/python/syncdaemon/gsyncd.py --path=/data/brick/gvol --monitor -c /var/lib/glusterd/geo-replication/gvol_xfs1_xvol/gsyncd.conf --iprefix=/var :gvol --glusterd-uuid=449f6672-fdcd-480b-870d-51e1ed92236c xfs1::xvol</div><div>root 6618 1.0 1.9 883944 19872 ? Ssl 11:27 0:00 python /usr/libexec/glusterfs/python/syncdaemon/gsyncd.py --path=/data/brick/gvol -c /var/lib/glusterd/geo-replication/gvol_xfs1_xvol/gsyncd.conf --iprefix=/var :gvol --glusterd-uuid=449f6672-fdcd-480b-870d-51e1ed92236c xfs1::xvol -N -p --slave-id ff6d57c8-cfb5-40b3-843f-bcd79cdd6164 --local-path /data/brick/gvol --agent --rpc-fd 8,11,10,9</div><div>root 6619 1.1 1.4 847548 15004 ? Sl 11:27 0:00 python /usr/libexec/glusterfs/python/syncdaemon/gsyncd.py --path=/data/brick/gvol -c /var/lib/glusterd/geo-replication/gvol_xfs1_xvol/gsyncd.conf --iprefix=/var :gvol --glusterd-uuid=449f6672-fdcd-480b-870d-51e1ed92236c xfs1::xvol -N -p --slave-id ff6d57c8-cfb5-40b3-843f-bcd79cdd6164 --feedback-fd 13 --local-path /data/brick/gvol --local-id .%2Fdata%2Fbrick%2Fgvol --rpc-fd 10,9,8,11 --subvol-num 1 --resource-remote ssh://root@xfs1:gluster://localhost:xvol</div><div>root 6631 0.3 0.4 80384 4240 ? S 11:27 0:00 ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-WIfjHQ/25f2a0dc75697352a40d6471e241edf7.sock root@xfs1 /usr/libexec/glusterfs/gsyncd --session-owner f77a024e-a865-493e-9ce2-d7dbe99ee6d5 -N --listen --timeout 120 gluster://localhost:xvol</div><div>root 6638 1.0 3.2 630408 33416 ? Ssl 11:27 0:00 /usr/sbin/glusterfs --aux-gfid-mount --acl --log-file=/var/log/glusterfs/geo-replication/gvol/ssh%3A%2F%2Froot%4054.172.172.245%3Agluster%3A%2F%2F127.0.0.1%3Axvol.%2Fdata%2Fbrick%2Fgvol.gluster.log --volfile-server=localhost --volfile-id=gvol --client-pid=-1 /tmp/gsyncd-aux-mount-o44DsN</div><div>root 6692 0.0 0.0 112640 960 pts/0 R+ 11:28 0:00 grep --color=auto gsyncd</div></div><div>---------------------</div><div><br></div><div><div>[root@xfs1 ~]# ps aux | grep gsyncd</div><div>root 2753 0.5 1.2 585232 12576 ? Ssl 11:28 0:00 /usr/bin/python /usr/libexec/glusterfs/python/syncdaemon/gsyncd.py --session-owner f77a024e-a865-493e-9ce2-d7dbe99ee6d5 -N --listen --timeout 120 gluster://localhost:xvol -c /var/lib/glusterd/geo-replication/gsyncd_template.conf</div><div>root 2773 0.3 3.4 630412 34728 ? Ssl 11:28 0:00 /usr/sbin/glusterfs --aux-gfid-mount --acl --log-file=/var/log/glusterfs/geo-replication-slaves/f77a024e-a865-493e-9ce2-d7dbe99ee6d5:gluster%3A%2F%2F127.0.0.1%3Axvol.gluster.log --volfile-server=localhost --volfile-id=xvol --client-pid=-1 /tmp/gsyncd-aux-mount-une5yr</div><div>root 2793 0.0 0.0 112640 956 pts/0 R+ 11:28 0:00 grep --color=auto gsyncd</div><div>[root@xfs1 ~]#</div></div><div><br></div><div>-----------------------</div><div><br></div><div><div>[root@xfs2 ec2-user]# ps aux | grep gsyncd</div><div>root 28921 0.0 1.2 585236 12668 ? Ssl 11:08 0:00 /usr/bin/python /usr/libexec/glusterfs/python/syncdaemon/gsyncd.py --session-owner f77a024e-a865-493e-9ce2-d7dbe99ee6d5 -N --listen --timeout 120 gluster://localhost:xvol -c /var/lib/glusterd/geo-replication/gsyncd_template.conf</div><div>root 28941 0.2 3.7 630412 38280 ? Ssl 11:08 0:02 /usr/sbin/glusterfs --aux-gfid-mount --acl --log-file=/var/log/glusterfs/geo-replication-slaves/f77a024e-a865-493e-9ce2-d7dbe99ee6d5:gluster%3A%2F%2F127.0.0.1%3Axvol.gluster.log --volfile-server=localhost --volfile-id=xvol --client-pid=-1 /tmp/gsyncd-aux-mount-cZvAEH</div><div>root 29029 0.0 0.0 112640 956 pts/0 R+ 11:29 0:00 grep --color=auto gsyncd</div><div>[root@xfs2 ec2-user]#</div></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Nov 17, 2015 at 12:39 AM, Aravinda <span dir="ltr"><<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
One status row should show Active and other should show Passive.
Please provide logs from gfs1 and gfs2
nodes(/var/log/glusterfs/geo-replication/gvol/*.log)<br>
<br>
Also please let us know,<br>
1. Gluster version and OS<br>
2. output of `ps aux | grep gsyncd` from Master nodes and Slave
nodes<br>
<pre cols="72">regards
Aravinda</pre><span class="">
<div>On 11/17/2015 02:09 AM, Deepak Ravi
wrote:<br>
</div>
</span><blockquote type="cite">
<pre>Hi all
I'm working on a Geo-replication setup that I'm having issues with.
Situation :
- In the east region of AWS, I Created a replicated volume between 2
nodes, lets call this volume *gvol*
-
*In the west region of AWS, I Created another replicated volume between 2
nodes, lets call this volume xvol *
- Geo replication was created and started successfully
-
[root@gfs1 mnt]# gluster volume geo-replication gvol xfs1::xvol status
MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE
SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED
-------------------------------------------------------------------------------------------------------------------------------
gfs1 gvol /data/brick/gvol root xfs1::xvol
N/A Passive N/A N/A
gfs2 gvol /data/brick/gvol root xfs1::xvol
N/A Passive N/A N/A
The data on nodes(gfs1 and gfs2) was not being replicated to xfs1 at all. I
tried restarting the services and it still didn't help. Looking at the log
files didn't help me much because I didn't know what I should be looking
for.
Can someone point me in the right direction?
Thanks
</pre>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Gluster-users mailing list
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><font face="tahoma, sans-serif"><b>~Deepak</b></font></div>
</div>