<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hi,<br>
+geo-rep team.<br>
<br>
Can you get the gluster version you are using?<br>
<br>
# For example: <br>
rpm -qa | grep gluster <br>
<br>
I hope you have same gluster version installed everywhere.<br>
Please double check and share the same.<br>
<br>
Thanks,<br>
Saravana<br>
<br>
<div class="moz-cite-prefix">On 05/19/2016 01:37 PM, vyyy杨雨阳 wrote:<br>
</div>
<blockquote
cite="mid:EC05CC47D8204349A9932E938B924BC8AF9A027F@SVR5096HP380.cn1.global.ctrip.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:宋体;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:等线;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Arial Unicode MS";
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:微软雅黑;
panose-1:2 11 5 3 2 2 4 2 2 4;}
@font-face
{font-family:"\@微软雅黑";
panose-1:2 11 5 3 2 2 4 2 2 4;}
@font-face
{font-family:"\@宋体";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@Arial Unicode MS";
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"\@等线";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:����;}
@font-face
{font-family:���ź�;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;}
h1
{mso-style-priority:9;
mso-style-link:"标题 1 字符";
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:24.0pt;
font-family:"Times New Roman",serif;
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML 预设格式 字符";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:����;
color:black;}
span.1
{mso-style-name:"标题 1 字符";
mso-style-priority:9;
mso-style-link:"标题 1";
font-family:"Calibri Light",sans-serif;
color:#2E74B5;}
p.10, li.10, div.10
{mso-style-name:"���� 1";
mso-style-link:"���� 1 �ַ�";
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;}
span.11
{mso-style-name:"���� 1 �ַ�";
mso-style-priority:9;
mso-style-link:"���� 1";
font-family:"Times New Roman",serif;
font-weight:bold;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.HTML
{mso-style-name:"HTML 预设格式 字符";
mso-style-priority:99;
mso-style-link:"HTML 预设格式";
font-family:Consolas;
color:black;}
span.EmailStyle25
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Hi,</span>
Saravana<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:等线">I have changed
log level to DEBUG. Then start geo-replication with log-file
option, attached the file.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:等线"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:等线">gluster volume
geo-replication filews
glusterfs01.sh3.ctripcorp.com::filews_slave start
--log-file=geo.log</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:等线"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:等线">I have checked
</span> /root/.ssh/authorized_keys in
glusterfs01.sh3.ctripcorp.com<span style="font-family:等线"> ,</span>
It have entries in
/var/lib/glusterd/geo-replication/common_secret.pem.pub<span
style="font-family:等线">. and I have removed the lines not
started with “command=” <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:等线"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New"">ssh
-i /var/lib/glusterd/geo-replication/secret.pem root@</span>
glusterfs01.sh3.ctripcorp.com<o:p></o:p></p>
<p class="MsoNormal">I can see gsyncd messages and no ssh error.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Attached etc-glusterfs-glusterd.vol.log
from faulty node, it shows :<span
style="font-size:10.0pt;font-family:"Courier New""><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[2016-05-19 06:39:23.405974] I
[glusterd-geo-rep.c:3516:glusterd_read_status_file] 0-: Using
passed config
template(/var/lib/glusterd/geo-replication/filews_glusterfs01.sh3.ctripcorp.com_filews_slave/gsyncd.conf).<o:p></o:p></p>
<p class="MsoNormal">[2016-05-19 06:39:23.541169] E
[glusterd-geo-rep.c:3200:glusterd_gsync_read_frm_status] 0-:
Unable to read gsyncd status file<o:p></o:p></p>
<p class="MsoNormal">[2016-05-19 06:39:23.541210] E
[glusterd-geo-rep.c:3603:glusterd_read_status_file] 0-: Unable
to read the statusfile for /export/sdb/filews brick for
filews(master),
glusterfs01.sh3.ctripcorp.com::filews_slave(slave) session<o:p></o:p></p>
<p class="MsoNormal">[2016-05-19 06:39:29.472047] I
[glusterd-geo-rep.c:1835:glusterd_get_statefile_name] 0-:
Using passed config
template(/var/lib/glusterd/geo-replication/filews_glusterfs01.sh3.ctripcorp.com_filews_slave/gsyncd.conf).<o:p></o:p></p>
<p class="MsoNormal">[2016-05-19 06:39:34.939709] I
[glusterd-geo-rep.c:3516:glusterd_read_status_file] 0-: Using
passed config
template(/var/lib/glusterd/geo-replication/filews_glusterfs01.sh3.ctripcorp.com_filews_slave/gsyncd.conf).<o:p></o:p></p>
<p class="MsoNormal">[2016-05-19 06:39:35.058520] E
[glusterd-geo-rep.c:3200:glusterd_gsync_read_frm_status] 0-:
Unable to read gsyncd status file<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">/var/log/glusterfs/geo-replication/filews/
ssh%3A%2F%2Froot%4010.15.65.66%3Agluster%3A%2F%2F127.0.0.1%3Afilews_slave.log
shows as following:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[2016-05-19 15:11:37.307755] I
[monitor(monitor):215:monitor] Monitor:
------------------------------------------------------------<o:p></o:p></p>
<p class="MsoNormal">[2016-05-19 15:11:37.308059] I
[monitor(monitor):216:monitor] Monitor: starting gsyncd worker<o:p></o:p></p>
<p class="MsoNormal">[2016-05-19 15:11:37.423320] D
[gsyncd(agent):627:main_i] <top>: rpc_fd: '7,11,10,9'<o:p></o:p></p>
<p class="MsoNormal">[2016-05-19 15:11:37.423882] I
[changelogagent(agent):72:__init__] ChangelogAgent: Agent
listining...<o:p></o:p></p>
<p class="MsoNormal">[2016-05-19 15:11:37.423906] I
[monitor(monitor):267:monitor] Monitor:
worker(/export/sdb/filews) died before establishing connection<o:p></o:p></p>
<p class="MsoNormal">[2016-05-19 15:11:37.424151] I
[repce(agent):92:service_loop] RepceServer: terminating on
reaching EOF.<o:p></o:p></p>
<p class="MsoNormal">[2016-05-19 15:11:37.424335] I
[syncdutils(agent):214:finalize] <top>: exiting.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"
style="text-align:justify;text-justify:inter-ideograph"><span
style="font-size:10.0pt;font-family:"微软雅黑",sans-
serif;color:#6E6E6F">Best Regards
</span><b><span
style="font-size:10.5pt;font-family:"Arial Unicode
MS",sans-serif;color:#1F497D"><o:p></o:p></span></b></p>
<table class="MsoNormalTable" style="width:394.9pt"
width="527" border="0" cellpadding="0" cellspacing="0">
<tbody>
<tr style="height:19.4pt">
<td style="width:394.9pt;padding:0in 0in 0in
0in;height:19.4pt" valign="top" width="527">
<p class="MsoNormal"
style="text-align:justify;text-justify:inter-ideograph"><b><span
style="font-size:10.0pt;font-family:宋体;
color:#6E6E6F">Yuyang Yang</span></b><b><span
style="font-size:16.0pt;font-family:"Tahoma",sans-serif;color:#6E6E6F"><o:p></o:p></span></b></p>
</td>
</tr>
<tr style="height:80.45pt">
<td style="width:394.9pt;padding:0in 0in 0in
0in;height:80.45pt" valign="top" width="527">
<p class="MsoNormal"
style="text-align:justify;text-justify:inter-ideograph"><span
style="font-size:9.0pt;font-family:"Tahoma",sans-serif"><o:p> </o:p></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"
style="text-align:justify;text-justify:inter-ideograph"><span
style="font-size:10.5pt;color:#1F497D"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-family:等线;color:windowtext" lang="ZH-CN">发
件人</span></b><b><span
style="font-family:等线;color:windowtext">:</span></b><span
style="font-family:等线;color:windowtext"> Saravanakumar
Arumugam [<a class="moz-txt-link-freetext" href="mailto:sarumuga@redhat.com">mailto:sarumuga@redhat.com</a>]
<br>
<b><span lang="ZH-CN">发送时间</span>:</b> Thursday, May 19,
2016 1:59 PM<br>
<b><span lang="ZH-CN">收件人</span>:</b> vyyy<span
lang="ZH-CN">杨雨阳</span> <a class="moz-txt-link-rfc2396E" href="mailto:yuyangyang@Ctrip.com"><yuyangyang@Ctrip.com></a>;
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<b><span lang="ZH-CN">主题</span>:</b> Re: [Gluster-users]
<span lang="ZH-CN">答复</span>: geo-replication status
partial faulty<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hi,<br>
<br>
There seems to be some issue in glusterfs01.sh3.ctripcorp.com
slave node.<br>
Can you share the complete logs ? <br>
<br>
You can increase verbosity of debug messages like this:<br>
gluster volume geo-replication <master volume> <slave
host>::<slave volume> config log-level DEBUG<br>
<br>
<br>
Also, check /root/.ssh/authorized_keys in
glusterfs01.sh3.ctripcorp.com<br>
It should have entries in
/var/lib/glusterd/geo-replication/common_secret.pem.pub
(present in master node).<br>
<br>
Have a look at this one for example:<br>
<a moz-do-not-send="true"
href="https://www.gluster.org/pipermail/gluster-users/2015-August/023174.html">https://www.gluster.org/pipermail/gluster-users/2015-August/023174.html</a><br>
<br>
Thanks,<br>
Saravana<span style="font-size:12.0pt"><o:p></o:p></span></p>
<div>
<p class="MsoNormal">On 05/19/2016 07:53 AM, vyyy<span
style="font-family:"微软雅黑",sans-serif"
lang="ZH-CN">杨雨阳</span> wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Hello,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I have tried to config a geo-replication
volume , all the master nodes configuration are the same,
When I start this volume, the status shows partial faulty as
following:<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">gluster volume geo-replication filews
glusterfs01.sh3.ctripcorp.com::filews_slave status<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">MASTER NODE MASTER VOL MASTER
BRICK
SLAVE STATUS
CHECKPOINT STATUS CRAWL STATUS
<o:p></o:p></p>
<p class="MsoNormal">-------------------------------------------------------------------------------------------------------------------------------------------------<o:p></o:p></p>
<p class="MsoNormal">SVR8048HW2285 filews
/export/sdb/filews
glusterfs01.sh3.ctripcorp.com::filews_slave faulty
N/A N/A
<o:p></o:p></p>
<p class="MsoNormal">SVR8050HW2285 filews
/export/sdb/filews
glusterfs03.sh3.ctripcorp.com::filews_slave Passive
N/A N/A
<o:p></o:p></p>
<p class="MsoNormal">SVR8047HW2285 filews
/export/sdb/filews
glusterfs01.sh3.ctripcorp.com::filews_slave Active
N/A Hybrid Crawl
<o:p></o:p></p>
<p class="MsoNormal">SVR8049HW2285 filews
/export/sdb/filews
glusterfs05.sh3.ctripcorp.com::filews_slave Active
N/A Hybrid Crawl
<o:p></o:p></p>
<p class="MsoNormal">SH02SVR5951 filews
/export/sdb/brick1
glusterfs06.sh3.ctripcorp.com::filews_slave Passive
N/A N/A
<o:p></o:p></p>
<p class="MsoNormal">SH02SVR5953 filews
/export/sdb/brick1
glusterfs01.sh3.ctripcorp.com::filews_slave faulty
N/A N/A
<o:p></o:p></p>
<p class="MsoNormal">SVR6995HW2285 filews
/export/sdb/filews
glusterfs01.sh3.ctripcorp.com::filews_slave faulty
N/A N/A
<o:p></o:p></p>
<p class="MsoNormal">SH02SVR5954 filews
/export/sdb/brick1
glusterfs01.sh3.ctripcorp.com::filews_slave faulty
N/A N/A
<o:p></o:p></p>
<p class="MsoNormal">SVR6994HW2285 filews
/export/sdb/filews
glusterfs02.sh3.ctripcorp.com::filews_slave Passive
N/A N/A
<o:p></o:p></p>
<p class="MsoNormal">SVR6993HW2285 filews
/export/sdb/filews
glusterfs01.sh3.ctripcorp.com::filews_slave faulty
N/A N/A
<o:p></o:p></p>
<p class="MsoNormal">SH02SVR5952 filews
/export/sdb/brick1
glusterfs01.sh3.ctripcorp.com::filews_slave faulty
N/A N/A
<o:p></o:p></p>
<p class="MsoNormal">SVR6996HW2285 filews
/export/sdb/filews
glusterfs04.sh3.ctripcorp.com::filews_slave Passive
N/A N/A
<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">On the faulty node, log file
/var/log/glusterfs/geo-replication/filews shows
worker(/export/sdb/filews) died before establishing
connection<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">[2016-05-18 16:55:46.402622] I
[monitor(monitor):215:monitor] Monitor:
------------------------------------------------------------<o:p></o:p></p>
<p class="MsoNormal">[2016-05-18 16:55:46.402930] I
[monitor(monitor):216:monitor] Monitor: starting gsyncd
worker<o:p></o:p></p>
<p class="MsoNormal">[2016-05-18 16:55:46.517460] I
[changelogagent(agent):72:__init__] ChangelogAgent: Agent
listining...<o:p></o:p></p>
<p class="MsoNormal">[2016-05-18 16:55:46.518066] I
[repce(agent):92:service_loop] RepceServer: terminating on
reaching EOF.<o:p></o:p></p>
<p class="MsoNormal">[2016-05-18 16:55:46.518279] I
[syncdutils(agent):214:finalize] <top>: exiting.<o:p></o:p></p>
<p class="MsoNormal">[2016-05-18 16:55:46.518194] I
[monitor(monitor):267:monitor] Monitor:
worker(/export/sdb/filews) died before establishing
connection<o:p></o:p></p>
<p class="MsoNormal">[2016-05-18 16:55:56.697036] I
[monitor(monitor):215:monitor] Monitor:
------------------------------------------------------------<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div style="border:none;border-bottom:solid #EEEEEE
1.0pt;padding:0in 0in 14.0pt 0in">
<p class="MsoNormal"
style="margin-bottom:15.0pt;line-height:30.0pt;background:white;vertical-align:baseline">Any
advice and suggestions will be greatly appreciated.<o:p></o:p></p>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"
style="text-align:justify;text-justify:inter-ideograph"><span
style="font-size:10.0pt;font-family:���ź�;color:#6E6E6F">Best
Regards
</span><o:p></o:p></p>
<table class="MsoNormalTable" style="width:394.9pt"
width="527" border="0" cellpadding="0" cellspacing="0">
<tbody>
<tr style="height:19.4pt">
<td style="width:394.9pt;padding:0in 0in 0in
0in;height:19.4pt" valign="top" width="527">
<p class="MsoNormal"
style="text-align:justify;text-justify:inter-ideograph"><b><span
style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:#6E6E6F">������</span></b><b><span
style="font-size:10.0pt;font-family:����;color:#6E6E6F"> Yuyang Yang</span></b><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><br>
<br>
<br>
<o:p></o:p></span></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Gluster-users mailing list<o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p> </o:p></span></p>
</div>
</blockquote>
<br>
</body>
</html>