<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:宋体;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:ç‰çº¿;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Arial Unicode MS";
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:微软雅黑;
panose-1:2 11 5 3 2 2 4 2 2 4;}
@font-face
{font-family:"\@微软雅黑";
panose-1:2 11 5 3 2 2 4 2 2 4;}
@font-face
{font-family:"\@宋体";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@Arial Unicode MS";
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"\@ç‰çº¿";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:����;}
@font-face
{font-family:΢���ź�;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;}
h1
{mso-style-priority:9;
mso-style-link:"æ ‡é¢˜ 1 å—符";
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:24.0pt;
font-family:"Times New Roman",serif;
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML é¢„è®¾æ ¼å¼ å—符";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:����;
color:black;}
span.1
{mso-style-name:"æ ‡é¢˜ 1 å—符";
mso-style-priority:9;
mso-style-link:"æ ‡é¢˜ 1";
font-family:"Calibri Light",sans-serif;
color:#2E74B5;}
p.10, li.10, div.10
{mso-style-name:"���� 1";
mso-style-link:"���� 1 �ַ�";
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;}
span.11
{mso-style-name:"���� 1 �ַ�";
mso-style-priority:9;
mso-style-link:"���� 1";
font-family:"Times New Roman",serif;
font-weight:bold;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.HTML
{mso-style-name:"HTML é¢„è®¾æ ¼å¼ å—符";
mso-style-priority:99;
mso-style-link:"HTML é¢„è®¾æ ¼å¼";
font-family:Consolas;
color:black;}
span.EmailStyle25
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Hi,</span> Saravana<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:ç‰çº¿">I have changed log level to DEBUG. Then start geo-replication with log-file option, attached the file.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:ç‰çº¿"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:ç‰çº¿">gluster volume geo-replication filews glusterfs01.sh3.ctripcorp.com::filews_slave start --log-file=geo.log</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:ç‰çº¿"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:ç‰çº¿">I have checked </span> /root/.ssh/authorized_keys in glusterfs01.sh3.ctripcorp.com<span style="font-family:ç‰çº¿"> ,</span> It have entries in /var/lib/glusterd/geo-replication/common_secret.pem.pub<span style="font-family:ç‰çº¿">.
and I have removed the lines not started with “command=†<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:ç‰çº¿"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">ssh -i /var/lib/glusterd/geo-replication/secret.pem root@</span> glusterfs01.sh3.ctripcorp.com<o:p></o:p></p>
<p class="MsoNormal">I can see gsyncd messages and no ssh error.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Attached etc-glusterfs-glusterd.vol.log from faulty node, it shows :<span style="font-size:10.0pt;font-family:"Courier New""><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[2016-05-19 06:39:23.405974] I [glusterd-geo-rep.c:3516:glusterd_read_status_file] 0-: Using passed config template(/var/lib/glusterd/geo-replication/filews_glusterfs01.sh3.ctripcorp.com_filews_slave/gsyncd.conf).<o:p></o:p></p>
<p class="MsoNormal">[2016-05-19 06:39:23.541169] E [glusterd-geo-rep.c:3200:glusterd_gsync_read_frm_status] 0-: Unable to read gsyncd status file<o:p></o:p></p>
<p class="MsoNormal">[2016-05-19 06:39:23.541210] E [glusterd-geo-rep.c:3603:glusterd_read_status_file] 0-: Unable to read the statusfile for /export/sdb/filews brick for filews(master), glusterfs01.sh3.ctripcorp.com::filews_slave(slave) session<o:p></o:p></p>
<p class="MsoNormal">[2016-05-19 06:39:29.472047] I [glusterd-geo-rep.c:1835:glusterd_get_statefile_name] 0-: Using passed config template(/var/lib/glusterd/geo-replication/filews_glusterfs01.sh3.ctripcorp.com_filews_slave/gsyncd.conf).<o:p></o:p></p>
<p class="MsoNormal">[2016-05-19 06:39:34.939709] I [glusterd-geo-rep.c:3516:glusterd_read_status_file] 0-: Using passed config template(/var/lib/glusterd/geo-replication/filews_glusterfs01.sh3.ctripcorp.com_filews_slave/gsyncd.conf).<o:p></o:p></p>
<p class="MsoNormal">[2016-05-19 06:39:35.058520] E [glusterd-geo-rep.c:3200:glusterd_gsync_read_frm_status] 0-: Unable to read gsyncd status file<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">/var/log/glusterfs/geo-replication/filews/ ssh%3A%2F%2Froot%4010.15.65.66%3Agluster%3A%2F%2F127.0.0.1%3Afilews_slave.log shows as following:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[2016-05-19 15:11:37.307755] I [monitor(monitor):215:monitor] Monitor: ------------------------------------------------------------<o:p></o:p></p>
<p class="MsoNormal">[2016-05-19 15:11:37.308059] I [monitor(monitor):216:monitor] Monitor: starting gsyncd worker<o:p></o:p></p>
<p class="MsoNormal">[2016-05-19 15:11:37.423320] D [gsyncd(agent):627:main_i] <top>: rpc_fd: '7,11,10,9'<o:p></o:p></p>
<p class="MsoNormal">[2016-05-19 15:11:37.423882] I [changelogagent(agent):72:__init__] ChangelogAgent: Agent listining...<o:p></o:p></p>
<p class="MsoNormal">[2016-05-19 15:11:37.423906] I [monitor(monitor):267:monitor] Monitor: worker(/export/sdb/filews) died before establishing connection<o:p></o:p></p>
<p class="MsoNormal">[2016-05-19 15:11:37.424151] I [repce(agent):92:service_loop] RepceServer: terminating on reaching EOF.<o:p></o:p></p>
<p class="MsoNormal">[2016-05-19 15:11:37.424335] I [syncdutils(agent):214:finalize] <top>: exiting.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph"><span style="font-size:10.0pt;font-family:"微软雅黑",sans-serif;color:#6E6E6F">Best Regards
</span><b><span style="font-size:10.5pt;font-family:"Arial Unicode MS",sans-serif;color:#1F497D"><o:p></o:p></span></b></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="527" style="width:394.9pt">
<tbody>
<tr style="height:19.4pt">
<td width="527" valign="top" style="width:394.9pt;padding:0in 0in 0in 0in;height:19.4pt">
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph"><b><span style="font-size:10.0pt;font-family:宋体;color:#6E6E6F">Yuyang Yang</span></b><b><span style="font-size:16.0pt;font-family:"Tahoma",sans-serif;color:#6E6E6F"><o:p></o:p></span></b></p>
</td>
</tr>
<tr style="height:80.45pt">
<td width="527" valign="top" style="width:394.9pt;padding:0in 0in 0in 0in;height:80.45pt">
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph"><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif"><o:p> </o:p></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph"><span style="font-size:10.5pt;color:#1F497D"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span lang="ZH-CN" style="font-family:ç‰çº¿;color:windowtext">å‘件人</span></b><b><span style="font-family:ç‰çº¿;color:windowtext">:</span></b><span style="font-family:ç‰çº¿;color:windowtext"> Saravanakumar Arumugam [mailto:sarumuga@redhat.com]
<br>
<b><span lang="ZH-CN">å‘逿—¶é—´</span>:</b> Thursday, May 19, 2016 1:59 PM<br>
<b><span lang="ZH-CN">收件人</span>:</b> vyyy<span lang="ZH-CN">æ¨é›¨é˜³</span> <yuyangyang@Ctrip.com>; Gluster-users@gluster.org<br>
<b><span lang="ZH-CN">主题</span>:</b> Re: [Gluster-users] <span lang="ZH-CN">ç”å¤</span>: geo-replication status partial faulty<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hi,<br>
<br>
There seems to be some issue in glusterfs01.sh3.ctripcorp.com slave node.<br>
Can you share the complete logs ? <br>
<br>
You can increase verbosity of debug messages like this:<br>
gluster volume geo-replication <master volume> <slave host>::<slave volume> config log-level DEBUG<br>
<br>
<br>
Also, check /root/.ssh/authorized_keys in glusterfs01.sh3.ctripcorp.com<br>
It should have entries in /var/lib/glusterd/geo-replication/common_secret.pem.pub (present in master node).<br>
<br>
Have a look at this one for example:<br>
<a href="https://www.gluster.org/pipermail/gluster-users/2015-August/023174.html">https://www.gluster.org/pipermail/gluster-users/2015-August/023174.html</a><br>
<br>
Thanks,<br>
Saravana<span style="font-size:12.0pt"><o:p></o:p></span></p>
<div>
<p class="MsoNormal">On 05/19/2016 07:53 AM, vyyy<span lang="ZH-CN" style="font-family:"微软雅黑",sans-serif">æ¨é›¨é˜³</span> wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Hello,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I have tried to config a geo-replication volume , all the master nodes configuration are the same, When I start this volume, the status shows partial faulty as following:<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">gluster volume geo-replication filews glusterfs01.sh3.ctripcorp.com::filews_slave status<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">MASTER NODE MASTER VOL MASTER BRICK SLAVE STATUS CHECKPOINT STATUS CRAWL STATUS
<o:p></o:p></p>
<p class="MsoNormal">-------------------------------------------------------------------------------------------------------------------------------------------------<o:p></o:p></p>
<p class="MsoNormal">SVR8048HW2285 filews /export/sdb/filews glusterfs01.sh3.ctripcorp.com::filews_slave faulty N/A N/A
<o:p></o:p></p>
<p class="MsoNormal">SVR8050HW2285 filews /export/sdb/filews glusterfs03.sh3.ctripcorp.com::filews_slave Passive N/A N/A
<o:p></o:p></p>
<p class="MsoNormal">SVR8047HW2285 filews /export/sdb/filews glusterfs01.sh3.ctripcorp.com::filews_slave Active N/A Hybrid Crawl
<o:p></o:p></p>
<p class="MsoNormal">SVR8049HW2285 filews /export/sdb/filews glusterfs05.sh3.ctripcorp.com::filews_slave Active N/A Hybrid Crawl
<o:p></o:p></p>
<p class="MsoNormal">SH02SVR5951 filews /export/sdb/brick1 glusterfs06.sh3.ctripcorp.com::filews_slave Passive N/A N/A
<o:p></o:p></p>
<p class="MsoNormal">SH02SVR5953 filews /export/sdb/brick1 glusterfs01.sh3.ctripcorp.com::filews_slave faulty N/A N/A
<o:p></o:p></p>
<p class="MsoNormal">SVR6995HW2285 filews /export/sdb/filews glusterfs01.sh3.ctripcorp.com::filews_slave faulty N/A N/A
<o:p></o:p></p>
<p class="MsoNormal">SH02SVR5954 filews /export/sdb/brick1 glusterfs01.sh3.ctripcorp.com::filews_slave faulty N/A N/A
<o:p></o:p></p>
<p class="MsoNormal">SVR6994HW2285 filews /export/sdb/filews glusterfs02.sh3.ctripcorp.com::filews_slave Passive N/A N/A
<o:p></o:p></p>
<p class="MsoNormal">SVR6993HW2285 filews /export/sdb/filews glusterfs01.sh3.ctripcorp.com::filews_slave faulty N/A N/A
<o:p></o:p></p>
<p class="MsoNormal">SH02SVR5952 filews /export/sdb/brick1 glusterfs01.sh3.ctripcorp.com::filews_slave faulty N/A N/A
<o:p></o:p></p>
<p class="MsoNormal">SVR6996HW2285 filews /export/sdb/filews glusterfs04.sh3.ctripcorp.com::filews_slave Passive N/A N/A
<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">On the faulty node, log file /var/log/glusterfs/geo-replication/filews shows worker(/export/sdb/filews) died before establishing connection<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">[2016-05-18 16:55:46.402622] I [monitor(monitor):215:monitor] Monitor: ------------------------------------------------------------<o:p></o:p></p>
<p class="MsoNormal">[2016-05-18 16:55:46.402930] I [monitor(monitor):216:monitor] Monitor: starting gsyncd worker<o:p></o:p></p>
<p class="MsoNormal">[2016-05-18 16:55:46.517460] I [changelogagent(agent):72:__init__] ChangelogAgent: Agent listining...<o:p></o:p></p>
<p class="MsoNormal">[2016-05-18 16:55:46.518066] I [repce(agent):92:service_loop] RepceServer: terminating on reaching EOF.<o:p></o:p></p>
<p class="MsoNormal">[2016-05-18 16:55:46.518279] I [syncdutils(agent):214:finalize] <top>: exiting.<o:p></o:p></p>
<p class="MsoNormal">[2016-05-18 16:55:46.518194] I [monitor(monitor):267:monitor] Monitor: worker(/export/sdb/filews) died before establishing connection<o:p></o:p></p>
<p class="MsoNormal">[2016-05-18 16:55:56.697036] I [monitor(monitor):215:monitor] Monitor: ------------------------------------------------------------<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div style="border:none;border-bottom:solid #EEEEEE 1.0pt;padding:0in 0in 14.0pt 0in">
<p class="MsoNormal" style="margin-bottom:15.0pt;line-height:30.0pt;background:white;vertical-align:baseline">
Any advice and suggestions will be greatly appreciated.<o:p></o:p></p>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph"><span style="font-size:10.0pt;font-family:΢���ź�;color:#6E6E6F">Best Regards
</span><o:p></o:p></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="527" style="width:394.9pt">
<tbody>
<tr style="height:19.4pt">
<td width="527" valign="top" style="width:394.9pt;padding:0in 0in 0in 0in;height:19.4pt">
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph"><b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:#6E6E6F">������</span></b><b><span style="font-size:10.0pt;font-family:����;color:#6E6E6F"> Yuyang Yang</span></b><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><br>
<br>
<br>
<o:p></o:p></span></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Gluster-users mailing list<o:p></o:p></pre>
<pre><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><o:p></o:p></pre>
<pre><a href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p>
</div>
</body>
</html>