<div dir="ltr"><div><div><div><div><div><div><div><div>Hi,<br><br></div>I have still facing this problem and analyzing the system as well as gluster logs.<br><br></div>From the system logs /var/log/messages I found that time stamp of <font face="Calibri" size="2">temporary n/w failure logs is very near and between to “volume status” command and before the remove-brick command where our brick is offline.<br><br></font></div><font face="Calibri" size="2">Please suggest me based on below logs is this could be possible reason for and stopping Brick to come online?<br><br></font></div><font face="Calibri" size="2">we have collect logs from two setup <br><br></font></div><font face="Calibri" size="2">Setup 1:<br><br></font><br><font face="Calibri" size="2">002500> cat /var/log/messages | grep -r "name lookup failed for"</font> <br>
<font face="Calibri" size="2">Apr 8 06:52:08 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[2361]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 8 06:52:09 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[2639]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 8 06:52:09 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[2657]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 8 06:52:09 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[2677]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 8 06:52:10 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[2878]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2"> </font> <br>
<font face="Calibri" size="2"># cat /var/log/glusterfs/cmd_history.log | grep -r "SUCCESS"</font> <br>
<font face="Calibri" size="2">[2016-04-08 06:52:13.687779] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-08 06:52:13.703139] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-08 06:52:14.041347] : volume remove-brick c_glusterfs replica 1 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-08 06:52:14.135737] : peer detach 10.32.1.144 : SUCCESS </font> <br>
<font face="Calibri" size="2">[2016-04-08 06:52:15.415575] : peer probe 10.32.1.144 : SUCCESS </font> <br>
<font face="Calibri" size="2">[2016-04-08 06:52:17.941650] : volume add-brick c_glusterfs replica 2 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS<br><br><br></font></div><font face="Calibri" size="2">Setup 2:<br><br></font><font face="Calibri" size="2"># cat /var/log/glusterfs/cmd_history.log | grep -r "SUCCESS"</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:48:00.544269] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:48:00.564007] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:48:01.624280] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:48:01.642542] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:48:02.699085] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:48:02.716108] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:48:03.782118] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:48:03.832059] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:22.709733] : volume set c_glusterfs nfs.disable on : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:24.037270] : volume start c_glusterfs force : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:34.326782] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:34.352975] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:35.441763] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:35.467474] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:36.544532] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:36.563667] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:37.633660] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:37.653251] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:38.726406] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:38.746097] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:39.805968] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:39.824601] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:40.886599] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:40.905728] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:41.963659] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:41.980006] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:43.037351] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:43.053252] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:44.112666] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:44.129299] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:45.186047] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:45.201741] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:46.259575] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:46.274823] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:47.332110] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:47.347350] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:48.408752] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:48.424115] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:49.481196] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:49.496417] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:50.556596] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:50.572201] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:51.645649] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:51.660980] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:52.718777] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:52.734715] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:53.798818] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:53.814556] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:54.873792] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:54.889273] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:55.949193] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:55.964678] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:57.026010] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:57.042384] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:58.104969] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:58.121515] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:59.177334] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:56:59.193038] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:00.249726] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:00.264757] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:01.321589] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:01.337437] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:02.394071] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:02.409723] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:03.465013] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:03.479919] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:04.535535] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:04.550799] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:05.607067] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:05.622387] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:06.677708] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:06.692804] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:07.749353] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:07.764955] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:08.821939] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:08.838049] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:09.892807] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:09.907671] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:10.962583] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:10.977470] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:12.032784] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:12.048140] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:13.102801] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:13.117916] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:14.174420] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:14.190000] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:15.245023] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:15.260118] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:16.316116] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:16.331088] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:17.385876] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:17.401176] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:18.456077] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:18.471002] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:19.526370] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:19.541316] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:20.599767] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:20.614622] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:21.670278] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:21.686137] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:22.741620] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:22.757029] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:23.814023] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:23.828713] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:24.885976] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:24.900984] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:25.956532] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:25.971524] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:27.027527] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:27.043078] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:28.098295] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:28.113567] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:29.176173] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:29.191290] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:30.246027] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:30.261358] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:31.319007] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:31.333841] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:32.388578] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:32.405997] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:33.460740] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:33.475143] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:34.529614] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:34.544573] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:35.601543] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:35.616131] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:36.671247] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:36.686742] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:37.742293] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:37.757473] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:38.815902] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:38.832081] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:39.887027] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:39.902269] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:40.957761] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:40.973322] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:42.028674] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:42.045765] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:43.104397] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:43.121970] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:44.183029] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:44.201168] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:45.256772] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:45.274251] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:46.331280] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:46.349052] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:47.408266] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:47.426468] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:48.483301] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:48.500622] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:49.556927] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:49.573825] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:50.630662] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:50.647136] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:51.705079] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:51.722046] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:52.777994] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:52.794681] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:53.853193] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:53.870088] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:54.928807] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:54.946332] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:56.003859] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:56.020624] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:57.076639] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:57.093430] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:58.148805] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:58.167321] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:59.225112] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:57:59.242433] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:00.300050] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:00.317123] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:01.373615] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:01.390987] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:02.448198] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:02.464742] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:03.522714] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:03.539341] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:04.595848] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:04.613142] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:05.670772] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:05.687886] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:06.744241] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:06.761412] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:07.818249] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:07.835174] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:08.891052] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:08.908118] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:09.963741] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:09.980837] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:11.036566] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:11.053823] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:12.111426] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:12.128725] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:13.184849] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:13.201892] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:14.257657] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:14.274758] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:15.330533] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:15.347758] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:16.405524] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:16.422845] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:17.480674] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:17.498446] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:18.555364] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:18.572318] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:19.629799] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:19.648091] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:20.710409] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:20.727520] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:21.785556] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:21.802865] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:22.018119] : volume remove-brick c_glusterfs replica 1 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:22.115762] : peer detach 10.32.1.144 : SUCCESS </font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:23.426499] : peer probe 10.32.1.144 : SUCCESS </font> <br>
<font face="Calibri" size="2">[2016-04-07 23:58:26.023473] : volume add-brick c_glusterfs replica 2 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-08 08:54:24.788298] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">[2016-04-08 08:54:24.804415] : volume status : SUCCESS</font> <br>
<font face="Calibri" size="2">#</font> <br>
<font face="Calibri" size="2"> </font> <br>
<font face="Calibri" size="2"> </font> <br>
<font face="Calibri" size="2">002500> cat /var/log/messages | grep -r "name lookup failed for"</font> <br>
<font face="Calibri" size="2">Apr 7 23:47:55 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[2428]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:47:56 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[2501]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:47:56 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[2517]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:47:56 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[2550]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:47:56 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[2665]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:47:56 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[2715]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:47:57 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[2740]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:47:57 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[2772]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:47:57 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[2797]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:47:57 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[2822]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:47:57 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[2837]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:48:01 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[2844]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:48:01 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[2851]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:48:01 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[2858]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:48:02 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[2881]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:48:02 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[2901]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:48:02 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[2912]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:48:02 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[2931]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:48:02 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[2938]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:48:02 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[2951]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:48:03 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[3032]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:56:33 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[10587]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:56:34 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[10713]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:56:34 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[10719]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:56:34 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[10726]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:56:35 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[10733]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:56:35 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[10749]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:56:35 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[10777]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:56:35 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[10805]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:56:35 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[10812]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:56:35 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[10822]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:56:36 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[10830]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:56:36 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[10869]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:56:36 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[10875]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:56:36 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[10892]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:56:36 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[10900]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:56:36 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[10906]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:56:36 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[10914]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:56:37 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[10920]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:56:37 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[10927]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:56:37 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[10934]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">Apr 7 23:56:38 2016 oamhost <a href="http://daemon.info">daemon.info</a> rsyncd[10980]: name lookup failed for <a href="http://10.32.0.48">10.32.0.48</a>: Temporary failure in name resolution</font> <br>
<font face="Calibri" size="2">002500><br><br></font></div><font face="Calibri" size="2">Regards,<br></font></div><font face="Calibri" size="2">Abhishek<br></font><div><div><div><div><div><font face="Calibri" size="2"><br><br></font></div></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 5, 2016 at 3:35 PM, ABHISHEK PALIWAL <span dir="ltr"><<a href="mailto:abhishpaliwal@gmail.com" target="_blank">abhishpaliwal@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Tue, Apr 5, 2016 at 2:22 PM, Atin Mukherjee <span dir="ltr"><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span><br>
<br>
On 04/05/2016 01:04 PM, ABHISHEK PALIWAL wrote:<br>
> Hi Team,<br>
><br>
> We are using Gluster 3.7.6 and facing one problem in which brick is not<br>
> comming online after restart the board.<br>
><br>
> To understand our setup, please look the following steps:<br>
> 1. We have two boards A and B on which Gluster volume is running in<br>
> replicated mode having one brick on each board.<br>
> 2. Gluster mount point is present on the Board A which is sharable<br>
> between number of processes.<br>
> 3. Till now our volume is in sync and everthing is working fine.<br>
> 4. Now we have test case in which we'll stop the glusterd, reboot the<br>
> Board B and when this board comes up, starts the glusterd again on it.<br>
> 5. We repeated Steps 4 multiple times to check the reliability of system.<br>
> 6. After the Step 4, sometimes system comes in working state (i.e. in<br>
> sync) but sometime we faces that brick of Board B is present in<br>
> “gluster volume status” command but not be online even waiting for<br>
> more than a minute.<br>
</span>As I mentioned in another email thread until and unless the log shows<br>
the evidence that there was a reboot nothing can be concluded. The last<br>
log what you shared with us few days back didn't give any indication<br>
that brick process wasn't running.<br></blockquote></span><div>How can we identify that the brick process is running in brick logs? <br></div><span class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<span>> 7. When the Step 4 is executing at the same time on Board A some<br>
> processes are started accessing the files from the Gluster mount point.<br>
><br>
> As a solution to make this brick online, we found some existing issues<br>
> in gluster mailing list giving suggestion to use “gluster volume start<br>
> <vol_name> force” to make the brick 'offline' to 'online'.<br>
><br>
> If we use “gluster volume start <vol_name> force” command. It will kill<br>
> the existing volume process and started the new process then what will<br>
> happen if other processes are accessing the same volume at the time when<br>
> volume process is killed by this command internally. Will it impact any<br>
> failure on these processes?<br>
</span>This is not true, volume start force will start the brick processes only<br>
if they are not running. Running brick processes will not be interrupted.<br></blockquote></span><div>we have tried and check the pid of process before force start and after force start.<br></div><div>the pid has been changed after force start.<br><br></div><div>Please find the logs at the time of failure attached once again with log-level=debug.<br><br></div><div>if you can give me the exact line where you are able to find out that the brick process<br></div><div>is running in brick log file please give me the line number of that file.<br></div><div><br></div><div>002500 - Board B that brick is offline<br></div><div>00300 - Board A logs<br></div><span class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
><br>
> *Question : What could be contributing to brick offline?*<br>
><br>
<span><font color="#888888">><br>
> --<br>
><br>
> Regards<br>
> Abhishek Paliwal<br>
><br>
><br>
> _______________________________________________<br>
> Gluster-devel mailing list<br>
> <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
><br>
</font></span></blockquote></span></div><br><br clear="all"><br></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature"><div dir="ltr"><br><br><br><br>Regards<br>
Abhishek Paliwal<br>
</div></div>
</div>