<div dir="ltr"><div><div><div><div>Hey Atul,<br><br></div>You&#39;d need to provide adequate information to us to get to the actual issue. I&#39;d recommend you to provide the following:<br><br></div><div>1. The detailed description of the problem statement.<br></div>2. Steps you did in the cluster.<br></div>3. log files from all the nodes in the cluster.<br><br></div><div>You can file a bug for the same if required.<br><br></div><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Nov 10, 2016 at 11:09 AM, Atul Yadav <span dir="ltr">&lt;<a href="mailto:atulyadavtech@gmail.com" target="_blank">atulyadavtech@gmail.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Team,<div><br></div><div>Can i have update on the issue..</div><div><br></div><div>Thank You</div><span class="HOEnZb"><font color="#888888"><div>Atul Yadav</div><div><br></div></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 5, 2016 at 4:56 PM, Atul Yadav <span dir="ltr">&lt;<a href="mailto:atulyadavtech@gmail.com" target="_blank">atulyadavtech@gmail.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span style="font-size:12.8px">Hi Team,</span><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Updated </div><div style="font-size:12.8px">No , gluster commands are not working from the same node.( in the below mention event)</div><div>df command will halt the server. <br></div></div></blockquote></div></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Note:- After rebooting entire cluster the gluster process is working fine.</div><div style="font-size:12.8px">But at the time of the failure(one node forcibly down), gluster peer still showing connected to that node.</div><div style="font-size:12.8px"><br></div></span><div style="font-size:12.8px">Looking for solution to fix this issue.</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Thank You</div><div style="font-size:12.8px"><div id="m_-1014848396031058734m_5920054098340404217:1ek"><img src="https://ssl.gstatic.com/ui/v1/icons/mail/images/cleardot.gif"></div></div></div><div class="m_-1014848396031058734HOEnZb"><div class="m_-1014848396031058734h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 5, 2016 at 4:48 PM, Atul Yadav <span dir="ltr">&lt;<a href="mailto:atulyadavtech@gmail.com" target="_blank">atulyadavtech@gmail.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Team,<div><br></div><div>Yes, gluster commands are working from the same node.</div><div><br></div><div>Note:- After rebooting entire cluster the gluster process is working fine.</div><div>But at the time of the failure(one node forcibly down), gluster peer still showing connected to that node.</div><div><br></div><div>Thank You</div><span class="m_-1014848396031058734m_5920054098340404217HOEnZb"><font color="#888888"><div>Atul Yadav</div><div> </div></font></span></div><div class="m_-1014848396031058734m_5920054098340404217HOEnZb"><div class="m_-1014848396031058734m_5920054098340404217h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 5, 2016 at 4:32 PM, Atin Mukherjee <span dir="ltr">&lt;<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>I could see that glusterd process is up from the log file. Aren&#39;t you able to run any gluster commands on the same node?<span><font color="#888888"><br><br></font></span></div><span><font color="#888888">~Atin<br></font></span></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 5, 2016 at 4:05 PM, Atul Yadav <span dir="ltr">&lt;<a href="mailto:atulyadavtech@gmail.com" target="_blank">atulyadavtech@gmail.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Team,<div><br></div><div>Please go through the attachment </div><div><br></div><div>Thank you</div><span><font color="#888888"><div>Atul Yadav</div></font></span></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 5, 2016 at 3:55 PM, Samikshan Bairagya <span dir="ltr">&lt;<a href="mailto:sbairagy@redhat.com" target="_blank">sbairagy@redhat.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span><br>
<br>
On 07/05/2016 03:15 PM, Atul Yadav wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Atin,<br>
<br>
please go through the attachment for log file.<br>
<br>
Waiting for your input....<br>
<br>
</blockquote>
<br></span>
Hi Atul,<br>
<br>
The glusterd log file can be found here: /var/log/glusterfs/usr-local-e<wbr>tc-glusterfs-glusterd.vol.log<br>
<br>
Sharing that instead of a RAR archive would make things easier for us.<div><div><br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
thank you<br>
Atul Yadav<br>
<br>
On Tue, Jul 5, 2016 at 2:43 PM, Atin Mukherjee &lt;<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>&gt; wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Why don&#39;t you share the glusterd log file?<br>
<br>
On Tue, Jul 5, 2016 at 12:53 PM, Atul Yadav &lt;<a href="mailto:atulyadavtech@gmail.com" target="_blank">atulyadavtech@gmail.com</a>&gt;<br>
wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi ,<br>
<br>
After restarting the service. service entered in to the fail state.<br>
[root@master1 ~]# /etc/init.d/glusterd restart<br>
Stopping glusterd:                                         [FAILED]<br>
Starting glusterd:                                         [FAILED]<br>
<br>
Note: This behavior only happening over rdma network. But with ethernet<br>
there is no issue.<br>
<br>
Thank you<br>
Atul Yadav<br>
<br>
<br>
<br>
On Tue, Jul 5, 2016 at 11:28 AM, Atin Mukherjee &lt;<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>&gt;<br>
wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
On Tue, Jul 5, 2016 at 11:01 AM, Atul Yadav &lt;<a href="mailto:atulyadavtech@gmail.com" target="_blank">atulyadavtech@gmail.com</a>&gt;<br>
wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi All,<br>
<br>
The glusterfs environment details are given below:-<br>
<br>
[root@master1 ~]# cat /etc/redhat-release<br>
CentOS release 6.7 (Final)<br>
[root@master1 ~]# uname -r<br>
2.6.32-642.1.1.el6.x86_64<br>
[root@master1 ~]# rpm -qa | grep -i gluster<br>
glusterfs-rdma-3.8rc2-1.el6.x8<wbr>6_64<br>
glusterfs-api-3.8rc2-1.el6.x86<wbr>_64<br>
glusterfs-3.8rc2-1.el6.x86_64<br>
glusterfs-cli-3.8rc2-1.el6.x86<wbr>_64<br>
glusterfs-client-xlators-3.8rc<wbr>2-1.el6.x86_64<br>
glusterfs-server-3.8rc2-1.el6.<wbr>x86_64<br>
glusterfs-fuse-3.8rc2-1.el6.x8<wbr>6_64<br>
glusterfs-libs-3.8rc2-1.el6.x8<wbr>6_64<br>
[root@master1 ~]#<br>
<br>
Volume Name: home<br>
Type: Replicate<br>
Volume ID: 2403ddf9-c2e0-4930-bc94-734772<wbr>ef099f<br>
Status: Stopped<br>
Number of Bricks: 1 x 2 = 2<br>
Transport-type: rdma<br>
Bricks:<br>
Brick1: master1-ib.dbt.au:/glusterfs/h<wbr>ome/brick1<br>
Brick2: master2-ib.dbt.au:/glusterfs/h<wbr>ome/brick2<br>
Options Reconfigured:<br>
network.ping-timeout: 20<br>
nfs.disable: on<br>
performance.readdir-ahead: on<br>
transport.address-family: inet<br>
config.transport: rdma<br>
cluster.server-quorum-type: server<br>
cluster.quorum-type: fixed<br>
cluster.quorum-count: 1<br>
locks.mandatory-locking: off<br>
cluster.enable-shared-storage: disable<br>
cluster.server-quorum-ratio: 51%<br>
<br>
When my single master node is up only, but other nodes are still<br>
showing connected mode ....<br>
gluster pool list<br>
UUID                                    Hostname                State<br>
89ccd72e-cb99-4b52-a2c0-388c99<wbr>e5c7b3    <a href="http://master2-ib.dbt.au" rel="noreferrer" target="_blank">master2-ib.dbt.au</a><br>
Connected<br>
d2c47fc2-f673-4790-b368-d214a5<wbr>8c59f4    <a href="http://compute01-ib.dbt.au" rel="noreferrer" target="_blank">compute01-ib.dbt.au</a><br>
Connected<br>
a5608d66-a3c6-450e-a239-108668<wbr>083ff2    localhost<br>
Connected<br>
[root@master1 ~]#<br>
<br>
<br>
Please advise us<br>
Is this normal behavior Or This is issue.<br>
<br>
</blockquote>
<br>
First of, we don&#39;t have any master slave configuration mode for gluster<br>
trusted storage pool i.e. peer list. Secondly, if master2 and compute01 are<br>
still reflecting as &#39;connected&#39; even though they are down it means that<br>
localhost here didn&#39;t receive disconnect events for some reason. Could you<br>
restart glusterd service on this node and check the output of gluster pool<br>
list again?<br>
<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Thank You<br>
Atul Yadav<br>
<br>
<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a><br>
<br>
</blockquote>
<br>
<br>
</blockquote>
<br>
</blockquote>
<br>
</blockquote>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a><br>
<br>
</blockquote>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><br></div><div>~ Atin (atinm)<br></div></div></div></div>
</div></div></div>