<div dir="ltr"><div>Seems I found a problem. The problem was in the gluster_shared_storage. All nodes had gluster_shared_storage and when I tried to remove a node from the cluster it hadn't took any effects in shared storage. In shared storage it still was in peer list. </div><div>I.e. this command won't work with eneabled shared storage:</div><div>gluster peer detach 192.168.0.124</div><div><br></div><div>So, I removed shared storage and then removed a node:</div><div>gluster volume set all cluster.enable-shared-storage disable</div><div>gluster peer detach 192.168.0.124</div><div>gluster volume set all cluster.enable-shared-storage enable</div><div><br></div><div>It isn't very convenient. Also, now the main problem is when I adding a new node. How to add shared_storage to that node without removing shared storage and creating a new shared storage?</div><div><br></div><div>Sincerely,</div><div>Alexandr</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Nov 27, 2016 at 3:45 PM, Lindsay Mathieson <span dir="ltr"><<a href="mailto:lindsay.mathieson@gmail.com" target="_blank">lindsay.mathieson@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div class="m_-1772427067462644155moz-cite-prefix">bricks are not peers and vica versa.<br>
<br>
Your peers are the nodes, bricks are the disks on the nodes. When
you remove a bricks from the cluster you don't remove the peer.<span class=""><br>
<br>
# gluster peer detach 192.168.0.124:/data/brick1<br>
<br></span>
Incorrect syntax, the command is for removing the peer, not the
brick. It should be:<span class=""><br>
<br>
# gluster peer detach 192.168.0.124<br>
<br>
<br></span><div><div class="h5">
On 27/11/2016 8:49 PM, Alexandr Porunov wrote:<br>
</div></div></div><div><div class="h5">
<blockquote type="cite">
<div dir="ltr"># gluster volume status gv0<br>
<div>
<div>Status of volume: gv0</div>
<div>Gluster process TCP Port
RDMA Port Online Pid</div>
<div>------------------------------<wbr>------------------------------<wbr>------------------</div>
<div>Brick 192.168.0.123:/data/brick1/gv0 N/A N/A
N N/A </div>
<div>Brick 192.168.0.125:/data/brick1/gv0 49152 0
Y 1396 </div>
<div>Self-heal Daemon on localhost N/A N/A
Y 3252 </div>
<div>Self-heal Daemon on 192.168.0.125 N/A N/A
Y 13339</div>
<div> </div>
<div>Task Status of Volume gv0</div>
<div>------------------------------<wbr>------------------------------<wbr>------------------</div>
<div>There are no active volume tasks</div>
</div>
<div><br>
</div>
<div>It doesn't show that 192.168.0.124 is in the volume but it
is in the cluster. Here is why:</div>
<div><br>
</div>
<div>When I try to add it back to peer list it doesn't do
anything. Because it says that it is already in a peer list:</div>
<div>
<div><br>
</div>
<div># gluster peer probe 192.168.0.124</div>
<div>peer probe: success. Host 192.168.0.124 port 24007
already in peer list</div>
</div>
<div><br>
</div>
<div>OK. I go to the machine 192.168.0.124 and try to show a
peer list:</div>
<div>
<div># gluster peer status</div>
<div>Number of Peers: 0</div>
</div>
<div><br>
</div>
<div>OK. I go to the machine 192.168.0.123 and try to show peer
status:</div>
<div>
<div># gluster peer status</div>
<div>Number of Peers: 2</div>
<div><br>
</div>
<div>Hostname: 192.168.0.125</div>
<div>Uuid: a6ed1da8-3027-4400-afed-<wbr>96429380fdc9</div>
<div>State: Peer in Cluster (Connected)</div>
<div><br>
</div>
<div>Hostname: 192.168.0.124</div>
<div>Uuid: b7d829f3-80d9-4a78-90b8-<wbr>f018bc758df0</div>
<div>State: Peer Rejected (Connected)</div>
</div>
<div><br>
</div>
<div>As we see machine with ip 192.168.0.123 thinks that
192.168.0.124 is in the cluster. OK lets remove it from the
cluster:</div>
<div>
<div><br>
</div>
<div># gluster peer detach 192.168.0.124:/data/brick1</div>
<div>peer detach: failed: 192.168.0.124:/data/brick1 is not
part of cluster</div>
</div>
<div><br>
</div>
<div>
<div># gluster peer detach 192.168.0.124</div>
<div>peer detach: failed: Brick(s) with the peer 192.168.0.124
exist in cluster</div>
</div>
<div><br>
</div>
<div>Isn't it strange? It is in the cluster and it isn't in the
cluster. I can't neither add machine with IP 192.168.0.124 nor
remove machine with IP 192.168.0.124</div>
<div><br>
</div>
<div>Do you know what is wrong with it?</div>
<div><br>
</div>
<div>Sincerely,</div>
<div>Alexandr</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sun, Nov 27, 2016 at 12:29 PM,
Lindsay Mathieson <span dir="ltr"><<a href="mailto:lindsay.mathieson@gmail.com" target="_blank">lindsay.mathieson@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On 27/11/2016 7:28 PM, Alexandr Porunov wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
# Above command showed success but in reality brick is
still in the cluster.<br>
</blockquote>
<br>
</span>
What makes you think this? what does a "gluster v gv0" show?<span class="m_-1772427067462644155HOEnZb"><font color="#888888"><br>
<br>
<br>
-- <br>
Lindsay Mathieson<br>
<br>
</font></span></blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<p><br>
</p>
</div></div><span class="HOEnZb"><font color="#888888"><pre class="m_-1772427067462644155moz-signature" cols="72">--
Lindsay Mathieson</pre>
</font></span></div>
</blockquote></div><br></div>