<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Thanks Atin, the files you asked :&nbsp;<a href="https://we.tl/XrOvFhffGq" class="">https://we.tl/XrOvFhffGq</a></div><div class=""><br class=""></div><div><blockquote type="cite" class=""><div class="">On 13 Dec 2016, at 19:08, Atin Mukherjee &lt;<a href="mailto:amukherj@redhat.com" class="">amukherj@redhat.com</a>&gt; wrote:</div><br class="Apple-interchange-newline"><div class=""><div class=""><div class="">Thanks, we will get back on this. In the mean time can you please also share glusterd statedump file from both the nodes? The way to take statedump is 'kill -SIGUSR1 $(pidof glusterd)' and the file can be found at /var/run/gluster directory.</div></div><div class=""><br class=""><div class="gmail_quote"><div class="">On Tue, 13 Dec 2016 at 22:11, Cedric Lemarchand &lt;<a href="mailto:yipikai7@gmail.com" class="">yipikai7@gmail.com</a>&gt; wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class="gmail_msg">1. sorry, 3.9.0-1<div class="gmail_msg">2. no it does nothing</div><div class="gmail_msg">3. here they are, from gl1 to gl6 :&nbsp;<a href="https://we.tl/EPaMs6geoR" class="gmail_msg" target="_blank">https://we.tl/EPaMs6geoR</a></div></div><div style="word-wrap:break-word" class="gmail_msg"><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><div class="gmail_msg"><br class="gmail_msg"><div class="gmail_msg"><blockquote type="cite" class="gmail_msg"><div class="gmail_msg">On 13 Dec 2016, at 16:49, Atin Mukherjee &lt;<a href="mailto:amukherj@redhat.com" class="gmail_msg" target="_blank">amukherj@redhat.com</a>&gt; wrote:</div><br class="gmail_msg m_-2010779676140610534Apple-interchange-newline"><div class="gmail_msg"><div class="gmail_msg">And 3. In case 2 doesn't work, please provide the glusterd log files from gl1 &amp; gl5<br class="gmail_msg"></div><div class="gmail_extra gmail_msg"><br class="gmail_msg"><div class="gmail_quote gmail_msg">On Tue, Dec 13, 2016 at 9:16 PM, Atin Mukherjee <span class="gmail_msg">&lt;<a href="mailto:amukherj@redhat.com" class="gmail_msg" target="_blank">amukherj@redhat.com</a>&gt;</span> wrote:<br class="gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_msg"><div class="gmail_msg">1. Could you mention which gluster version are you running with?<br class="gmail_msg"></div>2. Does restarting glusterd instance on gl1 &amp; gl5 solves the issue (after removing the volume-id xattr from the bricks) ?<br class="gmail_msg"></div><div class="gmail_extra gmail_msg"><div class="gmail_msg"><div class="gmail_msg m_-2010779676140610534h5"><br class="gmail_msg"><div class="gmail_quote gmail_msg">On Tue, Dec 13, 2016 at 8:56 PM, Cedric Lemarchand <span class="gmail_msg">&lt;<a href="mailto:yipikai7@gmail.com" class="gmail_msg" target="_blank">yipikai7@gmail.com</a>&gt;</span> wrote:<br class="gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">When I try to add 3 bricks to a working cluster composed of 3 nodes / 3 bricks in dispersed mode 2+1, it fails like this :<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">root@gl1:~# gluster volume add-brick vol1 gl4:/data/br1 gl5:/data/br1 gl6:/data/br1<br class="gmail_msg"><br class=""><br class="">volume add-brick: failed: Pre Validation failed on gl4. Host gl5 not connected<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">However all peers are connected and there aren't networking issues :<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">root@gl1:~# gluster peer status<br class="gmail_msg"><br class=""><br class="">Number of Peers: 5<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">Hostname: gl2<br class="gmail_msg"><br class=""><br class="">Uuid: 616f100f-a3f4-46e4-b161-ee5db5a60e26<br class="gmail_msg"><br class=""><br class="">State: Peer in Cluster (Connected)<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">Hostname: gl3<br class="gmail_msg"><br class=""><br class="">Uuid: acb828b8-f4b3-42ab-a9d2-b3e7b917dc9a<br class="gmail_msg"><br class=""><br class="">State: Peer in Cluster (Connected)<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">Hostname: gl4<br class="gmail_msg"><br class=""><br class="">Uuid: 813ad056-5e84-4fdb-ac13-38d24c748bc4<br class="gmail_msg"><br class=""><br class="">State: Peer in Cluster (Connected)<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">Hostname: gl5<br class="gmail_msg"><br class=""><br class="">Uuid: a7933aeb-b08b-4ebb-a797-b8ecbe5a03c6<br class="gmail_msg"><br class=""><br class="">State: Peer in Cluster (Connected)<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">Hostname: gl6<br class="gmail_msg"><br class=""><br class="">Uuid: 63c9a6c1-0adf-4cf5-af7b-b28a60911c99<br class="gmail_msg"><br class=""><br class="">State: Peer in Cluster (Connected)<br class="gmail_msg"><br class=""><br class="">&nbsp;:<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">When I try a second time, the error is different :<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">root@gl1:~# gluster volume add-brick vol1 gl4:/data/br1 gl5:/data/br1 gl6:/data/br1<br class="gmail_msg"><br class=""><br class="">volume add-brick: failed: Pre Validation failed on gl5. /data/br1 is already part of a volume<br class="gmail_msg"><br class=""><br class="">Pre Validation failed on gl6. /data/br1 is already part of a volume<br class="gmail_msg"><br class=""><br class="">Pre Validation failed on gl4. /data/br1 is already part of a volume<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">It seems the previous try, even if it has failed, have created the gluster attributes on file system as shown by attr on gl4/5/6 :<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">Attribute "glusterfs.volume-id" has a 16 byte value for /data/br1<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">I already purge gluster and reformat brick on gl4/5/6 but the issue persist, any ideas ? did I miss something ?<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">Some informations :<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">root@gl1:~# gluster volume info<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">Volume Name: vol1<br class="gmail_msg"><br class=""><br class="">Type: Disperse<br class="gmail_msg"><br class=""><br class="">Volume ID: bb563884-0e2a-4757-9fd5-cb851ba113c3<br class="gmail_msg"><br class=""><br class="">Status: Started<br class="gmail_msg"><br class=""><br class="">Snapshot Count: 0<br class="gmail_msg"><br class=""><br class="">Number of Bricks: 1 x (2 + 1) = 3<br class="gmail_msg"><br class=""><br class="">Transport-type: tcp<br class="gmail_msg"><br class=""><br class="">Bricks:<br class="gmail_msg"><br class=""><br class="">Brick1: gl1:/data/br1<br class="gmail_msg"><br class=""><br class="">Brick2: gl2:/data/br1<br class="gmail_msg"><br class=""><br class="">Brick3: gl3:/data/br1<br class="gmail_msg"><br class=""><br class="">Options Reconfigured:<br class="gmail_msg"><br class=""><br class="">features.scrub-freq: hourly<br class="gmail_msg"><br class=""><br class="">features.scrub: Inactive<br class="gmail_msg"><br class=""><br class="">features.bitrot: off<br class="gmail_msg"><br class=""><br class="">cluster.disperse-self-heal-daemon: enable<br class="gmail_msg"><br class=""><br class="">transport.address-family: inet<br class="gmail_msg"><br class=""><br class="">performance.readdir-ahead: on<br class="gmail_msg"><br class=""><br class="">nfs.disable: on, I have the following error :<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">root@gl1:~# gluster volume status<br class="gmail_msg"><br class=""><br class="">Status of volume: vol1<br class="gmail_msg"><br class=""><br class="">Gluster process&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;TCP Port&nbsp; RDMA Port&nbsp; Online&nbsp; Pid<br class="gmail_msg"><br class=""><br class="">------------------------------------------------------------------------------<br class="gmail_msg"><br class=""><br class="">Brick gl1:/data/br1&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;49152&nbsp; &nbsp; &nbsp;0&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Y&nbsp; &nbsp; &nbsp; &nbsp;23403<br class="gmail_msg"><br class=""><br class="">Brick gl2:/data/br1&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;49152&nbsp; &nbsp; &nbsp;0&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Y&nbsp; &nbsp; &nbsp; &nbsp;14545<br class="gmail_msg"><br class=""><br class="">Brick gl3:/data/br1&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;49152&nbsp; &nbsp; &nbsp;0&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Y&nbsp; &nbsp; &nbsp; &nbsp;11348<br class="gmail_msg"><br class=""><br class="">Self-heal Daemon on localhost&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;N/A&nbsp; &nbsp; &nbsp; &nbsp;N/A&nbsp; &nbsp; &nbsp; &nbsp; Y&nbsp; &nbsp; &nbsp; &nbsp;24766<br class="gmail_msg"><br class=""><br class="">Self-heal Daemon on gl4&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;N/A&nbsp; &nbsp; &nbsp; &nbsp;N/A&nbsp; &nbsp; &nbsp; &nbsp; Y&nbsp; &nbsp; &nbsp; &nbsp;1087<br class="gmail_msg"><br class=""><br class="">Self-heal Daemon on gl5&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;N/A&nbsp; &nbsp; &nbsp; &nbsp;N/A&nbsp; &nbsp; &nbsp; &nbsp; Y&nbsp; &nbsp; &nbsp; &nbsp;1080<br class="gmail_msg"><br class=""><br class="">Self-heal Daemon on gl3&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;N/A&nbsp; &nbsp; &nbsp; &nbsp;N/A&nbsp; &nbsp; &nbsp; &nbsp; Y&nbsp; &nbsp; &nbsp; &nbsp;12321<br class="gmail_msg"><br class=""><br class="">Self-heal Daemon on gl2&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;N/A&nbsp; &nbsp; &nbsp; &nbsp;N/A&nbsp; &nbsp; &nbsp; &nbsp; Y&nbsp; &nbsp; &nbsp; &nbsp;15496<br class="gmail_msg"><br class=""><br class="">Self-heal Daemon on gl6&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;N/A&nbsp; &nbsp; &nbsp; &nbsp;N/A&nbsp; &nbsp; &nbsp; &nbsp; Y&nbsp; &nbsp; &nbsp; &nbsp;1091<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">Task Status of Volume vol1<br class="gmail_msg"><br class=""><br class="">------------------------------------------------------------------------------<br class="gmail_msg"><br class=""><br class="">There are no active volume tasks<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">root@gl1:~# gluster volume info<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">Volume Name: vol1<br class="gmail_msg"><br class=""><br class="">Type: Disperse<br class="gmail_msg"><br class=""><br class="">Volume ID: bb563884-0e2a-4757-9fd5-cb851ba113c3<br class="gmail_msg"><br class=""><br class="">Status: Started<br class="gmail_msg"><br class=""><br class="">Snapshot Count: 0<br class="gmail_msg"><br class=""><br class="">Number of Bricks: 1 x (2 + 1) = 3<br class="gmail_msg"><br class=""><br class="">Transport-type: tcp<br class="gmail_msg"><br class=""><br class="">Bricks:<br class="gmail_msg"><br class=""><br class="">Brick1: gl1:/data/br1<br class="gmail_msg"><br class=""><br class="">Brick2: gl2:/data/br1<br class="gmail_msg"><br class=""><br class="">Brick3: gl3:/data/br1<br class="gmail_msg"><br class=""><br class="">Options Reconfigured:<br class="gmail_msg"><br class=""><br class="">features.scrub-freq: hourly<br class="gmail_msg"><br class=""><br class="">features.scrub: Inactive<br class="gmail_msg"><br class=""><br class="">features.bitrot: off<br class="gmail_msg"><br class=""><br class="">cluster.disperse-self-heal-daemon: enable<br class="gmail_msg"><br class=""><br class="">transport.address-family: inet<br class="gmail_msg"><br class=""><br class="">performance.readdir-ahead: on<br class="gmail_msg"><br class=""><br class="">nfs.disable: on<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">root@gl1:~# gluster peer status<br class="gmail_msg"><br class=""><br class="">Number of Peers: 5<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">Hostname: gl2<br class="gmail_msg"><br class=""><br class="">Uuid: 616f100f-a3f4-46e4-b161-ee5db5a60e26<br class="gmail_msg"><br class=""><br class="">State: Peer in Cluster (Connected)<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">Hostname: gl3<br class="gmail_msg"><br class=""><br class="">Uuid: acb828b8-f4b3-42ab-a9d2-b3e7b917dc9a<br class="gmail_msg"><br class=""><br class="">State: Peer in Cluster (Connected)<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">Hostname: gl4<br class="gmail_msg"><br class=""><br class="">Uuid: 813ad056-5e84-4fdb-ac13-38d24c748bc4<br class="gmail_msg"><br class=""><br class="">State: Peer in Cluster (Connected)<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">Hostname: gl5<br class="gmail_msg"><br class=""><br class="">Uuid: a7933aeb-b08b-4ebb-a797-b8ecbe5a03c6<br class="gmail_msg"><br class=""><br class="">State: Peer in Cluster (Connected)<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">Hostname: gl6<br class="gmail_msg"><br class=""><br class="">Uuid: 63c9a6c1-0adf-4cf5-af7b-b28a60911c99<br class="gmail_msg"><br class=""><br class="">State: Peer in Cluster (Connected)<br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class=""><br class="gmail_msg"><br class=""><br class="">_______________________________________________<br class="gmail_msg"><br class=""><br class="">Gluster-users mailing list<br class="gmail_msg"><br class=""><br class=""><a href="mailto:Gluster-users@gluster.org" class="gmail_msg" target="_blank">Gluster-users@gluster.org</a><br class="gmail_msg"><br class=""><br class=""><a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" class="gmail_msg" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br class="gmail_msg"><br class=""><br class=""></blockquote></div><br class="gmail_msg"><br clear="all" class="gmail_msg"><br class="gmail_msg"></div></div><span class="m_-2010779676140610534HOEnZb gmail_msg"><font color="#888888" class="gmail_msg">-- <br class="gmail_msg"><div class="gmail_msg m_-2010779676140610534m_-7665095754714168145gmail_signature" data-smartmail="gmail_signature"><div class="gmail_msg"><div class="gmail_msg"><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">~ Atin (atinm)<br class="gmail_msg"></div></div></div></div><br class=""><br class=""></font></span></div><br class=""><br class=""></blockquote></div><br class="gmail_msg"><br clear="all" class="gmail_msg"><br class="gmail_msg">-- <br class="gmail_msg"><div class="gmail_msg m_-2010779676140610534gmail_signature" data-smartmail="gmail_signature"><div class="gmail_msg"><div class="gmail_msg"><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">~ Atin (atinm)<br class="gmail_msg"></div></div></div></div><br class=""><br class=""></div><br class=""><br class=""></div></blockquote></div><br class="gmail_msg"></div></div></div></blockquote></div></div><div dir="ltr" class="">-- <br class=""></div><div data-smartmail="gmail_signature" class="">- Atin (atinm)</div>
</div></blockquote></div><br class=""></body></html>