<div dir="ltr">Saw the ping command, but I'm not familiar with that and port checking which is why I asked about telnet.<div><br></div><div>If you turn off the Ubuntu firewall as a test, does that help?</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jan 7, 2015 at 4:41 PM, Michael Bushey <span dir="ltr"><<a href="mailto:michael@realtymogul.com" target="_blank">michael@realtymogul.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi Ryan. Thanks for responding. :) Yes, as I mentioned fping3 shows all ports open to all machines. <br><br></div>root@web3:~# telnet web1 24007 <br>Trying 10.10.x.x...<br>Connected to web1.<br>Escape character is '^]'.<br><div><br>root@web3:~# traceroute web1<br>traceroute to web1 (10.10.x.x), 30 hops max, 60 byte packets<br> 1 10.11.0.1 (10.11.0.1) 0.967 ms 0.968 ms 0.960 ms<br> 2 web1 (10.10.x.x) 0.581 ms 0.621 ms 0.665 ms<br><br></div><div>ufw status<br>....<br>glusterfs ALLOW <a href="http://10.10.0.0/16" target="_blank">10.10.0.0/16</a><br>glusterfs ALLOW <a href="http://10.11.0.0/16" target="_blank">10.11.0.0/16</a><br><br><br>cat /etc/ufw/applications.d/ufw-glusterfs <br>[glusterfs]<br>title=GlusterFS<br>description=GlusterFS is an open source, distributed file system capable of scaling to several petabytes<br>ports=111,2049,24007,49152:49162/tcp|111/udp<br><br><br><br></div><div><br></div></div><div class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">On Wed, Jan 7, 2015 at 2:05 PM, Ryan Nix <span dir="ltr"><<a href="mailto:ryan.nix@gmail.com" target="_blank">ryan.nix@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Have you been able to telnet to the Gluster ports on the other peers? If you run a traceroute to the affected peers, what does it return?</div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Wed, Jan 7, 2015 at 3:44 PM, Michael Bushey <span dir="ltr"><<a href="mailto:michael@realtymogul.com" target="_blank">michael@realtymogul.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr"><div>I have two production machines (web1, web2) that are currently using Glusterfs. I added two new machines, web3 and web4. Web1 and web2 are peered and are running great. Web3 and web4 will peer with each other but web1 and web2 will not peer with web3 or web4 and vice versa.<br><br>All machines are running Gluster 3.6.1:<br>ii glusterfs-client 3.6.1-1 amd64 clustered file-system (client package)<br>ii glusterfs-common 3.6.1-1 amd64 GlusterFS common libraries and translator modules<br>ii glusterfs-server 3.6.1-1 amd64 clustered file-system (server package)<br><br>I ran "gluster volume set all cluster.op-version 30600" on web2, now all four machines have "operating-version=30600" in /var/lib/glusterd/<a href="http://glusterd.info" target="_blank">glusterd.info</a><br><br>All machines have the same kernel:<br>Linux web1 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt2-1 (2014-12-08) x86_64 GNU/Linux<br><br>/etc/hosts is managed with Ansible, all machines resolve by their names (web1, web2, web3, web4) to their internal IP.<br><br>"hping3 -8 111,2049,24007,49152-49162 -S [web1-web4]" shows all Gluster ports open from all servers.<br><br>root@web1:~# gluster peer probe web2<br>peer probe: success. Host web2 port 24007 already in peer list<br>root@web1:~# gluster peer probe web3<br>peer probe: failed: Probe returned with unknown errno 107<br>root@web1:~# gluster peer probe web4<br>peer probe: failed: Probe returned with unknown errno 107<br><br><br>root@web3:~# gluster peer probe web1<br>peer probe: failed: Probe returned with unknown errno 107<br>root@web3:~# gluster peer probe web2<br>peer probe: failed: Probe returned with unknown errno 107<br>root@web3:~# gluster peer probe web4<br>peer probe: success. Host web4 port 24007 already in peer list<br><br><br>from /var/log/glusterfs/cli.log:<br><br>[2015-01-07 20:38:33.978304] T [cli.c:264:cli_rpc_notify] 0-glusterfs: got RPC_CLNT_CONNECT<br>[2015-01-07 20:38:33.978322] T [cli-quotad-client.c:94:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_CONNECT<br>[2015-01-07 20:38:33.978333] I [socket.c:2344:socket_event_handler] 0-transport: disconnecting now<br>[2015-01-07 20:38:33.978354] T [cli-quotad-client.c:100:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_DISCONNECT<br>[2015-01-07 20:38:33.978622] T [rpc-clnt.c:1381:rpc_clnt_record] 0-glusterfs: Auth Info: pid: 0, uid: 0, gid: 0, owner:<br>[2015-01-07 20:38:33.978681] T [rpc-clnt.c:1238:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 144, payload: 80, rpc hdr: 64<br>[2015-01-07 20:38:33.979037] T [socket.c:2863:socket_connect] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1cd)[0x7f9e8ef0ac1d] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.6.1/rpc-transport/socket.so(+0x637c)[0x7f9e8bcc637c] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x3a4)[0x7f9e8e40fad4] (--> gluster(cli_submit_request+0x15f)[0x7f9e8f3c52df] (--> gluster(cli_cmd_submit+0x8b)[0x7f9e8f3c6eeb] ))))) 0-glusterfs: connect () called on transport already connected<br>[2015-01-07 20:38:33.979100] T [rpc-clnt.c:1573:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x1 Program: Gluster CLI, ProgVers: 2, Proc: 1) to rpc-transport (glusterfs)<br>[2015-01-07 20:38:33.979111] D [rpc-clnt-ping.c:231:rpc_clnt_start_ping] 0-glusterfs: ping timeout is 0, returning<br>[2015-01-07 20:38:33.986269] T [rpc-clnt.c:660:rpc_clnt_reply_init] 0-glusterfs: received rpc message (RPC XID: 0x1 Program: Gluster CLI, ProgVers: 2, Proc: 1) from rpc-transport (glusterfs)<br>[2015-01-07 20:38:33.986311] I [cli-rpc-ops.c:131:gf_cli_probe_cbk] 0-cli: Received resp to probe<br>[2015-01-07 20:38:33.986321] E [cli-rpc-ops.c:136:gf_cli_probe_cbk] 0-cli: Probe returned with unknown errno 107<br>[2015-01-07 20:38:33.986620] D [cli-cmd.c:384:cli_cmd_submit] 0-cli: Returning -1<br>[2015-01-07 20:38:33.986638] D [cli-rpc-ops.c:3021:gf_cli_probe] 0-cli: Returning -1<br>[2015-01-07 20:38:33.986650] D [cli-cmd-peer.c:96:cli_cmd_peer_probe_cbk] 0-cli: frame->local is not NULL (0x7f9e7c000980)<br>[2015-01-07 20:38:33.986667] I [input.c:36:cli_batch] 0-: Exiting with: -1<br><br><br><br></div>Thanks in advance for any help.<br><div><br><br></div></div>
<br></div></div>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>
</blockquote></div><br><br clear="all"><br></div></div>-- <br><div><div dir="ltr"><div><b>Michael Bushey</b><br></div><div>Director of Technology</div><div>Realty Mogul, Co.</div><div><a href="mailto:michael@realtymogul.com" target="_blank">michael@realtymogul.com</a></div><div>Cell: <a href="tel:%28310%29%20619-0414" value="+13106190414" target="_blank">(310) 619-0414</a></div><div>via Front Desk: <a href="tel:%28310%29%20907-7129" value="+13109077129" target="_blank">(310) 907-7129</a> (ext. 124)</div><div><br></div><div><br></div><div><b>Realty Mogul in 19 words:</b> Realty Mogul is a marketplace for accredited investors to pool money online and buy shares of pre-vetted investment properties.</div><div><br></div><div><font size="1">Please Note: This email, including attachments, is intended only for the addresses(s) indicated, and may include non-public, proprietary, confidential or legally privileged information. If you are not an intended recipient or an authorized agent of an intended recipient, you are hereby notified that any dissemination, distribution or copying of the information contained in or transmitted with this e-mail is unauthorized and strictly prohibited. If you have received this email in error, delete it and any attachments or copies immediately. </font></div><div><font size="1"><br></font></div><div><font size="1">Realty Mogul does not make investment recommendations, and this message should not be construed as such. This message is not an offer to sell or the solicitation of an offer to buy any security or interest in any fund, which only can be made through official offering documents that contain important information about risks, fees and expenses. Any investment information contained herein has been secured from sources Realty Mogul believes are reliable, but we make no representations or warranties as to the accuracy of such information and accept no liability therefor. RealtyMogul offers equity securities through <a href="http://www.wealthforgeholdings.com" target="_blank">WealthForge, LLC</a>, member <a href="http://www.finra.org" target="_blank">FINRA</a>/<a href="http://www.sipc.org" target="_blank">SIPC</a>. Realty Mogul operates in California under DRE license #01926613. We recommend that you consult with a financial advisor, attorney, accountant, and any other professional that can help you to understand and assess the risks associated with any investment opportunity.</font></div></div></div>
</div>
</blockquote></div><br></div>