<div dir="ltr"><div>Here are gluster peer status on the four nodes. The fifth peer is my laptop (10.214.70.62), which I have peer probed, but not done anything else with. The other news is that I have (since emailing yesterday) rebooted most of the nodes, which fixed the "missing files" problem. Before doing this, I noticed that not all peers agreed in their peer status. Shouldn't peer status of all-connected on one node mean that they are all connected?</div><br><div><div>wentworth:~# gluster peer status</div><div>Number of Peers: 4</div><div><br></div><div>Hostname: <a href="http://bennet.physics.oregonstate.edu">bennet.physics.oregonstate.edu</a></div><div>Uuid: 15dcda25-8043-4243-bbdc-168a73c91bcc</div><div>State: Peer in Cluster (Connected)</div><div>Other names:</div><div>128.193.96.83</div><div><br></div><div>Hostname: <a href="http://knightley.physics.oregonstate.edu">knightley.physics.oregonstate.edu</a></div><div>Uuid: 9f61673a-7215-4f07-9e75-4ed3dbeaff7e</div><div>State: Peer in Cluster (Connected)</div><div><br></div><div>Hostname: 10.214.70.62</div><div>Uuid: 7bdfc78b-82a9-45a3-94fd-a1f14c04a8ff</div><div>State: Peer in Cluster (Disconnected)</div><div><br></div><div>Hostname: 128.193.96.93</div><div>Uuid: 1b436ed4-44c9-414b-9cc6-74a38d97d66e</div><div>State: Peer in Cluster (Connected)</div><div>Other names:</div><div><a href="http://elliot.physics.oregonstate.edu">elliot.physics.oregonstate.edu</a></div></div><div><br></div><div><div>elliot:~# gluster peer status</div><div>Number of Peers: 4</div><div><br></div><div>Hostname: <a href="http://bennet.physics.oregonstate.edu">bennet.physics.oregonstate.edu</a></div><div>Uuid: 15dcda25-8043-4243-bbdc-168a73c91bcc</div><div>State: Peer in Cluster (Connected)</div><div>Other names:</div><div>128.193.96.83</div><div><br></div><div>Hostname: <a href="http://knightley.physics.oregonstate.edu">knightley.physics.oregonstate.edu</a></div><div>Uuid: 9f61673a-7215-4f07-9e75-4ed3dbeaff7e</div><div>State: Peer in Cluster (Connected)</div><div><br></div><div>Hostname: 10.214.70.62</div><div>Uuid: 7bdfc78b-82a9-45a3-94fd-a1f14c04a8ff</div><div>State: Peer in Cluster (Disconnected)</div><div><br></div><div>Hostname: <a href="http://wentworth.physics.oregonstate.edu">wentworth.physics.oregonstate.edu</a></div><div>Uuid: 6f463d9d-c32e-4973-bbc5-06b782678ee7</div><div>State: Peer in Cluster (Connected)</div></div><div><br></div><div><div>knightley:~# gluster peer status</div><div>Number of Peers: 4</div><div><br></div><div>Hostname: 10.214.70.62</div><div>Uuid: 7bdfc78b-82a9-45a3-94fd-a1f14c04a8ff</div><div>State: Peer in Cluster (Disconnected)</div><div><br></div><div>Hostname: <a href="http://wentworth.physics.oregonstate.edu">wentworth.physics.oregonstate.edu</a></div><div>Uuid: 6f463d9d-c32e-4973-bbc5-06b782678ee7</div><div>State: Peer in Cluster (Connected)</div><div><br></div><div>Hostname: 128.193.96.93</div><div>Uuid: 1b436ed4-44c9-414b-9cc6-74a38d97d66e</div><div>State: Peer in Cluster (Connected)</div><div>Other names:</div><div><a href="http://elliot.physics.oregonstate.edu">elliot.physics.oregonstate.edu</a></div><div><br></div><div>Hostname: <a href="http://bennet.physics.oregonstate.edu">bennet.physics.oregonstate.edu</a></div><div>Uuid: 15dcda25-8043-4243-bbdc-168a73c91bcc</div><div>State: Peer in Cluster (Connected)</div><div>Other names:</div><div>128.193.96.83</div></div><div><br></div><div><br></div><div><div>bennet:~# gluster peer status</div><div>Number of Peers: 4</div><div><br></div><div>Hostname: <a href="http://knightley.physics.oregonstate.edu">knightley.physics.oregonstate.edu</a></div><div>Uuid: 9f61673a-7215-4f07-9e75-4ed3dbeaff7e</div><div>State: Peer in Cluster (Connected)</div><div><br></div><div>Hostname: 128.193.96.93</div><div>Uuid: 1b436ed4-44c9-414b-9cc6-74a38d97d66e</div><div>State: Peer in Cluster (Connected)</div><div>Other names:</div><div><a href="http://elliot.physics.oregonstate.edu">elliot.physics.oregonstate.edu</a></div><div><br></div><div>Hostname: <a href="http://wentworth.physics.oregonstate.edu">wentworth.physics.oregonstate.edu</a></div><div>Uuid: 6f463d9d-c32e-4973-bbc5-06b782678ee7</div><div>State: Peer in Cluster (Connected)</div><div><br></div><div>Hostname: 10.214.70.62</div><div>Uuid: 7bdfc78b-82a9-45a3-94fd-a1f14c04a8ff</div><div>State: Peer in Cluster (Disconnected)</div></div><div><br></div><div><div>wentworth:~# gluster volume status</div><div>Status of volume: austen</div><div>Gluster process<span class="Apple-tab-span" style="white-space:pre"> </span>Port<span class="Apple-tab-span" style="white-space:pre"> </span>Online<span class="Apple-tab-span" style="white-space:pre"> </span>Pid</div><div>------------------------------------------------------------------------------</div><div>Brick elliot.physics.oregonstate.edu:/srv/brick<span class="Apple-tab-span" style="white-space:pre"> </span>49152<span class="Apple-tab-span" style="white-space:pre"> </span>Y<span class="Apple-tab-span" style="white-space:pre"> </span>2272</div><div>Brick wentworth.physics.oregonstate.edu:/srv/brick<span class="Apple-tab-span" style="white-space:pre"> </span>49152<span class="Apple-tab-span" style="white-space:pre"> </span>Y<span class="Apple-tab-span" style="white-space:pre"> </span>1566</div><div>Brick bennet:/srv/brick<span class="Apple-tab-span" style="white-space:pre"> </span>49152<span class="Apple-tab-span" style="white-space:pre"> </span>Y<span class="Apple-tab-span" style="white-space:pre"> </span>1516</div><div>Brick knightley:/srv/brick<span class="Apple-tab-span" style="white-space:pre"> </span>49152<span class="Apple-tab-span" style="white-space:pre"> </span>Y<span class="Apple-tab-span" style="white-space:pre"> </span>2153</div><div>NFS Server on localhost<span class="Apple-tab-span" style="white-space:pre"> </span>2049<span class="Apple-tab-span" style="white-space:pre"> </span>Y<span class="Apple-tab-span" style="white-space:pre"> </span>1568</div><div>Self-heal Daemon on localhost<span class="Apple-tab-span" style="white-space:pre"> </span>N/A<span class="Apple-tab-span" style="white-space:pre"> </span>Y<span class="Apple-tab-span" style="white-space:pre"> </span>1570</div><div>NFS Server on <a href="http://bennet.physics.oregonstate.edu">bennet.physics.oregonstate.edu</a><span class="Apple-tab-span" style="white-space:pre"> </span>2049<span class="Apple-tab-span" style="white-space:pre"> </span>Y<span class="Apple-tab-span" style="white-space:pre"> </span>1523</div><div>Self-heal Daemon on <a href="http://bennet.physics.oregonstate.edu">bennet.physics.oregonstate.edu</a><span class="Apple-tab-span" style="white-space:pre"> </span>N/A<span class="Apple-tab-span" style="white-space:pre"> </span>Y<span class="Apple-tab-span" style="white-space:pre"> </span>1528</div><div>NFS Server on 128.193.96.93<span class="Apple-tab-span" style="white-space:pre"> </span>2049<span class="Apple-tab-span" style="white-space:pre"> </span>Y<span class="Apple-tab-span" style="white-space:pre"> </span>2279</div><div>Self-heal Daemon on 128.193.96.93<span class="Apple-tab-span" style="white-space:pre"> </span>N/A<span class="Apple-tab-span" style="white-space:pre"> </span>Y<span class="Apple-tab-span" style="white-space:pre"> </span>2286</div><div>NFS Server on <a href="http://knightley.physics.oregonstate.edu">knightley.physics.oregonstate.edu</a><span class="Apple-tab-span" style="white-space:pre"> </span>2049<span class="Apple-tab-span" style="white-space:pre"> </span>Y<span class="Apple-tab-span" style="white-space:pre"> </span>2160</div><div>Self-heal Daemon on <a href="http://knightley.physics.oregonstate.edu">knightley.physics.oregonstate.edu</a><span class="Apple-tab-span" style="white-space:pre"> </span>N/A<span class="Apple-tab-span" style="white-space:pre"> </span>Y<span class="Apple-tab-span" style="white-space:pre"> </span>2165</div><div> </div><div>Task Status of Volume austen</div><div>------------------------------------------------------------------------------</div><div>There are no active volume tasks</div></div><div><br></div><div><br></div><div><div>wentworth:~# gluster --version</div><div>glusterfs 3.6.3 built on May 19 2015 19:27:14</div><div>Repository revision: git://<a href="http://git.gluster.com/glusterfs.git">git.gluster.com/glusterfs.git</a></div><div>Copyright (c) 2006-2011 Gluster Inc. <<a href="http://www.gluster.com">http://www.gluster.com</a>></div><div>GlusterFS comes with ABSOLUTELY NO WARRANTY.</div><div>You may redistribute copies of GlusterFS under the terms of the GNU General Public License.</div></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Jun 19, 2015 at 8:53 PM Atin Mukherjee <<a href="mailto:atin.mukherjee83@gmail.com">atin.mukherjee83@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">Could you paste output of :</p>
<p dir="ltr">gluster peer status from all the nodes?</p>
<p dir="ltr">gluster volume status</p>
<p dir="ltr">gluster --version<br></p>
<p dir="ltr">Atin</p>
<p dir="ltr">Sent from one plus one</p><p dir="ltr"><br>
On Jun 20, 2015 4:12 AM, "David Roundy" <<a href="mailto:roundyd@physics.oregonstate.edu" target="_blank">roundyd@physics.oregonstate.edu</a>> wrote:<br>
><br>
> Hi all,<br>
><br>
> I'm having some trouble where different nodes in my volume have differing ideas as to the state of the system. This is a 4-node (one brick per node) cluster, with replica 2.<br>
><br>
> I have identified two differing symptoms, I am hoping of the same problem.<br>
><br>
> The first symptom is that two of the four nodes don't show files in some of the directories of the volume, which are present in the other two nodes.<br>
><br>
> The second symptom is that I am having some compile commands (but not all) error with "No space left on device" when there is plenty of space on every brick.<br>
><br>
> The second symptom seems harder to track down than the first. Presumably my nodes are not properly synchronized in some way. How would I track something like this down?<br>
><br>
> A side note: I have had rebalance fix-layout seemingly hang with no evidence of progress, and a plain rebalance took a long time and I eventually stopped it. I created the cluster two nodes at a time, with the understanding that I would be able to add pairs of nodes without problems, but it doesn't seem to be working out that way so far.<br>
><br>
> Any suggestions would be appreciated! As with my last question, I'm not sure what further information will be helpful, and will be happy to provide more information!<br>
><br>
> David<br>
><br></p><p dir="ltr">
> _______________________________________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
</p>
</blockquote></div>