<div dir="ltr">Anyone who could help? We just ran into this exact same problem again. I just noticed we are running GlusterFS 3.7.1 on the clients (oVirt hosts/VDSM). Could this be an issue?</div><div class="gmail_extra"><br><div class="gmail_quote">On 8 June 2015 at 11:40, Tiemen Ruiten <span dir="ltr"><<a href="mailto:t.ruiten@rdmedia.com" target="_blank">t.ruiten@rdmedia.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Some extra points:<div><br></div><div>- 10.100.3.41 is one of the oVirt hosts.<div><br></div><div>- I only needed to restart glusterfsd & glusterd in one of the gluster nodes (also the one where I pulled the logs from) to get everything in working order.</div></div><div><br></div><div class="gmail_extra">- it's a separate gluster volume, not managed from oVirt engine.</div><div><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On 8 June 2015 at 11:35, Tiemen Ruiten <span dir="ltr"><<a href="mailto:t.ruiten@rdmedia.com" target="_blank">t.ruiten@rdmedia.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hello,<div><br></div><div>We are running an oVirt cluster on top of a 2 node replica 2 Gluster volume. Yesterday we suddenly noticed VMs were not responding and quickly found out the Gluster volume had issues. These errors were filling up the etc-glusterfs-glusterd.log file:</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">[2015-06-07 08:36:26.498012] W [rpcsvc.c:270:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for <a href="http://10.100.3.41:1022" target="_blank">10.100.3.41:1022</a><br>[2015-06-07 08:36:26.498073] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully</blockquote><div> </div><div>A restart of glusterfsd and glusterd resolved the issue, but triggered a lot of self-heals.</div><div><br></div><div>We are running glusterfs 3.7.0 on ZFS.</div><div><br></div><div>I have attached etc-glusterfs-glusterd.log, the brick log file and the glustershd.log. I would be grateful if anyone could shed any light on what happened here and if there's anything we can do to prevent it.</div><span><font color="#888888"><div><br></div><div>-- <br><div><div dir="ltr">Tiemen Ruiten<br>Systems Engineer<br>R&D Media<br></div></div>
</div></font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div><div dir="ltr">Tiemen Ruiten<br>Systems Engineer<br>R&D Media<br></div></div>
</div></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr">Tiemen Ruiten<br>Systems Engineer<br>R&D Media<br></div></div>
</div>