<div dir="ltr"><div><div>David, <br><br></div>I'm seeing similar behavior in my lab, but it has been caused by healing files in the gluster cluster, though I attribute my problems to problems with the storage fabric. See if 'gluster volume heal $VOL info' indicates files that are being healed, and if those reduce in number, can the VM start? <br><br></div>Dan<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 11, 2016 at 7:52 AM, David Gossage <span dir="ltr"><<a href="mailto:dgossage@carouselchecks.com" target="_blank">dgossage@carouselchecks.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Figure I would repost here as well. one client out of 3 complaining of stale file handles on a few new VM's I migrated over. No errors on storage nodes just client. Maybe just put that one in maintenance and restart gluster mount?<div><span class="HOEnZb"><font color="#888888"><br clear="all"><div><div data-smartmail="gmail_signature"><div dir="ltr"><span><font color="#888888"><span style="color:rgb(0,0,0)"><b><i>David Gossage</i></b></span><font><i><span style="color:rgb(51,51,51)"><b><br>
</b></span></i></font></font></span><div><span><font color="#888888"><font><i><span style="color:rgb(51,51,51)"></span></i><font size="1"><b style="color:rgb(153,0,0)">Carousel Checks Inc.<span style="color:rgb(204,204,204)"> | System Administrator</span></b></font></font><font style="color:rgb(153,153,153)"><font size="1"><br>
</font></font><font><font size="1"><span style="color:rgb(51,51,51)"><b style="color:rgb(153,153,153)">Office</b><span style="color:rgb(153,153,153)"> <a value="+17086132426">708.613.2284<font color="#888888"><font size="1"><br></font></font></a></span></span></font></font></font></span></div></div></div></div></font></span><div><div class="h5">
<br><div class="gmail_quote">---------- Forwarded message ----------<br>From: <b class="gmail_sendername">David Gossage</b> <span dir="ltr"><<a href="mailto:dgossage@carouselchecks.com" target="_blank">dgossage@carouselchecks.com</a>></span><br>Date: Thu, Aug 11, 2016 at 12:17 AM<br>Subject: vm paused unknown storage error one node out of 3 only<br>To: users <<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a>><br><br><br><div dir="ltr"><div><div>Out of a 3 node cluster running oVirt <span style="color:rgb(0,0,0);font-family:"Arial Unicode MS",Arial,sans-serif;line-height:21.6667px;text-align:-webkit-center">3.6.6.2-1.el7.centos with a 3 replicate gluster 3.7.14 starting a VM i just copied in on one node of the 3 gets the following errors. The other 2 the vm starts fine. All ovirt and gluster are centos 7 based. VM on start of the one node it tries to default to on its own accord immediately puts into paused for unknown reason. Telling it to start on different node starts ok. node with issue already has 5 VMs running fine on it same gluster storage plus the hosted engine on different volume.</span></div><div><span style="color:rgb(0,0,0);font-family:"Arial Unicode MS",Arial,sans-serif;line-height:21.6667px;text-align:-webkit-center"><br></span></div><div>gluster nodes logs did not have any errors for volume</div><div>nodes own gluster logs had this in log</div><div><br></div><div>dfb8777a-7e8c-40ff-8faa-252bea<wbr>bba5f8 couldnt find in .glusterfs .shard or images/<span style="color:rgb(0,0,0);font-family:"Arial Unicode MS",Arial,sans-serif;line-height:21.6667px;text-align:-webkit-center"><br></span></div><div><br></div><div>7919f4a0-125c-4b11-b5c9-fb50cc<wbr>195c43 is the gfid of the bootable drive of the vm<br></div><div><br></div><div>[2016-08-11 04:31:39.982952] W [MSGID: 114031] [client-rpc-fops.c:3050:client<wbr>3_3_readv_cbk] 0-GLUSTER1-client-2: remote operation failed [No such file or directory]</div><div>[2016-08-11 04:31:39.983683] W [MSGID: 114031] [client-rpc-fops.c:1572:client<wbr>3_3_fstat_cbk] 0-GLUSTER1-client-2: remote operation failed [No such file or directory]</div><div>[2016-08-11 04:31:39.984182] W [MSGID: 114031] [client-rpc-fops.c:1572:client<wbr>3_3_fstat_cbk] 0-GLUSTER1-client-0: remote operation failed [No such file or directory]</div><div>[2016-08-11 04:31:39.984221] W [MSGID: 114031] [client-rpc-fops.c:1572:client<wbr>3_3_fstat_cbk] 0-GLUSTER1-client-1: remote operation failed [No such file or directory]</div><div>[2016-08-11 04:31:39.985941] W [MSGID: 108008] [afr-read-txn.c:244:afr_read_t<wbr>xn] 0-GLUSTER1-replicate-0: Unreadable subvolume -1 found with event generation 3 for gfid dfb8777a-7e8c-40ff-8faa-252bea<wbr>bba5f8. (Possible split-brain)</div><div>[2016-08-11 04:31:39.986633] W [MSGID: 114031] [client-rpc-fops.c:1572:client<wbr>3_3_fstat_cbk] 0-GLUSTER1-client-2: remote operation failed [No such file or directory]</div><div>[2016-08-11 04:31:39.987644] E [MSGID: 109040] [dht-helper.c:1190:dht_migrati<wbr>on_complete_check_task] 0-GLUSTER1-dht: (null): failed to lookup the file on GLUSTER1-dht [Stale file handle]</div><div>[2016-08-11 04:31:39.987751] W [fuse-bridge.c:2227:fuse_readv<wbr>_cbk] 0-glusterfs-fuse: 15152930: READ => -1 gfid=7919f4a0-125c-4b11-b5c9-f<wbr>b50cc195c43 fd=0x7f00a80bdb64 (Stale file handle)</div><div>[2016-08-11 04:31:39.986567] W [MSGID: 114031] [client-rpc-fops.c:1572:client<wbr>3_3_fstat_cbk] 0-GLUSTER1-client-0: remote operation failed [No such file or directory]</div><div>[2016-08-11 04:31:39.986567] W [MSGID: 114031] [client-rpc-fops.c:1572:client<wbr>3_3_fstat_cbk] 0-GLUSTER1-client-1: remote operation failed [No such file or directory]</div><div>[2016-08-11 04:35:21.210145] W [MSGID: 108008] [afr-read-txn.c:244:afr_read_t<wbr>xn] 0-GLUSTER1-replicate-0: Unreadable subvolume -1 found with event generation 3 for gfid dfb8777a-7e8c-40ff-8faa-252bea<wbr>bba5f8. (Possible split-brain)</div><div>[2016-08-11 04:35:21.210873] W [MSGID: 114031] [client-rpc-fops.c:1572:client<wbr>3_3_fstat_cbk] 0-GLUSTER1-client-1: remote operation failed [No such file or directory]</div><div>[2016-08-11 04:35:21.210888] W [MSGID: 114031] [client-rpc-fops.c:1572:client<wbr>3_3_fstat_cbk] 0-GLUSTER1-client-2: remote operation failed [No such file or directory]</div><div>[2016-08-11 04:35:21.210947] W [MSGID: 114031] [client-rpc-fops.c:1572:client<wbr>3_3_fstat_cbk] 0-GLUSTER1-client-0: remote operation failed [No such file or directory]</div><div>[2016-08-11 04:35:21.213270] E [MSGID: 109040] [dht-helper.c:1190:dht_migrati<wbr>on_complete_check_task] 0-GLUSTER1-dht: (null): failed to lookup the file on GLUSTER1-dht [Stale file handle]</div><div>[2016-08-11 04:35:21.213345] W [fuse-bridge.c:2227:fuse_readv<wbr>_cbk] 0-glusterfs-fuse: 15156910: READ => -1 gfid=7919f4a0-125c-4b11-b5c9-f<wbr>b50cc195c43 fd=0x7f00a80bf6d0 (Stale file handle)</div><div>[2016-08-11 04:35:21.211516] W [MSGID: 108008] [afr-read-txn.c:244:afr_read_t<wbr>xn] 0-GLUSTER1-replicate-0: Unreadable subvolume -1 found with event generation 3 for gfid dfb8777a-7e8c-40ff-8faa-252bea<wbr>bba5f8. (Possible split-brain)</div><div>[2016-08-11 04:35:21.212013] W [MSGID: 114031] [client-rpc-fops.c:1572:client<wbr>3_3_fstat_cbk] 0-GLUSTER1-client-0: remote operation failed [No such file or directory]</div><div>[2016-08-11 04:35:21.212081] W [MSGID: 114031] [client-rpc-fops.c:1572:client<wbr>3_3_fstat_cbk] 0-GLUSTER1-client-1: remote operation failed [No such file or directory]</div><div>[2016-08-11 04:35:21.212121] W [MSGID: 114031] [client-rpc-fops.c:1572:client<wbr>3_3_fstat_cbk] 0-GLUSTER1-client-2: remote operation failed [No such file or directory]</div><div><br></div><div>I attached vdsm.log starting from when I spun up vm on offending node</div><div><br></div></div>
</div>
</div><br></div></div></div></div>
<br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>