<div dir="ltr"><div class="gmail_extra"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><br></div></div></div></div><div class="gmail_quote">On Thu, Jul 7, 2016 at 9:51 PM, David Gossage <span dir="ltr"><<a href="mailto:dgossage@carouselchecks.com" target="_blank">dgossage@carouselchecks.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">So a NIC on one of my nodes died today. Chance are good that a reboot would bring it back to life, but then I'd be down while 2-3TB of unsharded VM's healed which wouldn't be fun. I figured I'll run with 2 nodes while I get sharding enabled and then bring in the 3rd new node I was going to replace that one with anyway.<div><br></div><div>1) I am using oVirt3.6 with CentOS 7 and while I am going to confirm I believe it does its communications over fuse mount still. So I am thinking it would help for me to move from 3.7.11 to 3.7.12 as the ligpfapi issues shouldn't hit me.</div><div><br></div></div></blockquote><div><br></div><div>Came in this morning to update to 3.7.12 and noticed that 3.7.13 had been released. So shut down VM's and gluster volumes and updated.</div><div>update process itself went smoothly but on starting up oVirt engine the main gluster storage volume didn't activate. I manually activated and it came up but oVirt wouldn't report on how much space was used. However ovirt nodes did mount and allow me to start VM's. However after a few minutes it would claim to be inactive again even if the nodes themselevs still had access and mounted volumes and the VM's were still running. Found these errors flooding the gluster logs on nodes.</div><div><br></div><div><div>[2016-07-09 15:27:46.935694] I [fuse-bridge.c:4083:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.22</div><div>[2016-07-09 15:27:49.555466] W [MSGID: 114031] [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-1: remote operation failed [Operation not permitted]</div><div>[2016-07-09 15:27:49.556574] W [MSGID: 114031] [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-0: remote operation failed [Operation not permitted]</div><div>[2016-07-09 15:27:49.556659] W [fuse-bridge.c:2227:fuse_readv_cbk] 0-glusterfs-fuse: 80: READ => -1 gfid=deb61291-5176-4b81-8315-3f1cf8e3534d fd=0x7f5224002f68 (Operation not permitted)</div><div>[2016-07-09 15:27:59.612477] W [MSGID: 114031] [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-1: remote operation failed [Operation not permitted]</div><div>[2016-07-09 15:27:59.613700] W [MSGID: 114031] [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-0: remote operation failed [Operation not permitted]</div><div>[2016-07-09 15:27:59.613781] W [fuse-bridge.c:2227:fuse_readv_cbk] 0-glusterfs-fuse: 168: READ => -1 gfid=deb61291-5176-4b81-8315-3f1cf8e3534d fd=0x7f5224002f68 (Operation not permitted)</div></div><div><br></div><div><br></div><div>Downgrading everything to 3.7.12 still had same issues. Issues went away finally when back on 3.7.11 and manually chaned op back to 30710</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div><div>2) enable sharding. Do I need to completely move the VM(powered off) image off the mount then back on for it to shad or can I rename VM images on the fuse mount?</div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>Typical disk dir has 3 files. I'm thinking since only the large image will shard that is only one I would need to move as others wouldn't shard?</div><div><div>-rw-rw----. 1 vdsm kvm 25G Apr 15 14:21 e7818fd2-7e2e-46e8-92b5-bc036850d88b</div><div>-rw-rw----. 1 vdsm kvm 1.0M Dec 2 2015 e7818fd2-7e2e-46e8-92b5-bc036850d88b.lease</div><div>-rw-r--r--. 1 vdsm kvm 320 Dec 2 2015 e7818fd2-7e2e-46e8-92b5-bc036850d88b.meta</div><div><br></div><div>3) Since I don't plan to re-enable the server with NIC issues can I just rsync /var/lib/glusterd then give new server same IP other one used to peer with? Do I need to change UUID of new server? Can I manually update the info in /var/lib/glusterd/vols/*/info so options match after enabling shards?</div><div><br></div><div>Any glaring gotchas I am overlooking?</div><span class=""><font color="#888888"><div><br></div><div><div data-smartmail="gmail_signature"><div dir="ltr"><span><font color="#888888"><span style="color:rgb(0,0,0)"><b><i>David Gossage</i></b></span><font><i><span style="color:rgb(51,51,51)"><b><br>
</b></span></i></font></font></span><div><span><font color="#888888"><font><i><span style="color:rgb(51,51,51)"></span></i><font size="1"><b style="color:rgb(153,0,0)">Carousel Checks Inc.<span style="color:rgb(204,204,204)"> | System Administrator</span></b></font></font><font style="color:rgb(153,153,153)"><font size="1"><br>
</font></font><font><font size="1"><span style="color:rgb(51,51,51)"><b style="color:rgb(153,153,153)">Office</b><span style="color:rgb(153,153,153)"> <a value="+17086132426">708.613.2284<font color="#888888"><font size="1"><br></font></font></a></span></span></font></font></font></span></div></div></div></div>
</font></span></div></div>
</blockquote></div><br></div></div>