<div dir="ltr"><div><div>Frank,<br><br></div>Could you share your volume configuration (`gluster volume info <VOLNAME>`)?<br><br></div>-Krutika<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 12, 2016 at 3:44 PM, David Gossage <span dir="ltr"><<a href="mailto:dgossage@carouselchecks.com" target="_blank">dgossage@carouselchecks.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><div><div data-smartmail="gmail_signature"><div dir="ltr"><div><br></div></div></div></div><div class="gmail_quote"><span class="">On Tue, Jul 12, 2016 at 5:08 AM, Frank Rothenstein <span dir="ltr"><<a href="mailto:f.rothenstein@bodden-kliniken.de" target="_blank">f.rothenstein@bodden-kliniken.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">Hi David,<br>
<br>
As I'm on gluster I use<br>
<br>
glusterfs.x86_64 3.7.12-2.el7 @centos-gluster37<br>
<br>
I already tried to revert back to 3.7.11 but somehow this didn't help. <br>
<br>
Do you have more info on aio/caching?<br></blockquote><div><br></div></span><div>Not at the moment I really haven't had time to dig into it this week yet. I'll quote Lindsay Mathieson 's response and maybe she can chime in if she sees this chain.</div><div><br></div><div><br></div><div>--------------------------------------------------------------</div><div><span style="font-size:12.8px">Hi David, I did a quick test this morning with Proxmox and 3.7.13 and was able to get it working with the fuse mount *and* libgfapi.</span><br style="font-size:12.8px"><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">One caveat - you *have* to enable qemu caching, either write-back or write-through. 3.7.12 & 13 seem to now disable aio support, and qemu requires that when caching is turned off.</span><br style="font-size:12.8px"><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">There are setting for aio in gluster that I haven't played with yet.</span><br></div><div><span style="font-size:12.8px">-----------------------------------------------------------------</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Did you revert client and server's to 3.7.11? It could be also that we had 2 completely different issues. The only similarity being the recent upgrades and Operation not permitted messages.</span></div><div><div class="h5"><div><span style="font-size:12.8px"><br></span></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<br>
FrankAm Dienstag, den 12.07.2016, 04:25 -0500 schrieb David Gossage:<br>
><br>
> On Tue, Jul 12, 2016 at 4:06 AM, Frank Rothenstein <f.rothenstein@bod<br>
> <a href="http://den-kliniken.de" rel="noreferrer" target="_blank">den-kliniken.de</a>> wrote:<br>
> > Hi all,<br>
> ><br>
> > I am stuck with a strange problem using ovirt/gluster on<br>
> > ZFSonLinux.<br>
> ><br>
> > I detached a data-volume form my Cluster an reimported it to change<br>
> > from POSIX-type/glusterfs to GlusterFS-Storage. When I try to<br>
> > attach it<br>
> > to the Cluster I always get a Sanlock error. I checked the logs and<br>
> > what I found were some "operation not permitted"<br>
> > errors in the brick logs:<br>
> ><br>
> > [2016-07-08 09:19:24.542046] E [MSGID: 113040] [posix-<br>
> > helpers.c:1684:__posix_fd_ctx_get] 0-datadom-posix: Failed to get<br>
> > anonymous real_path:<br>
> > /tank2/datadom/brick1/br/.glusterfs/c1/43/c1433bab-cd72-4395-ac62-<br>
> > b278670c333a _fd = -1 [No such file or directory]<br>
> > [2016-07-08 09:19:24.542103] W [MSGID: 113006]<br>
> > [posix.c:3010:posix_readv] 0-datadom-posix: pfd is NULL from<br>
> > fd=0x7f15f0003b48 [Operation not permitted]<br>
> > [2016-07-08 09:19:24.542142] E [MSGID: 115068] [server-rpc-<br>
> > fops.c:1486:server_readv_cbk] 0-datadom-server: 4836: READV -2<br>
> > (c1433bab-cd72-4395-ac62-b278670c333a) ==> (Operation not<br>
> > permitted)<br>
> > [Operation not permitted]<br>
> ><br>
> > Anyone ever expierienced such problem?<br>
> ><br>
> > In fact I am able to create/change/delete files in the brick path<br>
> > itself and the mount points (NFS/Fuse).<br>
> > Ovirt itself seems to test the volumes with /usr/bin/dd, but uses<br>
> > iflag=direct. Using this parameter I'm also unable to create any<br>
> > file -<br>
> > is there any gluster-option that changed recently? Or is it ZFS,<br>
> > that<br>
> > introduced something that blocks now?<br>
><br>
> What version of Gluster are you on?<br>
><br>
> My errors aren't quite the same as yours, but I have a similar setup<br>
><br>
> Ovirt 3.6.6/Gluster3.7.11 on ZFSonLinux<br>
><br>
> Updating from 3.7.11 -> 12 or 13 this last weekend I had these issues<br>
> pop up for me which while not exactly the same did result<br>
> in [Operation not permitted]<br>
><br>
> [2016-07-09 15:27:46.935694] I [fuse-bridge.c:4083:fuse_init] 0-<br>
> glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22<br>
> kernel 7.22<br>
> [2016-07-09 15:27:49.555466] W [MSGID: 114031] [client-rpc-<br>
> fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-1: remote<br>
> operation failed [Operation not permitted]<br>
> [2016-07-09 15:27:49.556574] W [MSGID: 114031] [client-rpc-<br>
> fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-0: remote<br>
> operation failed [Operation not permitted]<br>
> [2016-07-09 15:27:49.556659] W [fuse-bridge.c:2227:fuse_readv_cbk] 0-<br>
> glusterfs-fuse: 80: READ => -1 gfid=deb61291-5176-4b81-8315-<br>
> 3f1cf8e3534d fd=0x7f5224002f68 (Operation not permitted)<br>
> [2016-07-09 15:27:59.612477] W [MSGID: 114031] [client-rpc-<br>
> fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-1: remote<br>
> operation failed [Operation not permitted]<br>
> [2016-07-09 15:27:59.613700] W [MSGID: 114031] [client-rpc-<br>
> fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-0: remote<br>
> operation failed [Operation not permitted]<br>
> [2016-07-09 15:27:59.613781] W [fuse-bridge.c:2227:fuse_readv_cbk] 0-<br>
> glusterfs-fuse: 168: READ => -1 gfid=deb61291-5176-4b81-8315-<br>
> 3f1cf8e3534d fd=0x7f5224002f68 (Operation not permitted) <br>
><br>
> Another gluster user suggested that in 3.7.12/13 aio support was<br>
> disabled and I may need to change disk caching method. So far I have<br>
> just stayed at 3.7.11 until I have time to get a better test setup.<br>
><br>
> > Thanks in advance, Frank<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > ___________________________________________________________________<br>
> > ___________<br>
> > BODDEN-KLINIKEN Ribnitz-Damgarten GmbH<br>
> > Sandhufe 2<br>
> > 18311 Ribnitz-Damgarten<br>
> ><br>
> > Telefon: 03821-700-0<br>
> > Fax: 03821-700-240<br>
> ><br>
> > E-Mail: <a href="mailto:info@bodden-kliniken.de" target="_blank">info@bodden-kliniken.de</a> Internet: <a href="http://www.bodden-klini" rel="noreferrer" target="_blank">http://www.bodden-klini</a><br>
> > <a href="http://ken.de" rel="noreferrer" target="_blank">ken.de</a><br>
> ><br>
> > Sitz: Ribnitz-Damgarten, Amtsgericht: Stralsund, HRB 2919, Steuer-<br>
> > Nr.: 079/133/40188<br>
> > Aufsichtsratsvorsitzende: Carmen Schröter, Geschäftsführer: Dr.<br>
> > Falko Milski<br>
> ><br>
> > Der Inhalt dieser E-Mail ist ausschließlich für den bezeichneten<br>
> > Adressaten bestimmt. Wenn Sie nicht der vorge-<br>
> > sehene Adressat dieser E-Mail oder dessen Vertreter sein sollten,<br>
> > beachten Sie bitte, dass jede Form der Veröf-<br>
> > fentlichung, Vervielfältigung oder Weitergabe des Inhalts dieser E-<br>
> > Mail unzulässig ist. Wir bitten Sie, sofort den<br>
> > Absender zu informieren und die E-Mail zu löschen.<br>
> ><br>
> ><br>
> > Bodden-Kliniken Ribnitz-Damgarten GmbH 2016<br>
> > *** Virenfrei durch Kerio Mail Server und Sophos Antivirus ***<br>
> ><br>
> > _______________________________________________<br>
> > Gluster-users mailing list<br>
> > <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
> > <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
><br>
<br>
<br>
<br>
<br>
<br>
______________________________________________________________________________<br>
BODDEN-KLINIKEN Ribnitz-Damgarten GmbH<br>
Sandhufe 2<br>
18311 Ribnitz-Damgarten<br>
<br>
Telefon: 03821-700-0<br>
Fax: 03821-700-240<br>
<br>
E-Mail: <a href="mailto:info@bodden-kliniken.de" target="_blank">info@bodden-kliniken.de</a> Internet: <a href="http://www.bodden-kliniken.de" rel="noreferrer" target="_blank">http://www.bodden-kliniken.de</a><br>
<br>
Sitz: Ribnitz-Damgarten, Amtsgericht: Stralsund, HRB 2919, Steuer-Nr.: 079/133/40188<br>
Aufsichtsratsvorsitzende: Carmen Schröter, Geschäftsführer: Dr. Falko Milski<br>
<br>
Der Inhalt dieser E-Mail ist ausschließlich für den bezeichneten Adressaten bestimmt. Wenn Sie nicht der vorge-<br>
sehene Adressat dieser E-Mail oder dessen Vertreter sein sollten, beachten Sie bitte, dass jede Form der Veröf-<br>
fentlichung, Vervielfältigung oder Weitergabe des Inhalts dieser E-Mail unzulässig ist. Wir bitten Sie, sofort den<br>
Absender zu informieren und die E-Mail zu löschen.<br>
<br>
<br>
Bodden-Kliniken Ribnitz-Damgarten GmbH 2016<br>
*** Virenfrei durch Kerio Mail Server und Sophos Antivirus ***<br>
<br>
</blockquote></div></div></div><br></div></div>
<br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>