<div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jul 14, 2016 at 4:53 AM, Manikandan Selvaganesh <span dir="ltr"><<a href="mailto:mselvaga@redhat.com" target="_blank">mselvaga@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi David,<div><br></div><div>As you have mentioned already, the issue is fixed already and</div><div>the patch is here[1]. It is also back ported to 3.8 and 3.7.12.</div><div><br></div><div>[1] <a href="http://review.gluster.org/#/c/13793/" target="_blank">http://review.gluster.org/#/c/13793/</a></div></div></blockquote><div><br></div><div>Good news. Sadly when I tested updating to 3.7.12 & 13 last weekend I ran into issues where oVirt would not keep storage active. I would be flooded with these </div><div><br></div><div><div style="font-size:12.8px">[2016-07-09 15:27:46.935694] I [fuse-bridge.c:4083:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.22</div><div style="font-size:12.8px">[2016-07-09 15:27:49.555466] W [MSGID: 114031] [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-1: remote operation failed [Operation not permitted]</div><div style="font-size:12.8px">[2016-07-09 15:27:49.556574] W [MSGID: 114031] [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-0: remote operation failed [Operation not permitted]</div><div style="font-size:12.8px">[2016-07-09 15:27:49.556659] W [fuse-bridge.c:2227:fuse_readv_cbk] 0-glusterfs-fuse: 80: READ => -1 gfid=deb61291-5176-4b81-8315-3f1cf8e3534d fd=0x7f5224002f68 (Operation not permitted)</div><div style="font-size:12.8px">[2016-07-09 15:27:59.612477] W [MSGID: 114031] [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-1: remote operation failed [Operation not permitted]</div><div style="font-size:12.8px">[2016-07-09 15:27:59.613700] W [MSGID: 114031] [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-0: remote operation failed [Operation not permitted]</div><div style="font-size:12.8px">[2016-07-09 15:27:59.613781] W [fuse-bridge.c:2227:fuse_readv_cbk] 0-glusterfs-fuse: 168: READ => -1 gfid=deb61291-5176-4b81-8315-3f1cf8e3534d fd=0x7f5224002f68 (Operation not permitted)</div></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">I haven't had time to dig into it yet, but Lindsay Mathieson suggested it had to do with aio support being removed in gluster and that disk image writeback settings would need to be changed. Which I can probably do from custom settings in oVirt I think, but the tests oVirt runs on storage nodes via dd to test if they are up I am not sure I can modify easily. And if those tests fail after few minutes node would be made inactive and even if the VM's themselves still see storage engine would likely keep pausing them thinking their is a storage issue.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jul 14, 2016 at 2:40 PM, David Gossage <span dir="ltr"><<a href="mailto:dgossage@carouselchecks.com" target="_blank">dgossage@carouselchecks.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><div class="gmail_quote"><span>On Thu, Jul 14, 2016 at 4:07 AM, David Gossage <span dir="ltr"><<a href="mailto:dgossage@carouselchecks.com" target="_blank">dgossage@carouselchecks.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jul 14, 2016 at 3:33 AM, Manikandan Selvaganesh <span dir="ltr"><<a href="mailto:mselvaga@redhat.com" target="_blank">mselvaga@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi David,<div><br></div><div>Which version are you using. Though the error seems superfluous, do you observe any functional failures.</div></div></blockquote><div><br></div><div>3.7.11 and no so far I have noticed no issues over past week as I have been enabling sharding on storage. VM's all seem to be running just fine. Been migrating disk images off and on to shard a few a night since Sunday and all have been behaving as expected.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>Also, there are quite some EINVAL bugs we fixed in 3.8, could you point out</div><div>to the one you find matching.</div></div></blockquote><div><br></div><div><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1325810" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1325810</a></div><div><br></div><div>This is one I found while searching a portion of my error message. </div></div></div></div></blockquote><div><br></div></span><div>Technically that was a duplicate report and the one it is covered in is at </div><div><br></div><div><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1319581" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1319581</a> </div><div><br></div><div>Though I do not have quota enabled (that I am aware of) as is described there.</div><div><div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jul 14, 2016 at 1:51 PM, David Gossage <span dir="ltr"><<a href="mailto:dgossage@carouselchecks.com" target="_blank">dgossage@carouselchecks.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><div><div data-smartmail="gmail_signature"><div dir="ltr"><div><span><font color="#888888"><font><font size="1"><span style="color:rgb(51,51,51)"><span style="color:rgb(153,153,153)"><a value="+17086132426"><font color="#888888"><font size="1"><br></font></font></a></span></span></font></font></font></span></div></div></div></div>
<br><div class="gmail_quote"><span>On Wed, Jul 13, 2016 at 11:02 PM, Atin Mukherjee <span dir="ltr"><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jul 14, 2016 at 8:02 AM, David Gossage <span dir="ltr"><<a href="mailto:dgossage@carouselchecks.com" target="_blank">dgossage@carouselchecks.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>M, David Gossage <span dir="ltr"><<a href="mailto:dgossage@carouselchecks.com" target="_blank">dgossage@carouselchecks.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Is their a way to reduce logging of informational spam?<div><br></div><div>/var/log/glusterfs/bricks/gluster1-BRICK1-1.log is now 3GB over past few days<br><div><br></div><div><div>[2016-07-14 00:54:35.267018] I [dict.c:473:dict_get] (-->/lib64/libglusterfs.so.0(default_getxattr_cbk+0xac) [0x7fcdc396dcbc] -->/usr/lib64/glusterfs/3.7.11/xlator/features/marker.so(marker_getxattr_cbk+0xa7) [0x7fcdafde5917] -->/lib64/libglusterfs.so.0(dict_get+0xac) [0x7fcdc395e0fc] ) 0-dict: !this || key=() [Invalid argument]</div><div>[2016-07-14 00:54:35.272945] I [dict.c:473:dict_get] (-->/lib64/libglusterfs.so.0(default_getxattr_cbk+0xac) [0x7fcdc396dcbc] -->/usr/lib64/glusterfs/3.7.11/xlator/features/marker.so(marker_getxattr_cbk+0xa7) [0x7fcdafde5917] -->/lib64/libglusterfs.so.0(dict_get+0xac) [0x7fcdc395e0fc] ) 0-dict: !this || key=() [Invalid argument]</div></div></div></div></blockquote></span></div></div></div></blockquote><div><br></div><div><br>@Mani, Is this something which get logged in a normal scenario? I doubt.<br><br></div></div></div></div></blockquote><div><br></div></span><div>I did find a bug report about it presumably being fixed in 3.8. </div><div><br></div><div>I also currently have a node down which may be triggering them.</div><span><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><span><font color="#888888"><div><br></div></font></span></div></div></div></blockquote><div><br></div></span><div>Believe I found it</div><div><br></div><div> gluster volume set testvol diagnostics.brick-log-level WARNING</div><div><br></div><div><a href="https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3/html/Administration_Guide/Configuring_the_Log_Level.html" target="_blank">https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3/html/Administration_Guide/Configuring_the_Log_Level.html</a><br></div><span><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><span><font color="#888888"><div></div><div><div data-smartmail="gmail_signature"><div dir="ltr"><span><font color="#888888"><span style="color:rgb(0,0,0)"><b><i>David Gossage</i></b></span><font><i><span style="color:rgb(51,51,51)"><b><br>
</b></span></i></font></font></span><div><span><font color="#888888"><font><i><span style="color:rgb(51,51,51)"></span></i><font size="1"><b style="color:rgb(153,0,0)">Carousel Checks Inc.<span style="color:rgb(204,204,204)"> | System Administrator</span></b></font></font><font style="color:rgb(153,153,153)"><font size="1"><br>
</font></font><font><font size="1"><span style="color:rgb(51,51,51)"><b style="color:rgb(153,153,153)">Office</b><span style="color:rgb(153,153,153)"> <a value="+17086132426">708.613.2284<font color="#888888"><font size="1"><br></font></font></a></span></span></font></font></font></span></div></div></div></div>
</font></span></div></div></div>
</blockquote></span></div><br></div></div>
<br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><span><font color="#888888"><br></font></span></blockquote></div><span><font color="#888888"><br><br clear="all"><span><font color="#888888"><span><font color="#888888"><br>-- <br><div data-smartmail="gmail_signature"><div dir="ltr"><div><br></div>--Atin<br></div></div>
</font></span></font></span></font></span></div></div><span><font color="#888888"><span><font color="#888888">
</font></span></font></span></blockquote></span></div><span><font color="#888888"><span><font color="#888888"><br></font></span></font></span></div></div><span><font color="#888888"><span><font color="#888888">
</font></span></font></span></blockquote></div><span><font color="#888888"><span><font color="#888888"><br><br clear="all"><span><font color="#888888"><div><br></div>-- <br><div data-smartmail="gmail_signature"><div dir="ltr"><div>Regards,<br></div>Manikandan Selvaganesh.<br></div></div>
</font></span></font></span></font></span></div></div><span><font color="#888888">
</font></span></blockquote></div><span><font color="#888888"><br></font></span></div></div><span><font color="#888888">
</font></span></blockquote></div></div></div><span><font color="#888888"><br></font></span></div></div><span><font color="#888888">
</font></span></blockquote></div><span><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div data-smartmail="gmail_signature"><div dir="ltr"><div>Regards,<br></div>Manikandan Selvaganesh.<br></div></div>
</font></span></div></div>
</blockquote></div><br></div></div>