<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Oct 21, 2016 at 6:36 PM, Soumya Koduri <span dir="ltr"><<a target="_blank" href="mailto:skoduri@redhat.com">skoduri@redhat.com</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><span class="gmail-"><br>
<br>
On 10/21/2016 02:03 PM, Xavier Hernandez wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">
Hi Niels,<br>
<br>
On 21/10/16 10:03, Niels de Vos wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">
On Fri, Oct 21, 2016 at 09:03:30AM +0200, Xavier Hernandez wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">
Hi,<br>
<br>
I've just tried Gluster 3.8.5 with Proxmox using gfapi and I<br>
consistently<br>
see a crash each time an attempt to connect to the volume is made.<br>
</blockquote>
<br>
Thanks, that likely is the same bug as<br>
<a target="_blank" rel="noreferrer" href="https://bugzilla.redhat.com/1379241">https://bugzilla.redhat.com/13<wbr>79241</a> .<br>
</blockquote>
<br>
I'm not sure it's the same problem. The crash on my case happens always<br>
and immediately. When creating an image, the file is created but size is<br>
0. The stack trace is quite different also.<br>
</blockquote>
<br></span>
Right. The issue reported in sug1379241 looks like the one we hit with client-io-threads enabled (already discussed in gluster-devel). Disabling that option may prevent the crash seen.<br></blockquote><div><br></div><div>Pranith has sent a fix <a href="http://review.gluster.org/#/c/15620/">http://review.gluster.org/#/c/15620/</a> for the same.<br> <br></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">
<br>
Thanks,<br>
Soumya<div class="gmail-HOEnZb"><div class="gmail-h5"><br>
<br>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">
<br>
Xavi<br>
<br>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">
<br>
Satheesaran, could you revert commit 7a50690 from the build that you<br>
were testing, and see if that causes the problem to go away again? Let<br>
me know of you want me to provide RPMs for testing.<br>
<br>
Niels<br>
<br>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">
<br>
The backtrace of the crash shows this:<br>
<br>
#0 pthread_spin_lock () at<br>
../nptl/sysdeps/x86_64/pthread<wbr>_spin_lock.S:24<br>
#1 0x00007fe5345776a5 in fd_unref (fd=0x7fe523f7205c) at fd.c:553<br>
#2 0x00007fe53482ba18 in glfs_io_async_cbk (op_ret=<optimized out>,<br>
op_errno=0, frame=<optimized out>, cookie=0x7fe526c67040,<br>
iovec=iovec@entry=0x0, count=count@entry=0)<br>
at glfs-fops.c:839<br>
#3 0x00007fe53482beed in glfs_fsync_async_cbk (frame=<optimized out>,<br>
cookie=<optimized out>, this=<optimized out>, op_ret=<optimized out>,<br>
op_errno=<optimized out>,<br>
prebuf=<optimized out>, postbuf=0x7fe5217fe890, xdata=0x0) at<br>
glfs-fops.c:1382<br>
#4 0x00007fe520be2eb7 in ?? () from<br>
/usr/lib/x86_64-linux-gnu/glus<wbr>terfs/3.8.5/xlator/debug/io-<wbr>stats.so<br>
#5 0x00007fe5345d118a in default_fsync_cbk (frame=0x7fe52ceef3ac,<br>
cookie=0x560ef95398e8, this=0x8, op_ret=0, op_errno=0, prebuf=0x1,<br>
postbuf=0x7fe5217fe890, xdata=0x0) at defaults.c:1508<br>
#6 0x00007fe5345d118a in default_fsync_cbk (frame=0x7fe52ceef204,<br>
cookie=0x560ef95398e8, this=0x8, op_ret=0, op_errno=0, prebuf=0x1,<br>
postbuf=0x7fe5217fe890, xdata=0x0) at defaults.c:1508<br>
#7 0x00007fe525f78219 in dht_fsync_cbk (frame=0x7fe52ceef2d8,<br>
cookie=0x560ef95398e8, this=0x0, op_ret=0, op_errno=0,<br>
prebuf=0x7fe5217fe820, postbuf=0x7fe5217fe890, xdata=0x0)<br>
at dht-inode-read.c:873<br>
#8 0x00007fe5261bbc7f in client3_3_fsync_cbk (req=0x7fe525f78030<br>
<dht_fsync_cbk>, iov=0x7fe526c61040, count=8, myframe=0x7fe52ceef130) at<br>
client-rpc-fops.c:975<br>
#9 0x00007fe5343201f0 in rpc_clnt_handle_reply (clnt=0x18,<br>
clnt@entry=0x7fe526fafac0, pollin=0x7fe526c3a1c0) at rpc-clnt.c:791<br>
#10 0x00007fe53432056c in rpc_clnt_notify (trans=<optimized out>,<br>
mydata=0x7fe526fafaf0, event=<optimized out>, data=0x7fe526c3a1c0) at<br>
rpc-clnt.c:962<br>
#11 0x00007fe53431c8a3 in rpc_transport_notify (this=<optimized out>,<br>
event=<optimized out>, data=<optimized out>) at rpc-transport.c:541<br>
#12 0x00007fe5283e8d96 in socket_event_poll_in (this=0x7fe526c69440) at<br>
socket.c:2267<br>
#13 0x00007fe5283eaf37 in socket_event_handler (fd=<optimized out>,<br>
idx=5,<br>
data=0x7fe526c69440, poll_in=1, poll_out=0, poll_err=0) at socket.c:2397<br>
#14 0x00007fe5345ab3f6 in event_dispatch_epoll_handler<br>
(event=0x7fe5217fecc0, event_pool=0x7fe526ca2040) at event-epoll.c:571<br>
#15 event_dispatch_epoll_worker (data=0x7fe527c0f0c0) at<br>
event-epoll.c:674<br>
#16 0x00007fe5324140a4 in start_thread (arg=0x7fe5217ff700) at<br>
pthread_create.c:309<br>
#17 0x00007fe53214962d in clone () at<br>
../sysdeps/unix/sysv/linux/x86<wbr>_64/clone.S:111<br>
<br>
The fd being unreferenced contains this:<br>
<br>
(gdb) print *fd<br>
$6 = {<br>
pid = 97649,<br>
flags = 2,<br>
refcount = 0,<br>
inode_list = {<br>
next = 0x7fe523f7206c,<br>
prev = 0x7fe523f7206c<br>
},<br>
inode = 0x0,<br>
lock = {<br>
spinlock = 1,<br>
mutex = {<br>
__data = {<br>
__lock = 1,<br>
__count = 0,<br>
__owner = 0,<br>
__nusers = 0,<br>
__kind = 0,<br>
__spins = 0,<br>
__elision = 0,<br>
__list = {<br>
__prev = 0x0,<br>
__next = 0x0<br>
}<br>
},<br>
__size = "\001", '\000' <repeats 38 times>,<br>
__align = 1<br>
}<br>
},<br>
_ctx = 0x7fe52ec31c40,<br>
xl_count = 11,<br>
lk_ctx = 0x7fe526c126a0,<br>
anonymous = _gf_false<br>
}<br>
<br>
fd->inode is NULL, explaining the cause of the crash. We also see that<br>
fd->refcount is already 0. So I'm wondering if this couldn't be an extra<br>
fd_unref() introduced by that patch.<br>
<br>
The crash seems to happen immediately after a graph switch.<br>
<br>
Xavi<br>
</blockquote></blockquote></blockquote>
______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a target="_blank" href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<a target="_blank" rel="noreferrer" href="http://www.gluster.org/mailman/listinfo/gluster-devel">http://www.gluster.org/mailman<wbr>/listinfo/gluster-devel</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature"><div dir="ltr"><div class="gmail_signature"><div dir="ltr"><br></div><div>~ Atin (atinm)<br></div></div></div></div>
</div></div>