<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN">
<html><body>
<p>Hi Jeff,</p>
<p>On 28.04.2016 15:20, Jeff Darcy wrote:</p>
<blockquote type="cite" style="padding-left:5px; border-left:#1010ff 2px solid; margin-left:5px; width:100%">
<blockquote type="cite" style="padding-left:5px; border-left:#1010ff 2px solid; margin-left:5px; width:100%">This happens with Gluster 3.7.11 accessed through Ganesha and gfapi. The volume is a distributed-disperse 4*(4+2). I'm able to reproduce the problem easily doing the following test: iozone -t2 -s10g -r1024k -i0 -w -F/iozone{1..2}.dat echo 3 >/proc/sys/vm/drop_caches iozone -t2 -s10g -r1024k -i1 -w -F/iozone{1..2}.dat The error happens soon after starting the read test. As can be seen in the data below, client3_3_readv_cbk() is processing an iovec of 116 bytes, however it should be of 154 bytes (the buffer in memory really seems to contain 154 bytes). The data on the network seems ok (at least I haven't been able to identify any problem), so this must be a processing error on the client side. The last field in cut buffer of the sequentialized data corresponds to the length of the xdata field: 0x26. So at least 38 more byte should be present.</blockquote>
<pre>Nice detective work, Xavi. It would be *very* interesting to see what
the value of the "count" parameter is (it's unfortunately optimized out).
I'll bet it's two, and iov[1].iov_len is 38. I have a weak memory of
some problems with how this iov is put together, a couple of years ago,
and it looks like you might have tripped over one more.</pre>
<pre> </pre>
</blockquote>
<pre>It seems you are right. The count is 2 and the first 38 bytes of the second vector contains the remaining data of xdata field. The rest of the data in the second vector seems the payload of the readv fop, plus a 2 bytes padding:</pre>
<pre> </pre>
<pre>(gdb) f 0<br />#0 client3_3_readv_cbk (req=0x7fdc4051a31c, iov=0x7fdc4051a35c, count=<optimized out>, myframe=0x7fdc520d505c) at client-rpc-fops.c:3021<br />3021 gf_msg (this->name, GF_LOG_ERROR, EINVAL,<br />(gdb) print *iov<br />$2 = {iov_base = 0x7fdc14b0d018, iov_len = 116}<br />(gdb) f 1<br />#1 0x00007fdc56dafab0 in rpc_clnt_handle_reply (clnt=clnt@entry=0x7fdc3c1f4bb0, pollin=pollin@entry=0x7fdc34010f20) at rpc-clnt.c:764<br />764 req->cbkfn (req, req->rsp, req->rspcnt, saved_frame->frame);<br />(gdb) print *pollin<br />$3 = {vector = {{iov_base = 0x7fdc14b0d000, iov_len = 140}, {iov_base = 0x7fdc14a4d000, iov_len = 32808}, {iov_base = 0x0, iov_len = 0} <repeats 14 times>}, count = 2,<br /> vectored = 1 '\001', private = 0x7fdc340106c0, iobref = 0x7fdc34006660, hdr_iobuf = 0x7fdc3c4c07c0, is_reply = 1 '\001'}<br />(gdb) f 0<br />#0 client3_3_readv_cbk (req=0x7fdc4051a31c, iov=0x7fdc4051a35c, count=<optimized out>, myframe=0x7fdc520d505c) at client-rpc-fops.c:3021<br />3021 gf_msg (this->name, GF_LOG_ERROR, EINVAL,<br />(gdb) print iov[1]<br />$4 = {iov_base = 0x7fdc14a4d000, iov_len = 32808}<br />(gdb) print iov[2]<br />$5 = {iov_base = 0x2, iov_len = 140583741974112}<br />(gdb) x/128xb 0x7fdc14a4d000<br />0x7fdc14a4d000: 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x17<br />0x7fdc14a4d008: 0x00 0x00 0x00 0x02 0x67 0x6c 0x75 0x73<br />0x7fdc14a4d010: 0x74 0x65 0x72 0x66 0x73 0x2e 0x69 0x6e<br />0x7fdc14a4d018: 0x6f 0x64 0x65 0x6c 0x6b 0x2d 0x63 0x6f<br />0x7fdc14a4d020: 0x75 0x6e 0x74 0x00 0x31 0x00 0x00 0x00<br />0x7fdc14a4d028: 0x5c 0x5c 0x5c 0x5c 0x5c 0x5c 0x5c 0x5c<br />0x7fdc14a4d030: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00<br />0x7fdc14a4d038: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00<br />0x7fdc14a4d040: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00<br />0x7fdc14a4d048: 0x5c 0x00 0x00 0x00 0x00 0x00 0x00 0x00<br />0x7fdc14a4d050: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00<br />0x7fdc14a4d058: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00<br />0x7fdc14a4d060: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00<br />0x7fdc14a4d068: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00<br />0x7fdc14a4d070: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00<br />0x7fdc14a4d078: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00</pre>
<pre> </pre>
<blockquote type="cite" style="padding-left:5px; border-left:#1010ff 2px solid; margin-left:5px; width:100%">
<pre> Maybe it's related to all that epoll stuff.</pre>
<pre> </pre>
</blockquote>
<pre>I'm currently using 4 epoll threads (this improves ec performance). I'll try to repeat the tests with a single epoll thread, but I'm not sure if this will be enough to get any conclusion if the problem doesn't manifest, since ec through fuse with 4 epoll threads doesn't seem to trigger the problem.</pre>
<pre> </pre>
<pre>Xavi</pre>
</body></html>