<div dir="ltr">I included you on a thread on users, let us see if he can help you out.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Aug 29, 2016 at 4:02 PM, Oleksandr Natalenko <span dir="ltr"><<a href="mailto:oleksandr@natalenko.name" target="_blank">oleksandr@natalenko.name</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">More info here.<br>
<br>
Massif puts the following warning on volume unmount:<br>
<br>
===<br>
valgrind: m_mallocfree.c:304 (get_bszB_as_is): Assertion 'bszB_lo == bszB_hi' failed.<br>
valgrind: Heap block lo/hi size mismatch: lo = 1, hi = 0.<br>
This is probably caused by your program erroneously writing past the<br>
end of a heap block and corrupting heap metadata. If you fix any<br>
invalid writes reported by Memcheck, this assertion failure will<br>
probably go away. Please try that before reporting this as a bug.<br>
...<br>
Thread 1: status = VgTs_Runnable<br>
==30590== at 0x4C29037: free (in /usr/lib64/valgrind/vgpreload_<wbr>massif-amd64-linux.so)<br>
==30590== by 0x67CE63B: __libc_freeres (in /usr/lib64/<a href="http://libc-2.17.so" rel="noreferrer" target="_blank">libc-2.17.so</a>)<br>
==30590== by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_<wbr>core-amd64-linux.so)<br>
==30590== by 0x66A2E2A: __run_exit_handlers (in /usr/lib64/<a href="http://libc-2.17.so" rel="noreferrer" target="_blank">libc-2.17.so</a>)<br>
==30590== by 0x66A2EB4: exit (in /usr/lib64/<a href="http://libc-2.17.so" rel="noreferrer" target="_blank">libc-2.17.so</a>)<br>
==30590== by 0x1117E9: cleanup_and_exit (glusterfsd.c:1308)<br>
==30590== by 0x669F66F: ??? (in /usr/lib64/<a href="http://libc-2.17.so" rel="noreferrer" target="_blank">libc-2.17.so</a>)<br>
==30590== by 0x606EEF4: pthread_join (in /usr/lib64/<a href="http://libpthread-2.17.so" rel="noreferrer" target="_blank">libpthread-2.17.so</a>)<br>
==30590== by 0x4EC2687: event_dispatch_epoll (event-epoll.c:762)<br>
==30590== by 0x10E876: main (glusterfsd.c:2370)<br>
...<br>
===<br>
<br>
I rechecked mount/ls/unmount with memcheck tool as suggested and got the following:<br>
<br>
===<br>
...<br>
==30315== Thread 8:<br>
==30315== Syscall param writev(vector[...]) points to uninitialised byte(s)<br>
==30315== at 0x675FEA0: writev (in /usr/lib64/<a href="http://libc-2.17.so" rel="noreferrer" target="_blank">libc-2.17.so</a>)<br>
==30315== by 0xE664795: send_fuse_iov (fuse-bridge.c:158)<br>
==30315== by 0xE6649B9: send_fuse_data (fuse-bridge.c:197)<br>
==30315== by 0xE666F7A: fuse_attr_cbk (fuse-bridge.c:753)<br>
==30315== by 0xE6671A6: fuse_root_lookup_cbk (fuse-bridge.c:783)<br>
==30315== by 0x14519937: io_stats_lookup_cbk (io-stats.c:1512)<br>
==30315== by 0x14300B3E: mdc_lookup_cbk (md-cache.c:867)<br>
==30315== by 0x13EE9226: qr_lookup_cbk (quick-read.c:446)<br>
==30315== by 0x13CD8B66: ioc_lookup_cbk (io-cache.c:260)<br>
==30315== by 0x1346405D: dht_revalidate_cbk (dht-common.c:985)<br>
==30315== by 0x1320EC60: afr_discover_done (afr-common.c:2316)<br>
==30315== by 0x1320EC60: afr_discover_cbk (afr-common.c:2361)<br>
==30315== by 0x12F9EE91: client3_3_lookup_cbk (client-rpc-fops.c:2981)<br>
==30315== Address 0x170b238c is on thread 8's stack<br>
==30315== in frame #3, created by fuse_attr_cbk (fuse-bridge.c:723)<br>
...<br>
==30315== Warning: invalid file descriptor -1 in syscall close()<br>
==30315== Thread 1:<br>
==30315== Invalid free() / delete / delete[] / realloc()<br>
==30315== at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_<wbr>memcheck-amd64-linux.so)<br>
==30315== by 0x67D663B: __libc_freeres (in /usr/lib64/<a href="http://libc-2.17.so" rel="noreferrer" target="_blank">libc-2.17.so</a>)<br>
==30315== by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_<wbr>core-amd64-linux.so)<br>
==30315== by 0x66AAE2A: __run_exit_handlers (in /usr/lib64/<a href="http://libc-2.17.so" rel="noreferrer" target="_blank">libc-2.17.so</a>)<br>
==30315== by 0x66AAEB4: exit (in /usr/lib64/<a href="http://libc-2.17.so" rel="noreferrer" target="_blank">libc-2.17.so</a>)<br>
==30315== by 0x1117E9: cleanup_and_exit (glusterfsd.c:1308)<br>
==30315== by 0x66A766F: ??? (in /usr/lib64/<a href="http://libc-2.17.so" rel="noreferrer" target="_blank">libc-2.17.so</a>)<br>
==30315== by 0x6076EF4: pthread_join (in /usr/lib64/<a href="http://libpthread-2.17.so" rel="noreferrer" target="_blank">libpthread-2.17.so</a>)<br>
==30315== by 0x4ECA687: event_dispatch_epoll (event-epoll.c:762)<br>
==30315== by 0x10E876: main (glusterfsd.c:2370)<br>
==30315== Address 0x6a2d3d0 is 0 bytes inside data symbol "noai6ai_cached"<br>
===<br>
<br>
It seems Massif crashes (?) because of invalid memory access in glusterfs process cleanup stage.<br>
<br>
Pranith? Nithya?<span class="im HOEnZb"><br>
<br>
29.08.2016 13:14, Oleksandr Natalenko wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
===<br>
valgrind --tool=massif --trace-children=yes /usr/sbin/glusterfs -N<br>
--volfile-server=<a href="http://server.example.com" rel="noreferrer" target="_blank">server.exampl<wbr>e.com</a> --volfile-id=test<br>
/mnt/net/glusterfs/test<br>
===<br>
</blockquote></span><div class="HOEnZb"><div class="h5">
______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-devel</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div>