<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jan 21, 2016 at 10:49 AM, Pranith Kumar Karampuri <span dir="ltr">&lt;<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""><br>
<br>
On 01/18/2016 02:28 PM, Oleksandr Natalenko wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
XFS. Server side works OK, I&#39;m able to mount volume again. Brick is 30% full.<br>
</blockquote>
<br></span>
Oleksandr,<br>
      Will it be possible to get the statedump of the client, bricks output next time it happens?<br>
<a href="https://github.com/gluster/glusterfs/blob/master/doc/debugging/statedump.md#how-to-generate-statedump" rel="noreferrer" target="_blank">https://github.com/gluster/glusterfs/blob/master/doc/debugging/statedump.md#how-to-generate-statedump</a></blockquote><div><br></div><div style="">We also need to dump inode information. To do that you&#39;ve to add &quot;all=yes&quot; to /var/run/gluster/glusterdump.options before you issue commands to get statedump.</div><div style=""> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="HOEnZb"><font color="#888888"><br>
<br>
Pranith</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
On понеділок, 18 січня 2016 р. 15:07:18 EET baul jianguo wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
What is your brick file system? and the glusterfsd process and all<br>
thread status?<br>
I met same issue when client app such as rsync stay in D status,and<br>
the brick process and relate thread also be in the D status.<br>
And the brick dev disk util is 100% .<br>
<br>
On Sun, Jan 17, 2016 at 6:13 AM, Oleksandr Natalenko<br>
<br>
&lt;<a href="mailto:oleksandr@natalenko.name" target="_blank">oleksandr@natalenko.name</a>&gt; wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Wrong assumption, rsync hung again.<br>
<br>
On субота, 16 січня 2016 р. 22:53:04 EET Oleksandr Natalenko wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
One possible reason:<br>
<br>
cluster.lookup-optimize: on<br>
cluster.readdir-optimize: on<br>
<br>
I&#39;ve disabled both optimizations, and at least as of now rsync still does<br>
its job with no issues. I would like to find out what option causes such<br>
a<br>
behavior and why. Will test more.<br>
<br>
On пʼятниця, 15 січня 2016 р. 16:09:51 EET Oleksandr Natalenko wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Another observation: if rsyncing is resumed after hang, rsync itself<br>
hangs a lot faster because it does stat of already copied files. So,<br>
the<br>
reason may be not writing itself, but massive stat on GlusterFS volume<br>
as well.<br>
<br>
15.01.2016 09:40, Oleksandr Natalenko написав:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
While doing rsync over millions of files from ordinary partition to<br>
GlusterFS volume, just after approx. first 2 million rsync hang<br>
happens, and the following info appears in dmesg:<br>
<br>
===<br>
[17075038.924481] INFO: task rsync:10310 blocked for more than 120<br>
seconds.<br>
[17075038.931948] &quot;echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs&quot;<br>
disables this message.<br>
[17075038.940748] rsync           D ffff88207fc13680     0 10310<br>
10309 0x00000080<br>
[17075038.940752]  ffff8809c578be18 0000000000000086 ffff8809c578bfd8<br>
0000000000013680<br>
[17075038.940756]  ffff8809c578bfd8 0000000000013680 ffff880310cbe660<br>
ffff881159d16a30<br>
[17075038.940759]  ffff881e3aa25800 ffff8809c578be48 ffff881159d16b10<br>
ffff88087d553980<br>
[17075038.940762] Call Trace:<br>
[17075038.940770]  [&lt;ffffffff8160a1d9&gt;] schedule+0x29/0x70<br>
[17075038.940797]  [&lt;ffffffffa023a53d&gt;]<br>
__fuse_request_send+0x13d/0x2c0<br>
[fuse]<br>
[17075038.940801]  [&lt;ffffffffa023db30&gt;] ?<br>
fuse_get_req_nofail_nopages+0xc0/0x1e0 [fuse]<br>
[17075038.940805]  [&lt;ffffffff81098350&gt;] ? wake_up_bit+0x30/0x30<br>
[17075038.940809]  [&lt;ffffffffa023a6d2&gt;] fuse_request_send+0x12/0x20<br>
[fuse]<br>
[17075038.940813]  [&lt;ffffffffa024382f&gt;] fuse_flush+0xff/0x150 [fuse]<br>
[17075038.940817]  [&lt;ffffffff811c45c4&gt;] filp_close+0x34/0x80<br>
[17075038.940821]  [&lt;ffffffff811e4ed8&gt;] __close_fd+0x78/0xa0<br>
[17075038.940824]  [&lt;ffffffff811c6103&gt;] SyS_close+0x23/0x50<br>
[17075038.940828]  [&lt;ffffffff81614de9&gt;]<br>
system_call_fastpath+0x16/0x1b<br>
===<br>
<br>
rsync blocks in D state, and to kill it, I have to do umount --lazy<br>
on<br>
GlusterFS mountpoint, and then kill corresponding client glusterfs<br>
process. Then rsync exits.<br>
<br>
Here is GlusterFS volume info:<br>
<br>
===<br>
Volume Name: asterisk_records<br>
Type: Distributed-Replicate<br>
Volume ID: dc1fe561-fa3a-4f2e-8330-ec7e52c75ba4<br>
Status: Started<br>
Number of Bricks: 3 x 2 = 6<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1:<br>
server1:/bricks/10_megaraid_0_3_9_x_0_4_3_hdd_r1_nolvm_hdd_storage_01<br>
/as<br>
te<br>
risk/records Brick2:<br>
server2:/bricks/10_megaraid_8_5_14_x_8_6_16_hdd_r1_nolvm_hdd_storage_<br>
01/<br>
as<br>
terisk/records Brick3:<br>
server1:/bricks/11_megaraid_0_5_4_x_0_6_5_hdd_r1_nolvm_hdd_storage_02<br>
/as<br>
te<br>
risk/records Brick4:<br>
server2:/bricks/11_megaraid_8_7_15_x_8_8_20_hdd_r1_nolvm_hdd_storage_<br>
02/<br>
as<br>
terisk/records Brick5:<br>
server1:/bricks/12_megaraid_0_7_6_x_0_13_14_hdd_r1_nolvm_hdd_storage_<br>
03/<br>
as<br>
terisk/records Brick6:<br>
server2:/bricks/12_megaraid_8_9_19_x_8_13_24_hdd_r1_nolvm_hdd_storage<br>
_03<br>
/a<br>
sterisk/records Options Reconfigured:<br>
cluster.lookup-optimize: on<br>
cluster.readdir-optimize: on<br>
client.event-threads: 2<br>
network.inode-lru-limit: 4096<br>
server.event-threads: 4<br>
performance.client-io-threads: on<br>
storage.linux-aio: on<br>
performance.write-behind-window-size: 4194304<br>
performance.stat-prefetch: on<br>
performance.quick-read: on<br>
performance.read-ahead: on<br>
performance.flush-behind: on<br>
performance.write-behind: on<br>
performance.io-thread-count: 2<br>
performance.cache-max-file-size: 1048576<br>
performance.cache-size: 33554432<br>
features.cache-invalidation: on<br>
performance.readdir-ahead: on<br>
===<br>
<br>
The issue reproduces each time I rsync such an amount of files.<br>
<br>
How could I debug this issue better?<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
</blockquote>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
</blockquote>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
</blockquote></blockquote>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote>
<br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a></div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">Raghavendra G<br></div>
</div></div>