<div dir="ltr"><br><div>It depends upon the memory available and the workload. In this case, the size of the files being copied are huge. So more I/O happens to completely copy the file. </div><div><br></div><div>Can you please give the o/p of "gluster volume info <volume name>"?</div><div><br></div><div>Regards,</div><div>Raghavendra</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Feb 3, 2016 at 4:54 PM, Taste-Of-IT <span dir="ltr"><<a href="mailto:kontakt@taste-of-it.de" target="_blank">kontakt@taste-of-it.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">Am 2016-02-03 21:24, schrieb Raghavendra Bhat:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
I think this is what is happening. Someone please correct me if I am<br>
wrong.<br>
<br>
I think this is happening because nfs client, nfs server and bricks<br>
being in the same machine. What happens is, when the large write<br>
comes, nfs client sends the request to the nfs server and the nfs<br>
server sends it to the brick. The brick process tries to write it via<br>
making the write system call and the call enters the kernel. Kernel<br>
might not find memory available for performing the operation and thus<br>
wants to free some memory. NFS client does heavy caching. It might<br>
have saved many things in its memory. So, it has to free some memory.<br>
But nfs client is stuck with the write operation. It is still waiting<br>
for a response from the server. So it will not be able to free the<br>
memory till it gets a response from the nfs server (which in turn is<br>
waiting for a response from the brick) for the write operation it<br>
sent. But brick cannot get a response from kernel until kernel is able<br>
to get some memory for the operation and perform write.<br>
<br>
Thus it is stuck in this deadlock. Thats why you see your setup<br>
blocked.<br>
<br>
Can you please mount your volume via nfs on a different node other<br>
than the gluster server, and see if the issue happens again?<br>
<br>
Regards,<br>
Raghavendra<br>
<br>
On Wed, Feb 3, 2016 at 2:32 PM, Taste-Of-IT <<a href="mailto:kontakt@taste-of-it.de" target="_blank">kontakt@taste-of-it.de</a>><br>
wrote:<br>
<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
Am 2016-02-03 20:09, schrieb Raghavendra Bhat:<br>
<br>
Hi,<br>
<br>
Is your nfs client mounted on one of the gluster serves? <br>
<br>
Regards,<br>
Raghavendra<br>
<br>
On Wed, Feb 3, 2016 at 10:08 AM, Taste-Of-IT<br>
<<a href="mailto:kontakt@taste-of-it.de" target="_blank">kontakt@taste-of-it.de</a>><br>
wrote:<br>
<br>
Hello,<br>
<br>
hope some expert can help. I have a 2 Brick 1 Volume Distributed<br>
GlusterFS in Version 3.7.6 on Debian. The volume is shared via nfs.<br>
If i copy via midnight commander large files (>30GB), i got<br>
following messages. I replace sata cable, checked memory but i<br>
didnt<br>
find an error. SMART Values on all disks seems ok. After 30-40<br>
minutes i can copy again. Any Idea?<br>
<br>
Feb 3 12:46:31 gluster01 kernel: [11186.588367] [sched_delayed]<br>
sched: RT throttling activated<br>
Feb 3 12:56:09 gluster01 kernel: [11764.932749] glusterfsd <br>
D ffff88040ca6d788 0 1150 1 0x00000000<br>
Feb 3 12:56:09 gluster01 kernel: [11764.932759] <br>
ffff88040ca6d330 0000000000000082 0000000000012f00 ffff88040ad1bfd8<br>
Feb 3 12:56:09 gluster01 kernel: [11764.932767] <br>
0000000000012f00 ffff88040ca6d330 ffff88040ca6d330 ffff88040ad1be88<br>
Feb 3 12:56:09 gluster01 kernel: [11764.932773] <br>
ffff88040e18d4b8 ffff88040e18d4a0 ffffffff00000000 ffff88040e18d4a8<br>
Feb 3 12:56:09 gluster01 kernel: [11764.932780] Call Trace:<br>
Feb 3 12:56:09 gluster01 kernel: [11764.932796] <br>
[<ffffffff81512cd5>] ? rwsem_down_write_failed+0x1d5/0x320<br>
Feb 3 12:56:09 gluster01 kernel: [11764.932807] <br>
[<ffffffff812b7d13>] ? call_rwsem_down_write_failed+0x13/0x20<br>
Feb 3 12:56:09 gluster01 kernel: [11764.932816] <br>
[<ffffffff812325b0>] ? proc_keys_show+0x3f0/0x3f0<br>
Feb 3 12:56:09 gluster01 kernel: [11764.932823] <br>
[<ffffffff81512649>] ? down_write+0x29/0x40<br>
Feb 3 12:56:09 gluster01 kernel: [11764.932830] <br>
[<ffffffff811592bc>] ? vm_mmap_pgoff+0x6c/0xc0<br>
Feb 3 12:56:09 gluster01 kernel: [11764.932838] <br>
[<ffffffff8116ea4e>] ? SyS_mmap_pgoff+0x10e/0x250<br>
Feb 3 12:56:09 gluster01 kernel: [11764.932844] <br>
[<ffffffff811a969a>] ? SyS_readv+0x6a/0xd0<br>
Feb 3 12:56:09 gluster01 kernel: [11764.932853] <br>
[<ffffffff81513ccd>] ? system_call_fast_compare_end+0x10/0x15<br>
Feb 3 12:58:09 gluster01 kernel: [11884.979935] glusterfsd <br>
D ffff88040ca6d788 0 1150 1 0x00000000<br>
Feb 3 12:58:09 gluster01 kernel: [11884.979945] <br>
ffff88040ca6d330 0000000000000082 0000000000012f00 ffff88040ad1bfd8<br>
Feb 3 12:58:09 gluster01 kernel: [11884.979952] <br>
0000000000012f00 ffff88040ca6d330 ffff88040ca6d330 ffff88040ad1be88<br>
Feb 3 12:58:09 gluster01 kernel: [11884.979959] <br>
ffff88040e18d4b8 ffff88040e18d4a0 ffffffff00000000 ffff88040e18d4a8<br>
Feb 3 12:58:09 gluster01 kernel: [11884.979966] Call Trace:<br>
Feb 3 12:58:09 gluster01 kernel: [11884.979982] <br>
[<ffffffff81512cd5>] ? rwsem_down_write_failed+0x1d5/0x320<br>
Feb 3 12:58:09 gluster01 kernel: [11884.979993] <br>
[<ffffffff812b7d13>] ? call_rwsem_down_write_failed+0x13/0x20<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980001] <br>
[<ffffffff812325b0>] ? proc_keys_show+0x3f0/0x3f0<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980008] <br>
[<ffffffff81512649>] ? down_write+0x29/0x40<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980015] <br>
[<ffffffff811592bc>] ? vm_mmap_pgoff+0x6c/0xc0<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980023] <br>
[<ffffffff8116ea4e>] ? SyS_mmap_pgoff+0x10e/0x250<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980030] <br>
[<ffffffff811a969a>] ? SyS_readv+0x6a/0xd0<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980038] <br>
[<ffffffff81513ccd>] ? system_call_fast_compare_end+0x10/0x15<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980351] mc <br>
D ffff88040e6d8fb8 0 5119 1447 0x00000000<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980358] <br>
ffff88040e6d8b60 0000000000000082 0000000000012f00 ffff88040d5dbfd8<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980365] <br>
0000000000012f00 ffff88040e6d8b60 ffff88041ec937b0 ffff88041efcc9e8<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980371] <br>
0000000000000002 ffffffff8113ce00 ffff88040d5dbcb0 ffff88040d5dbd98<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980377] Call Trace:<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980385] <br>
[<ffffffff8113ce00>] ? wait_on_page_read+0x60/0x60<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980392] <br>
[<ffffffff81510759>] ? io_schedule+0x99/0x120<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980399] <br>
[<ffffffff8113ce0a>] ? sleep_on_page+0xa/0x10<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980405] <br>
[<ffffffff81510adc>] ? __wait_on_bit+0x5c/0x90<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980412] <br>
[<ffffffff8113cbff>] ? wait_on_page_bit+0x7f/0x90<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980420] <br>
[<ffffffff810a7bd0>] ? autoremove_wake_function+0x30/0x30<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980426] <br>
[<ffffffff8114a17d>] ? pagevec_lookup_tag+0x1d/0x30<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980433] <br>
[<ffffffff8113cce0>] ? filemap_fdatawait_range+0xd0/0x160<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980442] <br>
[<ffffffff8113e7ca>] ? filemap_write_and_wait_range+0x3a/0x60<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980461] <br>
[<ffffffffa072363f>] ? nfs_file_fsync+0x7f/0x100 [nfs]<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980476] <br>
[<ffffffffa0723a2a>] ? nfs_file_write+0xda/0x1a0 [nfs]<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980484] <br>
[<ffffffff811a7e24>] ? new_sync_write+0x74/0xa0<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980492] <br>
[<ffffffff811a8562>] ? vfs_write+0xb2/0x1f0<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980500] <br>
[<ffffffff811a842d>] ? vfs_read+0xed/0x170<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980505] <br>
[<ffffffff811a90a2>] ? SyS_write+0x42/0xa0<br>
Feb 3 12:58:09 gluster01 kernel: [11884.980513] <br>
[<ffffffff81513ccd>] ? system_call_fast_compare_end+0x10/0x15<br>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
</div></div><a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a> [1] [1]<br>
<br>
Links:<br>
------<br>
[1] <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a> [1]<br>
</blockquote><span class="">
Hi Raghavendra,<br>
yes in this case i have to mount on one of the gluster server, but it<br>
doesnt matter on which i mount and its only a question of time when<br>
the trace came.<br>
Taste<br>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a> [1]<br>
<br>
<br>
Links:<br>
------<br>
[1] <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
</span></blockquote>
<br>
Hi,<br>
sounds logical. Is that a normal behavior? I tested it from a client and it looks fine, without trace. I tried 4 files about 30GB. The only thing i notice is, that the first file was copied with nearly full bandwidth, over both server, but the second was only with 20-30 Percent of possible bandwith. are there any perforamnce / stable option which i can use for nfs or glusterfs mount?<div class="HOEnZb"><div class="h5"><br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a></div></div></blockquote></div><br></div>