<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hi Folks,<br>
<br>
last week I upgraded gluster from 3.7.11 to 3.7.12. Unfortunately I
had two times the problem<br>
of a task being blocked for more than 120 seconds leaving me with
nothing more than a hard reset of the node!<br>
<br>
kernel: [1705322.676270] INFO: task apache2:3092 blocked for more
than 120 seconds.<br>
kernel: [1705322.682202] Not tainted 3.13.0-88-generic
#135-Ubuntu<br>
kernel: [1705322.682770] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.<br>
kernel: [1705322.683738] apache2 D ffff880a03d93180 0
3092 1800 0x00000000<br>
kernel: [1705322.683761] ffff88016f865b38 0000000000000082
ffff8809cbde8000 0000000000013180<br>
kernel: [1705322.683769] ffff88016f865fd8 0000000000013180
ffff8809cbde8000 ffff880a03d93a18<br>
kernel: [1705322.683775] ffff880a03f92d90 0000000000000002
ffffffffa029e670 ffff88016f865bb0<br>
kernel: [1705322.683782] Call Trace:<br>
kernel: [1705322.683978] [<ffffffffa029e670>] ?
nfs_free_request+0xb0/0xb0 [nfs]<br>
kernel: [1705322.684043] [<ffffffff8172e10d>]
io_schedule+0x9d/0x130<br>
kernel: [1705322.684096] [<ffffffffa029e67e>]
nfs_wait_bit_uninterruptible+0xe/0x20 [nfs]<br>
kernel: [1705322.684103] [<ffffffff8172e582>]
__wait_on_bit+0x62/0x90<br>
kernel: [1705322.684162] [<ffffffff81617018>] ?
sk_reset_timer+0x18/0x30<br>
kernel: [1705322.684193] [<ffffffffa029e670>] ?
nfs_free_request+0xb0/0xb0 [nfs]<br>
kernel: [1705322.684203] [<ffffffff8172e627>]
out_of_line_wait_on_bit+0x77/0x90<br>
kernel: [1705322.684234] [<ffffffff810adac0>] ?
autoremove_wake_function+0x40/0x40<br>
kernel: [1705322.684258] [<ffffffffa029ea33>]
nfs_wait_on_request+0x33/0x40 [nfs]<br>
kernel: [1705322.684276] [<ffffffffa02a3a40>]
nfs_updatepage+0x150/0x660 [nfs]<br>
kernel: [1705322.684290] [<ffffffffa0294dfb>]
nfs_write_end+0x5b/0x340 [nfs]<br>
kernel: [1705322.684318] [<ffffffff811522da>]
generic_file_buffered_write+0x16a/0x250<br>
kernel: [1705322.684336] [<ffffffff81153991>]
__generic_file_aio_write+0x1c1/0x3d0<br>
kernel: [1705322.684340] [<ffffffff81153bf8>]
generic_file_aio_write+0x58/0xa0<br>
kernel: [1705322.684354] [<ffffffffa029406b>]
nfs_file_write+0xbb/0x1d0 [nfs]<br>
kernel: [1705322.684387] [<ffffffff811c096a>]
do_sync_write+0x5a/0x90<br>
kernel: [1705322.684394] [<ffffffff811c10f4>]
vfs_write+0xb4/0x1f0<br>
kernel: [1705322.684399] [<ffffffff811c1b29>]
SyS_write+0x49/0xa0<br>
kernel: [1705322.684411] [<ffffffff8173a4dd>]
system_call_fastpath+0x1a/0x1f<br>
<br>
Some research came up with this links:<br>
*
<a class="moz-txt-link-freetext" href="https://www.gluster.org/pipermail/gluster-users/2016-July/027327.html">https://www.gluster.org/pipermail/gluster-users/2016-July/027327.html</a><br>
*
<a class="moz-txt-link-freetext" href="http://serverfault.com/questions/500222/kernel-3-8-apache2-with-wsgi-info-task-apache2-blocked-for-more-than-120-sec">http://serverfault.com/questions/500222/kernel-3-8-apache2-with-wsgi-info-task-apache2-blocked-for-more-than-120-sec</a><br>
* <a class="moz-txt-link-freetext" href="https://www.novell.com/support/kb/doc.php?id=7010287l">https://www.novell.com/support/kb/doc.php?id=7010287l</a><br>
<br>
The gluster volume serves a home directory for apache/php-fpm and
usually the server is quite busy in terms of requests.<br>
As with 3.7.11 I did not have any problems the last few weeks, I am
unsure if it is related to the 3.7.11 -> 3.7.12 upgrade.<br>
(or is just the file system blocking?)<br>
<br>
My setup is:<br>
# dpkg -l | grep gluster<br>
ii glusterfs-client
3.7.12-ubuntu1~trusty1 amd64
clustered file-system (client package)<br>
ii glusterfs-common
3.7.12-ubuntu1~trusty1 amd64
GlusterFS common libraries and translator modules<br>
ii glusterfs-server
3.7.12-ubuntu1~trusty1 amd64
clustered file-system (server package)<br>
The gluster Volume is mounted on the same host as the volume:<br>
XXX:/gldata on /home/gldata type nfs (rw,nfsvers=3,addr=XXX,_netdev)<br>
<br>
Are there any known race conditions with 3.7.12 and NFS?<br>
Should I apply the
<span style="color: rgb(0, 0, 0); font-family: Arial, Helvetica,
sans-serif; font-size: 12px; font-style: normal; font-variant:
normal; font-weight: normal; letter-spacing: normal; line-height:
normal; orphans: auto; text-align: left; text-indent: 0px;
text-transform: none; white-space: normal; widows: 1;
word-spacing: 0px; -webkit-text-stroke-width: 0px; display: inline
!important; float: none; background-color: rgb(255, 255, 255);">vm.dirty_ratio
settings mentioned in the links above?</span><br>
Should I downgrade to 3.7.11 or upgrade to 3.7.13?<br>
<br>
I appreciate any help,<br>
THX Georg<br>
<br>
<br>
</body>
</html>