<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
hi Glomski,<br>
This is the second time I am hearing about memory allocation
problems in 3.7.6 but this time on brick side. Are you able to
recreate this issue? Will it be possible to get statedumps of the
bricks processes just before they crash?<br>
<br>
Pranith <br>
<br>
<div class="moz-cite-prefix">On 12/22/2015 02:25 AM, Glomski,
Patrick wrote:<br>
</div>
<blockquote
cite="mid:CALkMjdB_eq2DrvW0WFV9gQWFPWDU07ipTxg4-g0uacRW6vz0Zw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>Hello,<br>
<br>
</div>
We've recently upgraded from gluster 3.6.6 to 3.7.6 and have
started encountering dmesg page allocation errors (stack trace
is appended). <br>
<br>
It appears that glusterfsd now sometimes fills up the cache
completely and crashes with a page allocation failure. I
*believe* it mainly happens when copying lots of new data to
the system, running a 'find', or similar. Hosts are all
Scientific Linux 6.6 and these errors occur consistently on
two separate gluster pools.<br>
</div>
<div><br>
Has anyone else seen this issue and are there any known fixes
for it via sysctl kernel parameters or other means?<br>
<br>
</div>
<div>Please let me know of any other diagnostic information that
would help.<br>
</div>
<div><br>
</div>
<div>Thanks,<br>
</div>
<div>Patrick<br>
</div>
<div><br>
<br>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px
solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">[1458118.134697]
glusterfsd: page allocation failure. order:5, mode:0x20<br>
[1458118.134701] Pid: 6010, comm: glusterfsd Not tainted
2.6.32-573.3.1.el6.x86_64 #1<br>
[1458118.134702] Call Trace:<br>
[1458118.134714] [<ffffffff8113770c>] ?
__alloc_pages_nodemask+0x7dc/0x950<br>
[1458118.134728] [<ffffffffa0321800>] ?
mlx4_ib_post_send+0x680/0x1f90 [mlx4_ib]<br>
[1458118.134733] [<ffffffff81176e92>] ?
kmem_getpages+0x62/0x170<br>
[1458118.134735] [<ffffffff81177aaa>] ?
fallback_alloc+0x1ba/0x270<br>
[1458118.134736] [<ffffffff811774ff>] ?
cache_grow+0x2cf/0x320<br>
[1458118.134738] [<ffffffff81177829>] ?
____cache_alloc_node+0x99/0x160<br>
[1458118.134743] [<ffffffff8145f732>] ?
pskb_expand_head+0x62/0x280<br>
[1458118.134744] [<ffffffff81178479>] ?
__kmalloc+0x199/0x230<br>
[1458118.134746] [<ffffffff8145f732>] ?
pskb_expand_head+0x62/0x280<br>
[1458118.134748] [<ffffffff8146001a>] ?
__pskb_pull_tail+0x2aa/0x360<br>
[1458118.134751] [<ffffffff8146f389>] ?
harmonize_features+0x29/0x70<br>
[1458118.134753] [<ffffffff8146f9f4>] ?
dev_hard_start_xmit+0x1c4/0x490<br>
[1458118.134758] [<ffffffff8148cf8a>] ?
sch_direct_xmit+0x15a/0x1c0<br>
[1458118.134759] [<ffffffff8146ff68>] ?
dev_queue_xmit+0x228/0x320<br>
[1458118.134762] [<ffffffff8147665d>] ?
neigh_connected_output+0xbd/0x100<br>
[1458118.134766] [<ffffffff814abc67>] ?
ip_finish_output+0x287/0x360<br>
[1458118.134767] [<ffffffff814abdf8>] ?
ip_output+0xb8/0xc0<br>
[1458118.134769] [<ffffffff814ab04f>] ?
__ip_local_out+0x9f/0xb0<br>
[1458118.134770] [<ffffffff814ab085>] ?
ip_local_out+0x25/0x30<br>
[1458118.134772] [<ffffffff814ab580>] ?
ip_queue_xmit+0x190/0x420<br>
[1458118.134773] [<ffffffff81137059>] ?
__alloc_pages_nodemask+0x129/0x950<br>
[1458118.134776] [<ffffffff814c0c54>] ?
tcp_transmit_skb+0x4b4/0x8b0<br>
[1458118.134778] [<ffffffff814c319a>] ?
tcp_write_xmit+0x1da/0xa90<br>
[1458118.134779] [<ffffffff81178cbd>] ?
__kmalloc_node+0x4d/0x60<br>
[1458118.134780] [<ffffffff814c3a80>] ?
tcp_push_one+0x30/0x40<br>
[1458118.134782] [<ffffffff814b410c>] ?
tcp_sendmsg+0x9cc/0xa20<br>
[1458118.134786] [<ffffffff8145836b>] ?
sock_aio_write+0x19b/0x1c0<br>
[1458118.134788] [<ffffffff814581d0>] ?
sock_aio_write+0x0/0x1c0<br>
[1458118.134791] [<ffffffff8119169b>] ?
do_sync_readv_writev+0xfb/0x140<br>
[1458118.134797] [<ffffffff810a14b0>] ?
autoremove_wake_function+0x0/0x40<br>
[1458118.134801] [<ffffffff8123e92f>] ?
selinux_file_permission+0xbf/0x150<br>
[1458118.134804] [<ffffffff812316d6>] ?
security_file_permission+0x16/0x20<br>
[1458118.134806] [<ffffffff81192746>] ?
do_readv_writev+0xd6/0x1f0<br>
[1458118.134807] [<ffffffff811928a6>] ?
vfs_writev+0x46/0x60<br>
[1458118.134809] [<ffffffff811929d1>] ?
sys_writev+0x51/0xd0<br>
[1458118.134812] [<ffffffff810e88ae>] ?
__audit_syscall_exit+0x25e/0x290<br>
[1458118.134816] [<ffffffff8100b0d2>] ?
system_call_fastpath+0x16/0x1b<br>
</blockquote>
<br>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-devel mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-devel">http://www.gluster.org/mailman/listinfo/gluster-devel</a></pre>
</blockquote>
<br>
</body>
</html>