<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
<br>
<div class="moz-cite-prefix">On 12/22/2015 09:10 PM, David Robinson
wrote:<br>
</div>
<blockquote
cite="mid:em9c481848-031a-4f62-8054-d914b352abac@dfrobins-vaio"
type="cite">
<style id="eMClientCss">blockquote.cite { margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right:0px; border-left: 1px solid #cccccc }
blockquote.cite2 {margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right:0px; border-left: 1px solid #cccccc; margin-top: 3px; padding-top: 0px; }
.plain pre, .plain tt { font-family: monospace; font-size: 100%; font-weight: normal; font-style: normal; white-space: pre-wrap; }
a img { border: 0px; }body {font-family: Times New Roman;font-size: 12pt;}
.plain pre, .plain tt {font-family: Times New Roman;font-size: 12pt;}
</style>
<style></style>
<div>Pranith, </div>
<div> </div>
<div>This issue continues to happen. If you could provide
instructions for getting you the statedump, I would be happy to
send that information.</div>
<div>I am not sure how to get a statedump just before the crash as
the crash is intermittent.</div>
</blockquote>
Command: gluster volume statedump <volname><br>
<br>
This generates statedump files in /var/run/gluster/ directory. Do
you think you can execute this command once every 'X' time until the
crash is hit? Post these files and hopefully that should be good
enough to fix the problem.<br>
<br>
Pranith<br>
<blockquote
cite="mid:em9c481848-031a-4f62-8054-d914b352abac@dfrobins-vaio"
type="cite">
<div> </div>
<div>David</div>
<div> </div>
<div> </div>
<div>------ Original Message ------</div>
<div>From: "Pranith Kumar Karampuri" <<a moz-do-not-send="true"
href="mailto:pkarampu@redhat.com">pkarampu@redhat.com</a>></div>
<div>To: "Glomski, Patrick" <<a moz-do-not-send="true"
href="mailto:patrick.glomski@corvidtec.com">patrick.glomski@corvidtec.com</a>>;
<a moz-do-not-send="true"
href="mailto:gluster-devel@gluster.org">gluster-devel@gluster.org</a>;
<a moz-do-not-send="true"
href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a></div>
<div>Cc: "David Robinson" <<a moz-do-not-send="true"
href="mailto:david.robinson@corvidtec.com">david.robinson@corvidtec.com</a>></div>
<div>Sent: 12/21/2015 11:59:33 PM</div>
<div>Subject: Re: [Gluster-devel] glusterfsd crash due to page
allocation failure</div>
<div> </div>
<div id="x78585c967b814e6d953499138e3d698c" style="COLOR: #000000">
<blockquote class="cite2" cite="5678D8B5.1080605@redhat.com"
type="cite">hi Glomski,<br>
This is the second time I am hearing about memory
allocation problems in 3.7.6 but this time on brick side. Are
you able to recreate this issue? Will it be possible to get
statedumps of the bricks processes just before they crash?<br>
<br>
Pranith <br>
<br>
<div class="moz-cite-prefix">On 12/22/2015 02:25 AM, Glomski,
Patrick wrote:<br>
</div>
<blockquote class="cite"
cite="mid:CALkMjdB_eq2DrvW0WFV9gQWFPWDU07ipTxg4-g0uacRW6vz0Zw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>Hello,<br>
<br>
</div>
We've recently upgraded from gluster 3.6.6 to 3.7.6 and
have started encountering dmesg page allocation errors
(stack trace is appended). <br>
<br>
It appears that glusterfsd now sometimes fills up the
cache completely and crashes with a page allocation
failure. I *believe* it mainly happens when copying lots
of new data to the system, running a 'find', or similar.
Hosts are all Scientific Linux 6.6 and these errors
occur consistently on two separate gluster pools.<br>
</div>
<div><br>
Has anyone else seen this issue and are there any known
fixes for it via sysctl kernel parameters or other
means?<br>
<br>
</div>
<div>Please let me know of any other diagnostic
information that would help.<br>
</div>
<div><br>
</div>
<div>Thanks,<br>
</div>
<div>Patrick<br>
</div>
<div><br>
<br>
<blockquote class="gmail_quote" style="PADDING-LEFT:
1ex; BORDER-LEFT: rgb(204,204,204) 1px solid; MARGIN:
0px 0px 0px 0.8ex">[1458118.134697] glusterfsd: page
allocation failure. order:5, mode:0x20<br>
[1458118.134701] Pid: 6010, comm: glusterfsd Not
tainted 2.6.32-573.3.1.el6.x86_64 #1<br>
[1458118.134702] Call Trace:<br>
[1458118.134714] [<ffffffff8113770c>] ?
__alloc_pages_nodemask+0x7dc/0x950<br>
[1458118.134728] [<ffffffffa0321800>] ?
mlx4_ib_post_send+0x680/0x1f90 [mlx4_ib]<br>
[1458118.134733] [<ffffffff81176e92>] ?
kmem_getpages+0x62/0x170<br>
[1458118.134735] [<ffffffff81177aaa>] ?
fallback_alloc+0x1ba/0x270<br>
[1458118.134736] [<ffffffff811774ff>] ?
cache_grow+0x2cf/0x320<br>
[1458118.134738] [<ffffffff81177829>] ?
____cache_alloc_node+0x99/0x160<br>
[1458118.134743] [<ffffffff8145f732>] ?
pskb_expand_head+0x62/0x280<br>
[1458118.134744] [<ffffffff81178479>] ?
__kmalloc+0x199/0x230<br>
[1458118.134746] [<ffffffff8145f732>] ?
pskb_expand_head+0x62/0x280<br>
[1458118.134748] [<ffffffff8146001a>] ?
__pskb_pull_tail+0x2aa/0x360<br>
[1458118.134751] [<ffffffff8146f389>] ?
harmonize_features+0x29/0x70<br>
[1458118.134753] [<ffffffff8146f9f4>] ?
dev_hard_start_xmit+0x1c4/0x490<br>
[1458118.134758] [<ffffffff8148cf8a>] ?
sch_direct_xmit+0x15a/0x1c0<br>
[1458118.134759] [<ffffffff8146ff68>] ?
dev_queue_xmit+0x228/0x320<br>
[1458118.134762] [<ffffffff8147665d>] ?
neigh_connected_output+0xbd/0x100<br>
[1458118.134766] [<ffffffff814abc67>] ?
ip_finish_output+0x287/0x360<br>
[1458118.134767] [<ffffffff814abdf8>] ?
ip_output+0xb8/0xc0<br>
[1458118.134769] [<ffffffff814ab04f>] ?
__ip_local_out+0x9f/0xb0<br>
[1458118.134770] [<ffffffff814ab085>] ?
ip_local_out+0x25/0x30<br>
[1458118.134772] [<ffffffff814ab580>] ?
ip_queue_xmit+0x190/0x420<br>
[1458118.134773] [<ffffffff81137059>] ?
__alloc_pages_nodemask+0x129/0x950<br>
[1458118.134776] [<ffffffff814c0c54>] ?
tcp_transmit_skb+0x4b4/0x8b0<br>
[1458118.134778] [<ffffffff814c319a>] ?
tcp_write_xmit+0x1da/0xa90<br>
[1458118.134779] [<ffffffff81178cbd>] ?
__kmalloc_node+0x4d/0x60<br>
[1458118.134780] [<ffffffff814c3a80>] ?
tcp_push_one+0x30/0x40<br>
[1458118.134782] [<ffffffff814b410c>] ?
tcp_sendmsg+0x9cc/0xa20<br>
[1458118.134786] [<ffffffff8145836b>] ?
sock_aio_write+0x19b/0x1c0<br>
[1458118.134788] [<ffffffff814581d0>] ?
sock_aio_write+0x0/0x1c0<br>
[1458118.134791] [<ffffffff8119169b>] ?
do_sync_readv_writev+0xfb/0x140<br>
[1458118.134797] [<ffffffff810a14b0>] ?
autoremove_wake_function+0x0/0x40<br>
[1458118.134801] [<ffffffff8123e92f>] ?
selinux_file_permission+0xbf/0x150<br>
[1458118.134804] [<ffffffff812316d6>] ?
security_file_permission+0x16/0x20<br>
[1458118.134806] [<ffffffff81192746>] ?
do_readv_writev+0xd6/0x1f0<br>
[1458118.134807] [<ffffffff811928a6>] ?
vfs_writev+0x46/0x60<br>
[1458118.134809] [<ffffffff811929d1>] ?
sys_writev+0x51/0xd0<br>
[1458118.134812] [<ffffffff810e88ae>] ?
__audit_syscall_exit+0x25e/0x290<br>
[1458118.134816] [<ffffffff8100b0d2>] ?
system_call_fastpath+0x16/0x1b<br>
</blockquote>
<br>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-devel mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-devel">http://www.gluster.org/mailman/listinfo/gluster-devel</a></pre>
</blockquote>
<br>
</blockquote>
</div>
</blockquote>
<br>
</body>
</html>