<HTML><HEAD>
<STYLE id=eMClientCss>
<![CDATA[blockquote.cite { margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right:0px; border-left: 1px solid #cccccc }
blockquote.cite2 {margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right:0px; border-left: 1px solid #cccccc; margin-top: 3px; padding-top: 0px; }
.plain pre, .plain tt { font-family: monospace; font-size: 100%; font-weight: normal; font-style: normal; white-space: pre-wrap; }
a img { border: 0px; }body {font-family: Times New Roman;font-size: 12pt;}
.plain pre, .plain tt {font-family: Times New Roman;font-size: 12pt;}
]]></STYLE>
<STYLE>#x304eabd0be90426389b5a65edcbd9930 BLOCKQUOTE.cite2
{MARGIN-TOP: 3px; PADDING-TOP: 0px; PADDING-LEFT: 10px; MARGIN-LEFT: 5px; BORDER-LEFT: #cccccc 1px solid; PADDING-RIGHT: 0px; MARGIN-RIGHT: 0px}
#x304eabd0be90426389b5a65edcbd9930 .plain PRE, #x304eabd0be90426389b5a65edcbd9930 .plain TT
{FONT-SIZE: 100%; FONT-FAMILY: monospace; WHITE-SPACE: pre-wrap; FONT-WEIGHT: normal; FONT-STYLE: normal}
#x304eabd0be90426389b5a65edcbd9930 A IMG
{BORDER-TOP: 0px; BORDER-RIGHT: 0px; BORDER-BOTTOM: 0px; BORDER-LEFT: 0px}
#x304eabd0be90426389b5a65edcbd9930 .plain PRE, #x304eabd0be90426389b5a65edcbd9930 .plain TT, #x304eabd0be90426389b5a65edcbd9930
{FONT-SIZE: 12pt; FONT-FAMILY: Times New Roman}
</STYLE>
</HEAD>
<BODY bgColor=#ffffff scroll=auto class>
<DIV>sure... I'll setup a watch command to run this at some interval and send the files.</DIV>
<DIV> </DIV>
<DIV>David</DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>------ Original Message ------</DIV>
<DIV>From: "Pranith Kumar Karampuri" <<A href="mailto:pkarampu@redhat.com">pkarampu@redhat.com</A>></DIV>
<DIV>To: "David Robinson" <<A href="mailto:drobinson@corvidtec.com">drobinson@corvidtec.com</A>>; "Glomski, Patrick" <<A href="mailto:patrick.glomski@corvidtec.com">patrick.glomski@corvidtec.com</A>>; <A href="mailto:gluster-devel@gluster.org">gluster-devel@gluster.org</A>; <A href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</A></DIV>
<DIV>Cc: "David Robinson" <<A href="mailto:david.robinson@corvidtec.com">david.robinson@corvidtec.com</A>></DIV>
<DIV>Sent: 12/22/2015 12:11:35 PM</DIV>
<DIV>Subject: Re: [Gluster-devel] glusterfsd crash due to page allocation failure</DIV>
<DIV> </DIV>
<DIV id=x304eabd0be90426389b5a65edcbd9930 style="COLOR: #000000">
<BLOCKQUOTE class=cite2 cite=56798447.9060300@redhat.com type="cite"><BR><BR>
<DIV class=moz-cite-prefix>On 12/22/2015 09:10 PM, David Robinson wrote:<BR></DIV>
<BLOCKQUOTE class=cite cite=mid:em9c481848-031a-4f62-8054-d914b352abac@dfrobins-vaio type="cite">
<STYLE></STYLE>
<DIV>Pranith, </DIV>
<DIV> </DIV>
<DIV>This issue continues to happen. If you could provide instructions for getting you the statedump, I would be happy to send that information.</DIV>
<DIV>I am not sure how to get a statedump just before the crash as the crash is intermittent.</DIV></BLOCKQUOTE>Command: gluster volume statedump <volname><BR><BR>This generates statedump files in /var/run/gluster/ directory. Do you think you can execute this command once every 'X' time until the crash is hit? Post these files and hopefully that should be good enough to fix the problem.<BR><BR>Pranith<BR>
<BLOCKQUOTE class=cite cite=mid:em9c481848-031a-4f62-8054-d914b352abac@dfrobins-vaio type="cite">
<DIV> </DIV>
<DIV>David</DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>------ Original Message ------</DIV>
<DIV>From: "Pranith Kumar Karampuri" <<A href="mailto:pkarampu@redhat.com" moz-do-not-send="true">pkarampu@redhat.com</A>></DIV>
<DIV>To: "Glomski, Patrick" <<A href="mailto:patrick.glomski@corvidtec.com" moz-do-not-send="true">patrick.glomski@corvidtec.com</A>>; <A href="mailto:gluster-devel@gluster.org" moz-do-not-send="true">gluster-devel@gluster.org</A>; <A href="mailto:gluster-users@gluster.org" moz-do-not-send="true">gluster-users@gluster.org</A></DIV>
<DIV>Cc: "David Robinson" <<A href="mailto:david.robinson@corvidtec.com" moz-do-not-send="true">david.robinson@corvidtec.com</A>></DIV>
<DIV>Sent: 12/21/2015 11:59:33 PM</DIV>
<DIV>Subject: Re: [Gluster-devel] glusterfsd crash due to page allocation failure</DIV>
<DIV> </DIV>
<DIV id=x78585c967b814e6d953499138e3d698c style="COLOR: #000000">
<BLOCKQUOTE class=cite2 cite=5678D8B5.1080605@redhat.com type="cite">hi Glomski,<BR> This is the second time I am hearing about memory allocation problems in 3.7.6 but this time on brick side. Are you able to recreate this issue? Will it be possible to get statedumps of the bricks processes just before they crash?<BR><BR>Pranith <BR><BR>
<DIV class=moz-cite-prefix>On 12/22/2015 02:25 AM, Glomski, Patrick wrote:<BR></DIV>
<BLOCKQUOTE class=cite cite=mid:CALkMjdB_eq2DrvW0WFV9gQWFPWDU07ipTxg4-g0uacRW6vz0Zw@mail.gmail.com type="cite">
<DIV dir=ltr>
<DIV>
<DIV>Hello,<BR><BR></DIV>We've recently upgraded from gluster 3.6.6 to 3.7.6 and have started encountering dmesg page allocation errors (stack trace is appended). <BR><BR>It appears that glusterfsd now sometimes fills up the cache completely and crashes with a page allocation failure. I *believe* it mainly happens when copying lots of new data to the system, running a 'find', or similar. Hosts are all Scientific Linux 6.6 and these errors occur consistently on two separate gluster pools.<BR></DIV>
<DIV><BR>Has anyone else seen this issue and are there any known fixes for it via sysctl kernel parameters or other means?<BR><BR></DIV>
<DIV>Please let me know of any other diagnostic information that would help.<BR></DIV>
<DIV><BR></DIV>
<DIV>Thanks,<BR></DIV>
<DIV>Patrick<BR></DIV>
<DIV><BR><BR>
<BLOCKQUOTE class=gmail_quote style="PADDING-LEFT: 1ex; BORDER-LEFT: rgb(204,204,204) 1px solid; MARGIN: 0px 0px 0px 0.8ex">[1458118.134697] glusterfsd: page allocation failure. order:5, mode:0x20<BR>[1458118.134701] Pid: 6010, comm: glusterfsd Not tainted 2.6.32-573.3.1.el6.x86_64 #1<BR>[1458118.134702] Call Trace:<BR>[1458118.134714] [<ffffffff8113770c>] ? __alloc_pages_nodemask+0x7dc/0x950<BR>[1458118.134728] [<ffffffffa0321800>] ? mlx4_ib_post_send+0x680/0x1f90 [mlx4_ib]<BR>[1458118.134733] [<ffffffff81176e92>] ? kmem_getpages+0x62/0x170<BR>[1458118.134735] [<ffffffff81177aaa>] ? fallback_alloc+0x1ba/0x270<BR>[1458118.134736] [<ffffffff811774ff>] ? cache_grow+0x2cf/0x320<BR>[1458118.134738] [<ffffffff81177829>] ? ____cache_alloc_node+0x99/0x160<BR>[1458118.134743] [<ffffffff8145f732>] ? pskb_expand_head+0x62/0x280<BR>[1458118.134744] [<ffffffff81178479>] ? __kmalloc+0x199/0x230<BR>[1458118.134746] [<ffffffff8145f732>] ? pskb_expand_head+0x62/0x280<BR>[1458118.134748] [<ffffffff8146001a>] ? __pskb_pull_tail+0x2aa/0x360<BR>[1458118.134751] [<ffffffff8146f389>] ? harmonize_features+0x29/0x70<BR>[1458118.134753] [<ffffffff8146f9f4>] ? dev_hard_start_xmit+0x1c4/0x490<BR>[1458118.134758] [<ffffffff8148cf8a>] ? sch_direct_xmit+0x15a/0x1c0<BR>[1458118.134759] [<ffffffff8146ff68>] ? dev_queue_xmit+0x228/0x320<BR>[1458118.134762] [<ffffffff8147665d>] ? neigh_connected_output+0xbd/0x100<BR>[1458118.134766] [<ffffffff814abc67>] ? ip_finish_output+0x287/0x360<BR>[1458118.134767] [<ffffffff814abdf8>] ? ip_output+0xb8/0xc0<BR>[1458118.134769] [<ffffffff814ab04f>] ? __ip_local_out+0x9f/0xb0<BR>[1458118.134770] [<ffffffff814ab085>] ? ip_local_out+0x25/0x30<BR>[1458118.134772] [<ffffffff814ab580>] ? ip_queue_xmit+0x190/0x420<BR>[1458118.134773] [<ffffffff81137059>] ? __alloc_pages_nodemask+0x129/0x950<BR>[1458118.134776] [<ffffffff814c0c54>] ? tcp_transmit_skb+0x4b4/0x8b0<BR>[1458118.134778] [<ffffffff814c319a>] ? tcp_write_xmit+0x1da/0xa90<BR>[1458118.134779] [<ffffffff81178cbd>] ? __kmalloc_node+0x4d/0x60<BR>[1458118.134780] [<ffffffff814c3a80>] ? tcp_push_one+0x30/0x40<BR>[1458118.134782] [<ffffffff814b410c>] ? tcp_sendmsg+0x9cc/0xa20<BR>[1458118.134786] [<ffffffff8145836b>] ? sock_aio_write+0x19b/0x1c0<BR>[1458118.134788] [<ffffffff814581d0>] ? sock_aio_write+0x0/0x1c0<BR>[1458118.134791] [<ffffffff8119169b>] ? do_sync_readv_writev+0xfb/0x140<BR>[1458118.134797] [<ffffffff810a14b0>] ? autoremove_wake_function+0x0/0x40<BR>[1458118.134801] [<ffffffff8123e92f>] ? selinux_file_permission+0xbf/0x150<BR>[1458118.134804] [<ffffffff812316d6>] ? security_file_permission+0x16/0x20<BR>[1458118.134806] [<ffffffff81192746>] ? do_readv_writev+0xd6/0x1f0<BR>[1458118.134807] [<ffffffff811928a6>] ? vfs_writev+0x46/0x60<BR>[1458118.134809] [<ffffffff811929d1>] ? sys_writev+0x51/0xd0<BR>[1458118.134812] [<ffffffff810e88ae>] ? __audit_syscall_exit+0x25e/0x290<BR>[1458118.134816] [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b<BR></BLOCKQUOTE><BR></DIV></DIV><BR>
<FIELDSET class=mimeAttachmentHeader></FIELDSET> <BR><PRE wrap="">_______________________________________________
Gluster-devel mailing list
<A class=moz-txt-link-abbreviated href="mailto:Gluster-devel@gluster.org" moz-do-not-send="true">Gluster-devel@gluster.org</A>
<A class=moz-txt-link-freetext href="http://www.gluster.org/mailman/listinfo/gluster-devel" moz-do-not-send="true">http://www.gluster.org/mailman/listinfo/gluster-devel</A></PRE></BLOCKQUOTE><BR></BLOCKQUOTE></DIV></BLOCKQUOTE><BR></BLOCKQUOTE></DIV></BODY></HTML>