<div style="font:14px/1.5 'Lucida Grande', '微软雅黑';color:#333;"><p style="font:18px/1.5 'Lucida Grande';margin:0;">Hi, <span style="font-size: 14px; line-height: 1.5;">Raghavendra,</span></p><p style="font:18px/1.5 'Lucida Grande';margin:0;"><span style="font-size: 14px; line-height: 1.5;"><br></span></p><p style="font:18px/1.5 'Lucida Grande';margin:0;"><span style="font-size: 14px; line-height: 1.5;">The typical file size is 2GB. I remember that the problem existed if I use 10MB file size.</span></p><p style="font:18px/1.5 'Lucida Grande';margin:0;"><span style="font-size: 14px; line-height: 1.5;"><br></span></p><p style="font:18px/1.5 'Lucida Grande';margin:0;"><span style="font-size: 14px; line-height: 1.5;">I have attached the logs. The attachment file includes the logs of one client and two servers.</span></p><p style="font:18px/1.5 'Lucida Grande';margin:0;"><span style="font-size: 14px; line-height: 1.5;"><br></span></p><p style="font-style: normal; font-variant: normal; font-weight: normal; font-family: 'Lucida Grande'; margin: 0px;">This time the client hung at 16:48 on June 24. </p><br><div><div style="font: 14px/1.5 'Lucida Grande';"><div style="font-family: Verdana; widows: 1;"><div style="font-family: 'lucida Grande', Verdana;">Thanks,</div><div style="font-family: 'lucida Grande', Verdana;">Paul</div></div></div></div><span style="font: 14px/1.5 'Lucida Grande';color:#333;"><br></span><div class="foxmail_blockquote_fromhere_element" style="font: 12px/1.5 'Lucida Grande';padding:2px 0 2px 0;"><span style="color:#333;text-decoration:line-through;white-space:pre-wrap;"> </span> Original Message <span style="color:#333;text-decoration:line-through;white-space:pre-wrap;"> </span></div><div style="font: 12px/1.5 'Lucida Grande';background:#efefef;color:#666666;padding:8px;"><div><b style="color:#999;">Sender:</b> FNU Raghavendra Manjunath<rabhat@redhat.com></div><div><b style="color:#999;">Recipient:</b> 冷波<lengbo@storswift.com></div><div><b style="color:#999;">Cc:</b> gluster-devel@gluster.org<gluster-devel@gluster.org>; gluster-users@gluster.org<gluster-users@gluster.org></div><div><b style="color:#999;">Date:</b> Friday, Jun 24, 2016 23:18</div><div><b style="color:#999;">Subject:</b> Re: [Gluster-devel] Fuse client hangs on doing multithreading IO tests</div></div><br><div class="mail_quote_A1CAF767BA954CE79A14856C0685EB34" style="font: 14px/1.5 'Lucida Grande';color:#333;"><div dir="ltr"><br><div>Hi,</div><div><br></div><div>Any idea how big were the files that were being read?</div><div><br></div><div>Can you please attach the logs from all the gluster server and client nodes? (the logs can be found in /var/log/glusterfs)</div><div><br></div><div>Also please provide the /var/log/messages from all the server and client nodes.</div><div><br></div><div>Regards,</div><div>Raghavendra</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jun 24, 2016 at 10:32 AM, 冷波 <span dir="ltr"><<a href="mailto:lengbo@storswift.com" target="_blank">lengbo@storswift.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="font:14px/1.5 'Lucida Grande','\005fae\008f6f\0096c5\009ed1';color:#333"><p style="font:18px/1.5 'Lucida Grande';margin:0">Hi,</p><p style="font:18px/1.5 'Lucida Grande';margin:0"><br></p><p style="font:18px/1.5 'Lucida Grande';margin:0">We found a problem when doing traffic tests. We created a replicated volume with two storage nodes (CentOS 6.5). There was one FUSE client (CentOS 6.7) which did multi-threading reads and writes. Most of IOs are reads for big files. All machines used 10Gbe NICs. And the typical read throught was 4-6Gbps (0.5-1.5GB/s). </p><p style="font:18px/1.5 'Lucida Grande';margin:0"><br></p><p style="font:18px/1.5 'Lucida Grande';margin:0">After the test ran several minutes, the test program hung. The throughput suddenly dropped to zero. Then there was no traffic any more. If we ran df, df would hang, too. But we could still read or write the volume from other clients.</p><p style="font:18px/1.5 'Lucida Grande';margin:0"><br></p><p style="font:18px/1.5 'Lucida Grande';margin:0">We tried several GlusterFS version from 3.7.5 to 3.8.0. Each version had this problem. We also tried to restore default GlusterFS options, but the problem still existed.</p><p style="font:18px/1.5 'Lucida Grande';margin:0"><br></p><p style="font:18px/1.5 'Lucida Grande';margin:0">The GlusterFS version was 3.7.11 for the following stacks.</p><p style="font:18px/1.5 'Lucida Grande';margin:0"><br></p><p style="font:18px/1.5 'Lucida Grande';margin:0">This was the stack of dd when hanging:</p><p style="font-size:18px;line-height:1.5;font-family:'Lucida Grande';margin:0px">[<ffffffffa046d211>] wait_answer_interruptible+0x81/0xc0 [fuse]</p><p style="font-size:18px;line-height:1.5;font-family:'Lucida Grande';margin:0px">[<ffffffffa046d42b>] __fuse_request_send+0x1db/0x2b0 [fuse]</p><p style="font-size:18px;line-height:1.5;font-family:'Lucida Grande';margin:0px">[<ffffffffa046d512>] fuse_request_send+0x12/0x20 [fuse]</p><p style="font-size:18px;line-height:1.5;font-family:'Lucida Grande';margin:0px">[<ffffffffa0477d4a>] fuse_statfs+0xda/0x150 [fuse]</p><p style="font-size:18px;line-height:1.5;font-family:'Lucida Grande';margin:0px">[<ffffffff811c2b64>] statfs_by_dentry+0x74/0xa0</p><p style="font-size:18px;line-height:1.5;font-family:'Lucida Grande';margin:0px">[<ffffffff811c2c9b>] vfs_statfs+0x1b/0xb0</p><p style="font-size:18px;line-height:1.5;font-family:'Lucida Grande';margin:0px">[<ffffffff811c2e97>] user_statfs+0x47/0xb0</p><p style="font-size:18px;line-height:1.5;font-family:'Lucida Grande';margin:0px">[<ffffffff811c2f9a>] sys_statfs+0x2a/0x50</p><p style="font-size:18px;line-height:1.5;font-family:'Lucida Grande';margin:0px">[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b</p><p style="font-size:18px;line-height:1.5;font-family:'Lucida Grande';margin:0px">[<ffffffffffffffff>] 0xffffffffffffffff</p><p style="font-size:18px;line-height:1.5;font-family:'Lucida Grande';margin:0px"><br></p><p style="font-size:18px;line-height:1.5;font-family:'Lucida Grande';margin:0px">This was the stack of gluster:</p><p style="font-size:18px;line-height:1.5;font-family:'Lucida Grande';margin:0px">[<ffffffff810b226a>] futex_wait_queue_me+0xba/0xf0</p><p style="font-size:18px;line-height:1.5;font-family:'Lucida Grande';margin:0px">[<ffffffff810b33a0>] futex_wait+0x1c0/0x310</p><p style="font-size:18px;line-height:1.5;font-family:'Lucida Grande';margin:0px">[<ffffffff810b4c91>] do_futex+0x121/0xae0</p><p style="font-size:18px;line-height:1.5;font-family:'Lucida Grande';margin:0px">[<ffffffff810b56cb>] sys_futex+0x7b/0x170</p><p style="font-size:18px;line-height:1.5;font-family:'Lucida Grande';margin:0px">[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b</p><p style="font-size:18px;line-height:1.5;font-family:'Lucida Grande';margin:0px">[<ffffffffffffffff>] 0xffffffffffffffff</p><p style="font-size:18px;line-height:1.5;font-family:'Lucida Grande';margin:0px"><br></p><p style="font-size:18px;line-height:1.5;font-family:'Lucida Grande';margin:0px">This was the stack of the test program:</p><p style="font-size:18px;line-height:1.5;font-family:'Lucida Grande';margin:0px">[<ffffffff810a3f74>] hrtimer_nanosleep+0xc4/0x180</p><p style="font-size:18px;line-height:1.5;font-family:'Lucida Grande';margin:0px">[<ffffffff810a409e>] sys_nanosleep+0x6e/0x80</p><p style="font-size:18px;line-height:1.5;font-family:'Lucida Grande';margin:0px">[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b</p><p style="font-size:18px;line-height:1.5;font-family:'Lucida Grande';margin:0px">[<ffffffffffffffff>] 0xffffffffffffffff</p><p style="font:18px/1.5 'Lucida Grande';margin:0"><br></p><p style="font:18px/1.5 'Lucida Grande';margin:0">Any clue?</p><br><div><div style="font:14px/1.5 'Lucida Grande'"><div style="font-family:Verdana"><div style="font-family:'lucida Grande',Verdana">Thanks,</div><div style="font-family:'lucida Grande',Verdana">Paul</div></div></div></div></div><br>_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br></blockquote></div><br></div>
</div></div>