<html><body><div style="font-family: garamond,new york,times,serif; font-size: 12pt; color: #000000"><div>Raghavendra,<br></div><div><br></div><div>The crash was due to bug(s) in clear-locks command implementation. Joe and I had had an offline discussion about this.<br></div><div><br></div><div>-Krutika<br></div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Raghavendra G" <raghavendra@gluster.com><br><b>To: </b>"Krutika Dhananjay" <kdhananj@redhat.com><br><b>Cc: </b>"Joe Julian" <joe@julianfamily.org>, "Gluster Devel" <gluster-devel@gluster.org><br><b>Sent: </b>Tuesday, February 23, 2016 8:54:01 PM<br><b>Subject: </b>Re: [Gluster-devel] 3.6.8 crashing a lot in production<br><div><br></div><div dir="ltr"><div style=""><span style="color:rgb(51,51,51);font-family:monospace;font-size:11px;line-height:19.59375px;white-space:pre-wrap">Came across a glibc bug which could've caused some corruptions. </span><span style="color:rgb(51,51,51);font-family:monospace;font-size:11px;line-height:19.59375px;white-space:pre-wrap">On googling about possible problems, we found that there is an issue (</span><a target="_blank" href="https://bugzilla.redhat.com/show_bug.cgi?id=1305406" style="color:rgb(0,136,206);text-decoration:none;font-family:monospace;font-size:11px;line-height:19.59375px;white-space:pre-wrap">https://bugzilla.redhat.com/show_bug.cgi?id=1305406</a><span style="color:rgb(51,51,51);font-family:monospace;font-size:11px;line-height:19.59375px;white-space:pre-wrap">) fixed in glibc-2.17-121.el7. From the bug we found the following test-script to determine if the glibc is buggy. And on running it, we ran it on the local setup using the following method given in the bug:</span></div><span style="color:rgb(51,51,51);font-family:monospace;font-size:11px;line-height:19.59375px;white-space:pre-wrap">
----------------
# objdump -r -d /lib64/libc.so.6 | grep -C 20 _int_free | grep -C 10 cmpxchg | head -21 | grep -A 3 cmpxchg | tail -1 | (grep '%r' && echo "Your libc is likely buggy." || echo "Your libc looks OK.")
7cc36: 48 85 c9 test %rcx,%rcx
Your libc is likely buggy.
----------------
Could you check if the above command on your setup gives the same output which says "Your libc is likely buggy."
</span>regards,</div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Feb 13, 2016 at 7:46 AM, Krutika Dhananjay <span dir="ltr"><<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:garamond,new york,times,serif;font-size:12pt;color:#000000"><div>Taking a look. Give me some time.<br></div><div><br></div><div>-Krutika<br></div><div><br></div><hr><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><b>From: </b>"Joe Julian" <<a href="mailto:joe@julianfamily.org" target="_blank">joe@julianfamily.org</a>><br><b>To: </b>"Krutika Dhananjay" <<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>>, <a href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a><br><b>Sent: </b>Saturday, February 13, 2016 6:02:13 AM<br><b>Subject: </b>Fwd: [Gluster-devel] 3.6.8 crashing a lot in production<div><div class="h5"><br><div><br></div>Could this be a regression from <a href="http://review.gluster.org/7981" target="_blank">http://review.gluster.org/7981</a> ?<br><div><br>-------- Forwarded Message --------<table class="mceItemTable" border="0" cellpadding="0" cellspacing="0"><tbody><tr><th align="RIGHT" nowrap="" valign="BASELINE">Subject:</th><td>[Gluster-devel] 3.6.8 crashing a lot in production</td></tr><tr><th align="RIGHT" nowrap="" valign="BASELINE">Date:</th><td>Fri, 12 Feb 2016 16:20:59 -0800</td></tr><tr><th align="RIGHT" nowrap="" valign="BASELINE">From:</th><td>Joe Julian <a href="mailto:joe@julianfamily.org" target="_blank"><joe@julianfamily.org></a><br></td></tr><tr><th align="RIGHT" nowrap="" valign="BASELINE">To:</th><td><a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>, <a href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a><br></td></tr></tbody></table><br> <br><pre>I have multiple bricks crashing in production. Any help would be greatly
appreciated.
The crash log is in this bug report:
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1307146" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1307146</a>
Looks like it's crashing in pl_inodelk_client_cleanup
_______________________________________________
Gluster-devel mailing list
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a>
</pre><br></div><br></div></div></blockquote><div><br></div></div></div><br>_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">Raghavendra G<br></div>
</div>
</blockquote><div><br></div></div></body></html>