<div dir="ltr">Hi Soumya,<div><br></div><div style="">Can you send a fix to this regression on upstream master too? This patch is merged there.</div><div style=""><br></div><div style="">regards,</div><div style="">Raghavendra</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 1, 2016 at 10:34 PM, Kotresh Hiremath Ravishankar <span dir="ltr"><<a href="mailto:khiremat@redhat.com" target="_blank">khiremat@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Soumya,<br>
<br>
I analysed the issue and found out that crash has happened because of the patch [1].<br>
<br>
The patch doesn't set transport object to NULL in 'rpc_clnt_disable' but instead does it on<br>
'rpc_clnt_trigger_destroy'. So if there are pending rpc invocations on the rpc object that<br>
is disabled (those instances are possible as happening now in changelog), it will trigger a<br>
CONNECT notify again with 'mydata' that is freed causing a crash. This happens because<br>
'rpc_clnt_submit' reconnects if rpc is not connected.<br>
<br>
rpc_clnt_submit (...) {<br>
...<br>
if (conn->connected == 0) {<br>
ret = rpc_transport_connect (conn->trans,<br>
conn->config.remote_port);<br>
}<br>
...<br>
}<br>
<br>
Without your patch, conn->trans was set NULL and hence CONNECT fails not resulting with<br>
CONNECT notify call. And also the cleanup happens in failure path.<br>
<br>
So the memory leak can happen, if there is no try for rpc invocation after DISCONNECT.<br>
It will be cleaned up otherwise.<br>
<br>
<br>
[1] <a href="http://review.gluster.org/#/c/13507/" rel="noreferrer" target="_blank">http://review.gluster.org/#/c/13507/</a><br>
<span class="im HOEnZb"><br>
Thanks and Regards,<br>
Kotresh H R<br>
<br>
----- Original Message -----<br>
</span><div class="HOEnZb"><div class="h5">> From: "Kotresh Hiremath Ravishankar" <<a href="mailto:khiremat@redhat.com">khiremat@redhat.com</a>><br>
> To: "Soumya Koduri" <<a href="mailto:skoduri@redhat.com">skoduri@redhat.com</a>><br>
> Cc: <a href="mailto:avishwan@redhat.com">avishwan@redhat.com</a>, "Gluster Devel" <<a href="mailto:gluster-devel@gluster.org">gluster-devel@gluster.org</a>><br>
> Sent: Monday, February 29, 2016 4:15:22 PM<br>
> Subject: Re: Cores generated with ./tests/geo-rep/georep-basic-dr-tarssh.t<br>
><br>
> Hi Soumya,<br>
><br>
> I just tested that it is reproducible only with your patch both in master and<br>
> 3.76 branch.<br>
> The geo-rep test cases are marked bad in master. So it's not hit in master.<br>
> rpc is introduced<br>
> in changelog xlator to communicate to applications via libgfchangelog.<br>
> Venky/Me will check<br>
> why is the crash happening and will update.<br>
><br>
><br>
> Thanks and Regards,<br>
> Kotresh H R<br>
><br>
> ----- Original Message -----<br>
> > From: "Soumya Koduri" <<a href="mailto:skoduri@redhat.com">skoduri@redhat.com</a>><br>
> > To: <a href="mailto:avishwan@redhat.com">avishwan@redhat.com</a>, "kotresh" <<a href="mailto:khiremat@redhat.com">khiremat@redhat.com</a>><br>
> > Cc: "Gluster Devel" <<a href="mailto:gluster-devel@gluster.org">gluster-devel@gluster.org</a>><br>
> > Sent: Monday, February 29, 2016 2:10:51 PM<br>
> > Subject: Cores generated with ./tests/geo-rep/georep-basic-dr-tarssh.t<br>
> ><br>
> > Hi Aravinda/Kotresh,<br>
> ><br>
> > With [1], I consistently see cores generated with the test<br>
> > './tests/geo-rep/georep-basic-dr-tarssh.t' in release-3.7 branch. From<br>
> > the cores, looks like we are trying to dereference a freed<br>
> > changelog_rpc_clnt_t(crpc) object in changelog_rpc_notify(). Strangely<br>
> > this was not reported in master branch.<br>
> ><br>
> > I tried debugging but couldn't find any possible suspects. I request you<br>
> > to take a look and let me know if [1] caused any regression.<br>
> ><br>
> > Thanks,<br>
> > Soumya<br>
> ><br>
> > [1] <a href="http://review.gluster.org/#/c/13507/" rel="noreferrer" target="_blank">http://review.gluster.org/#/c/13507/</a><br>
> ><br>
><br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">Raghavendra G<br></div>
</div>