<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi Sachin,<br>
<br>
Good to see you again in Gluster-devel.<br>
<br>
I had implemented those API which I mentioned for a POC. In fact I
couldn't push into master.<br>
<br>
Regarding your questions, my comments are inline.<br>
<br>
<div class="moz-cite-prefix">On 06/22/2016 05:50 AM, Sachin Pandit
wrote:<br>
</div>
<blockquote
cite="mid:91d83c5f6b7a44c284f0f4962f73d1fa@POST-2.gp.cv.commvault.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:191917745;
mso-list-type:hybrid;
mso-list-template-ids:1887760304 67698705 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1
{mso-list-id:650985096;
mso-list-type:hybrid;
mso-list-template-ids:2086970442 67698705 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l1:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l2
{mso-list-id:752169642;
mso-list-type:hybrid;
mso-list-template-ids:-2004723328 1321242568 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l2:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:38.25pt;
text-indent:-.25in;}
@list l2:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:74.25pt;
text-indent:-.25in;}
@list l2:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:110.25pt;
text-indent:-9.0pt;}
@list l2:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:146.25pt;
text-indent:-.25in;}
@list l2:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:182.25pt;
text-indent:-.25in;}
@list l2:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:218.25pt;
text-indent:-9.0pt;}
@list l2:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:254.25pt;
text-indent:-.25in;}
@list l2:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:290.25pt;
text-indent:-.25in;}
@list l2:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:326.25pt;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hey
Pranith, I am good, I hope you are doing good too.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Please
find the comments inline.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif">
Pranith Kumar Karampuri [<a class="moz-txt-link-freetext" href="mailto:pkarampu@redhat.com">mailto:pkarampu@redhat.com</a>]
<br>
<b>Sent:</b> Tuesday, June 21, 2016 5:58 AM<br>
<b>To:</b> Sachin Pandit <a class="moz-txt-link-rfc2396E" href="mailto:spandit@commvault.com"><spandit@commvault.com></a><br>
<b>Cc:</b> <a class="moz-txt-link-abbreviated" href="mailto:gluster-devel@gluster.org">gluster-devel@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-devel] Reduce memcpy in glfs
read and write<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">Hey!!<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">
Hope you are doing good. I took a look at the bt. So when
flush comes write-behind has to flush all the writes down.
I see the following frame hung in iob_unref:<br>
Thread 7 (Thread 0x7fa601a30700 (LWP 16218)): <br>
#0 0x00007fa60cc55225 in pthread_spin_lock () from
/lib64/libpthread.so.0 <<---- Does it always hang
there?<span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">---------------------------------<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">>>It
does always hang here.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span
style="color:#1F497D">---------------------------------</span><br>
#1 0x00007fa60e1f373e in iobref_unref (iobref=0x19dc7e0)
at iobuf.c:907 <br>
#2 0x00007fa60e246fb2 in args_wipe (args=0x19e70ec) at
default-args.c:1593 <br>
#3 0x00007fa60e1ea534 in call_stub_wipe_args
(stub=0x19e709c) at call-stub.c:2466
<br>
#4 0x00007fa60e1ea5de in call_stub_destroy
(stub=0x19e709c) at call-stub.c:2482<o:p></o:p></p>
</div>
<p class="MsoNormal">Is this on top of master branch? It seems
like we missed an unlock of the spin-lock or the iobref has
junk value which gives the feeling that it is in locked
state (May be double free?). Do you have any extra patches
you have in your repo which make changes in iobuf?<o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">----------------------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">>>I
have implemented a method to reduce memcpy in libgfapi (My
patch is on top of master branch), by making use of buffer
from iobuf pool and passing the buffer to application.
However, I have not made any changes in iobuf core
feature. I don’t think double free is happening anywhere
in the code (I did check this using logs)<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">
Method that I have implemented:<o:p></o:p></span></p>
<p class="MsoListParagraph"
style="text-indent:-.25in;mso-list:l1 level1 lfo1"><!--[if !supportLists]--><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><span
style="mso-list:Ignore">1)<span style="font:7.0pt
"Times New Roman"">
</span></span></span><!--[endif]--><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Application
asks for a buffer of specific size, and the buffer is
allocated from the iobuf pool.<o:p></o:p></span></p>
<p class="MsoListParagraph"
style="text-indent:-.25in;mso-list:l1 level1 lfo1"><!--[if !supportLists]--><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><span
style="mso-list:Ignore">2)<span style="font:7.0pt
"Times New Roman"">
</span></span></span><!--[endif]--><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Buffer
is passed on to application, and the application writes
the data into that buffer.
<o:p></o:p></span></p>
<p class="MsoListParagraph"
style="text-indent:-.25in;mso-list:l1 level1 lfo1"><!--[if !supportLists]--><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><span
style="mso-list:Ignore">3)<span style="font:7.0pt
"Times New Roman"">
</span></span></span><!--[endif]--><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Buffer
with data in it is passed from application to libgfapi and
the underlying translators (no memcpy in glfs_write)<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I
have couple of questions, and observations:<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Observations:<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">------------------<o:p></o:p></span></p>
<p class="MsoListParagraph"
style="margin-left:38.25pt;text-indent:-.25in;mso-list:l2
level1 lfo3">
<!--[if !supportLists]--><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><span
style="mso-list:Ignore">1)<span style="font:7.0pt
"Times New Roman"">
</span></span></span><!--[endif]--><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">For
every write if I get a fresh buffer then I don’t see any
problem. All the writes are going through.<o:p></o:p></span></p>
<p class="MsoListParagraph"
style="margin-left:38.25pt;text-indent:-.25in;mso-list:l2
level1 lfo3">
<!--[if !supportLists]--><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><span
style="mso-list:Ignore">2)<span style="font:7.0pt
"Times New Roman"">
</span></span></span><!--[endif]--><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">If
I try to make use of buffer for consecutive writes, then I
am seeing the hang in flush.
<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Question1:
Is it fine if I reuse the buffer for consecutive writes??</span></p>
</div>
</div>
</blockquote>
<br>
The answer is no, If IO-cache and write-behind are enabled. In case
of IO-cache, It just take a ref on the buffer to store into the
cache, which means the io-cache is still using the buffer,<br>
<br>
In case of write-behind, if it decided to aggregate multiple write
request , in this case also it will take a ref on the iobuf and will
lie to the appluca<br>
<br>
<br>
<blockquote
cite="mid:91d83c5f6b7a44c284f0f4962f73d1fa@POST-2.gp.cv.commvault.com"
type="cite">
<div class="WordSection1">
<div>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Question2:
Is it always ensured that the data is written to the file
when I get a response from syncop_writev.</span></p>
</div>
</div>
</blockquote>
<br>
As I explained the write-behind may prevent this.<br>
<br>
<br>
<blockquote
cite="mid:91d83c5f6b7a44c284f0f4962f73d1fa@POST-2.gp.cv.commvault.com"
type="cite">
<div class="WordSection1">
<div>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Sachin
Pandit.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">----------------------------------<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Tue, Jun 21, 2016 at 4:07 AM, Sachin
Pandit <<a moz-do-not-send="true"
href="mailto:spandit@commvault.com" target="_blank">spandit@commvault.com</a>>
wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC
1.0pt;padding:0in 0in 0in
6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi
all,<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I
bid adieu to you all with the hope of crossing path
again, and the time has come rather quickly. It
feels great to work on GlusterFS again.<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">
<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Currently
we are trying to write data backed up by Commvault
Simpana to glusterfs volume (Disperse volume). To
improve the performance, I have implemented the
proposal put forward my Rafi K C [1]. I have some
questions regarding libgfapi and iobuf pool. <o:p>
</o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">To
reduce an extra level of copy in glfs read and
write, I have implemented few APIs to request a
buffer (similar to the one represented in [1]) from
iobuf pool which can be used by the application to
write data to. With this implementation, when I try
to reuse the buffer for consecutive writes, I could
see a hang in syncop_flush of glfs_close (BT of the
hang can be found in [2]). I wanted to know if
reusing the buffer is recommended. If not, do we
need to request buffer for each writes?<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Setup
: Distributed-Disperse ( 4 * (2+1)). Bricks
scattered over 3 nodes.<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[1]
<a moz-do-not-send="true"
href="http://www.gluster.org/pipermail/gluster-devel/2015-February/043966.html"
target="_blank">
http://www.gluster.org/pipermail/gluster-devel/2015-February/043966.html</a><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[2]
Attached file - bt.txt<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks
& Regards,<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Sachin
Pandit.<o:p></o:p></p>
</div>
</div>
<pre><o:p> </o:p></pre>
<pre>***************************Legal Disclaimer***************************<o:p></o:p></pre>
<pre>"This communication may contain confidential and privileged material for the<o:p></o:p></pre>
<pre>sole use of the intended recipient. Any unauthorized review, use or distribution<o:p></o:p></pre>
<pre>by others is strictly prohibited. If you have received the message by mistake,<o:p></o:p></pre>
<pre>please advise the sender by reply email and delete the message. Thank you."<o:p></o:p></pre>
<pre>**********************************************************************<o:p></o:p></pre>
<p class="MsoNormal"><br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a moz-do-not-send="true"
href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<a moz-do-not-send="true"
href="http://www.gluster.org/mailman/listinfo/gluster-devel"
target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<br>
-- <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">Pranith<o:p></o:p></p>
</div>
</div>
</div>
</div>
<pre>
***************************Legal Disclaimer***************************
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**********************************************************************</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-devel mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-devel">http://www.gluster.org/mailman/listinfo/gluster-devel</a></pre>
</blockquote>
<br>
</body>
</html>