3) Why not have a separate iobuf pool for RDMA?<br><br><div class="gmail_quote">On Tue Jan 13 2015 at 6:30:09 AM Mohammed Rafi K C <<a href="mailto:rkavunga@redhat.com">rkavunga@redhat.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi All,<br>
<br>
When using RDMA protocol, we need to register the buffer which is going<br>
to send through rdma with rdma device. In fact, it is a costly<br>
operation, and a performance killer if it happened in I/O path. So our<br>
current plan is to register pre-allocated iobuf_arenas from iobuf_pool<br>
with rdma when rdma is getting initialized. The problem comes when all<br>
the iobufs are exhausted, then we need to dynamically allocate new<br>
arenas from libglusterfs module. Since it is created in libglusterfs, we<br>
can't make a call to rdma from libglusterfs. So we will force to<br>
register each of the iobufs from the newly created arenas with rdma in<br>
I/O path. If io-cache is turned on in client stack, then all the<br>
pre-registred arenas will use by io-cache as cache buffer. so we have to<br>
do the registration in rdma for each i/o call for every iobufs,<br>
eventually we cannot make use of pre registered arenas.<br>
<br>
To address the issue, we have two approaches in mind,<br>
<br>
1) Register each dynamically created buffers in iobuf by bringing<br>
transport layer together with libglusterfs.<br>
<br>
2) create a separate buffer for caching and offload the data from the<br>
read response to the cache buffer in background.<br>
<br>
If we could make use of preregister memory for every rdma call, then we<br>
will have approximately 20% increment for write and 25% of increment for<br>
read.<br>
<br>
Please give your thoughts to address the issue.<br>
<br>
Thanks & Regards<br>
Rafi KC<br>
______________________________<u></u>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://www.gluster.org/<u></u>mailman/listinfo/gluster-devel</a><br>
</blockquote></div>