<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
        {font-family:"Cambria Math";
        panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
        {font-family:Calibri;
        panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0cm;
        margin-bottom:.0001pt;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
        {mso-style-priority:99;
        color:blue;
        text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
        {mso-style-priority:99;
        color:#954F72;
        text-decoration:underline;}
.MsoChpDefault
        {mso-style-type:export-only;}
@page WordSection1
        {size:612.0pt 792.0pt;
        margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
        {page:WordSection1;}
--></style>
</head>
<body lang="DE" link="blue" vlink="#954F72">
<div class="WordSection1">
<p style="margin:0cm;margin-bottom:.0001pt">Dear All,</p>
<p style="margin:0cm;margin-bottom:.0001pt">&nbsp;</p>
<p style="margin:0cm;margin-bottom:.0001pt">i have a glusterfs pool with 3 servers with Centos7.3, Glusterfs 3.8.5, network is Infiniband.</p>
<p style="margin:0cm;margin-bottom:.0001pt">Pacemaker/Corosync and Ganesha-NFS is installed and all seems to be OK, no error logged.</p>
<p style="margin:0cm;margin-bottom:.0001pt">I created a replica 3 volume with transport rdma (without tcp!).</p>
<p style="margin:0cm;margin-bottom:.0001pt">When i mount this volume via glusterfs and do some IO, no errors are logged and everything seems to go pretty well.</p>
<p style="margin:0cm;margin-bottom:.0001pt">&nbsp;</p>
<p style="margin:0cm;margin-bottom:.0001pt">When i mount the volume via nfs and do some IO, nfs freezes immediatly and following logs are written to
</p>
<p style="margin:0cm;margin-bottom:.0001pt">ganesha-gfapi.log:</p>
<p style="margin:0cm;margin-bottom:.0001pt">2017-01-05 23:23:53.536526] W [MSGID: 103004] [rdma.c:452:gf_rdma_register_arena] 0-rdma: allocation of mr failed</p>
<p style="margin:0cm;margin-bottom:.0001pt">[2017-01-05 23:23:53.541519] W [MSGID: 103004] [rdma.c:1463:__gf_rdma_create_read_chunks_from_vector] 0-rpc-transport/rdma: memory registration failed (peer:10.40.1.1:49152) [Keine Berechtigung]</p>
<p style="margin:0cm;margin-bottom:.0001pt">[2017-01-05 23:23:53.541547] W [MSGID: 103029] [rdma.c:1558:__gf_rdma_create_read_chunks] 0-rpc-transport/rdma: cannot create read chunks from vector entry-&gt;prog_payload</p>
<p style="margin:0cm;margin-bottom:.0001pt">[2017-01-05 23:23:53.541553] W [MSGID: 103033] [rdma.c:2063:__gf_rdma_ioq_churn_request] 0-rpc-transport/rdma: creation of read chunks failed</p>
<p style="margin:0cm;margin-bottom:.0001pt">[2017-01-05 23:23:53.541557] W [MSGID: 103040] [rdma.c:2775:__gf_rdma_ioq_churn_entry] 0-rpc-transport/rdma: failed to process request ioq entry to peer(10.40.1.1:49152)</p>
<p style="margin:0cm;margin-bottom:.0001pt">[2017-01-05 23:23:53.541562] W [MSGID: 103040] [rdma.c:2859:gf_rdma_writev] 0-vmstor1-client-0: processing ioq entry destined to (10.40.1.1:49152) failed</p>
<p style="margin:0cm;margin-bottom:.0001pt">[2017-01-05 23:23:53.541569] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 0-rpc-transport/rdma: sending request to peer (10.40.1.1:49152) failed</p>
<p style="margin:0cm;margin-bottom:.0001pt">[…]</p>
<p style="margin:0cm;margin-bottom:.0001pt">&nbsp;</p>
<p style="margin:0cm;margin-bottom:.0001pt">Some additional info:</p>
<p style="margin:0cm;margin-bottom:.0001pt">Firewall is disabled, SELinux is disabled.</p>
<p style="margin:0cm;margin-bottom:.0001pt">Different hardware with Centos 7.1 and the Mellanox OFED 3.4 packages instead of the Centos Infiniband packages lead to the same results.</p>
<p style="margin:0cm;margin-bottom:.0001pt">Just to mention: I am not trying to do NFS over RDMA, the Ganesha FSAL is just configured to &quot;glusterfs&quot;.</p>
<p style="margin:0cm;margin-bottom:.0001pt">&nbsp;</p>
<p style="margin:0cm;margin-bottom:.0001pt">I hope someone could help me, i am running out of ideas…</p>
<p style="margin:0cm;margin-bottom:.0001pt">&nbsp;</p>
<p style="margin:0cm;margin-bottom:.0001pt">Kind regards,</p>
<p style="margin:0cm;margin-bottom:.0001pt">Andreas</p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
</body>
</html>