<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 12pt;
font-family:Calibri
}
--></style></head>
<body class='hmmessage'><div dir='ltr'>Thank you very much.<div>I just noticed even without ganesha nfs i see this kind of traffic to the lo address.</div><div>and the warning message about the health status only happen when i hit 100% brick utilization, so it should be fine anyway.</div><div>I'll keep digging.</div><div><br></div><div>Thanks again.<br><br><br>
<div class="ecxmoz-signature">-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br><br></font><font color="#3366ff"></font></div><br><br><div>> Subject: Re: [Gluster-users] NFS-Ganesha lo traffic<br>> To: mahdi.adnan@outlook.com<br>> CC: gluster-users@gluster.org; nfs-ganesha-devel@lists.sourceforge.net<br>> From: skoduri@redhat.com<br>> Date: Wed, 10 Aug 2016 11:05:50 +0530<br>> <br>> <br>> <br>> On 08/09/2016 09:06 PM, Mahdi Adnan wrote:<br>> > Hi,<br>> > Thank you for your reply.<br>> ><br>> > The traffic is related to GlusterFS;<br>> ><br>> > 18:31:20.419056 IP 192.168.208.134.49058 > 192.168.208.134.49153: Flags<br>> > [.], ack 3876, win 24576, options [nop,nop,TS val 247718812 ecr<br>> > 247718772], length 0<br>> > 18:31:20.419080 IP 192.168.208.134.49056 > 192.168.208.134.49154: Flags<br>> > [.], ack 11625, win 24576, options [nop,nop,TS val 247718812 ecr<br>> > 247718772], length 0<br>> > 18:31:20.419084 IP 192.168.208.134.49060 > 192.168.208.134.49152: Flags<br>> > [.], ack 9861, win 24576, options [nop,nop,TS val 247718812 ecr<br>> > 247718772], length 0<br>> > 18:31:20.419088 IP 192.168.208.134.49054 > 192.168.208.134.49155: Flags<br>> > [.], ack 4393, win 24568, options [nop,nop,TS val 247718812 ecr<br>> > 247718772], length 0<br>> > 18:31:20.420084 IP 192.168.208.134.49052 > 192.168.208.134.49156: Flags<br>> > [.], ack 5525, win 24576, options [nop,nop,TS val 247718813 ecr<br>> > 247718773], length 0<br>> > 18:31:20.420092 IP 192.168.208.134.49049 > 192.168.208.134.49158: Flags<br>> > [.], ack 6657, win 24576, options [nop,nop,TS val 247718813 ecr<br>> > 247718773], length 0<br>> > 18:31:20.421065 IP 192.168.208.134.49050 > 192.168.208.134.49157: Flags<br>> > [.], ack 4729, win 24570, options [nop,nop,TS val 247718814 ecr<br>> > 247718774], length 0<br>> ><br>> <br>> Looks like that is the traffic coming to the bricks local to that node <br>> (>4915* ports are used by glusterfs brick processes). It could be from <br>> nfs-ganesha or any other glusterfs client processes (like self-heal <br>> daemon etc). Do you see this traffic even when there is no active I/O <br>> from the nfs-client? If so, it could be from the self-heal daemon then. <br>> Verify if there are any files/directories to be healed.<br>> <br>> > Screenshot from wireshark can be found in the attachments.<br>> > 208.134 is the server IP address, and it's looks like it talking to<br>> > itself via the lo interface, im wondering if this is a normal behavior<br>> > or not.<br>> yes. It is the expected behavior when there are clients actively <br>> accessing the volumes.<br>> <br>> > and regarding the Ganesha server logs, how can i debug it to find why<br>> > the servers not responding to the requests on time ?<br>> <br>> I suggest again to take tcpdump. Sometimes nfs-ganesha server (glusterfs <br>> client) may have to communicate with all the bricks over the network <br>> (like LOOKUP) and that may result in delay if there are lots of bricks <br>> involved. Try capturing packets from the node where the nfs-ganesha <br>> server is running and examine the packets between any of the NFS-client <br>> request and its corresponding reply packet.<br>> <br>> I usually use below cmd to capture the packets on all the interfaces -<br>> #tcpdump -i any -s 0 -w /var/tmp/nfs.pcap tcp and not port 22<br>> <br>> Thanks,<br>> Soumya<br>> ><br>> ><br>> > --<br>> ><br>> > Respectfully*<br>> > **Mahdi A. Mahdi*<br>> ><br>> ><br>> ><br>> >> Subject: Re: [Gluster-users] NFS-Ganesha lo traffic<br>> >> To: mahdi.adnan@outlook.com<br>> >> From: skoduri@redhat.com<br>> >> CC: gluster-users@gluster.org; nfs-ganesha-devel@lists.sourceforge.net<br>> >> Date: Tue, 9 Aug 2016 18:02:01 +0530<br>> >><br>> >><br>> >><br>> >> On 08/09/2016 03:33 PM, Mahdi Adnan wrote:<br>> >> > Hi,<br>> >> ><br>> >> > Im using NFS-Ganesha to access my volume, it's working fine for now but<br>> >> > im seeing lots of traffic on the Loopback interface, in fact it's the<br>> >> > same amount of traffic on the bonding interface, can anyone please<br>> >> > explain to me why is this happening ?<br>> >><br>> >> Could you please capture packets on those interfaces using tcpdump and<br>> >> examine the traffic?<br>> >><br>> >> > also, i got the following error in the ganesha log file;<br>> >> ><br>> >> > 09/08/2016 11:35:54 : epoch 57a5da0c : gfs04 :<br>> >> > ganesha.nfsd-1646[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health<br>> >> > status is unhealthy. Not sending heartbeat<br>> >> > 09/08/2016 11:46:04 : epoch 57a5da0c : gfs04 :<br>> >> > ganesha.nfsd-1646[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health<br>> >> > status is unhealthy. Not sending heartbeat<br>> >> > 09/08/2016 11:54:39 : epoch 57a5da0c : gfs04 :<br>> >> > ganesha.nfsd-1646[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health<br>> >> > status is unhealthy. Not sending heartbeat<br>> >> > 09/08/2016 12:06:04 : epoch 57a5da0c : gfs04 :<br>> >> > ganesha.nfsd-1646[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health<br>> >> > status is unhealthy. Not sending heartbeat<br>> >> ><br>> >> > is it something i should care about ?<br>> >><br>> >> Above warnings are thrown when the outstanding rpc request queue count<br>> >> doesn't change within two heartbeats, in other words the server may be<br>> >> taking a while to process the requests and responding slowly to its<br>> > clients.<br>> >><br>> >> Thanks,<br>> >> Soumya<br>> >><br>> >> ><br>> >> > My ganesha config is the following;<br>> >> ><br>> >> ><br>> >> > EXPORT{<br>> >> > Export_Id = 1 ;<br>> >> > Path = "/vlm02";<br>> >> ><br>> >> > FSAL {<br>> >> > name = GLUSTER;<br>> >> > hostname = "gfs04";<br>> >> > volume = "vlm02";<br>> >> > }<br>> >> ><br>> >> > Access_type = RW;<br>> >> > Disable_ACL = TRUE;<br>> >> > Squash = No_root_squash;<br>> >> > Protocols = "3" ;<br>> >> > Transports = "TCP";<br>> >> > }<br>> >> ><br>> >> ><br>> >> > Im accessing it via a floating ip assigned by CTDB.<br>> >> ><br>> >> ><br>> >> > Thank you.<br>> >> > --<br>> >> ><br>> >> > Respectfully*<br>> >> > **Mahdi A. Mahdi*<br>> >> ><br>> >> ><br>> >> ><br>> >> > _______________________________________________<br>> >> > Gluster-users mailing list<br>> >> > Gluster-users@gluster.org<br>> >> > http://www.gluster.org/mailman/listinfo/gluster-users<br>> >> ><br></div></div> </div></body>
</html>