<div dir="ltr">here is the profile for about 30 seconds.. I didn't let it run a full 60:<div><br></div><div><div>Brick: media2-be:/gluster/brick1/<wbr>gluster_volume_0</div><div>------------------------------<wbr>-------------------</div><div>Cumulative Stats:</div><div> Block Size: 512b+ 1024b+ 2048b+ </div><div> No. of Reads: 0 0 0 </div><div>No. of Writes: 31133 37339 35573 </div><div> </div><div> Block Size: 4096b+ 8192b+ 16384b+ </div><div> No. of Reads: 0 0 0 </div><div>No. of Writes: 284535 91431 43838 </div><div> </div><div> Block Size: 32768b+ 65536b+ 131072b+ </div><div> No. of Reads: 0 0 181121 </div><div>No. of Writes: 27764 22258 226187 </div><div> </div><div> Block Size: 262144b+ </div><div> No. of Reads: 0 </div><div>No. of Writes: 7 </div><div> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div> --------- ----------- ----------- ----------- ------------ ----</div><div> 0.00 0.00 us 0.00 us 0.00 us 53 RELEASE</div><div> 0.00 0.00 us 0.00 us 0.00 us 272 RELEASEDIR</div><div> 0.08 217.02 us 17.00 us 46751.00 us 837 STAT</div><div> 0.54 487.22 us 5.00 us 150634.00 us 2675 FINODELK</div><div> 1.07 2591.53 us 24.00 us 186199.00 us 1001 READ</div><div> 1.77 3224.61 us 16.00 us 113361.00 us 1322 WRITE</div><div> 3.02 84.19 us 8.00 us 186102.00 us 86693 INODELK</div><div> 5.10 11293.23 us 20.00 us 153002.00 us 1090 FXATTROP</div><div> 88.42 395188.99 us 2771.00 us 2378742.00 us 540 FSYNC</div><div> </div><div> Duration: 82547 seconds</div><div> Data Read: 23739891712 bytes</div><div>Data Written: 36058159104 bytes</div><div> </div><div>Interval 1 Stats:</div><div> Block Size: 512b+ 1024b+ 2048b+ </div><div> No. of Reads: 0 0 0 </div><div>No. of Writes: 24 2 14 </div><div> </div><div> Block Size: 4096b+ 8192b+ 16384b+ </div><div> No. of Reads: 0 0 0 </div><div>No. of Writes: 167 28 4 </div><div> </div><div> Block Size: 32768b+ 65536b+ 131072b+ </div><div> No. of Reads: 0 0 309 </div><div>No. of Writes: 8 1 0 </div><div> </div><div> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div> --------- ----------- ----------- ----------- ------------ ----</div><div> 0.13 61.58 us 17.00 us 2074.00 us 224 STAT</div><div> 0.19 41.50 us 7.00 us 4143.00 us 498 FINODELK</div><div> 0.53 188.37 us 27.00 us 11377.00 us 309 READ</div><div> 3.04 1337.96 us 18.00 us 48765.00 us 248 WRITE</div><div> 5.28 2594.87 us 21.00 us 47939.00 us 222 FXATTROP</div><div> 14.58 20.41 us 8.00 us 47905.00 us 77937 INODELK</div><div> 76.25 74945.14 us 23687.00 us 199942.00 us 111 FSYNC</div><div> </div><div> Duration: 53 seconds</div><div> Data Read: 40501248 bytes</div><div>Data Written: 1512448 bytes</div><div> </div><div>Brick: media1-be:/gluster/brick1/<wbr>gluster_volume_0</div><div>------------------------------<wbr>-------------------</div><div>Cumulative Stats:</div><div> Block Size: 512b+ 1024b+ 2048b+ </div><div> No. of Reads: 0 0 0 </div><div>No. of Writes: 2831 4699 6142 </div><div> </div><div> Block Size: 4096b+ 8192b+ 16384b+ </div><div> No. of Reads: 0 0 0 </div><div>No. of Writes: 46751 16712 7972 </div><div> </div><div> Block Size: 32768b+ 65536b+ 131072b+ </div><div> No. of Reads: 0 0 0 </div><div>No. of Writes: 4462 2938 27952 </div><div> </div><div> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div> --------- ----------- ----------- ----------- ------------ ----</div><div> 0.00 0.00 us 0.00 us 0.00 us 7 RELEASE</div><div> 0.00 0.00 us 0.00 us 0.00 us 11 RELEASEDIR</div><div> 1.75 245.15 us 46.00 us 19886.00 us 1321 WRITE</div><div> 6.99 1191.45 us 114.00 us 215838.00 us 1089 FXATTROP</div><div> 10.07 698.36 us 18.00 us 286316.00 us 2674 FINODELK</div><div> 24.51 52.44 us 23.00 us 171166.00 us 86694 INODELK</div><div> 56.69 19472.96 us 1568.00 us 249274.00 us 540 FSYNC</div><div> </div><div> Duration: 2224 seconds</div><div> Data Read: 0 bytes</div><div>Data Written: 4669031424 bytes</div><div> </div><div>Interval 1 Stats:</div><div> Block Size: 512b+ 1024b+ 2048b+ </div><div> No. of Reads: 0 0 0 </div><div>No. of Writes: 24 2 14 </div><div> </div><div> Block Size: 4096b+ 8192b+ 16384b+ </div><div> No. of Reads: 0 0 0 </div><div>No. of Writes: 167 28 4 </div><div> </div><div> Block Size: 32768b+ 65536b+ </div><div> No. of Reads: 0 0 </div><div>No. of Writes: 8 1 </div><div> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div> --------- ----------- ----------- ----------- ------------ ----</div><div> 1.12 302.43 us 114.00 us 10132.00 us 222 FXATTROP</div><div> 1.18 285.74 us 56.00 us 7058.00 us 248 WRITE</div><div> 4.31 519.43 us 21.00 us 188427.00 us 498 FINODELK</div><div> 32.76 17714.02 us 5018.00 us 205904.00 us 111 FSYNC</div><div> 60.63 46.69 us 23.00 us 9550.00 us 77936 INODELK</div><div> </div><div> Duration: 53 seconds</div><div> Data Read: 0 bytes</div><div>Data Written: 1512448 bytes</div></div><div><br></div><div><br></div><div><div>[root@media2 ~]# gluster volume status</div><div>Status of volume: gvol0</div><div>Gluster process TCP Port RDMA Port Online Pid</div><div>------------------------------<wbr>------------------------------<wbr>------------------</div><div>Brick media1-be:/gluster/brick1/<wbr>gluster_vol</div><div>ume_0 49152 0 Y 2829 </div><div>Brick media2-be:/gluster/brick1/<wbr>gluster_vol</div><div>ume_0 49152 0 Y 1456 </div><div>NFS Server on localhost N/A N/A N N/A </div><div>Self-heal Daemon on localhost N/A N/A Y 1451 </div><div>NFS Server on media1 N/A N/A N N/A </div><div>Self-heal Daemon on media1 N/A N/A Y 2824 </div><div> </div><div>Task Status of Volume gvol0</div><div>------------------------------<wbr>------------------------------<wbr>------------------</div><div>There are no active volume tasks</div><div> </div></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 6, 2016 at 4:25 PM, Michael Ciccarelli <span dir="ltr"><<a href="mailto:mikecicc01@gmail.com" target="_blank">mikecicc01@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>this is the info file contents.. is there another file you would want to see for config?</div><div>type=2</div><div>count=2</div><div>status=1</div><div>sub_count=2</div><div>stripe_count=1</div><div>replica_count=2</div><div>disperse_count=0</div><div>redundancy_count=0</div><div>version=3</div><div>transport-type=0</div><div>volume-id=98c258e6-ae9e-4407-<wbr>8f25-7e3f7700e100</div><div>username=removed just cause</div><div>password=removed just cause</div><div>op-version=3</div><div>client-op-version=3</div><div>quota-version=0</div><div>parent_volname=N/A</div><div>restored_from_snap=00000000-<wbr>0000-0000-0000-000000000000</div><div>snap-max-hard-limit=256</div><div>diagnostics.count-fop-hits=on</div><div>diagnostics.latency-<wbr>measurement=on</div><div>performance.readdir-ahead=on</div><div>brick-0=media1-be:-gluster-<wbr>brick1-gluster_volume_0</div><div>brick-1=media2-be:-gluster-<wbr>brick1-gluster_volume_0</div><div><br></div><div>here are some log entries, etc-glusterfs-glusterd.vol.<wbr>log:</div><div><div>The message "I [MSGID: 106006] [glusterd-svc-mgmt.c:323:<wbr>glusterd_svc_common_rpc_<wbr>notify] 0-management: nfs has disconnected from glusterd." repeated 39 times between [2016-10-06 20:10:14.963402] and [2016-10-06 20:12:11.979684]</div></div><div><div>[2016-10-06 20:12:14.980203] I [MSGID: 106006] [glusterd-svc-mgmt.c:323:<wbr>glusterd_svc_common_rpc_<wbr>notify] 0-management: nfs has disconnected from glusterd.</div><div>[2016-10-06 20:13:50.993490] W [socket.c:596:__socket_rwv] 0-nfs: readv on /var/run/gluster/<wbr>360710d59bc4799f8c8a6374936d2b<wbr>1b.socket failed (Invalid argument)</div></div><div><br></div><div>I can provide any specific details you would like to see.. Last night I tried 1 more time and it appeared to be working ok for running 1 VM under VMware but as soon as I had 3 running the targets became unresponsive. I believe gluster volume is ok but for whatever reason the ISCSI target daemon seems to be having some issues...</div><div><br></div><div>here is from the messages file:</div><div><div>Oct 5 23:13:00 media2 kernel: MODE SENSE: unimplemented page/subpage: 0x1c/0x02</div><div>Oct 5 23:13:00 media2 kernel: MODE SENSE: unimplemented page/subpage: 0x1c/0x02</div><div>Oct 5 23:13:35 media2 kernel: iSCSI/iqn.1998-01.com.vmware:<wbr>vmware4-0941d552: Unsupported SCSI Opcode 0x4d, sending CHECK_CONDITION.</div><div>Oct 5 23:13:35 media2 kernel: iSCSI/iqn.1998-01.com.vmware:<wbr>vmware4-0941d552: Unsupported SCSI Opcode 0x4d, sending CHECK_CONDITION.</div></div><div><br></div><div>and here are some more VMware iscsi errors:</div><div><div>2016-10-06T20:22:11.496Z cpu2:32825)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x89 (0x412e808532c0, 32801) to dev "naa.<wbr>6001405c0d86944f3d2468d80c7d15<wbr>40" on</div><div>2016-10-06T20:22:11.635Z cpu2:32787)ScsiDeviceIO: 2338: Cmd(0x412e808532c0) 0x89, CmdSN 0x4f05 from world 32801 to dev "naa.<wbr>6001405c0d86944f3d2468d80c7d1</div><div>2016-10-06T20:22:11.635Z cpu3:35532)Fil3: 15389: Max timeout retries exceeded for caller Fil3_FileIO (status 'Timeout') </div><div>2016-10-06T20:22:11.635Z cpu2:196414)HBX: 2832: Waiting for timed out [HB state abcdef02 offset 3928064 gen 25 stampUS 49571997650 uuid 57f5c142-45632d75</div><div>2016-10-06T20:22:11.635Z cpu3:35532)HBX: 2832: Waiting for timed out [HB state abcdef02 offset 3928064 gen 25 stampUS 49571997650 uuid 57f5c142-45632d75-</div><div>2016-10-06T20:22:11.635Z cpu0:32799)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412e80848580, 32799) to dev "naa.<wbr>6001405c0d86944f3d2468d80c7d15<wbr>40" on</div><div>2016-10-06T20:22:11.635Z cpu0:32799)ScsiDeviceIO: 2325: Cmd(0x412e80848580) 0x28, CmdSN 0x4f06 from world 32799 to dev "naa.<wbr>6001405c0d86944f3d2468d80c7d1</div><div>2016-10-06T20:22:11.773Z cpu0:32843)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412e80848580, 32799) to dev "naa.<wbr>6001405c0d86944f3d2468d80c7d15<wbr>40" on</div><div>2016-10-06T20:22:11.916Z cpu0:35549)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412e80848580, 32799) to dev "naa.<wbr>6001405c0d86944f3d2468d80c7d15<wbr>40" on</div><div>2016-10-06T20:22:12.000Z cpu2:33431)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x410987bf0800 network resource pool netsched.pools.persist.iscsi associa</div><div>2016-10-06T20:22:12.000Z cpu2:33431)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x410987bf0800 network tracker id 16 tracker.iSCSI.172.16.1.40 associated</div><div>2016-10-06T20:22:12.056Z cpu0:35549)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412e80848580, 32799) to dev "naa.<wbr>6001405c0d86944f3d2468d80c7d15<wbr>40" on</div><div>2016-10-06T20:22:12.194Z cpu0:35549)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412e80848580, 32799) to dev "naa.<wbr>6001405c0d86944f3d2468d80c7d15<wbr>40" on</div><div>2016-10-06T20:22:12.253Z cpu2:33431)WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhba38:CH:1 T:1 CN:0: iSCSI connection is being marked "ONLINE" </div><div>2016-10-06T20:22:12.253Z cpu2:33431)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000004 TARGET: iqn.2016-09.iscsi.gluster:<wbr>shared TPGT:</div><div>2016-10-06T20:22:12.253Z cpu2:33431)WARNING: iscsi_vmk: iscsivmk_StartConnection: Conn [CID: 0 L: <a href="http://172.16.1.53:49959" target="_blank">172.16.1.53:49959</a> R: <a href="http://172.16.1.40:3260" target="_blank">172.16.1.40:3260</a>] </div></div><div><br></div><div>Is it that the gluster overhead is just killing LIO/target?</div><div><br></div><div>thanks,</div><div>Mike</div><div><br></div><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 6, 2016 at 12:22 PM, Vijay Bellur <span dir="ltr"><<a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Mike,<br>
<br>
Can you please share your gluster volume configuration?<br>
<br>
Also do you notice anything in client logs on the node where fileio<br>
backstore is configured?<br>
<br>
Thanks,<br>
Vijay<br>
<div><div class="m_333360447997162302h5"><br>
On Wed, Oct 5, 2016 at 8:56 PM, Michael Ciccarelli <<a href="mailto:mikecicc01@gmail.com" target="_blank">mikecicc01@gmail.com</a>> wrote:<br>
> So I have a fairly basic setup using glusterfs between 2 nodes. The nodes<br>
> have 10 gig connections and the bricks reside on SSD LVM LUNs:<br>
><br>
> Brick1: media1-be:/gluster/brick1/glus<wbr>ter_volume_0<br>
> Brick2: media2-be:/gluster/brick1/glus<wbr>ter_volume_0<br>
><br>
><br>
> On this volume I have a LIO iscsi target with 1 fileio backstore that's<br>
> being shared out to vmware ESXi hosts. The volume is around 900 gig and the<br>
> fileio store is around 850g:<br>
><br>
> -rw-r--r-- 1 root root 912680550400 Oct 5 20:47 iscsi.disk.3<br>
><br>
> I set the WWN to be the same so the ESXi hosts see the nodes as 2 paths to<br>
> the same target. I believe this is what I want. The issues I'm seeing is<br>
> that while the IO wait is low I'm seeing high CPU usage with only 3 VMs<br>
> running on only 1 of the ESX servers:<br>
><br>
> this is media2-be:<br>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>
> 1474 root 20 0 1396620 37912 5980 S 135.0 0.1 157:01.84<br>
> glusterfsd<br>
> 1469 root 20 0 747996 13724 5424 S 2.0 0.0 1:10.59<br>
> glusterfs<br>
><br>
> And this morning it seemed like I had to restart the LIO service on<br>
> media1-be as the VMware was seeing time-out issues. I'm seeing issues like<br>
> this on the VMware ESX servers:<br>
><br>
> 2016-10-06T00:51:41.100Z cpu0:32785)WARNING: ScsiDeviceIO: 1223: Device<br>
> naa.600140501ce79002e724ebdb66<wbr>a6756d performance has deteriorated. I/O<br>
> latency increased from average value of 33420 microseconds to 732696<br>
> microseconds.<br>
><br>
> Are there any special settings I need to have gluster+LIO+vmware to work?<br>
> Has anyone gotten this to work fairly well that it is stable? What am I<br>
> missing?<br>
><br>
> thanks,<br>
> Mike<br>
><br>
><br>
><br>
</div></div>> ______________________________<wbr>_________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a><br>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>