<div dir="ltr"><div>Gluster volume configuration, those bold entries are the initial settings i have<br></div><div><div><br></div><div>Volume Name: g37test</div><div>Type: Stripe</div><div>Volume ID: 3f9dae3d-08f9-4321-aeac-67f44c7eb1ac</div><div>Status: Created</div><div>Number of Bricks: 1 x 10 = 10</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: 192.168.123.4:/mnt/sdb_mssd/data</div><div>Brick2: 192.168.123.4:/mnt/sdc_mssd/data</div><div>Brick3: 192.168.123.4:/mnt/sdd_mssd/data</div><div>Brick4: 192.168.123.4:/mnt/sde_mssd/data</div><div>Brick5: 192.168.123.4:/mnt/sdf_mssd/data</div><div>Brick6: 192.168.123.4:/mnt/sdg_mssd/data</div><div>Brick7: 192.168.123.4:/mnt/sdh_mssd/data</div><div>Brick8: 192.168.123.4:/mnt/sdj_mssd/data</div><div>Brick9: 192.168.123.4:/mnt/sdm_mssd/data</div><div>Brick10: 192.168.123.4:/mnt/sdn_mssd/data</div><div>Options Reconfigured:</div><div><b>server.allow-insecure: on</b></div><div><b>storage.owner-uid: 165</b></div><div><b>storage.owner-gid: 165</b></div><div><b>performance.quick-read: off</b></div><div><b>performance.io-cache: off</b></div><div>performance.read-ahead: off</div><div>performance.stat-prefetch: off</div><div>cluster.eager-lock: enable</div><div>network.remote-dio: enable</div><div>cluster.quorum-type: auto</div><div>cluster.server-quorum-type: server</div><div><b>nfs.disable: true</b></div></div><div><br></div><div><br></div><div>Tried those suggested settings but Guest CPU utilization still high.</div><div><br></div><div>4k random write</div><div><br></div><div><table border="0" cellpadding="0" cellspacing="0" width="294" style="border-collapse:collapse;width:220pt">
<colgroup><col width="187" style="width:140pt">
<col width="107" style="width:80pt">
</colgroup><tbody><tr height="20" style="height:15pt">
<td height="20" width="187" style="height:15pt;width:140pt">IOPS</td>
<td align="right" width="107" style="width:80pt">5054.46</td>
</tr>
<tr height="20" style="height:15pt">
<td height="20" style="height:15pt">Avg. response time (ms)</td>
<td align="right">2.94</td>
</tr>
<tr height="20" style="height:15pt">
<td height="20" style="height:15pt">CPU utilization total (%)</td>
<td align="right">96.09</td>
</tr>
<tr height="20" style="height:15pt">
<td height="20" style="height:15pt">CPU Privilegde time (%)</td>
<td align="right">92.3</td>
</tr></tbody></table><br></div><div>4k random read</div><div><br></div><div><table border="0" cellpadding="0" cellspacing="0" width="294" style="border-collapse:collapse;width:220pt">
<colgroup><col width="187" style="width:140pt">
<col width="107" style="width:80pt">
</colgroup><tbody><tr height="20" style="height:15pt">
<td height="20" width="187" style="height:15pt;width:140pt"></td>
<td width="107" style="width:80pt">4k random read</td>
</tr>
<tr height="20" style="height:15pt">
<td height="20" style="height:15pt">IOPS</td>
<td align="right">24805.38</td>
</tr>
<tr height="20" style="height:15pt">
<td height="20" style="height:15pt">Avg. response time (ms)</td>
<td align="right">0.6</td>
</tr>
<tr height="20" style="height:15pt">
<td height="20" style="height:15pt">CPU utilization total (%)</td>
<td align="right">92.77</td>
</tr>
<tr height="20" style="height:15pt">
<td height="20" style="height:15pt">CPU Privilegde time (%)</td>
<td align="right">89.33</td>
</tr></tbody></table><br></div><div>1) disable write-behind, IOPS cannot go higher anymore due to guest CPU utilization, so will test this once i setup the ubuntu VM again.</div><div><br></div><div>2) This is what i got, capture during the Iometer test<br></div><div><br></div><div><div># ps -ef | grep qemu-kvm</div><div>qemu 12184 1 99 00:59 ? 00:24:15 /usr/libexec/qemu-kvm -name instance-000000b6 -S -machine pc-i440fx-rhel7.1.0,accel=kvm,usb=off -cpu Haswell,+abm,+pdpe1gb,+rdrand,+f16c,+osxsave,+dca,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme,-rtm,-hle -m 8192 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 0994b9e9-e356-459e-8ff8-c302576b3d7f -smbios type=1,manufacturer=Fedora Project,product=OpenStack Nova,version=12.0.1-1.el7,serial=423c213b-7142-4053-a119-d06ae77b432a,uuid=0994b9e9-e356-459e-8ff8-c302576b3d7f,family=Virtual Machine -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-instance-000000b6/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=gluster://ad11:24007/adtestOS/volume-2e04c86f-0225-41b6-aadc-67a3a72853d4,if=none,id=drive-virtio-disk0,format=raw,serial=2e04c86f-0225-41b6-aadc-67a3a72853d4,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=gluster://ad11:24007/adtestOS/volume-1d5c62af-09b0-44ea-b4e8-91d3a14859cc,if=none,id=drive-virtio-disk1,format=raw,serial=1d5c62af-09b0-44ea-b4e8-91d3a14859cc,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=gluster://ad11:24007/adtestOS/volume-7014729d-b6a4-4907-b50d-01f870eebb5e,if=none,id=drive-virtio-disk2,format=raw,serial=7014729d-b6a4-4907-b50d-01f870eebb5e,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk2,id=virtio-disk2 -netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:67:11:4a,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/0994b9e9-e356-459e-8ff8-c302576b3d7f/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc <a href="http://0.0.0.0:0">0.0.0.0:0</a> -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on</div><div><br></div><div># perf record -p 12184</div><div>^C[ perf record: Woken up 485 times to write data ]</div><div>[ perf record: Captured and wrote 125.231 MB perf.data (3252540 samples) ]</div><div><br></div><div>__memset_sse2 /usr/lib64/<a href="http://libc-2.17.so">libc-2.17.so</a></div><div> 0.01 │ lea next_state.7659+0x19c,%rcx</div><div> 0.01 │ movswq (%rcx,%r8,2),%rcx</div><div> 0.20 │ lea (%rcx,%r11,1),%r11</div><div> 0.00 │ ↓ jmpq fffffffffff77150</div><div> │ nop</div><div> 0.00 │920: cmp $0x0,%r9</div><div> │ ↑ je 8c0</div><div> │ ↓ jmp 930</div><div> │ nop</div><div> 0.08 │930: lea -0x80(%r8),%r8</div><div> 7.98 │ cmp $0x80,%r8</div><div> 1.73 │ movntd %xmm0,(%rdi)</div><div> 7.91 │ movntd %xmm0,0x10(%rdi)</div><div> 4.16 │ movntd %xmm0,0x20(%rdi)</div><div> 39.22 │ movntd %xmm0,0x30(%rdi)</div><div> 7.35 │ movntd %xmm0,0x40(%rdi)</div><div> 0.98 │ movntd %xmm0,0x50(%rdi)</div><div> 2.15 │ movntd %xmm0,0x60(%rdi)</div><div> 23.22 │ movntd %xmm0,0x70(%rdi)</div><div> 1.72 │ lea 0x80(%rdi),%rdi</div><div> 0.04 │ ↑ jae 930</div><div> 0.00 │ sfence</div><div> 0.00 │ add %r8,%rdi</div><div> 0.00 │ lea __GI_memset+0x43d,%r11</div><div> 0.00 │ lea next_state.7659+0x19c,%rcx</div><div> │ movswq (%rcx,%r8,2),%rcx</div><div> 0.03 │ lea (%rcx,%r11,1),%r11</div><div> 0.00 │ ↓ jmpq fffffffffff77150</div><div> │ nop</div><div>Press 'h' for help on key bindings</div></div><div><br></div><div>Thanks.</div><div><br></div><div>Cw</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Apr 20, 2016 at 7:18 AM, Vijay Bellur <span dir="ltr"><<a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On Tue, Apr 19, 2016 at 1:24 AM, qingwei wei <<a href="mailto:tchengwee@gmail.com">tchengwee@gmail.com</a>> wrote:<br>
> Hi Vijay,<br>
><br>
> I rerun the test with gluster 3.7.11and found that the utilization still<br>
> high when i use libgfapi. The write performance is also no good.<br>
><br>
> Below are the info and results:<br>
><br>
> Hypervisor host:<br>
> libvirt 1.2.17-13.el7_2.4<br>
> qemu-kvm 2.1.2-23.el7.1<br>
><br>
><br>
> WIndows VM:<br>
> Windows 2008R2<br>
> IOmeter<br>
><br>
> Ubuntu VM:<br>
> Ubuntu 14.04<br>
> fio 2.1.3<br>
><br>
><br>
> Windows VM (libgfapi)<br>
><br>
> 4k random read<br>
><br>
> IOPS 22920.76<br>
> Avg. response time (ms) 0.65<br>
> CPU utilization total (%) 82.02<br>
> CPU Privilegde time (%) 77.76<br>
><br>
> 4k random write<br>
><br>
> IOPS 4526.39<br>
> Avg. response time (ms) 3.26<br>
> CPU utilization total (%) 93.61<br>
> CPU Privilegde time (%) 90.24<br>
><br>
> Windows VM (fuse)<br>
><br>
> 4k random read<br>
><br>
> IOPS 14662.86<br>
> Avg. response time (ms) 1.08<br>
> CPU utilization total (%) 27.66<br>
> CPU Privilegde time (%) 24.45<br>
><br>
> 4k random write<br>
><br>
> IOPS 16911.66<br>
> Avg. response time (ms) 0.94<br>
> CPU utilization total (%) 26.74<br>
> CPU Privilegde time (%) 22.64<br>
><br>
> Ubuntu VM (libgfapi)<br>
><br>
> 4k random read<br>
><br>
> IOPS 34364<br>
> Avg. response time (ms) 0.46<br>
> CPU utilization total (%) 6.09<br>
><br>
> 4k random write<br>
><br>
> IOPS 4531<br>
> Avg. response time (ms) 3.53<br>
> CPU utilization total (%) 1.2<br>
><br>
> Ubuntu VM (fuse)<br>
><br>
> 4k random read<br>
><br>
> IOPS 17341<br>
> Avg. response time (ms) 0.92<br>
> CPU utilization total (%) 4.22<br>
><br>
> 4k random write<br>
><br>
> IOPS 17611<br>
> Avg. response time (ms) 0.91<br>
> CPU utilization total (%) 4.65<br>
><br>
> Any comments on this or things i should try?<br>
><br>
<br>
</div></div>Can you please share your gluster volume configuration? It might be<br>
worth checking if the tunables in profile virt<br>
(extras/group-virt.example) are applied on this volume.<br>
<br>
Additionally I would also try to:<br>
1 disable write-behind in gluster to see if there is any peformance<br>
difference for writes<br>
2 use perf record -p <qemu> followed by perf annotate to observe the<br>
hot threads.<br>
<br>
HTH,<br>
Vijay<br>
</blockquote></div><br></div>