<div dir="ltr">Hi Vijay,<div><br></div><div>I finally manage to do this test on the shared volume.</div><div><br></div><div>gluster volume info</div><div><br></div><div><div>Volume Name: abctest</div><div>Type: Distributed-Replicate</div><div>Volume ID: 0db494e2-51a3-4521-a1ba-5d3479cecba2</div><div>Status: Started</div><div>Number of Bricks: 3 x 3 = 9</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: abc11:/data/hdd1/abctest</div><div>Brick2: abc12:/data/hdd1/abctest</div><div>Brick3: abc14:/data/hdd1/abctest</div><div>Brick4: abc16:/data/hdd1/abctest</div><div>Brick5: abc17:/data/hdd1/abctest</div><div>Brick6: abc20:/data/hdd1/abctest</div><div>Brick7: abc22:/data/hdd1/abctest</div><div>Brick8: abc23:/data/hdd1/abctest</div><div>Brick9: abc24:/data/hdd1/abctest</div><div>Options Reconfigured:</div><div>features.shard-block-size: 16MB</div><div>features.shard: on</div><div>server.allow-insecure: on</div><div>storage.owner-uid: 165</div><div>storage.owner-gid: 165</div><div>nfs.disable: true</div><div>performance.quick-read: off</div><div>performance.io-cache: off</div><div>performance.read-ahead: off</div><div>performance.stat-prefetch: off</div><div>cluster.lookup-optimize: on</div><div>cluster.quorum-type: auto</div><div>cluster.server-quorum-type: server</div><div>transport.address-family: inet</div><div>performance.readdir-ahead: off</div></div><div><br></div><div>Result is still the same.</div><div><br></div><div><div style="font-size:12.8px">4k random write</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><table border="0" cellpadding="0" cellspacing="0" width="294" style="border-collapse:collapse;width:220pt"><colgroup><col width="187" style="width:140pt"><col width="107" style="width:80pt"></colgroup><tbody><tr height="20" style="height:15pt"><td height="20" width="187" style="height:15pt;width:140pt">IOPS</td><td align="right" width="107" style="width:80pt">5355.75</td></tr><tr height="20" style="height:15pt"><td height="20" style="height:15pt">Avg. response time (ms)</td><td align="right">2.79</td></tr><tr height="20" style="height:15pt"><td height="20" style="height:15pt">CPU utilization total (%)</td><td align="right">96.73</td></tr><tr height="20" style="height:15pt"><td height="20" style="height:15pt">CPU Privilegde time (%)</td><td align="right">92.49</td></tr></tbody></table><br></div><div style="font-size:12.8px">4k random read</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><table border="0" cellpadding="0" cellspacing="0" width="294" style="border-collapse:collapse;width:220pt"><colgroup><col width="187" style="width:140pt"><col width="107" style="width:80pt"></colgroup><tbody><tr height="20" style="height:15pt"><td height="20" width="187" style="height:15pt;width:140pt"></td><td width="107" style="width:80pt">4k random read</td></tr><tr height="20" style="height:15pt"><td height="20" style="height:15pt">IOPS</td><td align="right">16718.93</td></tr><tr height="20" style="height:15pt"><td height="20" style="height:15pt">Avg. response time (ms)</td><td align="right">0.9</td></tr><tr height="20" style="height:15pt"><td height="20" style="height:15pt">CPU utilization total (%)</td><td align="right">79.2</td></tr><tr height="20" style="height:15pt"><td height="20" style="height:15pt">CPU Privilegde time (%)</td><td align="right">75.43</td></tr></tbody></table></div></div><div class="gmail_extra"><br></div><div class="gmail_extra">The snapshot of top -H while running 4k random write</div><div class="gmail_extra"><br></div><div class="gmail_extra"><div class="gmail_extra"> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND</div><div>23850 qemu 20 0 10.294g 8.203g 11788 R 92.7 6.5 5:42.01 qemu-kvm<br></div></div><div class="gmail_extra"><div class="gmail_extra">24116 qemu 20 0 10.294g 8.203g 11788 S 34.2 6.5 1:27.91 qemu-kvm</div><div class="gmail_extra">26948 qemu 20 0 10.294g 8.203g 11788 R 33.6 6.5 1:26.85 qemu-kvm</div><div class="gmail_extra">24115 qemu 20 0 10.294g 8.203g 11788 S 32.9 6.5 1:27.72 qemu-kvm</div><div class="gmail_extra">26937 qemu 20 0 10.294g 8.203g 11788 S 32.9 6.5 1:27.87 qemu-kvm</div><div class="gmail_extra">27050 qemu 20 0 10.294g 8.203g 11788 R 32.9 6.5 1:17.14 qemu-kvm</div><div class="gmail_extra">27033 qemu 20 0 10.294g 8.203g 11788 S 31.6 6.5 1:19.40 qemu-kvm</div><div class="gmail_extra">24119 qemu 20 0 10.294g 8.203g 11788 S 26.6 6.5 1:32.16 qemu-kvm</div><div class="gmail_extra">24120 qemu 20 0 10.294g 8.203g 11788 S 25.9 6.5 1:32.02 qemu-kvm</div><div class="gmail_extra">23880 qemu 20 0 10.294g 8.203g 11788 S 8.3 6.5 2:31.11 qemu-kvm</div><div class="gmail_extra">23881 qemu 20 0 10.294g 8.203g 11788 S 8.0 6.5 2:58.75 qemu-kvm</div><div class="gmail_extra">23878 qemu 20 0 10.294g 8.203g 11788 S 7.6 6.5 2:04.15 qemu-kvm</div><div class="gmail_extra">23879 qemu 20 0 10.294g 8.203g 11788 S 7.6 6.5 2:36.50 qemu-kvm</div><div><br></div></div><div class="gmail_extra">Thanks.</div><div class="gmail_extra"><br></div><div class="gmail_extra">Cw</div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Apr 21, 2016 at 10:12 PM, Vijay Bellur <span dir="ltr"><<a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class=""><div class="h5">On Wed, Apr 20, 2016 at 4:17 AM, qingwei wei <<a href="mailto:tchengwee@gmail.com">tchengwee@gmail.com</a>> wrote:<br>
> Gluster volume configuration, those bold entries are the initial settings i<br>
> have<br>
><br>
> Volume Name: g37test<br>
> Type: Stripe<br>
> Volume ID: 3f9dae3d-08f9-4321-aeac-67f44c7eb1ac<br>
> Status: Created<br>
> Number of Bricks: 1 x 10 = 10<br>
> Transport-type: tcp<br>
> Bricks:<br>
> Brick1: 192.168.123.4:/mnt/sdb_mssd/data<br>
> Brick2: 192.168.123.4:/mnt/sdc_mssd/data<br>
> Brick3: 192.168.123.4:/mnt/sdd_mssd/data<br>
> Brick4: 192.168.123.4:/mnt/sde_mssd/data<br>
> Brick5: 192.168.123.4:/mnt/sdf_mssd/data<br>
> Brick6: 192.168.123.4:/mnt/sdg_mssd/data<br>
> Brick7: 192.168.123.4:/mnt/sdh_mssd/data<br>
> Brick8: 192.168.123.4:/mnt/sdj_mssd/data<br>
> Brick9: 192.168.123.4:/mnt/sdm_mssd/data<br>
> Brick10: 192.168.123.4:/mnt/sdn_mssd/data<br>
> Options Reconfigured:<br>
> server.allow-insecure: on<br>
> storage.owner-uid: 165<br>
> storage.owner-gid: 165<br>
> performance.quick-read: off<br>
> performance.io-cache: off<br>
> performance.read-ahead: off<br>
> performance.stat-prefetch: off<br>
> cluster.eager-lock: enable<br>
> network.remote-dio: enable<br>
> cluster.quorum-type: auto<br>
> cluster.server-quorum-type: server<br>
> nfs.disable: true<br>
><br>
<br>
</div></div>I notice that you are using a stripe volume. Would it be possible to<br>
test with a sharded volume? We will be focusing only on sharded<br>
volumes for VM disks going forward.<br>
<br>
Thanks,<br>
Vijay<br>
</blockquote></div><br></div></div>