<div>Hi Lindsay<br></div><div><br></div><div>Just noticed that you have your ZFS logs on a single disk, you like living dangerously ;-) a you should have a mirror for the slog to be on the safe side.<br></div><div><br></div><div>Cheers,<br></div><div>M.<br></div><div class="protonmail_signature_block-empty"><br></div><div class="protonmail_signature_block protonmail_signature_block-empty"><div class="protonmail_signature_block-user protonmail_signature_block-empty"><div><br></div></div><div class="protonmail_signature_block-proton protonmail_signature_block-empty"><br></div></div><div class="protonmail_signature_block-empty"><br></div><div><br></div><div><br></div><blockquote type="cite" class="protonmail_quote"><div>-------- Original Message --------<br></div><div>Subject: Re: [Gluster-users] Improving IOPS<br></div><div>Local Time: November 5, 2016 9:52 AM<br></div><div>UTC Time: November 5, 2016 8:52 AM<br></div><div>From: lindsay.mathieson@gmail.com<br></div><div>To: Darrell Budic <budic@onholyground.com><br></div><div>gluster-users <Gluster-users@gluster.org><br></div><div><br></div><div>On 5/11/2016 1:30 AM, Darrell Budic wrote:<br></div><div>
> What’s your CPU and disk layout for those? You’re close to what I’m running, curious how it compares.<br></div><div>
<br></div><div>
All my nodes are running RAIDZ10. I have SSD 5GB slog partion, 100GB Cache<br></div><div>
<br></div><div>
Cache is hardly used, I think you'll find with VM workload you're only <br></div><div>
getting around 4% hit rates. You're better off using the SSD for slog, <br></div><div>
it improves sync writes consdierably.<br></div><div>
<br></div><div>
I tried the Samsung 850 pro, found them pretty bad in practice. Their <br></div><div>
sustained seq writes were atrocious in production and their lifetime <br></div><div>
very limted. Gluster/VM usage resultys in very high writes, ours all <br></div><div>
packed it in under a year.<br></div><div>
<br></div><div>
We have Kingston Hyper somethings :) they have a TBW of 300TB which is <br></div><div>
much better and their uncompressed write speed is very high.<br></div><div>
<br></div><div>
<br></div><div>
> hat are you doing to benchmark your IO?<br></div><div>
<br></div><div>
bonnie++ on the ZFS pools and Crystal DiskMark in the VM's, plus test <br></div><div>
real world workloads.<br></div><div>
<br></div><div>
<br></div><div>
VNA:<br></div><div>
<br></div><div>
- 2 * Xeon E5-2660 2.2 Ghz<br></div><div>
- 64Gb RAM<br></div><div>
- 2*1G balance-alb (Gluster Network)<br></div><div>
- 1G Public network<br></div><div>
<br></div><div>
pool: tank<br></div><div>
config:<br></div><div>
NAME STATE READ WRITE CKSUM<br></div><div>
tank ONLINE 0 0 0<br></div><div>
mirror-0 ONLINE 0 0 0<br></div><div>
ata-WDC_WD6000HLHX-01JJPV0_WD-WX41E81ZU901 ONLINE <br></div><div>
0 0 0<br></div><div>
ata-WDC_WD6000HLHX-01JJPV0_WD-WX81E81AFWJ4 ONLINE <br></div><div>
0 0 0<br></div><div>
mirror-1 ONLINE 0 0 0<br></div><div>
ata-WDC_WD6000HLHX-01JJPV0_WD-WX41E81ZV240 ONLINE <br></div><div>
0 0 0<br></div><div>
ata-WDC_WD6000HLHX-01JJPV0_WD-WX41E81ZV027 ONLINE <br></div><div>
0 0 0<br></div><div>
mirror-2 ONLINE 0 0 0<br></div><div>
ata-WDC_WD6000HLHX-01JJPV0_WD-WX41E81ZU903 ONLINE <br></div><div>
0 0 0<br></div><div>
ata-WDC_WD6000HLHX-01JJPV0_WD-WXB1E81EFFT2 ONLINE <br></div><div>
0 0 0<br></div><div>
mirror-4 ONLINE 0 0 0<br></div><div>
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N1UFDFKA ONLINE 0 <br></div><div>
0 0<br></div><div>
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N7ZKLK52 ONLINE 0 <br></div><div>
0 0<br></div><div>
logs<br></div><div>
ata-KINGSTON_SHSS37A240G_50026B7267031966-part1 ONLINE <br></div><div>
0 0 0<br></div><div>
cache<br></div><div>
ata-KINGSTON_SHSS37A240G_50026B7267031966-part2 ONLINE <br></div><div>
0 0 0<br></div><div>
<br></div><div>
VNB, VNG<br></div><div>
- Xenon E5-2620 2Ghz<br></div><div>
- 64GB RAM<br></div><div>
- 2*1G balance-alb (Gluster Network)<br></div><div>
- 1G Public network<br></div><div>
pool: tank<br></div><div>
config:<br></div><div>
NAME STATE READ WRITE CKSUM<br></div><div>
tank ONLINE 0 0 0<br></div><div>
mirror-0 ONLINE 0 0 0<br></div><div>
ata-WDC_WD30EFRX-68EUZN0_WD-WMC4N2874892 ONLINE 0 <br></div><div>
0 0<br></div><div>
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N4TKR8C2 ONLINE 0 <br></div><div>
0 0<br></div><div>
mirror-1 ONLINE 0 0 0<br></div><div>
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N4TKR3Y0 ONLINE 0 <br></div><div>
0 0<br></div><div>
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N4TKR84T ONLINE 0 <br></div><div>
0 0<br></div><div>
logs<br></div><div>
ata-KINGSTON_SHSS37A240G_50026B7266074B8A-part1 ONLINE <br></div><div>
0 0 0<br></div><div>
cache<br></div><div>
ata-KINGSTON_SHSS37A240G_50026B7266074B8A-part2 ONLINE <br></div><div>
0 0 0<br></div><div>
<br></div><div>
><br></div><div>
> My prod cluster:<br></div><div>
> 3x E5-2609 @ 1.9G, 6 core, 32G RAM, 2x10G network, parts of 2x samsung 850 pro used for zfs cache, no zil<br></div><div>
> 2x 9 x 1G drives in straight zfs stripe<br></div><div>
> 1x 8 x 2G drives in straight zfs stripe<br></div><div>
><br></div><div>
> I use lz4 compressions on my stores. The underlying storage seems to be capable of ~400MB/s writes and 1-1.5GB/s reads, although the pair of 850s I’m caching on probably max out around 1.2GB/s.<br></div><div>
<br></div><div>
<br></div><div>
-- <br></div><div>
Lindsay Mathieson<br></div><div>
<br></div><div>
_______________________________________________<br></div><div>
Gluster-users mailing list<br></div><div>
Gluster-users@gluster.org<br></div><div>
http://www.gluster.org/mailman/listinfo/gluster-users<br></div></blockquote><div><br></div>