GlusterFS 1.3.1 - 64 Bricks Aggregated I/O Benchmark

From GlusterDocumentation

Jump to: navigation, search

Contents

[edit]

Benchmark Brief

GlusterFS Aggregated I/O Benchmark was performed on 64 bricks clustered storage system over 10 Gbps Infiniband interconnect. A cluster of 220 clients pounded the storage system with multiple dd (disk-dump) instances, each reading / writing a 1 GB file with 1MB block size. GlusterFS was configured with unify translator and round-robin scheduler.

Thanks to Intel Corporation for granting us dedicated access to a large compute cluster to perform this benchmark.

[edit]

GlusterFS Brick Config

  • Processor - 2 x dual-core Intel(R) Xeon(R) CPU @ 3.00GHz
  • RAM - 8GB FB-DIMM
  • Linux Kernel - 2.6.9-42.0.10.ELsmp (RHEL 4.4)
  • Disk: SATA 120GB
  • Interconnect: Infiniband 10 Gbps 4x SDR - Mellanox Technologies MT25208 InfiniHost III Ex (rev 20)
  • Disk Filesystem: Ext3
  • Cluster Filesystem: GlusterFS v1.3.1 (unify with round-robin scheduler)
[edit]

Client Config

  • Processor - 2 x dual-core Intel(R) Xeon(R) CPU @ 3.00GHz
  • RAM - 8GB FB-DIMM
  • Linux Kernel - 2.6.9-42.0.10.ELsmp (RHEL 4.4), Fuse 2.6.7-glfs3
  • Disk: SATA 120GB
  • Interconnect: Infiniband 10 Gbps 4x SDR - Mellanox Technologies MT25208 InfiniHost III Ex (rev 20)
[edit]

GlusterFS 64 bricks (with ib-verbs transport)

GlusterFS delivered a whopping 32 GBps of read and 23 GBps of write throughput.

Note: The storage cluster had very slow disks (22MBps hdparm read performance per brick). RAID arrays typically deliver 100s of MBps throughput. Because we had 512 GB of RAM and 80 GBps (10Gbps * 64 / 8) aggregated interconnect bandwidth, we proceeded with the benchmark. GlusterFS will deliver lot more performance on a similar 64-brick configuration with today's SATA or SAS arrays.

aggregate.png

[edit]

GlusterFS performance with 220 clients and varying servers

scale.png

  • With increased number of server, the overall performance increases linearly.
[edit]

Notes to improve performance further

  • Using GlusterFS IO-Cache translator would have created 384 GB of cache over 64 bricks with each dedicating 6 GB out of 8 GB memory. Cache has a dramatic effect on overall storage performance.
  • Using GlusterFS Booster translator would have increased I/O performance through kernel bypass (zero copy)
Retrieved from "http://www.gluster.org/docs/index.php/GlusterFS_1.3.1_-_64_Bricks_Aggregated_I/O_Benchmark"
Personal tools