From GlusterDocumentation

GlusterFS 1.3.pre2-VERGACION vs Lustre-1.4.9.1 Various Benchmarks on 10 brick server and 20 clients over Infiniband Verbs transport.

Contents 1 Benchmark Environment 1.1 GlusterFS Brick Config (10 bricks) 1.2 Client Config (20 clients) 1.3 Software Version 2 GlusterFS 1.3.pre2-VERGACION vs Lustre-1.4.9.1 2.1 Directory Listing Benchmark 2.2 File Copy Benchmark 2.2.1 Copy Local to Cluster File System 2.2.2 Copy Local from Cluster File System 2.3 Checksum Benchmark 2.4 Base64 Conversion Benchmark 2.5 Pattern Search Benchmark 2.6 Data Compression Benchmark 2.7 Archival Benchmark 2.7.1 Archive Creation 2.7.2 Archive Extraction 2.8 Aggregated Read Throughput Benchmark 2.9 Aggregated Write Throughput Benchmark 2.10 Apache Web Server Benchmark

Benchmark Environment

Cluster hardware was provided by Mellanox Technologies through their Cluster Center program. Thank you Mellanox!!.

GlusterFS Brick Config (10 bricks)

Processor        : 2 x AMD Dual-Core Opteron™ Model 275 processors
RAM              : 6 GB
Interconnect     : InfiniBand 20 Gb/s - Mellanox MT25208 InfiniHost III Ex
Hard disk        : Western Digital Corp. WD1200JB-00REA0, ATA DISK drive

Client Config (20 clients)

Processor        : 2 x AMD Dual-Core Opteron™ Model 275 processors
RAM              : 6 GB
Interconnect     : InfiniBand 20 Gb/s - Mellanox MT25208 InfiniHost III Ex
Hard disk        : Western Digital Corp. WD1200JB-00REA0, ATA DISK drive

Software Version

Operation System : Redhat Enterprise GNU/Linux 4 (Update 3)
Linux version    : 2.6.9-42
Lustre version   : 1.4.9.1
GlusterFS version: 1.3-pre2.3

GlusterFS 1.3.pre2-VERGACION vs Lustre-1.4.9.1

Directory Listing Benchmark

"find" command navigates across the directory tree structure and prints them to console. In this case, there were thirteen thousand binary files.

Note: Commands are same for both GlusterFS and Lustre, except the directory part.

$ find /mnt/glusterfs

	Time
Lustre	1.7 secs
GlusterFS	1.5 secs

Note: Lower means faster.

File Copy Benchmark

cp utility is used to copy files and directories.

Copy Local to Cluster File System

Copy 12039 files (595 MB) were copied into the cluster file system.

$ cp -r /local/* /mnt/glusterfs/

	Time
Lustre	37 secs
GlusterFS	26 secs

Note: Lower means faster.

Copy Local from Cluster File System

Copy 12039 files (595 MB) were copied from the cluster file system.

$ cp -r  /mnt/glusterfs/ /local/*

	Time
Lustre	45 secs
GlusterFS	18 secs

Note: Lower means faster.

Checksum Benchmark

Perform md5sum calculation for all files across your file system. In this case, there were thirteen thousand binary files.

$ find . -type f -exec md5sum {} \;

	Time
Lustre	45.1 secs
GlusterFS	44.4 secs

Note: Lower means faster.

Base64 Conversion Benchmark

Base64 is an algorithm for encoding binary to ASCII and vice-versa. This benchmark was performed on a 640 MB binary file.

$ base64 --encode big-file big-file.base64

	Time
Lustre	25.7 secs
GlusterFS	25.1 secs

Note: Lower means faster.

Pattern Search Benchmark

grep utility searches for a PATTERN on a file and prints the matching lines to console. This benchmark used 1GB ASCII BASE-64 file.

$ grep GNU big-file.base64

	Time
Lustre	54.3 secs
GlusterFS	52.1 secs

Note: Lower means faster.

Data Compression Benchmark

GNU gzip utility compresses files using Lempel-Ziv coding. This benchmark was performed on 1GB TAR binary file.

$ gzip big-file.tar
$ gunzip big-file.tar.gz

	Compression Time	Decompression Time
Lustre	18.3 secs	14.8 secs
GlusterFS	16.5 secs	10.1 secs

Note: Lower means faster.

Archival Benchmark

tar utility is used for archiving filesystem data

Archive Creation

'tar utility created an archive of 12039 files (595 MB) served through GlusterFS.

$ tar czf benchmark.tar.gz /mnt/glusterfs

	Time
Lustre	41 secs
GlusterFS	25 secs

Note: Lower means faster.

Archive Extraction

'tar utility extracted the archive on to GlusterFS filesystem.

$ tar xzf benchmark.tar.gz 

	Time
Lustre	FAILED
GlusterFS	43 secs

Note: Lower means faster.

Tar extraction failed under Lustre with no space left on the device error. Disk-free (df -h) utility revealed lot of free space. It appears like I/O scheduler performed a poor job in distributing the data evenly.

tar: lib/libtdsS.so.1: Cannot create symlink to `libtdsS.so.1.0.0': No space left on device
tar: lib/libg.a: Cannot open: No space left on device
tar: Error exit delayed from previous errors

Aggregated Read Throughput Benchmark

Multiple dd utility were executed simultaneously with different block sizes to read from GlusterFS filesystem.

	4KB	16KB	128KB	256KB	512KB	1024KB
Lustre	1,796 MB/s	5,782 MB/s	20,423 MB/s	21,582 MB/s	22,789 MB/s	23,731 MB/s
GlusterFS	11,415 MB/s	11,424 MB/s	11,427 MB/s	11,419 MB/s	11,411 MB/s	11,409 MB/s

Note: Higher means faster.

Aggregated Write Throughput Benchmark

Multiple dd utility were executed simultaneously with different block sizes to write to GlusterFS filesystem.

	4KB	16KB	128KB	256KB	512KB	1024KB
Lustre	969 MB/s	1,613 MB/s	1,988 MB/s	1,989 MB/s	1,984 MB/s	1,983 MB/s
GlusterFS	1,886 MB/s	2,191 MB/s	2,237 MB/s	2,231 MB/s	2,236 MB/s	2,223 MB/s

Note: Higher means faster.

Apache Web Server Benchmark

Apache served 12039 files (595 MB) over HTTP protocol. wget client fetched the files recursively.

	Time
Lustre	Failed after downloading 33 MB out of 585 MB in 11 mins.
GlusterFS	3 mins 11 secs

Note: Lower means faster.