Performance testing Linux software RAID
With the forthcoming release of Debian Lenny it happened to be a good time for me to re-evaluate my decision to use one monster RAID6 device to back all my iSCSI targets. I ran a semi-formal test on different disk configurations for software raid and came up with these results…
Our test system, for those who dont follow this blog regularly, is a Supermicro X7DVL-E motherboard with a single Intel Core2 Xeon 5130 @ 2Ghz with 2GB of RAM. The disks used are Seagate ES.2 1TB SATA-II models hooked up to two Supermicro AOC-SAT2-MV8 8-port PCI-X SATA controllers. More complete specs are outlined in my original post where I was building the chassis, which can be found here.
The Linux distribution used is, of course, Debian :-) In this case, its Debian 5.0 (Lenny) running kernel 2.6.26-1-686.
Test #1 - Individual Disks
In order to get some baseline numbers to compare against subsequent tests we will first check performance of all the attached disks individually…
echo $disk && \
dd if=/dev/zero of=/dev/disk/by-id/$disk bs=4M oflag=direct count=250 && \
dd if=/dev/disk/by-id/$disk of=/dev/null bs=4M count=250;
done
Which yielded these results…
5QJ09HZR - read:111MB/s, write:112MB/s
5QJ09JHQ - read:113MB/s, write:109MB/s
5QJ09JV3 - read:109MB/s, write:108MB/s
So our average raw device read speed is 110.5MB/s and writes are 109.5MB/s.
Test #2 - Bonnie++ on a single disk
Next we will do a bonnie++ test on a single disk (5QJ06913), formatted with default ext3 options
mount /dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ06913 /mnt
/usr/sbin/bonnie++ -q -d /mnt -s 4096 -x 5 -u root
This test produced average block reads of 105.55MB/s @ 11.4% CPU usage, block writes of 65.93MB/s @ 19%, and block rewrites of 36.95MB/s @ 8%.
Test #3 - RAID1 pair
For this test we use a RAID1 pair with one disk on each controller, formatted with default ext3 options
dd if=/dev/zero of=/dev/md2 bs=4M oflag=direct count=250
dd if=/dev/md2 of=/dev/null bs=4M count=250
This produced raw I/O reads of 111MB/s and writes of 107MB/s. Now for some bonnie++ tests…
mount /dev/md2 /mnt
/usr/sbin/bonnie++ -q -d /mnt -s 4096 -x 5 -u root
This test produced average block reads of 103.43MB/s @ 14.4% CPU usage, block writes of 60.84MB/s @ 18.6%, and block rewrites of 33.82MB/s @ 8.8%.
Test #4 - RAID10
For this test we use a RAID10 device consisting of two RAID1 devices, each with a disk on each controller.
dd if=/dev/zero of=/dev/md4 bs=4M oflag=direct count=250
dd if=/dev/md4 of=/dev/null bs=4M
This produced raw I/O reads of 219MB/s and writes of 207MB/s. A pretty significant improvement.
mount /dev/md4 /mnt
/usr/sbin/bonnie++ -q -d /mnt -s 4096 -x 5 -u root
This test produced average block reads of 198.91MB/s @ 27.8% CPU usage, block writes of 132.4MB/s @ 44%, and block rewrites of 61.75MB/s @ 14.4%.
Test #5 - RAID5
For this test we will use a RAID5 device with the first two disks on one controller and the third on the second controller
dd if=/dev/zero of=/dev/md2 bs=4M oflag=direct count=250
dd if=/dev/md2 of=/dev/null bs=4M
This produced raw I/O reads of 67.3MB/s and writes of 130MB/s.
mount /dev/md2 /mnt
/usr/sbin/bonnie++ -q -d /mnt -s 4096 -x 5 -u root
This test produced average block reads of 62.91MB/s @ 8.6% CPU usage, block writes of 65.73MB/s @ 19.6%, and block rewrites of 33.14MB/s @ 7.6%.
Those were all the RAID types I was inclined to test. I realize I left out RAID0 and RAID6. For the application I have in mind, RAID10 is the clear winner. You can download a spreadsheet with the actual bonnie++ output here. Enjoy!
January 3rd, 2009 at 5:13 am
You could also test the raid10 kernel and not a raid 1+0 which you use.
man mdam -l10 would give you some informations.
January 4th, 2009 at 9:24 am
Friedmann:
Hmm interesting. I’ll have to check that out. Thanks!