pixelchaos.net

With the forthcoming release of Debian Lenny it happened to be a good time for me to re-evaluate my decision to use one monster RAID6 device to back all my iSCSI targets. I ran a semi-formal test on different disk configurations for software raid and came up with these results…

Our test system, for those who dont follow this blog regularly, is a Supermicro X7DVL-E motherboard with a single Intel Core2 Xeon 5130 @ 2Ghz with 2GB of RAM. The disks used are Seagate ES.2 1TB SATA-II models hooked up to two Supermicro AOC-SAT2-MV8 8-port PCI-X SATA controllers. More complete specs are outlined in my original post where I was building the chassis, which can be found here.

The Linux distribution used is, of course, Debian :-) In this case, its Debian 5.0 (Lenny) running kernel 2.6.26-1-686.

Test #1 – Individual Disks
In order to get some baseline numbers to compare against subsequent tests we will first check performance of all the attached disks individually…
for disk in $(ls /dev/disk/by-id/ | grep -v part | grep scsi-SATA_ST); do
echo $disk && \
dd if=/dev/zero of=/dev/disk/by-id/$disk bs=4M oflag=direct count=250 && \
dd if=/dev/disk/by-id/$disk of=/dev/null bs=4M count=250;
done

Which yielded these results…

5QJ06913 - read:109MB/s, write:109MB/s
5QJ09HZR - read:111MB/s, write:112MB/s
5QJ09JHQ - read:113MB/s, write:109MB/s
5QJ09JV3 - read:109MB/s, write:108MB/s


So our average raw device read speed is 110.5MB/s and writes are 109.5MB/s.

Test #2 – Bonnie++ on a single disk
Next we will do a bonnie++ test on a single disk (5QJ06913), formatted with default ext3 options

mkfs.ext3 /dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ06913
mount /dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ06913 /mnt
/usr/sbin/bonnie++ -q -d /mnt -s 4096 -x 5 -u root

This test produced average block reads of 105.55MB/s @ 11.4% CPU usage, block writes of 65.93MB/s @ 19%, and block rewrites of 36.95MB/s @ 8%.

Test #3 – RAID1 pair
For this test we use a RAID1 pair with one disk on each controller, formatted with default ext3 options

mdadm --create /dev/md2 --level=1 --raid-devices=2 /dev/sdf /dev/sdg
dd if=/dev/zero of=/dev/md2 bs=4M oflag=direct count=250
dd if=/dev/md2 of=/dev/null bs=4M count=250


This produced raw I/O reads of 111MB/s and writes of 107MB/s. Now for some bonnie++ tests…


mkfs.ext3 /dev/md2
mount /dev/md2 /mnt
/usr/sbin/bonnie++ -q -d /mnt -s 4096 -x 5 -u root


This test produced average block reads of 103.43MB/s @ 14.4% CPU usage, block writes of 60.84MB/s @ 18.6%, and block rewrites of 33.82MB/s @ 8.8%.

Test #4 – RAID10
For this test we use a RAID10 device consisting of two RAID1 devices, each with a disk on each controller.


mdadm --create /dev/md4 --level=0 --raid-devices=2 /dev/md2 /dev/md3
dd if=/dev/zero of=/dev/md4 bs=4M oflag=direct count=250
dd if=/dev/md4 of=/dev/null bs=4M


This produced raw I/O reads of 219MB/s and writes of 207MB/s. A pretty significant improvement.


mkfs.ext3 /dev/md4
mount /dev/md4 /mnt
/usr/sbin/bonnie++ -q -d /mnt -s 4096 -x 5 -u root


This test produced average block reads of 198.91MB/s @ 27.8% CPU usage, block writes of 132.4MB/s @ 44%, and block rewrites of 61.75MB/s @ 14.4%.

Test #5 – RAID5
For this test we will use a RAID5 device with the first two disks on one controller and the third on the second controller


mdadm --create /dev/md2 --level=5 --raid-devices=3 /dev/sdf1 /dev/sde1 /dev/sdg1
dd if=/dev/zero of=/dev/md2 bs=4M oflag=direct count=250
dd if=/dev/md2 of=/dev/null bs=4M


This produced raw I/O reads of 67.3MB/s and writes of 130MB/s.


mkfs.ext3 /dev/md2
mount /dev/md2 /mnt
/usr/sbin/bonnie++ -q -d /mnt -s 4096 -x 5 -u root


This test produced average block reads of 62.91MB/s @ 8.6% CPU usage, block writes of 65.73MB/s @ 19.6%, and block rewrites of 33.14MB/s @ 7.6%.

Those were all the RAID types I was inclined to test. I realize I left out RAID0 and RAID6. For the application I have in mind, RAID10 is the clear winner. You can download a spreadsheet with the actual bonnie++ output here. Enjoy!

This entry was posted on Friday, September 12th, 2008 at 1:39 pm and is filed under Debian, Hardware, Linux. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.