Now, there are quite a few ways around this problem. You can install to a ‘real’ computer and migrate the installation to a network block device. You can use debootstrap in place of the actual Debian installation system. You can use a combination of these two methods, NFS root filesystems, TFTP hacks, etc. All of these solutions are lacking in my opinion. I want to run the ‘real’ debian installer against a network block device and boot my physical hardware using only the built in PXE booting capability of the BIOS.

Taking all these issues as a personal challenge, I’ve outlined below how to go about using the regular old Debian Lenny installer directly against an AoE block device.

First of all, youll need the full installer disc 1 for Lenny. We need one of the .deb’s thats on that CD. The netinst or businesscard installers will not work with the steps below. (NOTE: If you want to be really creative, I suppose you could download the .deb from within the installer system via wget or even get it before hand and put it on a thumb drive or something…)

1. Boot the lenny install cd just to the point that /cdrom is mounted. For me, that was the “Configure the network” prompt.

2. Switch to a virtual terminal by hitting alt-f2.

3. Unpack the .deb that contains the aoe kernel module

udpkg --unpack /cdrom/pool/main/l/linux-2.6/linux-image-2.6.26-2-486_2.6.26-15_i386.deb

4. Now load the kernel module we just unpacked

insmod /lib/modules/2.6.26-2-486/kernel/drivers/block/aoe/aoe.ko

5. Now we create a symlink to the AoE block device. Its important that we use a symlink at this point in the process since it will cause the installer to set up a lot of paths in the correct fashion.

ln -s /dev/etherd/eX.X /dev/sda

6. Hit alt-f1 and continue with installation until the point that grub fails to install, then switch back to terminal 2.

7. Now chroot into the mostly installed system

chroot /target

8. Install the aoetools package

apt-get -y install aoetools

9. Listed below are two scripts that youll need to get into your chroot environment. They are used to build a new custom initrd image. I find it easiest to put them on a webserver somewhere and wget them.

wget http://www.pixelchaos.net/aoe/hooks/aoetools -O /etc/initramfs-tools/hooks/aoetools
chmod 755 /etc/initramfs-tools/hooks/aoetools
wget http://www.pixelchaos.net/aoe/scripts/local-top/aoetools -O /etc/initramfs-tools/scripts/local-top/aoetools
chmod 755 /etc/initramfs-tools/scripts/local-top/aoetools

10. Now we regenerate the initrd image

update-initramfs -u -k all

11. Now that all the above is done, we can manually install grub – which the debian installer was unable to do.

apt-get -y install grub
mkdir /boot/grub
cp -r /usr/lib/grub/i386-pc/. /boot/grub

12. Now we can remove our old symlink and create some hard links instead. This is required to allow grub to modify the MBR of our block device.

rm /dev/sda
ln /dev/etherd/eX.X /dev/sda
ln /dev/etherd/eX.Xp1 /dev/sda1
ln /dev/etherd/eX.Xp5 /dev/sda5

13. Now update grubs device.map

echo "(hd0) /dev/sda" > /boot/grub/device.map

14. Now run “grub” and issue the following commands

device (hd0) /dev/sda
root (hd0,0)
setup (hd0)

15. Now issue the “update-grub” command which will generate /boot/grub/menu.lst for the first time

16. Remember that grub has been using the hard link to /dev/sda this whole time. Change the root=/dev/sda1 line in /boot/grub/menu.lst to root=/dev/etherd/eX.Xp1, then issue the “update-grub” command again to commit the change to the boot stanzas.

17. For the sake of completeness, lets change grubs device.map back to its original contents.

echo "(hd0) /dev/etherd/eX.X" > /boot/grub/device.map

18. Get out of the chroot environment by issuing the “exit” command.

19. Go back to installer and choose “Continue without boot loader” to finish the installation and reboot the computer.

Thats it! Assuming you have properly set up a gPXE boot environment everything should “just work”. Setting up gPXE is outside the scope of this document but some good notes are available on the etherboot web site (see the links below).

Please note that there is one caveat to this whole process. Grub does not support installation onto AoE block devices. There very well may be some issues the next time you try to upgrade your kernel if some post-inst script calls grub-install against your root device. Keep that in mind.

aoe-hooks script

#!/bin/sh

set  -e

PREREQ=""

prereqs()
{
echo "$PREREQ"
}

case $1 in
prereqs)
prereqs
exit 0
;;
esac

. /usr/share/initramfs-tools/hook-functions

[ -x /sbin/aoe-discover ] && copy_exec /sbin/aoe-discover /sbin
manual_add_modules aoe

aoe-local-top script

#!/bin/sh

set -e

PREREQ="udev"

prereqs()
{
echo "$PREREQ"
}

case $1 in
prereqs)
prereqs
exit 0
;;
esac

case $ROOT in
/dev/etherd/e*)
INTERFACES=`sed -ne '/eth.*:/{s/:.*$//;p;}' < /proc/net/dev`
for i in $INTERFACES; do
echo Bringing up interface $i for AoE
ifconfig $i up
done
# Make sure udev has processed all events from adding the NIC
# modules before loading aoe
[ -x /sbin/udevsettle ] && /sbin/udevsettle --timeout=30
modprobe aoe
# Wait until aoe device files have been generated.
[ -x /sbin/udevsettle ] && /sbin/udevsettle --timeout=30
aoe-discover
;;
esac

references
http://maht0×0r.blogspot.com/2009/03/installing-debian-5-for-migration-to.html
http://www.etherboot.org/wiki/sanboot/debian_and_ubuntu
http://mike.neir.org/wiki/articles/AoE_Root_Fedora

In keeping with my character, the first thing I did was start all over again from scratch by reinstalling the operating system. This time around I set up /dev/md0 as /boot (255 MB) and /dev/md1 as an LVM physical volume (the remainder of the disk), within which /, /home, /usr and friends reside as logical volumes. Its something I’ve wanted to start doing with all my systems for a long time now and shouldn’t have any bearing on the performance tests we are about to do.

Regardless of which protocol will be used we need to enable jumbo frames on all the involved devices. For my setup that means the target (stor01), the initiator (node02), and the switch (a Cisco Catalyst 2970).

First, we turn on jumbo frames for gigabit ethernet at the switch. Beware that this requires a reset (aka reboot) of the switch to take effect:


c2970# system mtu jumbo 9000


Now we enable an MTU of 9000 on both the target and the initiator:


root@stor01:~# ifconfig bond0 mtu 9000



root@node02:~# ifconfig eth0 mtu 9000


For the sake of comparison, here is an iperf test done between the target and initiator with the standard MTU of 1500, and then with an MTU of 9000:


root@stor01:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 1.00 MByte (default)
------------------------------------------------------------
[ 4] local 65.171.150.4 port 5001 connected with 65.171.150.161 port 58731
[ 4] 0.0-10.0 sec 780 MBytes 654 Mbits/sec
[ 5] local 65.171.150.4 port 5001 connected with 65.171.150.161 port 58732
[ 5] 0.0-10.0 sec 916 MBytes 768 Mbits/sec


As you can see, just enabling jumbo frames produces a raw throughput increase of 17.43%. Nothing to sneeze at.

At this point I tried enabling flow control on the catalyst switch (it is already enabled for both send and receive by default in the e1000 driver) but it did not have any effect on iperf numbers. I turned it back off for now.

So now we set up a 20GB LVM volume on the target and export it using vblade to be mounted on the initiator. We then run a simple dd test to check throughput:


root@node02:~# dd if=/dev/zero of=/mnt/test oflag=direct bs=4M
419+0 records in
419+0 records out
1757413376 bytes (1.8 GB) copied, 133.476 seconds, 13.2 MB/s


CPU load on the target was 10-15% during the dd operation. Now we try writing direct to the (unmounted) block device to rule out any performance penalties of the filesystem itself…


root@node02:~# dd if=/dev/zero of=/dev/etherd/e0.1 oflag=direct bs=4M
513+0 records in
512+0 records out
2147483648 bytes (2.1 GB) copied, 170.991 seconds, 12.6 MB/s


CPU usage was slightly higher in that test, running 15-20%. So some slight difference but nothing to be too concerned about.

Now we take that same LVM device and share it via iSCSI for the same dd tests:


ladmin@node02:~$ dd if=/dev/zero of=/mnt/test oflag=direct bs=4M
462+0 records in
461+0 records out
1933574144 bytes (1.9 GB) copied, 38.2375 seconds, 50.6 MB/s


CPU load was 6-8% during that test. We also run that same test with flow control enabled at the switch:


ladmin@node02:~$ dd if=/dev/zero of=/mnt/test oflag=direct bs=4M
463+0 records in
462+0 records out
1937768448 bytes (1.9 GB) copied, 38.1851 seconds, 50.7 MB/s


Essentially the same…

Now this raises the question of why AoE is so much slower than iSCSI on an essentially default install of Debian Etch. To AoE’s credit, many people report getting just as good (50MB/s or better) of performance from AoE on their systems as I’m seeing with iSCSI. I spent quite a large amount of time playing with flow control, kernel ring buffer values, filesystem options, etc. and was unable to determine why performance is so terrible for me. I did find a pretty high number (half a dozen at least) of recent posts to the AoE mailing list by other people having essentially identical problems so I’m certainly not alone. In the interest of completing my testing, I’ve decided to move forward with iSCSI.

Now we try reformatting with the stride option to mkfs:


mkfs.ext3 -E stride=16


The results of several more tests are shown here…


root@node02:~# dd if=/dev/zero of=/mnt/test oflag=direct bs=4M
2038431744 bytes (2.0 GB) copied, 41.8938 seconds, 48.7 MB/s
2038431744 bytes (2.0 GB) copied, 42.2633 seconds, 48.2 MB/s
2038431744 bytes (2.0 GB) copied, 41.3756 seconds, 49.3 MB/s


So we don’t see any appreciable difference when using a combination of the stride= option and flow control, at least with a simple dd test.

Next we turn flow control back off, and reformat again without the stride= option. We are now back to our baseline setup for a new test with bonnie++.


ladmin@node02:~$ /usr/sbin/bonnie++ -d /mnt -s 4096Mb -n 10 -x 5 -q


This test produced block writes of about 82MB/s and block reads of about 37MB/s. The cause for the difference in write speed between the dd and bonnie++ tests is still unclear to me. There also appears to be a known issue where writes are much faster than reads which is apparently due to interrupt handling. This is further evidenced by running a quick dd test that does a read instead of a write:


ladmin@node02:~$ dd if=/mnt/test of=/dev/null bs=4M
3602907136 bytes (3.6 GB) copied, 96.2053 seconds, 37.5 MB/s
3602907136 bytes (3.6 GB) copied, 90.5524 seconds, 39.8 MB/s
3602907136 bytes (3.6 GB) copied, 90.0425 seconds, 40.0 MB/s
3602907136 bytes (3.6 GB) copied, 88.5708 seconds, 40.7 MB/s


As you can see, read operations are about 20% slower than write operations which goes against common thinking with regard to stripped disk arrays.

So there you have it. In my particular situation, with no tuning/optimizing done, iSCSI performs much better than AoE. Even in the event that I were to go to the trouble to performance tune AoE and get it as good as, or even better than, iSCSI I would still be inclined to standardize around iSCSI. Authentication, routing, NAT, etc. can all be done very easily on iSCSI protocol with all the standard TCP/IP tools that are out there. For me that’s a pretty big advantage.

Next up will be our final piece of the puzzle, left over from the initial system setup – getting hot swap working with the sata_mv module!

The Debian Etch installer supports building and installing onto software RAID arrays. Because of that…

during installation I configured the initial RAID1 boot volume with hot spare, consisting of three WD 160GB SATA 3Gbps disks. mdadm sees / and swap as /dev/md0 and /dev/md1 respectively. There was some remaining space on the drives which I set up as /dev/md2 for future use. The remaining arrays I decided to create manually using mdadm after getting a usable system up and running.

First of all we need figure out what devices we want in the arrays by probing /dev. The ultimate goal here is to build three arrays: 1 x RAID1 with 1 hot spare (8GB root, 1GB swap, 151GB extra), 1 x RAID6 with 1 hot spare (4TB Xen LVM’s + 2TB Bacula LVM), 1 x RAID5 (2TB offsite mirror).

I was forward thinking enough to label all the drive carriers with the serial number of the disk in it so all we need to do is get the disk-by-id name from /dev/disk/by-id/ and then build our test array like so:


mdadm --create /dev/md3 --level=6 --raid-devices=8 \
/dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ09HZR \
/dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ09JV3 \
/dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ06913 \
/dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ08XHB \
/dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ09KNM \
/dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ09JHQ \
/dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ0817T \
/dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ09B9P \
--spare-devices=1 \
/dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ07CF3


This assembles our /dev/md3 device with the default options for RAID6 (64k chunk, left-symmetric parity)


stor01:~# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md3 : active raid6 sdl[8](S) sdk[7] sdj[6] sdf[5] sdh[4] sdg[3] sdi[2] sde[1] sdd[0]
5860574976 blocks level 6, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]
[>....................] resync = 0.8% (8790864/976762496) finish=217.3min speed=74238K/sec


Here you can see that CPU load is moderate doing the first full sync of that array:


Cpu0 : 0.0%us, 33.3%sy, 0.0%ni, 63.3%id, 0.0%wa, 1.3%hi, 2.0%si, 0.0%st
Cpu1 : 0.0%us, 7.0%sy, 0.0%ni, 91.0%id, 0.0%wa, 0.3%hi, 1.7%si, 0.0%st


So, we wait about 4 hours… and its done :-)

For formatting the volume, I decided to use the stride= option to mkfs.ext3. This provides for optimal striping across a raid array. The secret here is to make $stride = $chunks / $block_size. In our case, thats 4096 byte (4k) blocks divided by 64k chunks. So 16 would be our optimal stride value.


root@stor01:~# mkfs.ext3 -E stride=16 /dev/md3


So how does it perform? Well enough to saturate the dual gigabit NIC ports – which is all that matters :-)


root@stor01:~# dd bs=4M if=/dev/zero of=/dev/md3
4469+0 records in
4469+0 records out
18744344576 bytes (19 GB) copied, 97.583 seconds, 192 MB/s


I also ran a more real-world test with bonnie++


/usr/sbin/bonnie++ -d /mnt -s 4096Mb -n 100 -x 10 -q


This test showed approximately 335MB/s read, 170MB/s write. You can download the actual data here if you wish.

For failure testing the array, its as simple as removing and inserting disks while the array is up and running. Both tests work fine for the onboard SATA II controller but alas, the sata_mv kernel module does not yet support hotplug so all we can do on the remaining drives is simulate a drive failure by removing one disk. This does work fine but I need to see if there is a way to refresh the SATA bus to get any replacement drive to show up and be added back into the running array. Otherwise, we will need to power down the array to replace a faulty disk which kind of ruins the whole project, dont you think? ;-)

So while we wait for sata_mv to start working (or find a different SATA controller to use in this project) we will move on to the remaining issues. Up next; getting port trunking working with a cisco switch…

First of all, hotplug. I have discovered that…

hotplug doesn’t seem to work on the Supermicro AOC-SAT2-MV8 cards. The controllers do work, but unless a drive is plugged in during bootup they will not be subsequently detected…


root@stor01:~# lspci
05:01.0 SCSI storage controller: Marvell Technology Group Ltd. MV88SX6081 8-port SATA II PCI-X Controller (rev 09)
05:02.0 SCSI storage controller: Marvell Technology Group Ltd. MV88SX6081 8-port SATA II PCI-X Controller (rev 09)


You can see that the Linux kernel sees these cards as Marvell MV88SX6081 devices. Further probing reveals that the sata_mv driver is claiming them:


root@stor01:~$ dmesg | grep 05:01.0
sata_mv 0000:05:01.0: version 0.7
ACPI: PCI Interrupt 0000:05:01.0[A] -> GSI 24 (level, low) -> IRQ 233
sata_mv 0000:05:01.0: 32 slots 8 ports SCSI mode IRQ via INTx


So off we go lkml.org and other sources to find out why… Basically, hotplug support in the sata_mv module is still a work in progress. A post to the lkml claims that its slated for inclusion in the 2.6.26 kernel. There is also more information here.

So hotplug isnt going to work for us today :-( On the bright side, 2.6.25-rc7 is the current development kernel so I can’t imagine that 2.6.26 is too far off.

The second (perceived) issue of getting all the hot swap bays to be mapped to /dev/sd* entries in a particular order doesnt seem to be as much of an issue as I’d thought.

The Linux kernel will initialize PCI devices in a strange fashion depending on what is plugged in to the bus. For example, if all fifteen bays have drives plugged into them, the leftmost bay always shows up at /dev/sda. Unplug just one drive and reboot the machine and it will be mapped to /dev/sdl for example. The reason this isnt really an issue is that newer Linux kernels have not only udev (which is great but not really a solution for this particular problem) but the /dev/disk/by-* tree. You can find all your disks in there with handy names like /dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ06913. That name belongs to that particular drive and only that particular drive, ever.


root@stor01:~# ls -l /dev/disk/by-id/ | grep scsi | grep -v part
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ06913 -> ../../sde
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ07CF3 -> ../../sdi
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ0817T -> ../../sdb
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ08T21 -> ../../sdh
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ08XHB -> ../../sdc
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09B9P -> ../../sda
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09GMW -> ../../sdd
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09HZR -> ../../sdg
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09JHQ -> ../../sdk
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09JV3 -> ../../sdj
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09KNM -> ../../sdl
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09LLA -> ../../sdf
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_WDC_WD1600AAJS-_WD-WCAP93952468 -> ../../sdm
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_WDC_WD1600AAJS-_WD-WMAS20341446 -> ../../sdo
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_WDC_WD1600AAJS-_WD-WMAS20366839 -> ../../sdn


Cool, huh?

So we will just let mdadm do its job of assembling /dev/md* devices on the fly at bootup and use the /dev/disk/by-id entries to do stuff to particular disks (like temperature monitoring, etc.).

Coming up in my next post – performance and failure testing the array.

There happens to be a company (Coraid) that makes a turnkey AoE device. Its far cheaper than a true fibre channel SAN or something similar. Perfect for setting up a SAN over Ethernet device that can serve Xen domU filesystems out to “thin” dom0’s on the network.

Well that’s all well and good but you see I’m always looking to save a buck…

So I asked myself; why not build my own “coraid” with off the shelf parts and save 50% in the process? Herein I’ll do my best to chronicle my adventures in getting this thing built, tested, and (hopefully) deployed.

First of all the hardware. Thanks to the ever-brilliant Mike Neir for helping me figure out what Coraid uses to build these suckers. Here are the parts I ended up ordering:

• 2 x Supermicro 8-Port SATA Card (AOC-SAT2-MV8)
• 1 x Supermicro Xeon Dual-Core Blackford VS ServerBoard (X7DVL-E)
• 1 x Supermicro Black 3U Rackmount Case 760W (SC933T-R760B)
• 1 x Kingston 2GB 240-Pin DDR2 FB-DIMM ECC (KVR667D2D8F5K2/2G)
• 1 x Intel Xeon 5130 Woodcrest 2.0GHz 4M (BX805565130A)
• 15 x Seagate Barracuda ES.2 1TB (ST31000340NS)
• 3 x Supermicro Drive Carrier (CSE-PT39B)
• 3 x Western Digital 7K 8M SATA2 160GB (WD1600AAJS)

Now you may ask, why buy 18 drives for a chassis that has 15 hot swap bays? Well, I’ll be using the three WD drives as a RAID1 boot device with one hot spare, then setting nine of the 1TB drives up into a RAID6 device with one hot spare. The remaining three drives will be in RAID5 and swapped with the last three drives on a regular basis to go off site as part of a hard disk based backup system.

Now that we have all of that out of the way; how did the assembly go? Flawless almost ;-)

There are two gotcha’s to be dealt with in the assembly of the aforementioned motherboard and server chassis; first of all, the EPS12V connector for CPU power that comes out of the power distribution block wont reach its molex connector on the motherboard. Second, the ribbon cable that connects the front panel controls to the their motherboard header just *barely* reaches which makes me uncomfortable.

To resolve the first problem I ordered up a StarTech 8″ EPS 8 Pin Power Extension Cable (EPS8EXT) which did the job nicely. To fix the second issue, I ordered twenty feet of 16 conductor ribbon cable (might need extra!), some cable ends and a crimper. Problem solved.

With all that taken care of its time to get an operating system installed and test this thing out. I did go ahead and power up the system and test the redundant PSU’s and fans. The alarm features of both seem to work fine with audible alarms when any of them are removed from the system. After that I plugged an old CD-ROM into the ATA header on the mobo and installed a rockin Linux distro.

Some things I discovered:

• The BIOS and the Linux kernel both initialize the two SATA PCI cards first. Some fiddling will need to be done to make the three WD drives (plugged into the motherboard SATA headers) show up as sda, sdb, and sdc.
• You must turn off “compatibility mode” in the BIOS for the onboard SATA controllers or hotplug will not work for drives on those headers. After turning it off, hotplug works just dandy. Yay!
• Hotplug isnt working for the two PCI SATA cards at this point. I suspect its something to do with the sata_mv kernel module but havent gotten to look into it much. That will be the first order of business when next I work on this project.

So thats it for now. Things are coming together nicely. My next post will detail getting hotplug working for the two PCI SATA cards and getting device detection and mapping working correctly in the Linux kernel (I smell some udev tomfoolery most likely).