Now, there are quite a few ways around this problem. You can install to a ‘real’ computer and migrate the installation to a network block device. You can use debootstrap in place of the actual Debian installation system. You can use a combination of these two methods, NFS root filesystems, TFTP hacks, etc. All of these solutions are lacking in my opinion. I want to run the ‘real’ debian installer against a network block device and boot my physical hardware using only the built in PXE booting capability of the BIOS.

Taking all these issues as a personal challenge, I’ve outlined below how to go about using the regular old Debian Lenny installer directly against an AoE block device.

First of all, youll need the full installer disc 1 for Lenny. We need one of the .deb’s thats on that CD. The netinst or businesscard installers will not work with the steps below. (NOTE: If you want to be really creative, I suppose you could download the .deb from within the installer system via wget or even get it before hand and put it on a thumb drive or something…)

1. Boot the lenny install cd just to the point that /cdrom is mounted. For me, that was the “Configure the network” prompt.

2. Switch to a virtual terminal by hitting alt-f2.

3. Unpack the .deb that contains the aoe kernel module

udpkg --unpack /cdrom/pool/main/l/linux-2.6/linux-image-2.6.26-2-486_2.6.26-15_i386.deb

4. Now load the kernel module we just unpacked

insmod /lib/modules/2.6.26-2-486/kernel/drivers/block/aoe/aoe.ko

5. Now we create a symlink to the AoE block device. Its important that we use a symlink at this point in the process since it will cause the installer to set up a lot of paths in the correct fashion.

ln -s /dev/etherd/eX.X /dev/sda

6. Hit alt-f1 and continue with installation until the point that grub fails to install, then switch back to terminal 2.

7. Now chroot into the mostly installed system

chroot /target

8. Install the aoetools package

apt-get -y install aoetools

9. Listed below are two scripts that youll need to get into your chroot environment. They are used to build a new custom initrd image. I find it easiest to put them on a webserver somewhere and wget them.

wget http://www.pixelchaos.net/aoe/hooks/aoetools -O /etc/initramfs-tools/hooks/aoetools
chmod 755 /etc/initramfs-tools/hooks/aoetools
wget http://www.pixelchaos.net/aoe/scripts/local-top/aoetools -O /etc/initramfs-tools/scripts/local-top/aoetools
chmod 755 /etc/initramfs-tools/scripts/local-top/aoetools

10. Now we regenerate the initrd image

update-initramfs -u -k all

11. Now that all the above is done, we can manually install grub – which the debian installer was unable to do.

apt-get -y install grub
mkdir /boot/grub
cp -r /usr/lib/grub/i386-pc/. /boot/grub

12. Now we can remove our old symlink and create some hard links instead. This is required to allow grub to modify the MBR of our block device.

rm /dev/sda
ln /dev/etherd/eX.X /dev/sda
ln /dev/etherd/eX.Xp1 /dev/sda1
ln /dev/etherd/eX.Xp5 /dev/sda5

13. Now update grubs device.map

echo "(hd0) /dev/sda" > /boot/grub/device.map

14. Now run “grub” and issue the following commands

device (hd0) /dev/sda
root (hd0,0)
setup (hd0)

15. Now issue the “update-grub” command which will generate /boot/grub/menu.lst for the first time

16. Remember that grub has been using the hard link to /dev/sda this whole time. Change the root=/dev/sda1 line in /boot/grub/menu.lst to root=/dev/etherd/eX.Xp1, then issue the “update-grub” command again to commit the change to the boot stanzas.

17. For the sake of completeness, lets change grubs device.map back to its original contents.

echo "(hd0) /dev/etherd/eX.X" > /boot/grub/device.map

18. Get out of the chroot environment by issuing the “exit” command.

19. Go back to installer and choose “Continue without boot loader” to finish the installation and reboot the computer.

Thats it! Assuming you have properly set up a gPXE boot environment everything should “just work”. Setting up gPXE is outside the scope of this document but some good notes are available on the etherboot web site (see the links below).

Please note that there is one caveat to this whole process. Grub does not support installation onto AoE block devices. There very well may be some issues the next time you try to upgrade your kernel if some post-inst script calls grub-install against your root device. Keep that in mind.

aoe-hooks script

#!/bin/sh

set  -e

PREREQ=""

prereqs()
{
echo "$PREREQ"
}

case $1 in
prereqs)
prereqs
exit 0
;;
esac

. /usr/share/initramfs-tools/hook-functions

[ -x /sbin/aoe-discover ] && copy_exec /sbin/aoe-discover /sbin
manual_add_modules aoe

aoe-local-top script

#!/bin/sh

set -e

PREREQ="udev"

prereqs()
{
echo "$PREREQ"
}

case $1 in
prereqs)
prereqs
exit 0
;;
esac

case $ROOT in
/dev/etherd/e*)
INTERFACES=`sed -ne '/eth.*:/{s/:.*$//;p;}' < /proc/net/dev`
for i in $INTERFACES; do
echo Bringing up interface $i for AoE
ifconfig $i up
done
# Make sure udev has processed all events from adding the NIC
# modules before loading aoe
[ -x /sbin/udevsettle ] && /sbin/udevsettle --timeout=30
modprobe aoe
# Wait until aoe device files have been generated.
[ -x /sbin/udevsettle ] && /sbin/udevsettle --timeout=30
aoe-discover
;;
esac

references
http://maht0×0r.blogspot.com/2009/03/installing-debian-5-for-migration-to.html
http://www.etherboot.org/wiki/sanboot/debian_and_ubuntu
http://mike.neir.org/wiki/articles/AoE_Root_Fedora

After some testing I’m happy to say it appears to work smashingly. What follows is a somewhat more complete version of how to achieve the same results on Debian Lenny. Enjoy :)

The first thing that needs to be done is to get the debian sources for clvm and modify then to use openais. After that we will recompile new packages from that source, then set up openais on our cluster nodes.

Install all the dependencies we need to compile clvm

root@host:~# apt-get build-dep clvm
root@host:~# apt-get install libopenais-dev

Now download the source files and cd into our working directory

root@host:~# cd /usr/src/
root@host:/usr/src# apt-get source clvm
root@host:/usr/src# cd lvm2-2.02.39

Now we’ll modify a few files in the source:

• The first is debian/clvm.init. You’ll need to remove any references to cman or cluster.conf. You can download an already edited version here.
• The next is debian/control. Modify the dependies (lvm2 without the version number, openais in place of cman) and modify the comments accordingly. A pre-edited version is here.
• The last file is debian/rules. Replace cman by openais in the configure options, and add the PATH where to find the openais libs. Again, a pre-made version is here.

For clarities sake, here is the actual code block from debian/rules:

$(STAMPS_DIR)/setup-deb: SOURCE_DIR = $(BUILD_DIR)/source
$(STAMPS_DIR)/setup-deb: DIR = $(BUILD_DIR)/build-deb
$(STAMPS_DIR)/setup-deb: $(STAMPS_DIR)/source
        rm -rf $(DIR)
        cp -al $(SOURCE_DIR) $(DIR)
        cd $(DIR); \
        ./configure CFLAGS="$(CFLAGS)" \
		LDFLAGS="-L/usr/lib/openais" \
                $(CONFIGURE_FLAGS) \
                --with-optimisation="" \
                --with-clvmd=openais \
                --enable-readline
        touch $@

The last thing to do is update the internal version number of the clvm package and add some comments to the changelog:

root@host:/usr/src/lm2-2.02.39# dch -i

Now go ahead and compile the package:

root@host:/usr/src/lm2-2.02.39# dpkg-buildpackage -rfakeroot -uc -b

After the compliation completes you should have some shiny new .deb files in /usr/src. The one we are interested in is clvm_2.02.39-7.1_i386.deb (the actual version of yours may vary depending on what you put in the debian changelog in the previous step).

So now that weve got our custom version of clvm compiled, its time to move on the cluster nodes. On each node in the cluster, do the following…

Install openais and add a user for it:

root@node:~# apt-get install openais
root@node:~# mkdir -p /etc/ais
root@node:~# adduser --no-create-home --disabled-password --disabled-login --gecos openAIS ais

Now create the following config in /etc/ais/openais.conf. This is the most basic config you can have. All you need to do is set 192.168.1.0 to be your actual network address.

totem {
        version: 2
        secauth: off
        threads: 0
        interface {
                ringnumber: 0
                bindnetaddr: 192.168.1.0
                mcastaddr: 226.94.1.1
                mcastport: 5405
        }
}

openais does not include a proper debian init sript so you can download one here and save it as /etc/init.d/openais. After that is done, add it to the proper runlevels by issuing:

root@node:~# update-rc.d openais start 62 S . start 50 0 6 .

Now we can install our custom deb and turn on LVM clustering

root@node:~# dpkg -i clvm_2.02.39-7.1_i386.deb
root@node:~# sed -i 's/^    locking_type = 1$/    locking_type = 3/' /etc/lvm/lvm.conf

Thats it! Reboot your cluster nodes and they should all be cluster aware now :)

Setting the device up to work with Debian Etch is fairly straightforward once you know what to do

Obviously, you need a motherboard with built-in Bluetooth or a USB Bluetooth receiver. Once you have that, lets install the software you’ll need:

apt-get install bluez-utils bluez-gnome

Now we need to figure out the Bluetooth address of the keyboard:

user@host:~$ hcitool scan
Scanning ...
        00:23:41:D4:42:78       laptop
        00:27:E2:A1:3F:E5       Apple Wireless Keyboard

Now that you have the Bluetooth address of the keyboard, lets modify the following lines in /etc/default/bluetooth:

HIDD_ENABLED=1
HIDD_OPTIONS="--master --connect $(bluetooth addr) --server"

Next you’ll add a new section to /etc/bluetooth/hcid.conf for the keyboard:

device $(bluetooth addr) {
    name "Apple Wireless Keyboard";
    auth enable;
    encrypt enable;
}

At this point you can pair the keyboard. Since I use fluxbox on my MythTV frontend computers I manually start the required applets, etc.

1. Start window manager
2. As the logged in user, run “bluetooth-applet”
3. Power cycle the Bluetooth keyboard
4. As root, execute “hidd –connect $(keyboard addr)
5. Type a PIN number on bluetooth keyboard (i.e. 1234) then hit return
6. The bluetooth applet will start flashing in the window manager.
7. Click the bluetooth applet and enter the same PIN in the dialog that pops up

That should be it. You now have a working wireless keyboard :-)

If anyone knows of a way to do this entirely on the command line, bypassing the window manager and bluetooth-applet entirely, I’d love to hear it.

Enjoy!

Our test system, for those who dont follow this blog regularly, is a Supermicro X7DVL-E motherboard with a single Intel Core2 Xeon 5130 @ 2Ghz with 2GB of RAM. The disks used are Seagate ES.2 1TB SATA-II models hooked up to two Supermicro AOC-SAT2-MV8 8-port PCI-X SATA controllers. More complete specs are outlined in my original post where I was building the chassis, which can be found here.

The Linux distribution used is, of course, Debian :-) In this case, its Debian 5.0 (Lenny) running kernel 2.6.26-1-686.

Test #1 – Individual Disks
In order to get some baseline numbers to compare against subsequent tests we will first check performance of all the attached disks individually…
for disk in $(ls /dev/disk/by-id/ | grep -v part | grep scsi-SATA_ST); do
echo $disk && \
dd if=/dev/zero of=/dev/disk/by-id/$disk bs=4M oflag=direct count=250 && \
dd if=/dev/disk/by-id/$disk of=/dev/null bs=4M count=250;
done

Which yielded these results…

5QJ06913 - read:109MB/s, write:109MB/s
5QJ09HZR - read:111MB/s, write:112MB/s
5QJ09JHQ - read:113MB/s, write:109MB/s
5QJ09JV3 - read:109MB/s, write:108MB/s


So our average raw device read speed is 110.5MB/s and writes are 109.5MB/s.

Test #2 – Bonnie++ on a single disk
Next we will do a bonnie++ test on a single disk (5QJ06913), formatted with default ext3 options

mkfs.ext3 /dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ06913
mount /dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ06913 /mnt
/usr/sbin/bonnie++ -q -d /mnt -s 4096 -x 5 -u root

This test produced average block reads of 105.55MB/s @ 11.4% CPU usage, block writes of 65.93MB/s @ 19%, and block rewrites of 36.95MB/s @ 8%.

Test #3 – RAID1 pair
For this test we use a RAID1 pair with one disk on each controller, formatted with default ext3 options

mdadm --create /dev/md2 --level=1 --raid-devices=2 /dev/sdf /dev/sdg
dd if=/dev/zero of=/dev/md2 bs=4M oflag=direct count=250
dd if=/dev/md2 of=/dev/null bs=4M count=250


This produced raw I/O reads of 111MB/s and writes of 107MB/s. Now for some bonnie++ tests…


mkfs.ext3 /dev/md2
mount /dev/md2 /mnt
/usr/sbin/bonnie++ -q -d /mnt -s 4096 -x 5 -u root


This test produced average block reads of 103.43MB/s @ 14.4% CPU usage, block writes of 60.84MB/s @ 18.6%, and block rewrites of 33.82MB/s @ 8.8%.

Test #4 – RAID10
For this test we use a RAID10 device consisting of two RAID1 devices, each with a disk on each controller.


mdadm --create /dev/md4 --level=0 --raid-devices=2 /dev/md2 /dev/md3
dd if=/dev/zero of=/dev/md4 bs=4M oflag=direct count=250
dd if=/dev/md4 of=/dev/null bs=4M


This produced raw I/O reads of 219MB/s and writes of 207MB/s. A pretty significant improvement.


mkfs.ext3 /dev/md4
mount /dev/md4 /mnt
/usr/sbin/bonnie++ -q -d /mnt -s 4096 -x 5 -u root


This test produced average block reads of 198.91MB/s @ 27.8% CPU usage, block writes of 132.4MB/s @ 44%, and block rewrites of 61.75MB/s @ 14.4%.

Test #5 – RAID5
For this test we will use a RAID5 device with the first two disks on one controller and the third on the second controller


mdadm --create /dev/md2 --level=5 --raid-devices=3 /dev/sdf1 /dev/sde1 /dev/sdg1
dd if=/dev/zero of=/dev/md2 bs=4M oflag=direct count=250
dd if=/dev/md2 of=/dev/null bs=4M


This produced raw I/O reads of 67.3MB/s and writes of 130MB/s.


mkfs.ext3 /dev/md2
mount /dev/md2 /mnt
/usr/sbin/bonnie++ -q -d /mnt -s 4096 -x 5 -u root


This test produced average block reads of 62.91MB/s @ 8.6% CPU usage, block writes of 65.73MB/s @ 19.6%, and block rewrites of 33.14MB/s @ 7.6%.

Those were all the RAID types I was inclined to test. I realize I left out RAID0 and RAID6. For the application I have in mind, RAID10 is the clear winner. You can download a spreadsheet with the actual bonnie++ output here. Enjoy!

The idea here is to end up with (at least) two VLANs on the network with the dom0 and domU’s being able to choose one or both networks on which to exist. In the case of both, you can set up a handy domU firewall/gateway :)

As you can see from the diagram above, we will end up with three bridges in the dom0 with all the appropriate glue to tie everything together. Best of all, this is all assembled on the fly during bootup.

Now, you obviously need to have a Xen dom0 set up and running properly. Once you are at that point, adding VLAN bridging is fairly straightforward:

First we need to install the vlan package to get the tools for manipulating vlans. Crazy, huh?

apt-get install vlan

Now we set up the interfaces we will need in /etc/network/interfaces. The seasoned admins out there will note that we only bring up the eth0 interface automatically but leave it unconfigured. It will not have an IP or any of the other usual information. The other two (VLAN) interfaces are defined but are NOT brought up during boot. This is important!

In this example the “vlan100″ interface is the replacement for your old “eth0″ interface. Its the interface you’ll be able to reach the dom0 on from the internal VLAN. The “vlan99″ interface is defined but is left unconfigured since there will be no publicly routeable path to the dom0.

auto eth0
iface eth0 inet manual
        pre-up ifconfig $IFACE up
        post-down ifconfig $IFACE down

iface vlan99 inet manual
        mtu 1500
        vlan_raw_device eth0

iface vlan100 inet static
        address 192.168.1.100
        netmask 255.255.255.0
        network 192.168.1.0
        broadcast 192.168.1.255
        gateway 192.168.1.1
        dns-nameservers 192.168.1.254
        dns-search pixelchaos.net
        mtu 1500
        vlan_raw_device eth0

Now we create a new network script for xen located at /etc/xen/scripts/network-multi:

#!/bin/sh
#============================================================================
# Xen vlan bridge start/stop script.
# Xend calls a network script when it starts.
# The script name to use is defined in /etc/xen/xend-config.sxp
# in the network-script field.
#
# This script creates multiple bridges to segregate individual domUs to
# separate VLANs. Customize to fit your needs.
#
# Usage:
#
# network-multi (start|stop|status)
#
#============================================================================

dir=$(dirname "$0")

##
# To make the tagged interface available to some DomUs, create the default
# bridge. Comment this out to only make vlan-based bridges available.
"$dir/network-bridge" "$@" vifnum=0 netdev=eth0 bridge="xbr_trunk"

# Once the "normal" bridge is active, bring up the vlan interfaces.
# They will be attached to the virtual eth0 interface, and not the
# physical one.
ifup vlan99
ifup vlan100

# Create any vlan-based briges.
"$dir/network-bridge" "$@" vifnum=1 bridge="xbr_vl99" netdev="vlan99"
"$dir/network-bridge" "$@" vifnum=2 bridge="xbr_vl100" netdev="vlan100"

# bring up the interfaces again? why do we need to do this? they should have been brought back up by the previous network-bridge call…
ip link set vlan100 up
route add default gw 192.168.1.1 vlan100

That script basically just calls the standard “network-bridge” script supplied with Xen three times – once for each VLAN we are setting up. You’ll notice some interesting bits at the end – We call “ip link” and “route” to bring the vlan100 interface up again. I’m not sure yet why the previous “network-bridge” call doesnt do that on its own… it should as far as I know but I havent dug into why it doesnt. This is just a quick hack to bring “vlan100″ back up again and fix the routes.

Go ahead and add the call to that script in the place of network-bridge in /etc/xen/xend-config.sxp.

Now we need to do one final thing to another Xen script. By default, the xen-network-common.sh script calls “ifdown” directly to take an interface down. Unfortunately, this has the undesirable side effect of destroying any VLANs associated with that interface. This causes all kinds of problems when we call our “network-multi” script so we need to patch /etc/xen/scripts/xen-network-common.sh with the following section:

# do not call ifdown directly
  ifdown()  {
    ip addr flush $1
    ip link set $1 down
    true
  }

To see exactly where that chunk of code goes you can download my already modified script here, or get a diff here if you prefer.

After youve done all that, reboot your dom0. You should now be up and running on the vlan100 interface. The output of “brctl show” should show this:

root@atlas:~# brctl show
bridge name  bridge id          STP enabled    interfaces
xbr_trunk    8000.feffffffffff  no             vif0.0
                                               peth0
xbr_vl100    8000.feffffffffff  no             vif0.2
                                               pvlan100
xbr_vl99     8000.feffffffffff  no             vif0.1
                                               pvlan99

If so, you should be all ready to go. Now edit the vif line in your config file for a given domU to link its ethernet interface to a bridge. My domU firewall is linked to two bridges so it looks like this:

vif= [ 'mac=00:16:3E:44:F9:9C, bridge=xbr_vl99', 'mac=00:16:3E:41:16:51, bridge=xbr_vl100' ]

Thats it! Easy, huh?

Many, many thanks go to Mike Neir for holding my hand through figuring this all out since he’d already set up a functional system that is very similar. Thanks also go to firstserved.net for the ifdown patch and renial.net for other ideas on ways to set up VLAN bridging in Xen.

First of all, install open-iscsi
dpkg -i open-iscsi_2.0.869.2-2_i386.deb

Now compile and install iscsi-target
tar -xzvf iscsitarget-0.4.16.tar.gz
cd iscsitarget-0.4.16
make
make install

And now the important bits. iscsi-target needs to be added to the sysv init scripts in the correct places, which a source install obviously doesnt do:
update-rc.d iscsi-target start 44 S . stop 82 0 6 .

That will make it start just before open-iscsi and stop just after it. Thats it! Any iscsi target that you configure with with the “automatic” option should be correctly connected to during bootup
iscsiadm -m node -T iqn.2008-01.net.pixelchaos:mytarget –op update -n node.conn[0].startup -v automatic

In keeping with my character, the first thing I did was start all over again from scratch by reinstalling the operating system. This time around I set up /dev/md0 as /boot (255 MB) and /dev/md1 as an LVM physical volume (the remainder of the disk), within which /, /home, /usr and friends reside as logical volumes. Its something I’ve wanted to start doing with all my systems for a long time now and shouldn’t have any bearing on the performance tests we are about to do.

Regardless of which protocol will be used we need to enable jumbo frames on all the involved devices. For my setup that means the target (stor01), the initiator (node02), and the switch (a Cisco Catalyst 2970).

First, we turn on jumbo frames for gigabit ethernet at the switch. Beware that this requires a reset (aka reboot) of the switch to take effect:


c2970# system mtu jumbo 9000


Now we enable an MTU of 9000 on both the target and the initiator:


root@stor01:~# ifconfig bond0 mtu 9000



root@node02:~# ifconfig eth0 mtu 9000


For the sake of comparison, here is an iperf test done between the target and initiator with the standard MTU of 1500, and then with an MTU of 9000:


root@stor01:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 1.00 MByte (default)
------------------------------------------------------------
[ 4] local 65.171.150.4 port 5001 connected with 65.171.150.161 port 58731
[ 4] 0.0-10.0 sec 780 MBytes 654 Mbits/sec
[ 5] local 65.171.150.4 port 5001 connected with 65.171.150.161 port 58732
[ 5] 0.0-10.0 sec 916 MBytes 768 Mbits/sec


As you can see, just enabling jumbo frames produces a raw throughput increase of 17.43%. Nothing to sneeze at.

At this point I tried enabling flow control on the catalyst switch (it is already enabled for both send and receive by default in the e1000 driver) but it did not have any effect on iperf numbers. I turned it back off for now.

So now we set up a 20GB LVM volume on the target and export it using vblade to be mounted on the initiator. We then run a simple dd test to check throughput:


root@node02:~# dd if=/dev/zero of=/mnt/test oflag=direct bs=4M
419+0 records in
419+0 records out
1757413376 bytes (1.8 GB) copied, 133.476 seconds, 13.2 MB/s


CPU load on the target was 10-15% during the dd operation. Now we try writing direct to the (unmounted) block device to rule out any performance penalties of the filesystem itself…


root@node02:~# dd if=/dev/zero of=/dev/etherd/e0.1 oflag=direct bs=4M
513+0 records in
512+0 records out
2147483648 bytes (2.1 GB) copied, 170.991 seconds, 12.6 MB/s


CPU usage was slightly higher in that test, running 15-20%. So some slight difference but nothing to be too concerned about.

Now we take that same LVM device and share it via iSCSI for the same dd tests:


ladmin@node02:~$ dd if=/dev/zero of=/mnt/test oflag=direct bs=4M
462+0 records in
461+0 records out
1933574144 bytes (1.9 GB) copied, 38.2375 seconds, 50.6 MB/s


CPU load was 6-8% during that test. We also run that same test with flow control enabled at the switch:


ladmin@node02:~$ dd if=/dev/zero of=/mnt/test oflag=direct bs=4M
463+0 records in
462+0 records out
1937768448 bytes (1.9 GB) copied, 38.1851 seconds, 50.7 MB/s


Essentially the same…

Now this raises the question of why AoE is so much slower than iSCSI on an essentially default install of Debian Etch. To AoE’s credit, many people report getting just as good (50MB/s or better) of performance from AoE on their systems as I’m seeing with iSCSI. I spent quite a large amount of time playing with flow control, kernel ring buffer values, filesystem options, etc. and was unable to determine why performance is so terrible for me. I did find a pretty high number (half a dozen at least) of recent posts to the AoE mailing list by other people having essentially identical problems so I’m certainly not alone. In the interest of completing my testing, I’ve decided to move forward with iSCSI.

Now we try reformatting with the stride option to mkfs:


mkfs.ext3 -E stride=16


The results of several more tests are shown here…


root@node02:~# dd if=/dev/zero of=/mnt/test oflag=direct bs=4M
2038431744 bytes (2.0 GB) copied, 41.8938 seconds, 48.7 MB/s
2038431744 bytes (2.0 GB) copied, 42.2633 seconds, 48.2 MB/s
2038431744 bytes (2.0 GB) copied, 41.3756 seconds, 49.3 MB/s


So we don’t see any appreciable difference when using a combination of the stride= option and flow control, at least with a simple dd test.

Next we turn flow control back off, and reformat again without the stride= option. We are now back to our baseline setup for a new test with bonnie++.


ladmin@node02:~$ /usr/sbin/bonnie++ -d /mnt -s 4096Mb -n 10 -x 5 -q


This test produced block writes of about 82MB/s and block reads of about 37MB/s. The cause for the difference in write speed between the dd and bonnie++ tests is still unclear to me. There also appears to be a known issue where writes are much faster than reads which is apparently due to interrupt handling. This is further evidenced by running a quick dd test that does a read instead of a write:


ladmin@node02:~$ dd if=/mnt/test of=/dev/null bs=4M
3602907136 bytes (3.6 GB) copied, 96.2053 seconds, 37.5 MB/s
3602907136 bytes (3.6 GB) copied, 90.5524 seconds, 39.8 MB/s
3602907136 bytes (3.6 GB) copied, 90.0425 seconds, 40.0 MB/s
3602907136 bytes (3.6 GB) copied, 88.5708 seconds, 40.7 MB/s


As you can see, read operations are about 20% slower than write operations which goes against common thinking with regard to stripped disk arrays.

So there you have it. In my particular situation, with no tuning/optimizing done, iSCSI performs much better than AoE. Even in the event that I were to go to the trouble to performance tune AoE and get it as good as, or even better than, iSCSI I would still be inclined to standardize around iSCSI. Authentication, routing, NAT, etc. can all be done very easily on iSCSI protocol with all the standard TCP/IP tools that are out there. For me that’s a pretty big advantage.

Next up will be our final piece of the puzzle, left over from the initial system setup – getting hot swap working with the sata_mv module!

This actually turns out to be much easier than expected…

The first thing we need to do is install the ifenslave utility:


root@stor01:~# apt-get install ifenslave
Reading package lists... Done
Building dependency tree... Done
The following extra packages will be installed:
ifenslave-2.6
The following NEW packages will be installed:
ifenslave ifenslave-2.6
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 12.4kB of archives.
After unpacking 57.3kB of additional disk space will be used.
Do you want to continue [Y/n]?


Now we create /etc/modprobe.d/bonding and add to it:


alias bond0 bonding
options bonding mode=802.3ad miimon=100 downdelay=200 updelay=200


Once that file is created, run “update-modules” to, well, update our module parameters. Of special note is the mode parameter. It tells the ethernet bonding modules to use 802.3ad bonding mode, which is supported by my Cisco Catalyst 2900 series gigabit switch.

Now add a new bond0 interface to /etc/network/interfaces (removing all the “slave” interfaces from that file at the same time):


auto bond0
iface bond0 inet static
address 192.168.1.4
netmask 255.255.255.0
network 192.168.1.0
broadcast 192.168.1.255
gateway 192.168.1.1
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 192.168.1.254
dns-search pixelchaos.net
up /sbin/ifenslave bond0 eth0
up /sbin/ifenslave bond0 eth1


Thats it! For the sake of simplicity, I just rebooted my server at this point. I’m of course leaving out the configuration of my Cisco Catalyst 2900 series switch to enable an 802.3ad etherchannel on the two ports we are plugged in to.

Next up will be performance testing AoE vs iSCSI…

The Debian Etch installer supports building and installing onto software RAID arrays. Because of that…

during installation I configured the initial RAID1 boot volume with hot spare, consisting of three WD 160GB SATA 3Gbps disks. mdadm sees / and swap as /dev/md0 and /dev/md1 respectively. There was some remaining space on the drives which I set up as /dev/md2 for future use. The remaining arrays I decided to create manually using mdadm after getting a usable system up and running.

First of all we need figure out what devices we want in the arrays by probing /dev. The ultimate goal here is to build three arrays: 1 x RAID1 with 1 hot spare (8GB root, 1GB swap, 151GB extra), 1 x RAID6 with 1 hot spare (4TB Xen LVM’s + 2TB Bacula LVM), 1 x RAID5 (2TB offsite mirror).

I was forward thinking enough to label all the drive carriers with the serial number of the disk in it so all we need to do is get the disk-by-id name from /dev/disk/by-id/ and then build our test array like so:


mdadm --create /dev/md3 --level=6 --raid-devices=8 \
/dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ09HZR \
/dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ09JV3 \
/dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ06913 \
/dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ08XHB \
/dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ09KNM \
/dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ09JHQ \
/dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ0817T \
/dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ09B9P \
--spare-devices=1 \
/dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ07CF3


This assembles our /dev/md3 device with the default options for RAID6 (64k chunk, left-symmetric parity)


stor01:~# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md3 : active raid6 sdl[8](S) sdk[7] sdj[6] sdf[5] sdh[4] sdg[3] sdi[2] sde[1] sdd[0]
5860574976 blocks level 6, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]
[>....................] resync = 0.8% (8790864/976762496) finish=217.3min speed=74238K/sec


Here you can see that CPU load is moderate doing the first full sync of that array:


Cpu0 : 0.0%us, 33.3%sy, 0.0%ni, 63.3%id, 0.0%wa, 1.3%hi, 2.0%si, 0.0%st
Cpu1 : 0.0%us, 7.0%sy, 0.0%ni, 91.0%id, 0.0%wa, 0.3%hi, 1.7%si, 0.0%st


So, we wait about 4 hours… and its done :-)

For formatting the volume, I decided to use the stride= option to mkfs.ext3. This provides for optimal striping across a raid array. The secret here is to make $stride = $chunks / $block_size. In our case, thats 4096 byte (4k) blocks divided by 64k chunks. So 16 would be our optimal stride value.


root@stor01:~# mkfs.ext3 -E stride=16 /dev/md3


So how does it perform? Well enough to saturate the dual gigabit NIC ports – which is all that matters :-)


root@stor01:~# dd bs=4M if=/dev/zero of=/dev/md3
4469+0 records in
4469+0 records out
18744344576 bytes (19 GB) copied, 97.583 seconds, 192 MB/s


I also ran a more real-world test with bonnie++


/usr/sbin/bonnie++ -d /mnt -s 4096Mb -n 100 -x 10 -q


This test showed approximately 335MB/s read, 170MB/s write. You can download the actual data here if you wish.

For failure testing the array, its as simple as removing and inserting disks while the array is up and running. Both tests work fine for the onboard SATA II controller but alas, the sata_mv kernel module does not yet support hotplug so all we can do on the remaining drives is simulate a drive failure by removing one disk. This does work fine but I need to see if there is a way to refresh the SATA bus to get any replacement drive to show up and be added back into the running array. Otherwise, we will need to power down the array to replace a faulty disk which kind of ruins the whole project, dont you think? ;-)

So while we wait for sata_mv to start working (or find a different SATA controller to use in this project) we will move on to the remaining issues. Up next; getting port trunking working with a cisco switch…

First of all, hotplug. I have discovered that…

hotplug doesn’t seem to work on the Supermicro AOC-SAT2-MV8 cards. The controllers do work, but unless a drive is plugged in during bootup they will not be subsequently detected…


root@stor01:~# lspci
05:01.0 SCSI storage controller: Marvell Technology Group Ltd. MV88SX6081 8-port SATA II PCI-X Controller (rev 09)
05:02.0 SCSI storage controller: Marvell Technology Group Ltd. MV88SX6081 8-port SATA II PCI-X Controller (rev 09)


You can see that the Linux kernel sees these cards as Marvell MV88SX6081 devices. Further probing reveals that the sata_mv driver is claiming them:


root@stor01:~$ dmesg | grep 05:01.0
sata_mv 0000:05:01.0: version 0.7
ACPI: PCI Interrupt 0000:05:01.0[A] -> GSI 24 (level, low) -> IRQ 233
sata_mv 0000:05:01.0: 32 slots 8 ports SCSI mode IRQ via INTx


So off we go lkml.org and other sources to find out why… Basically, hotplug support in the sata_mv module is still a work in progress. A post to the lkml claims that its slated for inclusion in the 2.6.26 kernel. There is also more information here.

So hotplug isnt going to work for us today :-( On the bright side, 2.6.25-rc7 is the current development kernel so I can’t imagine that 2.6.26 is too far off.

The second (perceived) issue of getting all the hot swap bays to be mapped to /dev/sd* entries in a particular order doesnt seem to be as much of an issue as I’d thought.

The Linux kernel will initialize PCI devices in a strange fashion depending on what is plugged in to the bus. For example, if all fifteen bays have drives plugged into them, the leftmost bay always shows up at /dev/sda. Unplug just one drive and reboot the machine and it will be mapped to /dev/sdl for example. The reason this isnt really an issue is that newer Linux kernels have not only udev (which is great but not really a solution for this particular problem) but the /dev/disk/by-* tree. You can find all your disks in there with handy names like /dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ06913. That name belongs to that particular drive and only that particular drive, ever.


root@stor01:~# ls -l /dev/disk/by-id/ | grep scsi | grep -v part
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ06913 -> ../../sde
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ07CF3 -> ../../sdi
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ0817T -> ../../sdb
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ08T21 -> ../../sdh
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ08XHB -> ../../sdc
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09B9P -> ../../sda
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09GMW -> ../../sdd
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09HZR -> ../../sdg
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09JHQ -> ../../sdk
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09JV3 -> ../../sdj
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09KNM -> ../../sdl
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09LLA -> ../../sdf
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_WDC_WD1600AAJS-_WD-WCAP93952468 -> ../../sdm
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_WDC_WD1600AAJS-_WD-WMAS20341446 -> ../../sdo
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_WDC_WD1600AAJS-_WD-WMAS20366839 -> ../../sdn


Cool, huh?

So we will just let mdadm do its job of assembling /dev/md* devices on the fly at bootup and use the /dev/disk/by-id entries to do stuff to particular disks (like temperature monitoring, etc.).

Coming up in my next post – performance and failure testing the array.