After some testing I’m happy to say it appears to work smashingly. What follows is a somewhat more complete version of how to achieve the same results on Debian Lenny. Enjoy :)

The first thing that needs to be done is to get the debian sources for clvm and modify then to use openais. After that we will recompile new packages from that source, then set up openais on our cluster nodes.

Install all the dependencies we need to compile clvm

root@host:~# apt-get build-dep clvm
root@host:~# apt-get install libopenais-dev

Now download the source files and cd into our working directory

root@host:~# cd /usr/src/
root@host:/usr/src# apt-get source clvm
root@host:/usr/src# cd lvm2-2.02.39

Now we’ll modify a few files in the source:

• The first is debian/clvm.init. You’ll need to remove any references to cman or cluster.conf. You can download an already edited version here.
• The next is debian/control. Modify the dependies (lvm2 without the version number, openais in place of cman) and modify the comments accordingly. A pre-edited version is here.
• The last file is debian/rules. Replace cman by openais in the configure options, and add the PATH where to find the openais libs. Again, a pre-made version is here.

For clarities sake, here is the actual code block from debian/rules:

$(STAMPS_DIR)/setup-deb: SOURCE_DIR = $(BUILD_DIR)/source
$(STAMPS_DIR)/setup-deb: DIR = $(BUILD_DIR)/build-deb
$(STAMPS_DIR)/setup-deb: $(STAMPS_DIR)/source
        rm -rf $(DIR)
        cp -al $(SOURCE_DIR) $(DIR)
        cd $(DIR); \
        ./configure CFLAGS="$(CFLAGS)" \
		LDFLAGS="-L/usr/lib/openais" \
                $(CONFIGURE_FLAGS) \
                --with-optimisation="" \
                --with-clvmd=openais \
                --enable-readline
        touch $@

The last thing to do is update the internal version number of the clvm package and add some comments to the changelog:

root@host:/usr/src/lm2-2.02.39# dch -i

Now go ahead and compile the package:

root@host:/usr/src/lm2-2.02.39# dpkg-buildpackage -rfakeroot -uc -b

After the compliation completes you should have some shiny new .deb files in /usr/src. The one we are interested in is clvm_2.02.39-7.1_i386.deb (the actual version of yours may vary depending on what you put in the debian changelog in the previous step).

So now that weve got our custom version of clvm compiled, its time to move on the cluster nodes. On each node in the cluster, do the following…

Install openais and add a user for it:

root@node:~# apt-get install openais
root@node:~# mkdir -p /etc/ais
root@node:~# adduser --no-create-home --disabled-password --disabled-login --gecos openAIS ais

Now create the following config in /etc/ais/openais.conf. This is the most basic config you can have. All you need to do is set 192.168.1.0 to be your actual network address.

totem {
        version: 2
        secauth: off
        threads: 0
        interface {
                ringnumber: 0
                bindnetaddr: 192.168.1.0
                mcastaddr: 226.94.1.1
                mcastport: 5405
        }
}

openais does not include a proper debian init sript so you can download one here and save it as /etc/init.d/openais. After that is done, add it to the proper runlevels by issuing:

root@node:~# update-rc.d openais start 62 S . start 50 0 6 .

Now we can install our custom deb and turn on LVM clustering

root@node:~# dpkg -i clvm_2.02.39-7.1_i386.deb
root@node:~# sed -i 's/^    locking_type = 1$/    locking_type = 3/' /etc/lvm/lvm.conf

Thats it! Reboot your cluster nodes and they should all be cluster aware now :)

The first thing youll need is a TFTP server. This will serve out the kernel and initrd images to clients on the network.

root@pxe:~$ apt-get install tftpd-hpa

That will install a good tftp server daemon and create the tftp root for you in /var/lib/tftpboot. Some HOWTO’s on the Internet suggest installing the syslinux package as well which provides a needed file (pxelinux.0). I prefer to download the entire netboot environment for debian since this will also provide a network bootable installer for Debian on top of the other goodness we are creating today.

root@pxe:~$ cd /var/lib/tftpboot
root@pxe:~$ wget http://ftp.debian.org/debian/dists/lenny/main/installer-i386/current/images/netboot/netboot.tar.gz
root@pxe:~$ tar xzvf netboot.tar.gz

The kernel and initrd you are going to use to boot the mythbox must reside on the TFTP server. Furthermore, the initrd must be updated to support NFS root. On the mythbox I was migrating to the NFS root, it looked like this:

root@mythbox:~$ sed -i 's/BOOT=local/BOOT=nfs/'  /etc/initramfs-tools/initramfs.conf
root@mythbox:~$ /usr/sbin/mkinitramfs -o /boot/initrd.img-2.6.26-1-686_netboot

Now just copy the /boot/vmlinuz-2.6.26-1-686 and /boot/initrd.img-2.6.26-1-686_netboot to /var/lib/tftpboot/ on the TFTP server.

Last step for the TFTP server is to create a config file for our mythbox. If you dont create one specifically for it, it will instead boot the network installer as mentioned above. Create the file /var/lib/tftpboot/pxelinux.cfg/01-$(hardware ethernet address). Including the hardware address in the file name binds that config to the particular workstation we are working on today. In my case, the file name is 01-00-19-db-4a-6a-0e.

LABEL linux
KERNEL vmlinuz-2.6.26-1-686
APPEND root=/dev/nfs initrd=initrd.img-2.6.26-1-686_netboot nfsroot=192.168.1.108:/srv/nfs/mythbox.pixelchaos.net ip=dhcp rw

Now that our TFTP server is up and running we will next need to configure a DHCP server to provide IP addresses to netboot clients, as well as point them to the TFTP server to continue the boot process.

root@dhcp:~$ apt-get install dhcp3-server

That will install the DHCP service for you. Now we just need to add one little section to /etc/dhcp3/dhcpd.conf for the workstation we are going to netboot.

host mythbox {
hardware ethernet 00:19:db:4a:6a:0e;
fixed-address mythbox.pixelchaos.net;
filename "pxelinux.0";
next-server 192.168.1.115;
}

host mythbox: mythbox is just the descriptive name for this configuration block .You can use any name you like.
hardware ethernet 00:19:db:4a:6a:0e: This is the hardware ethernet address of the workstation we are netbooting.
fixed-address mythbox.pixelchaos.net: This tells the DHCP server what address to assign the mythbox. You can use a FQDN (if its resolvable!) or an IP address.
filename “pxelinux.0″: This is the PXE boot loader file that we installed onto our TFTP server earlier.
next-server 192.168.1.115: This is the IP address of our TFTP server. It tells the mythbox where the “next server” is after it gets its IP address from the DHCP server. The “next server” can be any machine on the network. It can even be the same one running DHCP.

Once all that is done, restart your DHCP server.

root@dhcp:~$ /etc/init.d/dhcp3-server restart

Now for the third piece of the puzzle, our NFS root filesystem. You’ll first need to create a directory to share.

root@nfs:~$ mkdir /srv/nfs/mythbox.pixelchaos.net

Now add that directory to /etc/exports so that it gets shared via NFS with this line:

/srv/nfs/mythbox.pixelchaos.net mythbox(rw,async,no_root_squash)

The rw, async, and no_root_squash options are important. Dont omit them. At this point we can reload all our NFS exports to active this new share point.

root@nfs:~$ exportfs -rv

At this point the entire TFTP, DHCP, and NFS parts of the environment are ready to go. Now we will migrate our existing installation of Linux from the mythbox onto its new nfsroot. First we make a directory to mount the NFS share on and then mount it.

root@mythbox:~$ mkdir /mnt/nfs
root@mythbox:~$ mount -t nfs -o nolock 192.168.1.108:/srv/nfs/mythbox.pixelchaos.net /mnt/nfs

Now we run a cp command on every locally mounted partition, as well as /dev. On my mythbox it looked like this:

root@mythbox:~$ cp -axv /. /mnt/nfs/.
root@mythbox:~$ cp -axv /boot/. /mnt/nfs/boot/.
root@mythbox:~$ cp -axv /home/. /mnt/nfs/home/.
root@mythbox:~$ cp -axv /dev/. /mnt/nfs/dev/.

The last thing to do is edit two files that we copied up to /mnt/nfs. The first is /mnt/nfs/etc/network/interfaces. You’ll need to remove any lines that bring the interface up automatically. You want to remove anything like “allow-hotplug eth0″ and just leave a single line like “iface eth0 inet dhcp”. The network interface will have already been brought up by netboot normally and leaving in any “hotplug” or “auto” lines will bring it up again, which has the effect of destroying it.

Now we remove all the old filesystems from /mnt/nfs/etc/fstab so that it looks like this:

# /etc/fstab: static file system information.
#
# <file system> 		<mount point>   	<type>  <options>       <dump>  <pass>
/dev/nfs        /               nfs     defaults 0 0
none            /tmp            tmpfs   defaults 0 0
none            /var/run        tmpfs   defaults 0 0
none            /var/lock       tmpfs   defaults 0 0
none            /var/tmp        tmpfs   defaults 0 0
none            /media          tmpfs   defaults 0 0

That should be it. Now just set the mythbox to use network booting in its BIOS and reboot.

Cool, huh? :)

The idea here is to end up with (at least) two VLANs on the network with the dom0 and domU’s being able to choose one or both networks on which to exist. In the case of both, you can set up a handy domU firewall/gateway :)

As you can see from the diagram above, we will end up with three bridges in the dom0 with all the appropriate glue to tie everything together. Best of all, this is all assembled on the fly during bootup.

Now, you obviously need to have a Xen dom0 set up and running properly. Once you are at that point, adding VLAN bridging is fairly straightforward:

First we need to install the vlan package to get the tools for manipulating vlans. Crazy, huh?

apt-get install vlan

Now we set up the interfaces we will need in /etc/network/interfaces. The seasoned admins out there will note that we only bring up the eth0 interface automatically but leave it unconfigured. It will not have an IP or any of the other usual information. The other two (VLAN) interfaces are defined but are NOT brought up during boot. This is important!

In this example the “vlan100″ interface is the replacement for your old “eth0″ interface. Its the interface you’ll be able to reach the dom0 on from the internal VLAN. The “vlan99″ interface is defined but is left unconfigured since there will be no publicly routeable path to the dom0.

auto eth0
iface eth0 inet manual
        pre-up ifconfig $IFACE up
        post-down ifconfig $IFACE down

iface vlan99 inet manual
        mtu 1500
        vlan_raw_device eth0

iface vlan100 inet static
        address 192.168.1.100
        netmask 255.255.255.0
        network 192.168.1.0
        broadcast 192.168.1.255
        gateway 192.168.1.1
        dns-nameservers 192.168.1.254
        dns-search pixelchaos.net
        mtu 1500
        vlan_raw_device eth0

Now we create a new network script for xen located at /etc/xen/scripts/network-multi:

#!/bin/sh
#============================================================================
# Xen vlan bridge start/stop script.
# Xend calls a network script when it starts.
# The script name to use is defined in /etc/xen/xend-config.sxp
# in the network-script field.
#
# This script creates multiple bridges to segregate individual domUs to
# separate VLANs. Customize to fit your needs.
#
# Usage:
#
# network-multi (start|stop|status)
#
#============================================================================

dir=$(dirname "$0")

##
# To make the tagged interface available to some DomUs, create the default
# bridge. Comment this out to only make vlan-based bridges available.
"$dir/network-bridge" "$@" vifnum=0 netdev=eth0 bridge="xbr_trunk"

# Once the "normal" bridge is active, bring up the vlan interfaces.
# They will be attached to the virtual eth0 interface, and not the
# physical one.
ifup vlan99
ifup vlan100

# Create any vlan-based briges.
"$dir/network-bridge" "$@" vifnum=1 bridge="xbr_vl99" netdev="vlan99"
"$dir/network-bridge" "$@" vifnum=2 bridge="xbr_vl100" netdev="vlan100"

# bring up the interfaces again? why do we need to do this? they should have been brought back up by the previous network-bridge call…
ip link set vlan100 up
route add default gw 192.168.1.1 vlan100

That script basically just calls the standard “network-bridge” script supplied with Xen three times – once for each VLAN we are setting up. You’ll notice some interesting bits at the end – We call “ip link” and “route” to bring the vlan100 interface up again. I’m not sure yet why the previous “network-bridge” call doesnt do that on its own… it should as far as I know but I havent dug into why it doesnt. This is just a quick hack to bring “vlan100″ back up again and fix the routes.

Go ahead and add the call to that script in the place of network-bridge in /etc/xen/xend-config.sxp.

Now we need to do one final thing to another Xen script. By default, the xen-network-common.sh script calls “ifdown” directly to take an interface down. Unfortunately, this has the undesirable side effect of destroying any VLANs associated with that interface. This causes all kinds of problems when we call our “network-multi” script so we need to patch /etc/xen/scripts/xen-network-common.sh with the following section:

# do not call ifdown directly
  ifdown()  {
    ip addr flush $1
    ip link set $1 down
    true
  }

To see exactly where that chunk of code goes you can download my already modified script here, or get a diff here if you prefer.

After youve done all that, reboot your dom0. You should now be up and running on the vlan100 interface. The output of “brctl show” should show this:

root@atlas:~# brctl show
bridge name  bridge id          STP enabled    interfaces
xbr_trunk    8000.feffffffffff  no             vif0.0
                                               peth0
xbr_vl100    8000.feffffffffff  no             vif0.2
                                               pvlan100
xbr_vl99     8000.feffffffffff  no             vif0.1
                                               pvlan99

If so, you should be all ready to go. Now edit the vif line in your config file for a given domU to link its ethernet interface to a bridge. My domU firewall is linked to two bridges so it looks like this:

vif= [ 'mac=00:16:3E:44:F9:9C, bridge=xbr_vl99', 'mac=00:16:3E:41:16:51, bridge=xbr_vl100' ]

Thats it! Easy, huh?

Many, many thanks go to Mike Neir for holding my hand through figuring this all out since he’d already set up a functional system that is very similar. Thanks also go to firstserved.net for the ifdown patch and renial.net for other ideas on ways to set up VLAN bridging in Xen.