pixelchaos.net » Xen

openais: an alternative to clvm with cman

jcl — Thu, 23 Apr 2009 17:24:29 +0000

I’ve been battling lately with a lot of problems with cman, part of Red Hat Cluster Suite. Specifically, the fencing tool (fenced) is pretty much junk when you try to start using it with Xen dom0’s. After much searching and gnashing of teeth I happened upon this mailing list post. The promise there is that you could take clvm and compile it against openais and get a cluster aware LVM which doesnt require the rest of Red Hat Cluster Suite (and its crappy documentation, crappy fencing, and general all around crappiness). A little more searching turned up this web site from Olivier Le Cam which pretty much did 90% of the work for me.

After some testing I’m happy to say it appears to work smashingly. What follows is a somewhat more complete version of how to achieve the same results on Debian Lenny. Enjoy :)

The first thing that needs to be done is to get the debian sources for clvm and modify then to use openais. After that we will recompile new packages from that source, then set up openais on our cluster nodes.

Install all the dependencies we need to compile clvm

root@host:~# apt-get build-dep clvm
root@host:~# apt-get install libopenais-dev

Now download the source files and cd into our working directory

root@host:~# cd /usr/src/
root@host:/usr/src# apt-get source clvm
root@host:/usr/src# cd lvm2-2.02.39

Now we’ll modify a few files in the source:

The first is debian/clvm.init. You’ll need to remove any references to cman or cluster.conf. You can download an already edited version here.
The next is debian/control. Modify the dependies (lvm2 without the version number, openais in place of cman) and modify the comments accordingly. A pre-edited version is here.
The last file is debian/rules. Replace cman by openais in the configure options, and add the PATH where to find the openais libs. Again, a pre-made version is here.

For clarities sake, here is the actual code block from debian/rules:

$(STAMPS_DIR)/setup-deb: SOURCE_DIR = $(BUILD_DIR)/source
$(STAMPS_DIR)/setup-deb: DIR = $(BUILD_DIR)/build-deb
$(STAMPS_DIR)/setup-deb: $(STAMPS_DIR)/source
        rm -rf $(DIR)
        cp -al $(SOURCE_DIR) $(DIR)
        cd $(DIR); \
        ./configure CFLAGS="$(CFLAGS)" \
		LDFLAGS="-L/usr/lib/openais" \
                $(CONFIGURE_FLAGS) \
                --with-optimisation="" \
                --with-clvmd=openais \
                --enable-readline
        touch $@

The last thing to do is update the internal version number of the clvm package and add some comments to the changelog:

root@host:/usr/src/lm2-2.02.39# dch -i

Now go ahead and compile the package:

root@host:/usr/src/lm2-2.02.39# dpkg-buildpackage -rfakeroot -uc -b

After the compliation completes you should have some shiny new .deb files in /usr/src. The one we are interested in is clvm_2.02.39-7.1_i386.deb (the actual version of yours may vary depending on what you put in the debian changelog in the previous step).

So now that weve got our custom version of clvm compiled, its time to move on the cluster nodes. On each node in the cluster, do the following…

Install openais and add a user for it:

root@node:~# apt-get install openais
root@node:~# mkdir -p /etc/ais
root@node:~# adduser --no-create-home --disabled-password --disabled-login --gecos openAIS ais

Now create the following config in /etc/ais/openais.conf. This is the most basic config you can have. All you need to do is set 192.168.1.0 to be your actual network address.

totem {
        version: 2
        secauth: off
        threads: 0
        interface {
                ringnumber: 0
                bindnetaddr: 192.168.1.0
                mcastaddr: 226.94.1.1
                mcastport: 5405
        }
}

openais does not include a proper debian init sript so you can download one here and save it as /etc/init.d/openais. After that is done, add it to the proper runlevels by issuing:

root@node:~# update-rc.d openais start 62 S . start 50 0 6 .

Now we can install our custom deb and turn on LVM clustering

root@node:~# dpkg -i clvm_2.02.39-7.1_i386.deb
root@node:~# sed -i 's/^    locking_type = 1$/    locking_type = 3/' /etc/lvm/lvm.conf

Thats it! Reboot your cluster nodes and they should all be cluster aware now :)

VLAN Bridging in Xen

jcl — Thu, 17 Jul 2008 00:47:22 +0000

Recently I came upon the need to do all my network routing and firewalling inside a Xen domU. I am not the first to do this but I thought I’d do a little write up on it to help others trying to accomplish the same thing in Debian.

The idea here is to end up with (at least) two VLANs on the network with the dom0 and domU’s being able to choose one or both networks on which to exist. In the case of both, you can set up a handy domU firewall/gateway :)

As you can see from the diagram above, we will end up with three bridges in the dom0 with all the appropriate glue to tie everything together. Best of all, this is all assembled on the fly during bootup.

Now, you obviously need to have a Xen dom0 set up and running properly. Once you are at that point, adding VLAN bridging is fairly straightforward:

First we need to install the vlan package to get the tools for manipulating vlans. Crazy, huh?

apt-get install vlan

Now we set up the interfaces we will need in /etc/network/interfaces. The seasoned admins out there will note that we only bring up the eth0 interface automatically but leave it unconfigured. It will not have an IP or any of the other usual information. The other two (VLAN) interfaces are defined but are NOT brought up during boot. This is important!

In this example the “vlan100″ interface is the replacement for your old “eth0″ interface. Its the interface you’ll be able to reach the dom0 on from the internal VLAN. The “vlan99″ interface is defined but is left unconfigured since there will be no publicly routeable path to the dom0.

auto eth0
iface eth0 inet manual
        pre-up ifconfig $IFACE up
        post-down ifconfig $IFACE down

iface vlan99 inet manual
        mtu 1500
        vlan_raw_device eth0

iface vlan100 inet static
        address 192.168.1.100
        netmask 255.255.255.0
        network 192.168.1.0
        broadcast 192.168.1.255
        gateway 192.168.1.1
        dns-nameservers 192.168.1.254
        dns-search pixelchaos.net
        mtu 1500
        vlan_raw_device eth0

Now we create a new network script for xen located at /etc/xen/scripts/network-multi:

#!/bin/sh
#============================================================================
# Xen vlan bridge start/stop script.
# Xend calls a network script when it starts.
# The script name to use is defined in /etc/xen/xend-config.sxp
# in the network-script field.
#
# This script creates multiple bridges to segregate individual domUs to
# separate VLANs. Customize to fit your needs.
#
# Usage:
#
# network-multi (start|stop|status)
#
#============================================================================

dir=$(dirname "$0")

##
# To make the tagged interface available to some DomUs, create the default
# bridge. Comment this out to only make vlan-based bridges available.
"$dir/network-bridge" "$@" vifnum=0 netdev=eth0 bridge="xbr_trunk"

# Once the "normal" bridge is active, bring up the vlan interfaces.
# They will be attached to the virtual eth0 interface, and not the
# physical one.
ifup vlan99
ifup vlan100

# Create any vlan-based briges.
"$dir/network-bridge" "$@" vifnum=1 bridge="xbr_vl99" netdev="vlan99"
"$dir/network-bridge" "$@" vifnum=2 bridge="xbr_vl100" netdev="vlan100"

# bring up the interfaces again? why do we need to do this? they should have been brought back up by the previous network-bridge call…
ip link set vlan100 up
route add default gw 192.168.1.1 vlan100

That script basically just calls the standard “network-bridge” script supplied with Xen three times – once for each VLAN we are setting up. You’ll notice some interesting bits at the end – We call “ip link” and “route” to bring the vlan100 interface up again. I’m not sure yet why the previous “network-bridge” call doesnt do that on its own… it should as far as I know but I havent dug into why it doesnt. This is just a quick hack to bring “vlan100″ back up again and fix the routes.

Go ahead and add the call to that script in the place of network-bridge in /etc/xen/xend-config.sxp.

Now we need to do one final thing to another Xen script. By default, the xen-network-common.sh script calls “ifdown” directly to take an interface down. Unfortunately, this has the undesirable side effect of destroying any VLANs associated with that interface. This causes all kinds of problems when we call our “network-multi” script so we need to patch /etc/xen/scripts/xen-network-common.sh with the following section:

# do not call ifdown directly
  ifdown()  {
    ip addr flush $1
    ip link set $1 down
    true
  }

To see exactly where that chunk of code goes you can download my already modified script here, or get a diff here if you prefer.

After youve done all that, reboot your dom0. You should now be up and running on the vlan100 interface. The output of “brctl show” should show this:

root@atlas:~# brctl show
bridge name  bridge id          STP enabled    interfaces
xbr_trunk    8000.feffffffffff  no             vif0.0
                                               peth0
xbr_vl100    8000.feffffffffff  no             vif0.2
                                               pvlan100
xbr_vl99     8000.feffffffffff  no             vif0.1
                                               pvlan99

If so, you should be all ready to go. Now edit the vif line in your config file for a given domU to link its ethernet interface to a bridge. My domU firewall is linked to two bridges so it looks like this:

vif= [ 'mac=00:16:3E:44:F9:9C, bridge=xbr_vl99', 'mac=00:16:3E:41:16:51, bridge=xbr_vl100' ]

Thats it! Easy, huh?

Many, many thanks go to Mike Neir for holding my hand through figuring this all out since he’d already set up a functional system that is very similar. Thanks also go to firstserved.net for the ifdown patch and renial.net for other ideas on ways to set up VLAN bridging in Xen.

Coraid Odyssey: Part 3 (performance testing)

jcl — Tue, 08 Apr 2008 16:15:46 +0000

Performance and failure testing are next up in building our kickin’ iSCSI/AoE device.

The Debian Etch installer supports building and installing onto software RAID arrays. Because of that…

during installation I configured the initial RAID1 boot volume with hot spare, consisting of three WD 160GB SATA 3Gbps disks. mdadm sees / and swap as /dev/md0 and /dev/md1 respectively. There was some remaining space on the drives which I set up as /dev/md2 for future use. The remaining arrays I decided to create manually using mdadm after getting a usable system up and running.

First of all we need figure out what devices we want in the arrays by probing /dev. The ultimate goal here is to build three arrays: 1 x RAID1 with 1 hot spare (8GB root, 1GB swap, 151GB extra), 1 x RAID6 with 1 hot spare (4TB Xen LVM’s + 2TB Bacula LVM), 1 x RAID5 (2TB offsite mirror).

I was forward thinking enough to label all the drive carriers with the serial number of the disk in it so all we need to do is get the disk-by-id name from /dev/disk/by-id/ and then build our test array like so:

mdadm --create /dev/md3 --level=6 --raid-devices=8 \ /dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ09HZR \ /dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ09JV3 \ /dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ06913 \ /dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ08XHB \ /dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ09KNM \ /dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ09JHQ \ /dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ0817T \ /dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ09B9P \ --spare-devices=1 \ /dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ07CF3

This assembles our /dev/md3 device with the default options for RAID6 (64k chunk, left-symmetric parity)

stor01:~# cat /proc/mdstat Personalities : [raid1] [raid6] [raid5] [raid4] md3 : active raid6 sdl[8](S) sdk[7] sdj[6] sdf[5] sdh[4] sdg[3] sdi[2] sde[1] sdd[0] 5860574976 blocks level 6, 64k chunk, algorithm 2 [8/8] [UUUUUUUU] [>....................] resync = 0.8% (8790864/976762496) finish=217.3min speed=74238K/sec

Here you can see that CPU load is moderate doing the first full sync of that array:

Cpu0 : 0.0%us, 33.3%sy, 0.0%ni, 63.3%id, 0.0%wa, 1.3%hi, 2.0%si, 0.0%st Cpu1 : 0.0%us, 7.0%sy, 0.0%ni, 91.0%id, 0.0%wa, 0.3%hi, 1.7%si, 0.0%st

So, we wait about 4 hours… and its done :-)

For formatting the volume, I decided to use the stride= option to mkfs.ext3. This provides for optimal striping across a raid array. The secret here is to make $stride = $chunks / $block_size. In our case, thats 4096 byte (4k) blocks divided by 64k chunks. So 16 would be our optimal stride value.

root@stor01:~# mkfs.ext3 -E stride=16 /dev/md3

So how does it perform? Well enough to saturate the dual gigabit NIC ports – which is all that matters :-)

root@stor01:~# dd bs=4M if=/dev/zero of=/dev/md3 4469+0 records in 4469+0 records out 18744344576 bytes (19 GB) copied, 97.583 seconds, 192 MB/s

I also ran a more real-world test with bonnie++

/usr/sbin/bonnie++ -d /mnt -s 4096Mb -n 100 -x 10 -q

This test showed approximately 335MB/s read, 170MB/s write. You can download the actual data here if you wish.

For failure testing the array, its as simple as removing and inserting disks while the array is up and running. Both tests work fine for the onboard SATA II controller but alas, the sata_mv kernel module does not yet support hotplug so all we can do on the remaining drives is simulate a drive failure by removing one disk. This does work fine but I need to see if there is a way to refresh the SATA bus to get any replacement drive to show up and be added back into the running array. Otherwise, we will need to power down the array to replace a faulty disk which kind of ruins the whole project, dont you think? ;-)

So while we wait for sata_mv to start working (or find a different SATA controller to use in this project) we will move on to the remaining issues. Up next; getting port trunking working with a cisco switch…

Coraid Odyssey: Part 2 (sata_mv hotplug)

jcl — Tue, 01 Apr 2008 15:33:28 +0000

Todays adventure with building a SAN on the cheap involves attempting to get hotplug working and changing device mappings.

First of all, hotplug. I have discovered that…

hotplug doesn’t seem to work on the Supermicro AOC-SAT2-MV8 cards. The controllers do work, but unless a drive is plugged in during bootup they will not be subsequently detected…

root@stor01:~# lspci 05:01.0 SCSI storage controller: Marvell Technology Group Ltd. MV88SX6081 8-port SATA II PCI-X Controller (rev 09) 05:02.0 SCSI storage controller: Marvell Technology Group Ltd. MV88SX6081 8-port SATA II PCI-X Controller (rev 09)

You can see that the Linux kernel sees these cards as Marvell MV88SX6081 devices. Further probing reveals that the sata_mv driver is claiming them:

root@stor01:~$ dmesg | grep 05:01.0 sata_mv 0000:05:01.0: version 0.7 ACPI: PCI Interrupt 0000:05:01.0[A] -> GSI 24 (level, low) -> IRQ 233 sata_mv 0000:05:01.0: 32 slots 8 ports SCSI mode IRQ via INTx

So off we go lkml.org and other sources to find out why… Basically, hotplug support in the sata_mv module is still a work in progress. A post to the lkml claims that its slated for inclusion in the 2.6.26 kernel. There is also more information here.

So hotplug isnt going to work for us today :-( On the bright side, 2.6.25-rc7 is the current development kernel so I can’t imagine that 2.6.26 is too far off.

The second (perceived) issue of getting all the hot swap bays to be mapped to /dev/sd* entries in a particular order doesnt seem to be as much of an issue as I’d thought.

The Linux kernel will initialize PCI devices in a strange fashion depending on what is plugged in to the bus. For example, if all fifteen bays have drives plugged into them, the leftmost bay always shows up at /dev/sda. Unplug just one drive and reboot the machine and it will be mapped to /dev/sdl for example. The reason this isnt really an issue is that newer Linux kernels have not only udev (which is great but not really a solution for this particular problem) but the /dev/disk/by-* tree. You can find all your disks in there with handy names like /dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ06913. That name belongs to that particular drive and only that particular drive, ever.

root@stor01:~# ls -l /dev/disk/by-id/ | grep scsi | grep -v part lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ06913 -> ../../sde lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ07CF3 -> ../../sdi lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ0817T -> ../../sdb lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ08T21 -> ../../sdh lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ08XHB -> ../../sdc lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09B9P -> ../../sda lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09GMW -> ../../sdd lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09HZR -> ../../sdg lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09JHQ -> ../../sdk lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09JV3 -> ../../sdj lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09KNM -> ../../sdl lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09LLA -> ../../sdf lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_WDC_WD1600AAJS-_WD-WCAP93952468 -> ../../sdm lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_WDC_WD1600AAJS-_WD-WMAS20341446 -> ../../sdo lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_WDC_WD1600AAJS-_WD-WMAS20366839 -> ../../sdn

Cool, huh?

So we will just let mdadm do its job of assembling /dev/md* devices on the fly at bootup and use the /dev/disk/by-id entries to do stuff to particular disks (like temperature monitoring, etc.).

Coming up in my next post – performance and failure testing the array.

Coraid Odyssey: Part 1 (building the chassis)

jcl — Fri, 28 Mar 2008 21:17:03 +0000

AoE (ATA over Ethernet) and iSCSI are the hot new things. Xen is the hot new thing. I like using hot new things as long as they can be made rock solid.

There happens to be a company (Coraid) that makes a turnkey AoE device. Its far cheaper than a true fibre channel SAN or something similar. Perfect for setting up a SAN over Ethernet device that can serve Xen domU filesystems out to “thin” dom0’s on the network.

Well that’s all well and good but you see I’m always looking to save a buck…

So I asked myself; why not build my own “coraid” with off the shelf parts and save 50% in the process? Herein I’ll do my best to chronicle my adventures in getting this thing built, tested, and (hopefully) deployed.

First of all the hardware. Thanks to the ever-brilliant Mike Neir for helping me figure out what Coraid uses to build these suckers. Here are the parts I ended up ordering:

2 x Supermicro 8-Port SATA Card (AOC-SAT2-MV8)
1 x Supermicro Xeon Dual-Core Blackford VS ServerBoard (X7DVL-E)
1 x Supermicro Black 3U Rackmount Case 760W (SC933T-R760B)
1 x Kingston 2GB 240-Pin DDR2 FB-DIMM ECC (KVR667D2D8F5K2/2G)
1 x Intel Xeon 5130 Woodcrest 2.0GHz 4M (BX805565130A)
15 x Seagate Barracuda ES.2 1TB (ST31000340NS)
3 x Supermicro Drive Carrier (CSE-PT39B)
3 x Western Digital 7K 8M SATA2 160GB (WD1600AAJS)

Now you may ask, why buy 18 drives for a chassis that has 15 hot swap bays? Well, I’ll be using the three WD drives as a RAID1 boot device with one hot spare, then setting nine of the 1TB drives up into a RAID6 device with one hot spare. The remaining three drives will be in RAID5 and swapped with the last three drives on a regular basis to go off site as part of a hard disk based backup system.

Now that we have all of that out of the way; how did the assembly go? Flawless almost ;-)

There are two gotcha’s to be dealt with in the assembly of the aforementioned motherboard and server chassis; first of all, the EPS12V connector for CPU power that comes out of the power distribution block wont reach its molex connector on the motherboard. Second, the ribbon cable that connects the front panel controls to the their motherboard header just *barely* reaches which makes me uncomfortable.

To resolve the first problem I ordered up a StarTech 8″ EPS 8 Pin Power Extension Cable (EPS8EXT) which did the job nicely. To fix the second issue, I ordered twenty feet of 16 conductor ribbon cable (might need extra!), some cable ends and a crimper. Problem solved.

With all that taken care of its time to get an operating system installed and test this thing out. I did go ahead and power up the system and test the redundant PSU’s and fans. The alarm features of both seem to work fine with audible alarms when any of them are removed from the system. After that I plugged an old CD-ROM into the ATA header on the mobo and installed a rockin Linux distro.

Some things I discovered:

The BIOS and the Linux kernel both initialize the two SATA PCI cards first. Some fiddling will need to be done to make the three WD drives (plugged into the motherboard SATA headers) show up as sda, sdb, and sdc.
You must turn off “compatibility mode” in the BIOS for the onboard SATA controllers or hotplug will not work for drives on those headers. After turning it off, hotplug works just dandy. Yay!
Hotplug isnt working for the two PCI SATA cards at this point. I suspect its something to do with the sata_mv kernel module but havent gotten to look into it much. That will be the first order of business when next I work on this project.

So thats it for now. Things are coming together nicely. My next post will detail getting hotplug working for the two PCI SATA cards and getting device detection and mapping working correctly in the Linux kernel (I smell some udev tomfoolery most likely).