There had to be a better way.

After some research and design discussions I came up with a pretty elegant solution that solved all the above problems and was radically cheaper and simpler.

The Solution
The final implementation used a simple, and at this point well proven, technology: AnyCast. At the time AnyCast was fairly new on the infrastructure scene but it had a number of advantages. First of all it’s simple. You need no special hardware beyond the layer 3 switches you probably already have. The implementation is just a few lines in IOS and you’re up and running with a route and SLA check. Since the switches handle all the load balancing you can get rid of all that expensive load balancing gear with its added complexity.

Now that you have no load balancers to worry about you can just throw cheap, entry level nodes around every datacenter and point the AnyCast routes to them. In our case we just used cheap dual core boxes with 2GB of RAM each. Nothing special. This is horizontal scaling at it’s finest.

The final trick was to get rid of ISC BIND and replace it with unbound. Now, don’t get me wrong, ISC BIND works great and we could have continued to use it. There were a couple considerations that drove the decision for unbound however. First of all it performs nearly an order of magnitude better on the same hardware. Second, it does one thing and does it well – recursive queries and caching. Because of that its configuration is much simpler as well.

The deployment today consists of 16 AnyCast endpoints that are servicing an aggregate load of about 80,000 queries per second and could easily support much more than that. Initial performance testing showed that those cheap dual core hosts can support a query load of about 20,000 queries per second each.

A nice, clean setup that is simple and cheap. Perfect.

Design Considerations
There are a few things to be aware of when designing a system like this however.

• CEF and XOR : Cisco gear has to make a decision on where to route inbound queries when there are multiple endpoints that have identical route distances. This ends up being a pseudo-load balancing in practice because it is not round robin. The switch decides where to route packets by XOR’ing the last two octets of the source and destination IP’s. The balance of traffic across a pool of endpoints ends up being pretty close to even but it’s not perfect. You have to be aware of this slight traffic imbalance when capacity planning.
• Number of endpoints : CEF on Cisco devices currently only supports a maximum of 16 endpoints per device. In practice this isnt a practical problem though. It’s just something to be aware of.
• More general capacity planning : If you were to ever lose the route to a switch, however unlikely, AnyCast will fail ALL the traffic destined to those endpoints to the next lowest cost route. If you dont plan for that you’ll send too much traffic to the next cluster of nodes which will DoS it and make the SLA checks fail. AnyCast will then send all the original traffic, and all the traffic for cluster number 2, on to the third cluster and so forth. Cascading failure of your whole infrastructure can happen.
• Troubleshooting : It’s somewhat more complicated to know where traffic from a given DNS query is being routed. You have to dig around a lot to figure this out if there is a problem. It’s not impossible… just not as straightforward as designs that have a single cluster with a single virtual IP taking in all the inbound queries.

Beyond those few considerations though, a setup like this is quite reliable, endlessly scalable, and offers the ability to have a single DNS query target for all hosts across all datacenters.

Pretty nice.

Say you have thousands of hosts and want to collect tens of metrics from each one for analysis and alerting. Tools like cacti and munin simply dont scale up these levels. The very paradigm they operate under (a centralized database and data pull) is inherently flawed when working with these kinds of data sets. Furthermore, they are fairly inflexible when considering the almost daily changing requirements of engineers and developers. Generating customized graphs for one-off views of interesting metrics is difficult at best.

At my employer we currently monitor about 18,000 hosts and the number is constantly growing. Centralized polling systems like cacti and munin are in use but only on subsets of hosts for the very reasons already stated. Try plugging 10,000 devices into cacti and it will die in a fire pretty quickly no matter how good the hardware is. Some modest numbers:

18000 hosts
10 metrics per host
10 second collection interval
13 months retention
------------------------------
606,528,000,000 data points

Approximately six hundred billion data points to store, index, search, and somehow render on demand. Using RRD type databases (as cacti and other tools do) you can get that number way down if you want to sacrifice granularity on older data points. Lets assume that will decrease our data set by two orders of magnitude. Thats still 6,000,000,000 data points. No small challenge.

The Goal

The ideal statistics collection system would be completely distributed. Independent collection agents run on each host using data push, or dedicated hosts use data pull where that isnt possible (SNMP devices). Those agents send metrics into a data store that has no central controller, is distributed with no single point of failure, replicates and mirrors seamlessly, and scales linearly. To generate graphs the ideal tool would also work in a distributed fashion with no single point of failure and use a robust API to render graphs or serve up other data formats (JSON, CSV, etc.) on demand.

The Solution

There are many, many tools out there in the Open Source world for doing one or more of these things. Some scale, some dont, they have varying levels of maturity, and are written in a wide array and languages. You know the story.

All of the tools you’ll find in the Open Source world fall into one or more of the following five categories;

Collection
Transport
Processing
Storage
Presentation

Collection
collectd is particularly well suited to the role of collection. It is written in quite clean and well designed C. It uses a data push model and everything is done through plugins. There are hooks for perl, python, and other languages. As an added bonus it can push nagios alerts too.

Transport
While collectd is perfectly capable of sending data to multicast addresses or even relational databases that isnt a good fit for this problem. The primary concern is that a pool of hosts running carbon-cache may or may not receive the data. This creates consistency issues across the data store. While pushing the data into, say, cassandra would be pretty elegant here there is nothing readily written to do that. It’s also an open question whether the chosen front end can interface well with the cassandra DB. A more straightforward solution in this case would be to use a message queuing system like RabbitMQ. All the collectd daemons push their data to RabbitMQ which then decides which carbon-cache hosts to push given metrics to. You can then make your own decisions on replication logic, datacenter boundaries, etc.

Storage
There is a great application stack out there that handles storing and retrieving metrics from collectd called graphite. The storage part of that stack is handled by carbon and whisper. Carbon collects inbound data and writes it to whisper databases which are quite similar in structure to RRD. While carbon has mechanisms to handle replication and sharding built into it, using a message queue is a more elegant solution and offers some interesting advantages for data pool consistency.

Presentation
Graphite is the obvious choice here. It requires no pre-existing knowledge of the data sets it is going to graph. It has a simple URL API for graph rendering or retrieving data via raw, JSON, CSV, etc. It also allows you to apply useful functions against the rendered data set.

Conclusions

Using the above stack provides a linearly scaleable, cross-datacenter solution to collecting, storing, and demand-fetching very large numbers of metrics for any operational use. A pilot installation is being turned up as I write this. I will come back with updates and more detailed information if things deviate greatly.

Other Interesting Tools
OpenTSDB
reconnoiter
esper
d3
statsd
hadoop
hbase

1. I no longer have access to Mac OS X 10.4 running on an Intel chip
2. I am actively transitioning to BackupPC

Below youll find the method I have been using to construct all the .pkg installers youll find on this site. If you have any questions about the process please post them in the comments.

You will need at least three installations of Mac OS X – one 10.4 on PPC, one 10.4 on Intel, and one 10.5 on any architecture. On each 10.4 installation, download the bacula source code, decompress it, and run the following command to compile it in a way that plays nice with the rest of Mac OS X

./configure --prefix=/usr/local --sysconfdir=/usr/local/etc \
--mandir=/usr/local/share/man --enable-client-only \
--with-openssl --with-tcp-wrappers --disable-dependency-tracking \
--with-working-dir=/var/bacula/working
make

With that done, copy the compiled binaries from each architecture to your Mac OS X 10.5 installation to build the pkg installer. The files in question are;

• src/filed/bacula-fd
• src/filed/bacula-fd.conf
• src/console/bconsole
• src/console/bconsole.conf

I use the architecture type to give them unique names… bacula-fd-ppc and bacula-fd-intel for example. After thats done, use lipo to combine the two binaries into one fat multi-arch binary

lipo -create bacula-fd-intel bacula-fd-ppc -output bacula-fd

Now do the same for bconsole

lipo -create bconsole-intel bconsole-ppc -output bconsole

Having done that you now need to set up a mock directory structure to place the binaries into. You can then use PackageMaker on Mac OS X to build the pkg installer from that mock directory. My directory looks like this;

Library
Library/LaunchDaemons
Library/LaunchDaemons/bacula-fd.plist
usr
usr/local
usr/local/bin
usr/local/bin/bconsole
usr/local/etc
usr/local/etc/bacula-fd.conf
usr/local/etc/bconsole.conf
usr/local/sbin
usr/local/sbin/bacula-fd
usr/local/share
usr/local/share/man
usr/local/share/man/man8
usr/local/share/man/man8/bacula-fd.8.gz
usr/local/share/man/man8/bconsole.8.gz
var
var/bacula
var/bacula/working

You can see where I’ve placed bacula-fd and bconsole as well as their conf files. There are some gzipped man pages in there too, which you can grab out of the compiled source tree and gzip manually. Finally there is a .plist file which is required to make auto startup/shutdown of bacula-fd happen on Mac OS X.

The last important bit to note is that I use some modified .conf files which have special placeholders for host names. I then use pre-inst and post-inst scripts in PackageMaker to look for previous versions of packages, insert the current hostname into the .conf files, etc. I have included my source tree and PackageMaker files and scripts below for the last version I released (2.4.4) for your reference.

After all the above is done and the directory structure is all set up you should be able to just fire up PackageMaker, change a few variables in it (set permissions, set version numbers, make sure proper pre and post-inst scripts are being used, etc.) and then build the pkg installer.

Enjoy :)

bacula_244_macosx_source.zip

find . -type f -exec test -s {}/..namedfork/rsrc \; -print

Extract the contents of the image

gunzip < your-initrd.img | cpio -i --make-directories

Now make your edits and then repackage the initrd

find . | cpio -o -H newc | gzip -9 > your-new-initrd.img

Now, there are quite a few ways around this problem. You can install to a ‘real’ computer and migrate the installation to a network block device. You can use debootstrap in place of the actual Debian installation system. You can use a combination of these two methods, NFS root filesystems, TFTP hacks, etc. All of these solutions are lacking in my opinion. I want to run the ‘real’ debian installer against a network block device and boot my physical hardware using only the built in PXE booting capability of the BIOS.

Taking all these issues as a personal challenge, I’ve outlined below how to go about using the regular old Debian Lenny installer directly against an AoE block device.

First of all, youll need the full installer disc 1 for Lenny. We need one of the .deb’s thats on that CD. The netinst or businesscard installers will not work with the steps below. (NOTE: If you want to be really creative, I suppose you could download the .deb from within the installer system via wget or even get it before hand and put it on a thumb drive or something…)

1. Boot the lenny install cd just to the point that /cdrom is mounted. For me, that was the “Configure the network” prompt.

2. Switch to a virtual terminal by hitting alt-f2.

3. Unpack the .deb that contains the aoe kernel module

udpkg --unpack /cdrom/pool/main/l/linux-2.6/linux-image-2.6.26-2-486_2.6.26-15_i386.deb

4. Now load the kernel module we just unpacked

insmod /lib/modules/2.6.26-2-486/kernel/drivers/block/aoe/aoe.ko

5. Now we create a symlink to the AoE block device. Its important that we use a symlink at this point in the process since it will cause the installer to set up a lot of paths in the correct fashion.

ln -s /dev/etherd/eX.X /dev/sda

6. Hit alt-f1 and continue with installation until the point that grub fails to install, then switch back to terminal 2.

7. Now chroot into the mostly installed system

chroot /target

8. Install the aoetools package

apt-get -y install aoetools

9. Listed below are two scripts that youll need to get into your chroot environment. They are used to build a new custom initrd image. I find it easiest to put them on a webserver somewhere and wget them.

wget http://www.pixelchaos.net/stuff/aoe/hooks/aoetools -O /etc/initramfs-tools/hooks/aoetools
chmod 755 /etc/initramfs-tools/hooks/aoetools
wget http://www.pixelchaos.net/stuff/aoe/scripts/local-top/aoetools -O /etc/initramfs-tools/scripts/local-top/aoetools
chmod 755 /etc/initramfs-tools/scripts/local-top/aoetools

10. Now we regenerate the initrd image

update-initramfs -u -k all

11. Now that all the above is done, we can manually install grub – which the debian installer was unable to do.

apt-get -y install grub
mkdir /boot/grub
cp -r /usr/lib/grub/i386-pc/. /boot/grub

12. Now we can remove our old symlink and create some hard links instead. This is required to allow grub to modify the MBR of our block device.

rm /dev/sda
ln /dev/etherd/eX.X /dev/sda
ln /dev/etherd/eX.Xp1 /dev/sda1
ln /dev/etherd/eX.Xp5 /dev/sda5

13. Now update grubs device.map

echo "(hd0) /dev/sda" > /boot/grub/device.map

14. Now run “grub” and issue the following commands

device (hd0) /dev/sda
root (hd0,0)
setup (hd0)

15. Now issue the “update-grub” command which will generate /boot/grub/menu.lst for the first time

16. Remember that grub has been using the hard link to /dev/sda this whole time. Change the root=/dev/sda1 line in /boot/grub/menu.lst to root=/dev/etherd/eX.Xp1, then issue the “update-grub” command again to commit the change to the boot stanzas.

17. For the sake of completeness, lets change grubs device.map back to its original contents.

echo "(hd0) /dev/etherd/eX.X" > /boot/grub/device.map

18. Get out of the chroot environment by issuing the “exit” command.

19. Go back to installer and choose “Continue without boot loader” to finish the installation and reboot the computer.

Thats it! Assuming you have properly set up a gPXE boot environment everything should “just work”. Setting up gPXE is outside the scope of this document but some good notes are available on the etherboot web site (see the links below).

Please note that there is one caveat to this whole process. Grub does not support installation onto AoE block devices. There very well may be some issues the next time you try to upgrade your kernel if some post-inst script calls grub-install against your root device. Keep that in mind.

aoe-hooks script

#!/bin/sh

set  -e

PREREQ=""

prereqs()
{
echo "$PREREQ"
}

case $1 in
prereqs)
prereqs
exit 0
;;
esac

. /usr/share/initramfs-tools/hook-functions

[ -x /sbin/aoe-discover ] && copy_exec /sbin/aoe-discover /sbin
manual_add_modules aoe

aoe-local-top script

#!/bin/sh

set -e

PREREQ="udev"

prereqs()
{
echo "$PREREQ"
}

case $1 in
prereqs)
prereqs
exit 0
;;
esac

case $ROOT in
/dev/etherd/e*)
INTERFACES=`sed -ne '/eth.*:/{s/:.*$//;p;}' < /proc/net/dev`
for i in $INTERFACES; do
echo Bringing up interface $i for AoE
ifconfig $i up
done
# Make sure udev has processed all events from adding the NIC
# modules before loading aoe
[ -x /sbin/udevsettle ] && /sbin/udevsettle --timeout=30
modprobe aoe
# Wait until aoe device files have been generated.
[ -x /sbin/udevsettle ] && /sbin/udevsettle --timeout=30
aoe-discover
;;
esac

references
http://maht0x0r.blogspot.com/2009/03/installing-debian-5-for-migration-to.html
http://www.etherboot.org/wiki/sanboot/debian_and_ubuntu
http://mike.neir.org/wiki/articles/AoE_Root_Fedora

After some testing I’m happy to say it appears to work smashingly. What follows is a somewhat more complete version of how to achieve the same results on Debian Lenny. Enjoy :)

The first thing that needs to be done is to get the debian sources for clvm and modify then to use openais. After that we will recompile new packages from that source, then set up openais on our cluster nodes.

Install all the dependencies we need to compile clvm

root@host:~# apt-get build-dep clvm
root@host:~# apt-get install libopenais-dev

Now download the source files and cd into our working directory

root@host:~# cd /usr/src/
root@host:/usr/src# apt-get source clvm
root@host:/usr/src# cd lvm2-2.02.39

Now we’ll modify a few files in the source:

• The first is debian/clvm.init. You’ll need to remove any references to cman or cluster.conf. You can download an already edited version here.
• The next is debian/control. Modify the dependies (lvm2 without the version number, openais in place of cman) and modify the comments accordingly. A pre-edited version is here.
• The last file is debian/rules. Replace cman by openais in the configure options, and add the PATH where to find the openais libs. Again, a pre-made version is here.

For clarities sake, here is the actual code block from debian/rules:

$(STAMPS_DIR)/setup-deb: SOURCE_DIR = $(BUILD_DIR)/source
$(STAMPS_DIR)/setup-deb: DIR = $(BUILD_DIR)/build-deb
$(STAMPS_DIR)/setup-deb: $(STAMPS_DIR)/source
        rm -rf $(DIR)
        cp -al $(SOURCE_DIR) $(DIR)
        cd $(DIR); \
        ./configure CFLAGS="$(CFLAGS)" \
		LDFLAGS="-L/usr/lib/openais" \
                $(CONFIGURE_FLAGS) \
                --with-optimisation="" \
                --with-clvmd=openais \
                --enable-readline
        touch $@

The last thing to do is update the internal version number of the clvm package and add some comments to the changelog:

root@host:/usr/src/lm2-2.02.39# dch -i

Now go ahead and compile the package:

root@host:/usr/src/lm2-2.02.39# dpkg-buildpackage -rfakeroot -uc -b

After the compliation completes you should have some shiny new .deb files in /usr/src. The one we are interested in is clvm_2.02.39-7.1_i386.deb (the actual version of yours may vary depending on what you put in the debian changelog in the previous step).

So now that weve got our custom version of clvm compiled, its time to move on the cluster nodes. On each node in the cluster, do the following…

Install openais and add a user for it:

root@node:~# apt-get install openais
root@node:~# mkdir -p /etc/ais
root@node:~# adduser --no-create-home --disabled-password --disabled-login --gecos openAIS ais

Now create the following config in /etc/ais/openais.conf. This is the most basic config you can have. All you need to do is set 192.168.1.0 to be your actual network address.

totem {
        version: 2
        secauth: off
        threads: 0
        interface {
                ringnumber: 0
                bindnetaddr: 192.168.1.0
                mcastaddr: 226.94.1.1
                mcastport: 5405
        }
}

openais does not include a proper debian init sript so you can download one here and save it as /etc/init.d/openais. After that is done, add it to the proper runlevels by issuing:

root@node:~# update-rc.d openais start 62 S . start 50 0 6 .

Now we can install our custom deb and turn on LVM clustering

root@node:~# dpkg -i clvm_2.02.39-7.1_i386.deb
root@node:~# sed -i 's/^    locking_type = 1$/    locking_type = 3/' /etc/lvm/lvm.conf

Thats it! Reboot your cluster nodes and they should all be cluster aware now :)

See the lkml.org mailing list post here

… and the townspeople rejoiced :)

The first thing youll need is a TFTP server. This will serve out the kernel and initrd images to clients on the network.

root@pxe:~$ apt-get install tftpd-hpa

That will install a good tftp server daemon and create the tftp root for you in /var/lib/tftpboot. Some HOWTO’s on the Internet suggest installing the syslinux package as well which provides a needed file (pxelinux.0). I prefer to download the entire netboot environment for debian since this will also provide a network bootable installer for Debian on top of the other goodness we are creating today.

root@pxe:~$ cd /var/lib/tftpboot
root@pxe:~$ wget http://ftp.debian.org/debian/dists/lenny/main/installer-i386/current/images/netboot/netboot.tar.gz
root@pxe:~$ tar xzvf netboot.tar.gz

The kernel and initrd you are going to use to boot the mythbox must reside on the TFTP server. Furthermore, the initrd must be updated to support NFS root. On the mythbox I was migrating to the NFS root, it looked like this:

root@mythbox:~$ sed -i 's/BOOT=local/BOOT=nfs/'  /etc/initramfs-tools/initramfs.conf
root@mythbox:~$ /usr/sbin/mkinitramfs -o /boot/initrd.img-2.6.26-1-686_netboot

Now just copy the /boot/vmlinuz-2.6.26-1-686 and /boot/initrd.img-2.6.26-1-686_netboot to /var/lib/tftpboot/ on the TFTP server.

Last step for the TFTP server is to create a config file for our mythbox. If you dont create one specifically for it, it will instead boot the network installer as mentioned above. Create the file /var/lib/tftpboot/pxelinux.cfg/01-$(hardware ethernet address). Including the hardware address in the file name binds that config to the particular workstation we are working on today. In my case, the file name is 01-00-19-db-4a-6a-0e.

LABEL linux
KERNEL vmlinuz-2.6.26-1-686
APPEND root=/dev/nfs initrd=initrd.img-2.6.26-1-686_netboot nfsroot=192.168.1.108:/srv/nfs/mythbox.pixelchaos.net ip=dhcp rw

Now that our TFTP server is up and running we will next need to configure a DHCP server to provide IP addresses to netboot clients, as well as point them to the TFTP server to continue the boot process.

root@dhcp:~$ apt-get install dhcp3-server

That will install the DHCP service for you. Now we just need to add one little section to /etc/dhcp3/dhcpd.conf for the workstation we are going to netboot.

host mythbox {
hardware ethernet 00:19:db:4a:6a:0e;
fixed-address mythbox.pixelchaos.net;
filename "pxelinux.0";
next-server 192.168.1.115;
}

host mythbox: mythbox is just the descriptive name for this configuration block .You can use any name you like.
hardware ethernet 00:19:db:4a:6a:0e: This is the hardware ethernet address of the workstation we are netbooting.
fixed-address mythbox.pixelchaos.net: This tells the DHCP server what address to assign the mythbox. You can use a FQDN (if its resolvable!) or an IP address.
filename “pxelinux.0″: This is the PXE boot loader file that we installed onto our TFTP server earlier.
next-server 192.168.1.115: This is the IP address of our TFTP server. It tells the mythbox where the “next server” is after it gets its IP address from the DHCP server. The “next server” can be any machine on the network. It can even be the same one running DHCP.

Once all that is done, restart your DHCP server.

root@dhcp:~$ /etc/init.d/dhcp3-server restart

Now for the third piece of the puzzle, our NFS root filesystem. You’ll first need to create a directory to share.

root@nfs:~$ mkdir /srv/nfs/mythbox.pixelchaos.net

Now add that directory to /etc/exports so that it gets shared via NFS with this line:

/srv/nfs/mythbox.pixelchaos.net mythbox(rw,async,no_root_squash)

The rw, async, and no_root_squash options are important. Dont omit them. At this point we can reload all our NFS exports to active this new share point.

root@nfs:~$ exportfs -rv

At this point the entire TFTP, DHCP, and NFS parts of the environment are ready to go. Now we will migrate our existing installation of Linux from the mythbox onto its new nfsroot. First we make a directory to mount the NFS share on and then mount it.

root@mythbox:~$ mkdir /mnt/nfs
root@mythbox:~$ mount -t nfs -o nolock 192.168.1.108:/srv/nfs/mythbox.pixelchaos.net /mnt/nfs

Now we run a cp command on every locally mounted partition, as well as /dev. On my mythbox it looked like this:

root@mythbox:~$ cp -axv /. /mnt/nfs/.
root@mythbox:~$ cp -axv /boot/. /mnt/nfs/boot/.
root@mythbox:~$ cp -axv /home/. /mnt/nfs/home/.
root@mythbox:~$ cp -axv /dev/. /mnt/nfs/dev/.

The last thing to do is edit two files that we copied up to /mnt/nfs. The first is /mnt/nfs/etc/network/interfaces. You’ll need to remove any lines that bring the interface up automatically. You want to remove anything like “allow-hotplug eth0″ and just leave a single line like “iface eth0 inet dhcp”. The network interface will have already been brought up by netboot normally and leaving in any “hotplug” or “auto” lines will bring it up again, which has the effect of destroying it.

Now we remove all the old filesystems from /mnt/nfs/etc/fstab so that it looks like this:

# /etc/fstab: static file system information.
#
# <file system> 		<mount point>   	<type>  <options>       <dump>  <pass>
/dev/nfs        /               nfs     defaults 0 0
none            /tmp            tmpfs   defaults 0 0
none            /var/run        tmpfs   defaults 0 0
none            /var/lock       tmpfs   defaults 0 0
none            /var/tmp        tmpfs   defaults 0 0
none            /media          tmpfs   defaults 0 0

That should be it. Now just set the mythbox to use network booting in its BIOS and reboot.

Cool, huh? :)

Enjoy!
bacula-client-244-1