pixelchaos.net

Recently I came upon the need to do all my network routing and firewalling inside a Xen domU. I am not the first to do this but I thought I’d do a little write up on it to help others trying to accomplish the same thing in Debian.

The idea here is to end up with (at least) two VLANs on the network with the dom0 and domU’s being able to choose one or both networks on which to exist. In the case of both, you can set up a handy domU firewall/gateway :)

As you can see from the diagram above, we will end up with three bridges in the dom0 with all the appropriate glue to tie everything together. Best of all, this is all assembled on the fly during bootup.

Now, you obviously need to have a Xen dom0 set up and running properly. Once you are at that point, adding VLAN bridging is fairly straightforward:

First we need to install the vlan package to get the tools for manipulating vlans. Crazy, huh?

apt-get install vlan

Now we set up the interfaces we will need in /etc/network/interfaces. The seasoned admins out there will note that we only bring up the eth0 interface automatically but leave it unconfigured. It will not have an IP or any of the other usual information. The other two (VLAN) interfaces are defined but are NOT brought up during boot. This is important!

In this example the “vlan100″ interface is the replacement for your old “eth0″ interface. Its the interface you’ll be able to reach the dom0 on from the internal VLAN. The “vlan99″ interface is defined but is left unconfigured since there will be no publicly routeable path to the dom0.

auto eth0
iface eth0 inet manual
        pre-up ifconfig $IFACE up
        post-down ifconfig $IFACE down

iface vlan99 inet manual
        mtu 1500
        vlan_raw_device eth0

iface vlan100 inet static
        address 192.168.1.100
        netmask 255.255.255.0
        network 192.168.1.0
        broadcast 192.168.1.255
        gateway 192.168.1.1
        dns-nameservers 192.168.1.254
        dns-search pixelchaos.net
        mtu 1500
        vlan_raw_device eth0

Now we create a new network script for xen located at /etc/xen/scripts/network-multi:

#!/bin/sh
#============================================================================
# Xen vlan bridge start/stop script.
# Xend calls a network script when it starts.
# The script name to use is defined in /etc/xen/xend-config.sxp
# in the network-script field.
#
# This script creates multiple bridges to segregate individual domUs to
# separate VLANs. Customize to fit your needs.
#
# Usage:
#
# network-multi (start|stop|status)
#
#============================================================================

dir=$(dirname "$0")

##
# To make the tagged interface available to some DomUs, create the default
# bridge. Comment this out to only make vlan-based bridges available.
"$dir/network-bridge" "$@" vifnum=0 netdev=eth0 bridge="xbr_trunk"

# Once the "normal" bridge is active, bring up the vlan interfaces.
# They will be attached to the virtual eth0 interface, and not the
# physical one.
ifup vlan99
ifup vlan100

# Create any vlan-based briges.
"$dir/network-bridge" "$@" vifnum=1 bridge="xbr_vl99" netdev="vlan99"
"$dir/network-bridge" "$@" vifnum=2 bridge="xbr_vl100" netdev="vlan100"

# bring up the interfaces again? why do we need to do this? they should have been brought back up by the previous network-bridge call…
ip link set vlan100 up
route add default gw 192.168.1.1 vlan100

That script basically just calls the standard “network-bridge” script supplied with Xen three times – once for each VLAN we are setting up. You’ll notice some interesting bits at the end – We call “ip link” and “route” to bring the vlan100 interface up again. I’m not sure yet why the previous “network-bridge” call doesnt do that on its own… it should as far as I know but I havent dug into why it doesnt. This is just a quick hack to bring “vlan100″ back up again and fix the routes.

Go ahead and add the call to that script in the place of network-bridge in /etc/xen/xend-config.sxp.

Now we need to do one final thing to another Xen script. By default, the xen-network-common.sh script calls “ifdown” directly to take an interface down. Unfortunately, this has the undesirable side effect of destroying any VLANs associated with that interface. This causes all kinds of problems when we call our “network-multi” script so we need to patch /etc/xen/scripts/xen-network-common.sh with the following section:

# do not call ifdown directly
  ifdown()  {
    ip addr flush $1
    ip link set $1 down
    true
  }

To see exactly where that chunk of code goes you can download my already modified script here, or get a diff here if you prefer.

After youve done all that, reboot your dom0. You should now be up and running on the vlan100 interface. The output of “brctl show” should show this:

root@atlas:~# brctl show
bridge name  bridge id          STP enabled    interfaces
xbr_trunk    8000.feffffffffff  no             vif0.0
                                               peth0
xbr_vl100    8000.feffffffffff  no             vif0.2
                                               pvlan100
xbr_vl99     8000.feffffffffff  no             vif0.1
                                               pvlan99

If so, you should be all ready to go. Now edit the vif line in your config file for a given domU to link its ethernet interface to a bridge. My domU firewall is linked to two bridges so it looks like this:

vif= [ 'mac=00:16:3E:44:F9:9C, bridge=xbr_vl99', 'mac=00:16:3E:41:16:51, bridge=xbr_vl100' ]

Thats it! Easy, huh?

Many, many thanks go to Mike Neir for holding my hand through figuring this all out since he’d already set up a functional system that is very similar. Thanks also go to firstserved.net for the ifdown patch and renial.net for other ideas on ways to set up VLAN bridging in Xen.

This entry was posted on Wednesday, July 16th, 2008 at 8:47 pm and is filed under Debian, HowTo, Networking, Xen. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.