For the impatient, the important bit is this:

Well, if your disk image file happens to contain partitions its a little bit trickier. Here is what you need to do:

First you attach the disk image to the first available loopback device

Next, we get a listing of the partition table it contains:

Now for the juicy part. You attach one the partitions contained in the disk image to yet another loopback device by specifying the offset of the partition.

The -o option to losetup tells it the number of bytes to skip at the beginning of the file. To get that number, you multiply the number of bytes per unit (in this case, 512) by the ‘Start” value of the parititon (in this case, 62). So our -o value will be 31744.

Looks good so far. Now mount it.

Thats it. You should now be able to get at all the files on that partition.

Setting the device up to work with Debian Etch is fairly straightforward once you know what to do

Obviously, you need a motherboard with built-in Bluetooth or a USB Bluetooth receiver. Once you have that, lets install the software you’ll need:

Now we need to figure out the Bluetooth address of the keyboard:

Now that you have the Bluetooth address of the keyboard, lets modify the following lines in /etc/default/bluetooth:

Next you’ll add a new section to /etc/bluetooth/hcid.conf for the keyboard:

At this point you can pair the keyboard. Since I use fluxbox on my MythTV frontend computers I manually start the required applets, etc.

1. Start window manager
2. As the logged in user, run “bluetooth-applet”
3. Power cycle the Bluetooth keyboard
4. As root, execute “hidd –connect $(keyboard addr)
5. Type a PIN number on bluetooth keyboard (i.e. 1234) then hit return
6. The bluetooth applet will start flashing in the window manager.
7. Click the bluetooth applet and enter the same PIN in the dialog that pops up

That should be it. You now have a working wireless keyboard :-)

If anyone knows of a way to do this entirely on the command line, bypassing the window manager and bluetooth-applet entirely, I’d love to hear it.

Enjoy!

Our test system, for those who dont follow this blog regularly, is a Supermicro X7DVL-E motherboard with a single Intel Core2 Xeon 5130 @ 2Ghz with 2GB of RAM. The disks used are Seagate ES.2 1TB SATA-II models hooked up to two Supermicro AOC-SAT2-MV8 8-port PCI-X SATA controllers. More complete specs are outlined in my original post where I was building the chassis, which can be found here.

The Linux distribution used is, of course, Debian :-) In this case, its Debian 5.0 (Lenny) running kernel 2.6.26-1-686.

Test #1 - Individual Disks
In order to get some baseline numbers to compare against subsequent tests we will first check performance of all the attached disks individually…

Which yielded these results…

So our average raw device read speed is 110.5MB/s and writes are 109.5MB/s.

Test #2 - Bonnie++ on a single disk
Next we will do a bonnie++ test on a single disk (5QJ06913), formatted with default ext3 options

This test produced average block reads of 105.55MB/s @ 11.4% CPU usage, block writes of 65.93MB/s @ 19%, and block rewrites of 36.95MB/s @ 8%.

Test #3 - RAID1 pair
For this test we use a RAID1 pair with one disk on each controller, formatted with default ext3 options

This produced raw I/O reads of 111MB/s and writes of 107MB/s. Now for some bonnie++ tests…

This test produced average block reads of 103.43MB/s @ 14.4% CPU usage, block writes of 60.84MB/s @ 18.6%, and block rewrites of 33.82MB/s @ 8.8%.

Test #4 - RAID10
For this test we use a RAID10 device consisting of two RAID1 devices, each with a disk on each controller.

This produced raw I/O reads of 219MB/s and writes of 207MB/s. A pretty significant improvement.

This test produced average block reads of 198.91MB/s @ 27.8% CPU usage, block writes of 132.4MB/s @ 44%, and block rewrites of 61.75MB/s @ 14.4%.

Test #5 - RAID5
For this test we will use a RAID5 device with the first two disks on one controller and the third on the second controller

This produced raw I/O reads of 67.3MB/s and writes of 130MB/s.

This test produced average block reads of 62.91MB/s @ 8.6% CPU usage, block writes of 65.73MB/s @ 19.6%, and block rewrites of 33.14MB/s @ 7.6%.

Those were all the RAID types I was inclined to test. I realize I left out RAID0 and RAID6. For the application I have in mind, RAID10 is the clear winner. You can download a spreadsheet with the actual bonnie++ output here. Enjoy!

Enjoy!
bacula-client-242-1

The idea here is to end up with (at least) two VLANs on the network with the dom0 and domU’s being able to choose one or both networks on which to exist. In the case of both, you can set up a handy domU firewall/gateway :)

As you can see from the diagram above, we will end up with three bridges in the dom0 with all the appropriate glue to tie everything together. Best of all, this is all assembled on the fly during bootup.

Now, you obviously need to have a Xen dom0 set up and running properly. Once you are at that point, adding VLAN bridging is fairly straightforward:

First we need to install the vlan package to get the tools for manipulating vlans. Crazy, huh?

Now we set up the interfaces we will need in /etc/network/interfaces. The seasoned admins out there will note that we only bring up the eth0 interface automatically but leave it unconfigured. It will not have an IP or any of the other usual information. The other two (VLAN) interfaces are defined but are NOT brought up during boot. This is important!

In this example the “vlan100″ interface is the replacement for your old “eth0″ interface. Its the interface you’ll be able to reach the dom0 on from the internal VLAN. The “vlan99″ interface is defined but is left unconfigured since there will be no publicly routeable path to the dom0.

Now we create a new network script for xen located at /etc/xen/scripts/network-multi:

That script basically just calls the standard “network-bridge” script supplied with Xen three times - once for each VLAN we are setting up. You’ll notice some interesting bits at the end;

1. We call “ip link” and “route” to bring the vlan100 interface up again. I’m not sure yet why the previous “network-bridge” call doesnt do that on its own… it should as far as I know but I havent dug into why it doesnt. This is just a quick hack to bring “vlan100″ back up again and fix the routes.
2. We call two init.d scripts to restart both the iscsi-target service, as well as open-iscsi. I use iSCSI on my Xen dom0’s to access the filesystems of domU’s. If you do that too, youll need to restart the service here since it fails when the dom0 initially boots (since there is no active network interface, remember?)

Go ahead and add the call to that script in the place of network-bridge in /etc/xen/xend-config.sxp.

Now we need to do one final thing to another Xen script. By default, the xen-network-common.sh script calls “ifdown” directly to take an interface down. Unfortunately, this has the undesirable side effect of destroying any VLANs associated with that interface. This causes all kinds of problems when we call our “network-multi” script so we need to patch /etc/xen/scripts/xen-network-common.sh with the following section:

To see exactly where that chunk of code goes you can download my already modified script here, or get a diff here if you prefer.

After youve done all that, reboot your dom0. You should now be up and running on the vlan100 interface. The output of “brctl show” should show this:

If so, you should be all ready to go. Now edit the vif line in your config file for a given domU to link its ethernet interface to a bridge. My domU firewall is linked to two bridges so it looks like this:

Thats it! Easy, huh?

Many, many thanks go to Mike Neir for holding my hand through figuring this all out since he’d already set up a functional system that is very similar. Thanks also go to firstserved.net for the ifdown patch and renial.net for other ideas on ways to set up VLAN bridging in Xen.

First of all, install open-iscsi

Now compile and install iscsi-target

And now the important bits. iscsi-target needs to be added to the sysv init scripts in the correct places, which a source install obviously doesnt do:

That will make it start just before open-iscsi and stop just after it. Thats it! Any iscsi target that you configure with with the “automatic” option should be correctly connected to during bootup

Please report any issues and I will resolve them ASAP. Thanks and enjoy!

bacula-client-240-1

Thanks go to Mark Lord, Marvell Corporation, EMC Corporation and Red Hat, Inc. for the coding magic.

In keeping with my character, the first thing I did was start all over again from scratch by reinstalling the operating system. This time around I set up /dev/md0 as /boot (255 MB) and /dev/md1 as an LVM physical volume (the remainder of the disk), within which /, /home, /usr and friends reside as logical volumes. Its something I’ve wanted to start doing with all my systems for a long time now and shouldn’t have any bearing on the performance tests we are about to do.

Regardless of which protocol will be used we need to enable jumbo frames on all the involved devices. For my setup that means the target (stor01), the initiator (node02), and the switch (a Cisco Catalyst 2970).

First, we turn on jumbo frames for gigabit ethernet at the switch. Beware that this requires a reset (aka reboot) of the switch to take effect:

Now we enable an MTU of 9000 on both the target and the initiator:

For the sake of comparison, here is an iperf test done between the target and initiator with the standard MTU of 1500, and then with an MTU of 9000:

As you can see, just enabling jumbo frames produces a raw throughput increase of 17.43%. Nothing to sneeze at.

At this point I tried enabling flow control on the catalyst switch (it is already enabled for both send and receive by default in the e1000 driver) but it did not have any effect on iperf numbers. I turned it back off for now.

So now we set up a 20GB LVM volume on the target and export it using vblade to be mounted on the initiator. We then run a simple dd test to check throughput:

CPU load on the target was 10-15% during the dd operation. Now we try writing direct to the (unmounted) block device to rule out any performance penalties of the filesystem itself…

CPU usage was slightly higher in that test, running 15-20%. So some slight difference but nothing to be too concerned about.

Now we take that same LVM device and share it via iSCSI for the same dd tests:

CPU load was 6-8% during that test. We also run that same test with flow control enabled at the switch:

Essentially the same…

Now this raises the question of why AoE is so much slower than iSCSI on an essentially default install of Debian Etch. To AoE’s credit, many people report getting just as good (50MB/s or better) of performance from AoE on their systems as I’m seeing with iSCSI. I spent quite a large amount of time playing with flow control, kernel ring buffer values, filesystem options, etc. and was unable to determine why performance is so terrible for me. I did find a pretty high number (half a dozen at least) of recent posts to the AoE mailing list by other people having essentially identical problems so I’m certainly not alone. In the interest of completing my testing, I’ve decided to move forward with iSCSI.

Now we try reformatting with the stride option to mkfs:

The results of several more tests are shown here…

So we don’t see any appreciable difference when using a combination of the stride= option and flow control, at least with a simple dd test.

Next we turn flow control back off, and reformat again without the stride= option. We are now back to our baseline setup for a new test with bonnie++.

This test produced block writes of about 82MB/s and block reads of about 37MB/s. The cause for the difference in write speed between the dd and bonnie++ tests is still unclear to me. There also appears to be a known issue where writes are much faster than reads which is apparently due to interrupt handling. This is further evidenced by running a quick dd test that does a read instead of a write:

As you can see, read operations are about 20% slower than write operations which goes against common thinking with regard to stripped disk arrays.

So there you have it. In my particular situation, with no tuning/optimizing done, iSCSI performs much better than AoE. Even in the event that I were to go to the trouble to performance tune AoE and get it as good as, or even better than, iSCSI I would still be inclined to standardize around iSCSI. Authentication, routing, NAT, etc. can all be done very easily on iSCSI protocol with all the standard TCP/IP tools that are out there. For me that’s a pretty big advantage.

Next up will be our final piece of the puzzle, left over from the initial system setup - getting hot swap working with the sata_mv module!