Extract the contents of the image

gunzip < your-initrd.img | cpio -i --make-directories

Now make your edits and then repackage the initrd

find . | cpio -o -H newc | gzip -9 > your-new-initrd.img

See the lkml.org mailing list post here

… and the townspeople rejoiced :)

The first thing youll need is a TFTP server. This will serve out the kernel and initrd images to clients on the network.

root@pxe:~$ apt-get install tftpd-hpa

That will install a good tftp server daemon and create the tftp root for you in /var/lib/tftpboot. Some HOWTO’s on the Internet suggest installing the syslinux package as well which provides a needed file (pxelinux.0). I prefer to download the entire netboot environment for debian since this will also provide a network bootable installer for Debian on top of the other goodness we are creating today.

root@pxe:~$ cd /var/lib/tftpboot
root@pxe:~$ wget http://ftp.debian.org/debian/dists/lenny/main/installer-i386/current/images/netboot/netboot.tar.gz
root@pxe:~$ tar xzvf netboot.tar.gz

The kernel and initrd you are going to use to boot the mythbox must reside on the TFTP server. Furthermore, the initrd must be updated to support NFS root. On the mythbox I was migrating to the NFS root, it looked like this:

root@mythbox:~$ sed -i 's/BOOT=local/BOOT=nfs/'  /etc/initramfs-tools/initramfs.conf
root@mythbox:~$ /usr/sbin/mkinitramfs -o /boot/initrd.img-2.6.26-1-686_netboot

Now just copy the /boot/vmlinuz-2.6.26-1-686 and /boot/initrd.img-2.6.26-1-686_netboot to /var/lib/tftpboot/ on the TFTP server.

Last step for the TFTP server is to create a config file for our mythbox. If you dont create one specifically for it, it will instead boot the network installer as mentioned above. Create the file /var/lib/tftpboot/pxelinux.cfg/01-$(hardware ethernet address). Including the hardware address in the file name binds that config to the particular workstation we are working on today. In my case, the file name is 01-00-19-db-4a-6a-0e.

LABEL linux
KERNEL vmlinuz-2.6.26-1-686
APPEND root=/dev/nfs initrd=initrd.img-2.6.26-1-686_netboot nfsroot=192.168.1.108:/srv/nfs/mythbox.pixelchaos.net ip=dhcp rw

Now that our TFTP server is up and running we will next need to configure a DHCP server to provide IP addresses to netboot clients, as well as point them to the TFTP server to continue the boot process.

root@dhcp:~$ apt-get install dhcp3-server

That will install the DHCP service for you. Now we just need to add one little section to /etc/dhcp3/dhcpd.conf for the workstation we are going to netboot.

host mythbox {
hardware ethernet 00:19:db:4a:6a:0e;
fixed-address mythbox.pixelchaos.net;
filename "pxelinux.0";
next-server 192.168.1.115;
}

host mythbox: mythbox is just the descriptive name for this configuration block .You can use any name you like.
hardware ethernet 00:19:db:4a:6a:0e: This is the hardware ethernet address of the workstation we are netbooting.
fixed-address mythbox.pixelchaos.net: This tells the DHCP server what address to assign the mythbox. You can use a FQDN (if its resolvable!) or an IP address.
filename “pxelinux.0″: This is the PXE boot loader file that we installed onto our TFTP server earlier.
next-server 192.168.1.115: This is the IP address of our TFTP server. It tells the mythbox where the “next server” is after it gets its IP address from the DHCP server. The “next server” can be any machine on the network. It can even be the same one running DHCP.

Once all that is done, restart your DHCP server.

root@dhcp:~$ /etc/init.d/dhcp3-server restart

Now for the third piece of the puzzle, our NFS root filesystem. You’ll first need to create a directory to share.

root@nfs:~$ mkdir /srv/nfs/mythbox.pixelchaos.net

Now add that directory to /etc/exports so that it gets shared via NFS with this line:

/srv/nfs/mythbox.pixelchaos.net mythbox(rw,async,no_root_squash)

The rw, async, and no_root_squash options are important. Dont omit them. At this point we can reload all our NFS exports to active this new share point.

root@nfs:~$ exportfs -rv

At this point the entire TFTP, DHCP, and NFS parts of the environment are ready to go. Now we will migrate our existing installation of Linux from the mythbox onto its new nfsroot. First we make a directory to mount the NFS share on and then mount it.

root@mythbox:~$ mkdir /mnt/nfs
root@mythbox:~$ mount -t nfs -o nolock 192.168.1.108:/srv/nfs/mythbox.pixelchaos.net /mnt/nfs

Now we run a cp command on every locally mounted partition, as well as /dev. On my mythbox it looked like this:

root@mythbox:~$ cp -axv /. /mnt/nfs/.
root@mythbox:~$ cp -axv /boot/. /mnt/nfs/boot/.
root@mythbox:~$ cp -axv /home/. /mnt/nfs/home/.
root@mythbox:~$ cp -axv /dev/. /mnt/nfs/dev/.

The last thing to do is edit two files that we copied up to /mnt/nfs. The first is /mnt/nfs/etc/network/interfaces. You’ll need to remove any lines that bring the interface up automatically. You want to remove anything like “allow-hotplug eth0″ and just leave a single line like “iface eth0 inet dhcp”. The network interface will have already been brought up by netboot normally and leaving in any “hotplug” or “auto” lines will bring it up again, which has the effect of destroying it.

Now we remove all the old filesystems from /mnt/nfs/etc/fstab so that it looks like this:

# /etc/fstab: static file system information.
#
# <file system> 		<mount point>   	<type>  <options>       <dump>  <pass>
/dev/nfs        /               nfs     defaults 0 0
none            /tmp            tmpfs   defaults 0 0
none            /var/run        tmpfs   defaults 0 0
none            /var/lock       tmpfs   defaults 0 0
none            /var/tmp        tmpfs   defaults 0 0
none            /media          tmpfs   defaults 0 0

That should be it. Now just set the mythbox to use network booting in its BIOS and reboot.

Cool, huh? :)

Well, if your disk image file happens to contain partitions its a little bit trickier. Here is what you need to do:

First you attach the disk image to the first available loopback device
losetup -f -v disk.img

Next, we get a listing of the partition table it contains:

fdisk -ul /dev/loop0

Device Boot Start End Blocks Id System
/dev/loop0p1 62 7998619 3999279 83 Linux
/dev/loop0p2 7998620 8386181 193781 82 Linux swap / Solaris

Now for the juicy part. You attach one the partitions contained in the disk image to yet another loopback device by specifying the offset of the partition.

The -o option to losetup tells it the number of bytes to skip at the beginning of the file. To get that number, you multiply the number of bytes per unit (in this case, 512) by the ‘Start” value of the parititon (in this case, 62). So our -o value will be 31744.

losetup -o 31744 -f -v /dev/loop0
Loop device is /dev/loop2

Looks good so far. Now mount it.

mount /dev/loop2 /mnt

Thats it. You should now be able to get at all the files on that partition.

EDIT 2009-01-13
Since writing this I have discovered that you can supply the ‘offset’ argument to mount directly and bypass the entire process of using loopback devices…

fdisk -ul disk.img # to get your offset value as noted above
mount -o loop,offset=31744 disk.img /mnt # to mount a given partition on /mnt


Our test system, for those who dont follow this blog regularly, is a Supermicro X7DVL-E motherboard with a single Intel Core2 Xeon 5130 @ 2Ghz with 2GB of RAM. The disks used are Seagate ES.2 1TB SATA-II models hooked up to two Supermicro AOC-SAT2-MV8 8-port PCI-X SATA controllers. More complete specs are outlined in my original post where I was building the chassis, which can be found here.

The Linux distribution used is, of course, Debian :-) In this case, its Debian 5.0 (Lenny) running kernel 2.6.26-1-686.

Test #1 – Individual Disks
In order to get some baseline numbers to compare against subsequent tests we will first check performance of all the attached disks individually…
for disk in $(ls /dev/disk/by-id/ | grep -v part | grep scsi-SATA_ST); do
echo $disk && \
dd if=/dev/zero of=/dev/disk/by-id/$disk bs=4M oflag=direct count=250 && \
dd if=/dev/disk/by-id/$disk of=/dev/null bs=4M count=250;
done

Which yielded these results…

5QJ06913 - read:109MB/s, write:109MB/s
5QJ09HZR - read:111MB/s, write:112MB/s
5QJ09JHQ - read:113MB/s, write:109MB/s
5QJ09JV3 - read:109MB/s, write:108MB/s


So our average raw device read speed is 110.5MB/s and writes are 109.5MB/s.

Test #2 – Bonnie++ on a single disk
Next we will do a bonnie++ test on a single disk (5QJ06913), formatted with default ext3 options

mkfs.ext3 /dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ06913
mount /dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ06913 /mnt
/usr/sbin/bonnie++ -q -d /mnt -s 4096 -x 5 -u root

This test produced average block reads of 105.55MB/s @ 11.4% CPU usage, block writes of 65.93MB/s @ 19%, and block rewrites of 36.95MB/s @ 8%.

Test #3 – RAID1 pair
For this test we use a RAID1 pair with one disk on each controller, formatted with default ext3 options

mdadm --create /dev/md2 --level=1 --raid-devices=2 /dev/sdf /dev/sdg
dd if=/dev/zero of=/dev/md2 bs=4M oflag=direct count=250
dd if=/dev/md2 of=/dev/null bs=4M count=250


This produced raw I/O reads of 111MB/s and writes of 107MB/s. Now for some bonnie++ tests…


mkfs.ext3 /dev/md2
mount /dev/md2 /mnt
/usr/sbin/bonnie++ -q -d /mnt -s 4096 -x 5 -u root


This test produced average block reads of 103.43MB/s @ 14.4% CPU usage, block writes of 60.84MB/s @ 18.6%, and block rewrites of 33.82MB/s @ 8.8%.

Test #4 – RAID10
For this test we use a RAID10 device consisting of two RAID1 devices, each with a disk on each controller.


mdadm --create /dev/md4 --level=0 --raid-devices=2 /dev/md2 /dev/md3
dd if=/dev/zero of=/dev/md4 bs=4M oflag=direct count=250
dd if=/dev/md4 of=/dev/null bs=4M


This produced raw I/O reads of 219MB/s and writes of 207MB/s. A pretty significant improvement.


mkfs.ext3 /dev/md4
mount /dev/md4 /mnt
/usr/sbin/bonnie++ -q -d /mnt -s 4096 -x 5 -u root


This test produced average block reads of 198.91MB/s @ 27.8% CPU usage, block writes of 132.4MB/s @ 44%, and block rewrites of 61.75MB/s @ 14.4%.

Test #5 – RAID5
For this test we will use a RAID5 device with the first two disks on one controller and the third on the second controller


mdadm --create /dev/md2 --level=5 --raid-devices=3 /dev/sdf1 /dev/sde1 /dev/sdg1
dd if=/dev/zero of=/dev/md2 bs=4M oflag=direct count=250
dd if=/dev/md2 of=/dev/null bs=4M


This produced raw I/O reads of 67.3MB/s and writes of 130MB/s.


mkfs.ext3 /dev/md2
mount /dev/md2 /mnt
/usr/sbin/bonnie++ -q -d /mnt -s 4096 -x 5 -u root


This test produced average block reads of 62.91MB/s @ 8.6% CPU usage, block writes of 65.73MB/s @ 19.6%, and block rewrites of 33.14MB/s @ 7.6%.

Those were all the RAID types I was inclined to test. I realize I left out RAID0 and RAID6. For the application I have in mind, RAID10 is the clear winner. You can download a spreadsheet with the actual bonnie++ output here. Enjoy!

Thanks go to Mark Lord, Marvell Corporation, EMC Corporation and Red Hat, Inc. for the coding magic.

In keeping with my character, the first thing I did was start all over again from scratch by reinstalling the operating system. This time around I set up /dev/md0 as /boot (255 MB) and /dev/md1 as an LVM physical volume (the remainder of the disk), within which /, /home, /usr and friends reside as logical volumes. Its something I’ve wanted to start doing with all my systems for a long time now and shouldn’t have any bearing on the performance tests we are about to do.

Regardless of which protocol will be used we need to enable jumbo frames on all the involved devices. For my setup that means the target (stor01), the initiator (node02), and the switch (a Cisco Catalyst 2970).

First, we turn on jumbo frames for gigabit ethernet at the switch. Beware that this requires a reset (aka reboot) of the switch to take effect:


c2970# system mtu jumbo 9000


Now we enable an MTU of 9000 on both the target and the initiator:


root@stor01:~# ifconfig bond0 mtu 9000



root@node02:~# ifconfig eth0 mtu 9000


For the sake of comparison, here is an iperf test done between the target and initiator with the standard MTU of 1500, and then with an MTU of 9000:


root@stor01:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 1.00 MByte (default)
------------------------------------------------------------
[ 4] local 65.171.150.4 port 5001 connected with 65.171.150.161 port 58731
[ 4] 0.0-10.0 sec 780 MBytes 654 Mbits/sec
[ 5] local 65.171.150.4 port 5001 connected with 65.171.150.161 port 58732
[ 5] 0.0-10.0 sec 916 MBytes 768 Mbits/sec


As you can see, just enabling jumbo frames produces a raw throughput increase of 17.43%. Nothing to sneeze at.

At this point I tried enabling flow control on the catalyst switch (it is already enabled for both send and receive by default in the e1000 driver) but it did not have any effect on iperf numbers. I turned it back off for now.

So now we set up a 20GB LVM volume on the target and export it using vblade to be mounted on the initiator. We then run a simple dd test to check throughput:


root@node02:~# dd if=/dev/zero of=/mnt/test oflag=direct bs=4M
419+0 records in
419+0 records out
1757413376 bytes (1.8 GB) copied, 133.476 seconds, 13.2 MB/s


CPU load on the target was 10-15% during the dd operation. Now we try writing direct to the (unmounted) block device to rule out any performance penalties of the filesystem itself…


root@node02:~# dd if=/dev/zero of=/dev/etherd/e0.1 oflag=direct bs=4M
513+0 records in
512+0 records out
2147483648 bytes (2.1 GB) copied, 170.991 seconds, 12.6 MB/s


CPU usage was slightly higher in that test, running 15-20%. So some slight difference but nothing to be too concerned about.

Now we take that same LVM device and share it via iSCSI for the same dd tests:


ladmin@node02:~$ dd if=/dev/zero of=/mnt/test oflag=direct bs=4M
462+0 records in
461+0 records out
1933574144 bytes (1.9 GB) copied, 38.2375 seconds, 50.6 MB/s


CPU load was 6-8% during that test. We also run that same test with flow control enabled at the switch:


ladmin@node02:~$ dd if=/dev/zero of=/mnt/test oflag=direct bs=4M
463+0 records in
462+0 records out
1937768448 bytes (1.9 GB) copied, 38.1851 seconds, 50.7 MB/s


Essentially the same…

Now this raises the question of why AoE is so much slower than iSCSI on an essentially default install of Debian Etch. To AoE’s credit, many people report getting just as good (50MB/s or better) of performance from AoE on their systems as I’m seeing with iSCSI. I spent quite a large amount of time playing with flow control, kernel ring buffer values, filesystem options, etc. and was unable to determine why performance is so terrible for me. I did find a pretty high number (half a dozen at least) of recent posts to the AoE mailing list by other people having essentially identical problems so I’m certainly not alone. In the interest of completing my testing, I’ve decided to move forward with iSCSI.

Now we try reformatting with the stride option to mkfs:


mkfs.ext3 -E stride=16


The results of several more tests are shown here…


root@node02:~# dd if=/dev/zero of=/mnt/test oflag=direct bs=4M
2038431744 bytes (2.0 GB) copied, 41.8938 seconds, 48.7 MB/s
2038431744 bytes (2.0 GB) copied, 42.2633 seconds, 48.2 MB/s
2038431744 bytes (2.0 GB) copied, 41.3756 seconds, 49.3 MB/s


So we don’t see any appreciable difference when using a combination of the stride= option and flow control, at least with a simple dd test.

Next we turn flow control back off, and reformat again without the stride= option. We are now back to our baseline setup for a new test with bonnie++.


ladmin@node02:~$ /usr/sbin/bonnie++ -d /mnt -s 4096Mb -n 10 -x 5 -q


This test produced block writes of about 82MB/s and block reads of about 37MB/s. The cause for the difference in write speed between the dd and bonnie++ tests is still unclear to me. There also appears to be a known issue where writes are much faster than reads which is apparently due to interrupt handling. This is further evidenced by running a quick dd test that does a read instead of a write:


ladmin@node02:~$ dd if=/mnt/test of=/dev/null bs=4M
3602907136 bytes (3.6 GB) copied, 96.2053 seconds, 37.5 MB/s
3602907136 bytes (3.6 GB) copied, 90.5524 seconds, 39.8 MB/s
3602907136 bytes (3.6 GB) copied, 90.0425 seconds, 40.0 MB/s
3602907136 bytes (3.6 GB) copied, 88.5708 seconds, 40.7 MB/s


As you can see, read operations are about 20% slower than write operations which goes against common thinking with regard to stripped disk arrays.

So there you have it. In my particular situation, with no tuning/optimizing done, iSCSI performs much better than AoE. Even in the event that I were to go to the trouble to performance tune AoE and get it as good as, or even better than, iSCSI I would still be inclined to standardize around iSCSI. Authentication, routing, NAT, etc. can all be done very easily on iSCSI protocol with all the standard TCP/IP tools that are out there. For me that’s a pretty big advantage.

Next up will be our final piece of the puzzle, left over from the initial system setup – getting hot swap working with the sata_mv module!

dd if=image.bin of=image.trx bs=32 skip=1

That will strip the first 32 bytes off the image, thus making it a valid .trx formatted firmware image file. Neat, huh?

Simply start out within an ssh session on the machine youre sending data FROM:


tar cjvf - /home | ssh root@backup.pixelchaos.net "cat > /backup/home.tar.bz2"

You can also use dd (or any other data/pipe handling commands):


tar cjvf - /home | ssh root@backup.pixelchaos.net "dd of=/backup/home.tar.bz2"


To restore that same file over ssh:


ssh root@backup.pixelchaos.net "cat /backup/home.tar.bz2" | tar xjvf -


Enjoy!