pixelchaos.net

AoE (ATA over Ethernet) and iSCSI are the hot new things. Xen is the hot new thing. I like using hot new things as long as they can be made rock solid.

There happens to be a company (Coraid) that makes a turnkey AoE device. Its far cheaper than a true fibre channel SAN or something similar. Perfect for setting up a SAN over Ethernet device that can serve Xen domU filesystems out to “thin” dom0’s on the network.

Well that’s all well and good but you see I’m always looking to save a buck…

So I asked myself; why not build my own “coraid” with off the shelf parts and save 50% in the process? Herein I’ll do my best to chronicle my adventures in getting this thing built, tested, and (hopefully) deployed.

First of all the hardware. Thanks to the ever-brilliant Mike Neir for helping me figure out what Coraid uses to build these suckers. Here are the parts I ended up ordering:

• 2 x Supermicro 8-Port SATA Card (AOC-SAT2-MV8)
• 1 x Supermicro Xeon Dual-Core Blackford VS ServerBoard (X7DVL-E)
• 1 x Supermicro Black 3U Rackmount Case 760W (SC933T-R760B)
• 1 x Kingston 2GB 240-Pin DDR2 FB-DIMM ECC (KVR667D2D8F5K2/2G)
• 1 x Intel Xeon 5130 Woodcrest 2.0GHz 4M (BX805565130A)
• 15 x Seagate Barracuda ES.2 1TB (ST31000340NS)
• 3 x Supermicro Drive Carrier (CSE-PT39B)
• 3 x Western Digital 7K 8M SATA2 160GB (WD1600AAJS)

Now you may ask, why buy 18 drives for a chassis that has 15 hot swap bays? Well, I’ll be using the three WD drives as a RAID1 boot device with one hot spare, then setting nine of the 1TB drives up into a RAID6 device with one hot spare. The remaining three drives will be in RAID5 and swapped with the last three drives on a regular basis to go off site as part of a hard disk based backup system.

Now that we have all of that out of the way; how did the assembly go? Flawless almost ;-)

There are two gotcha’s to be dealt with in the assembly of the aforementioned motherboard and server chassis; first of all, the EPS12V connector for CPU power that comes out of the power distribution block wont reach its molex connector on the motherboard. Second, the ribbon cable that connects the front panel controls to the their motherboard header just *barely* reaches which makes me uncomfortable.

To resolve the first problem I ordered up a StarTech 8″ EPS 8 Pin Power Extension Cable (EPS8EXT) which did the job nicely. To fix the second issue, I ordered twenty feet of 16 conductor ribbon cable (might need extra!), some cable ends and a crimper. Problem solved.

With all that taken care of its time to get an operating system installed and test this thing out. I did go ahead and power up the system and test the redundant PSU’s and fans. The alarm features of both seem to work fine with audible alarms when any of them are removed from the system. After that I plugged an old CD-ROM into the ATA header on the mobo and installed a rockin Linux distro.

Some things I discovered:

• The BIOS and the Linux kernel both initialize the two SATA PCI cards first. Some fiddling will need to be done to make the three WD drives (plugged into the motherboard SATA headers) show up as sda, sdb, and sdc.
• You must turn off “compatibility mode” in the BIOS for the onboard SATA controllers or hotplug will not work for drives on those headers. After turning it off, hotplug works just dandy. Yay!
• Hotplug isnt working for the two PCI SATA cards at this point. I suspect its something to do with the sata_mv kernel module but havent gotten to look into it much. That will be the first order of business when next I work on this project.

So thats it for now. Things are coming together nicely. My next post will detail getting hotplug working for the two PCI SATA cards and getting device detection and mapping working correctly in the Linux kernel (I smell some udev tomfoolery most likely).

This entry was posted on Friday, March 28th, 2008 at 5:17 pm and is filed under AoE, Debian, Hardware, HowTo, Xen, iSCSI. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.