Coraid Odyssey: Part 2 (sata_mv hotplug)
Todays adventure with building a SAN on the cheap involves attempting to get hotplug working and changing device mappings.
First of all, hotplug. I have discovered that…
hotplug doesn’t seem to work on the Supermicro AOC-SAT2-MV8 cards. The controllers do work, but unless a drive is plugged in during bootup they will not be subsequently detected…
root@stor01:~# lspci
05:01.0 SCSI storage controller: Marvell Technology Group Ltd. MV88SX6081 8-port SATA II PCI-X Controller (rev 09)
05:02.0 SCSI storage controller: Marvell Technology Group Ltd. MV88SX6081 8-port SATA II PCI-X Controller (rev 09)
You can see that the Linux kernel sees these cards as Marvell MV88SX6081 devices. Further probing reveals that the sata_mv driver is claiming them:
root@stor01:~$ dmesg | grep 05:01.0
sata_mv 0000:05:01.0: version 0.7
ACPI: PCI Interrupt 0000:05:01.0[A] -> GSI 24 (level, low) -> IRQ 233
sata_mv 0000:05:01.0: 32 slots 8 ports SCSI mode IRQ via INTx
So off we go lkml.org and other sources to find out why… Basically, hotplug support in the sata_mv module is still a work in progress. A post to the lkml claims that its slated for inclusion in the 2.6.26 kernel. There is also more information here.
So hotplug isnt going to work for us today :-( On the bright side, 2.6.25-rc7 is the current development kernel so I can’t imagine that 2.6.26 is too far off.
The second (perceived) issue of getting all the hot swap bays to be mapped to /dev/sd* entries in a particular order doesnt seem to be as much of an issue as I’d thought.
The Linux kernel will initialize PCI devices in a strange fashion depending on what is plugged in to the bus. For example, if all fifteen bays have drives plugged into them, the leftmost bay always shows up at /dev/sda. Unplug just one drive and reboot the machine and it will be mapped to /dev/sdl for example. The reason this isnt really an issue is that newer Linux kernels have not only udev (which is great but not really a solution for this particular problem) but the /dev/disk/by-* tree. You can find all your disks in there with handy names like /dev/disk/by-id/scsi-SATA_ST31000340NS_5QJ06913. That name belongs to that particular drive and only that particular drive, ever.
root@stor01:~# ls -l /dev/disk/by-id/ | grep scsi | grep -v part
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ06913 -> ../../sde
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ07CF3 -> ../../sdi
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ0817T -> ../../sdb
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ08T21 -> ../../sdh
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ08XHB -> ../../sdc
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09B9P -> ../../sda
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09GMW -> ../../sdd
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09HZR -> ../../sdg
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09JHQ -> ../../sdk
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09JV3 -> ../../sdj
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09KNM -> ../../sdl
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_ST31000340NS_5QJ09LLA -> ../../sdf
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_WDC_WD1600AAJS-_WD-WCAP93952468 -> ../../sdm
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_WDC_WD1600AAJS-_WD-WMAS20341446 -> ../../sdo
lrwxrwxrwx 1 root root 9 2008-04-01 11:15 scsi-SATA_WDC_WD1600AAJS-_WD-WMAS20366839 -> ../../sdn
Cool, huh?
So we will just let mdadm do its job of assembling /dev/md* devices on the fly at bootup and use the /dev/disk/by-id entries to do stuff to particular disks (like temperature monitoring, etc.).
Coming up in my next post – performance and failure testing the array.