looks like it was at MFN early on while I was at google (before that it was at VA Linux)

moved from MFN to via.net in 2003

When I got home to fix it, the server was in a sad state because one of the raid5 drives was dead (you could hear the head scraping the disk when you turned it on) and a second drive would die as soon as you accessed some portions of it. So, when I managed to manually bring the raid5 array on 4 drives instead of 5, that second drive would go offline as soon as I accessed some of its sectors (as /usr was being mounted).
Unfortunately, all the partitions were all logical volumes on a big volume group, and the boot process would try to mount /usr from /dev/intraid5/usr and bring the raid5 array down, along with the volume group and all other logical volumes.
Now, I already had a backup of almost all the data there, but not my ftp site, as it was too big to bother backing up. This was what I was trying to get back.
After a little effort, I actually got it to work: I manually brought a not up to date raid with
mdadm -A --run --force /dev/md5 /dev/sd{a,b,c,d}3
(--run because one drive was missing and --force because /dev/sdc3 was not up to date with the rest of the drives, but at that point, I didn't give a shit, I just wanted it to come up with whatever data it did have).

At that point, I would try to mount /usr to get some binaries and everything would go down and fail. Eventually, I managed to rescue it by mounting just /var/ftp without touching/mounting any other logical volume, and I was able to bring networking up and make use of tar and nc which were both in /bin to copy the data off and save the day (rescuing a system without any binary in /usr can be a bit challenging :) )

While I was working working on bringing magic back up, I decided to build a new server: I decided against fixing the old server as is, just because all my SCSI/SCA drives were old, and replacement ones would be bound to fail again (not counting the fact that they are low capacity too). The old motherboard was also a dual P3 800Mhz with one failed CPU, and RAM that still worked, but wasn't new either.
Instead, I used a lesser VA Linux 2230 system which didn't have any hotswappable drives or SCA midplane (compared to the previous 2250 I had), and used a free server motherboard that a coworker gave me. In hindsight, maybe I should have bought a new server board and populated it, but I eventually made it work: the major challenge was that it didn't have SCSI onboard, and only one PCI slot (also, it unfortunately does not allow for bios console redirection on serial port).

I ended up with a frankstein machine with:

The PCI bus was a problem since it only allowed one card, but I was able to find a PCI extender with bus selector for the second card from logicsupply.com. This was challenging to get working (and only one of the two cards will post at boot time), but in the end both cards show up in linux and work, which is all I cared about.

In the process, I also managed to smoke a SATA card by putting it in the wrong direction in the PCI slot (don't laugh, it has no header bracket and it's very hard to see its front from its back, especially on an angled bracket), I also got some smoke to get off the motherboard when it wasn't screwed to the chassis and it slid while powered on and touched a chassis bolt which shorted a solder trace on the back of the board.
Amazingly, I'm now stress testing the board and it seems to still work fine, for all of CPU, Ram, and PCI. I've restored my last backup of magic on it, ready for going back in production tomorrow.

After that, I went to via.net to rack the new frankenstein magic, after having just finished copying live data back from moremagic and switching mail/web/everything back over. This is what it looked, racked up a few months later, still running that huge disk tray filled with obsolete SCSI/SCA drives :)

We're now back up with: 2TB of disk, 2GB of Ram, and 2x1.4Ghz CPUs (ok, not stellar, but it'll do). The 2TB was through an external SCA/SCSI disk array I had before and could still connect to.

Here are a few pictures of the new server:

this was the pci riser/doubler with idsel

The power connector was also 24 pin instead of 20, and too short. Thankfully I had a power connector extension that came in very handy

I had to adapt the motherboard specific front panel connector (for intel 550GX) to a power connector that worked with a regular motherboard

4 sata drives on the right, in proper slots, and 2 sata drives on the left, one in the CD-Rom slot and one in the floppy slot, with a jerry-rigged fan I added

those IDE to sata convertors (which actually have a sata controller that takes IDE commands and translates them into brand new sata commands), came in quite handy to add 2 more drives when I only had 4 sata slots on the controller card

As I write this after the fact, I do wonder "what on earth was I doing?", that was a fair amount of complex engineering to re-use free hardware that wasn't really meant for the job, but once I factor in my engineering time to make it all work, including the PCI doubler config, or making Sata drives work on IDE ports because I didn't have enough ports on my free Sata card and probably could have bought an 8 port Sata card, even back then

For version 3, it looks like I eventually removed the SCSI card to run that obsolete external array with power hungry obsolete drives, and somehow managed to connect a single 1 or 2TB drive, sitting on the motherboard with a CPU cooler and fan:

some of my best work, haha

This allowed me to get rid of the obsolete disk array filled with drives that got replaced by a single drive sitting on top of the motherboard :) which was non redundant but contained non essential data. It still had 6 sata drives I also got for free (250GB each) running Raid5 for the main system.

That new system went on to run from 2008 to 2019, not a bad run for such a weirdly pieced up machine that blew smoke from a short when I built it until the motherboard finally died in Aug 2019, apparently from a failed capacitor.

Further reading