Marc's Public Blog - Linux Hacking

Upgrading a 29 year old linux install and 16 year old server

Old pictures:

Dual PIIi Xeon with 10x SCSI drives

Then I upgraded it to 10x hot swappable Sata drives which I upgraded over time

Old motherboard with dual CPU before dual core existed, WinTV capture card and sound card

then it was time to upgrade that old board

new version with single dual core I3 2100T (so obsolete today) and later upgraded to a quad core i7-3770

It's impressive that the server has run non stop since 2010, and outside of the CPU and RAM upgrade, maxed to what that motherboard would allow, I only upgraded hard drives and that's it. It had a few failures, pretty much all related to drives, and I think the power supply actually survived the entire time.
That said, it was literally a 32bit debian install (upgraded many times) from 1997, so it's literally going to turn 30 years next year. 30 years without ever re-installing, and just upgrading userland and kernels on a per need basis. I could have done a full userland upgrade to 64bit, but to save myself some upgrade work, I kept most things 32bit while having a 64bit kernel and a few utilities like btrfs-tools in 64bit, as required by some kernel interfaces. So, who here has installed windows (or hell, even linux) and kept it upgraded without ever re-installing for 30 years?

That said, the hardware being more than 15 years old, did a good job, but it's really old and not power efficient anymore. At the same time, after 25+ years of having big disk arrays (I had up to 26 drives spinning early on, read up here, and settled on 2 arrays of 5 drives, or 10 drives running), it was finally time to switch to solid state drives (aka flash drives). Of course, those had been around for quite a while, but I had to get enough of them at a reasonable enough price. It wasn't my original plan, but after some ebay buying, I got another 12 SSDs to supplement the 6 4TB ones I already had, and ended up with 18 4TB drives. Now, I just needed a new server :)

Attempt to upgrade gargamel with an rPi5

Now, trying an rPi5 was not entirely crazy as I successfully upgraded my 2 internet facing servers magic and moremagic to rPi5 and while I did encounter some power issues with high performance Samsung M2 PRo SSDs crashing due to lack of M2 slot power (which given that all that power comes from a tiny ribbon cable, is not very surprising), the Pi5s were otherwise pretty stable and replaced some huge power hungry (200 to 400W) obsolete dell servers.

Here are the related posts:

Long story short, I was able to run 2 SSDs and 18 drives on a rPi5, but that was too much PCI traffic to go through a quad bus splitter, it worked until it didn't a bus timeout reset all the ports and crashed the Sata controllers:

So, no, running 2 or 3 Sata controllers and 18 SATA drives on a rPi5 is not reasonable sounding, and indeed does not work.

Moving from 2x 5 drive Sata Raid5s to 18 Sata SSDs

I loved those 5 drive triple height hot swappable sata carriers

First, I used a Sata Port Multiplier which had performance issues and obviously split the Sata bandwidth by 5, but at the time it was acceptable although array rebuilds did suck. As I increased the drive sizes from 1TB to 12TB, array rebuilds became more and more unbearable, so I tried to find reasonably priced (less than $500) Sata cards with 10 ports or more, but turns out those were rare, and some like and adaptec card I found, had terrible bios and firmware and insisted in trying to do its crappy RAID which I was never going to use (that was Adaptec Series 7 2274500-R 71605ESinglePCI-Express 3.0 x8 Low Profile SATA / SAS RAID Controller Card). That Adaptec card had 16 ports, awesome, but no real direct access to each port, it was very hard to even query the drive's SMART status due to the hardware raid crap in the middle...
Eventually I found the RocketRAID 2720 which only supported 8 ports, but with another silicon image 3124 which supported another 4 ports, allowed me to run the 10 ports as direct connect. It did require some special expensive splitter cables, but it worked overall and wasn't terribly priced.

For switching to SSDs, I knew switching to NVMEs was not going to be reasonable, first because they cost a fortune, and second because I wouldn't find a way to connect enough of them to a motherboard, so I Went to the Sata SSD route. While they are limited to 600MB/s, with 18 of them, that's still 10.8GB/s of bandwidth or 86Gbit/s, which is more than I'd be able to get from a motherboard anyway, so going to the SATA route was not terrible. I didn't need insane bandwidth, just 60TB of SSD storage at decent enough speed (faster than the hard drives they were replacing)
So, I settled with 3 Sata controllers running at PCIe 3.0 speeds, which around 1GB/s per card, only the speed of 2 SSDs shen I'd plug 6 of them in each card, but that's still 3GB/s or 18Gbit/s of combined bandwidth across 3 PCIe slots, which is plenty.

tried 3 different sata cards

it's a bit crowded, so it's good to label the cables to know where they go

Moving from an rPi5 to a quad M2 lane N355 StoneStorm server

I asked Google Gemini was was the next best low power solution to run all my drives, and it first suggested a N150 system. I quickly found out that N355 as a more recent and more powerful than the older N150. Once I had settled on N355, gemini recommended a few systems that had 4 NVME slots, and one of them was from CWWK ( https://cwwkpc.com ). Their StoneStorm N355 server is amazing in how much it packs in such little space. After picking it and setting it up, I did struggle with working on it and wiring, as things are very cramped and most things are hidden under the quad NVME daughter board.

This is how the server is meant to be used (although I found out later, that the daugherboard would not have enough power to deal with the peak demands of PRO drives):

And instead, I did this (I used 3 cards to spread out the I/O load over 3 PCI lanes, whereas on rPi5 it was a single lane split 4 ways, so that wouldn't work)

spreading the I/O load across 3 M2/PCI sata cards

At this point, I need to note that on a money standpoint, those M2 sata cards are incredibly cheap (around $40 or less) while getting a full size PCIe Sata card that can do 9 drives, costs up to 10 times more, and often includes some crap raid firmware I didn't want. So, on top of being so compact, those cards are amazing value compared to using a bigger server that doesn't offer 4 NVME slots, but actual PCI slots.

one downside is the chips get warm, so I had to add a fan to keep them cool so they didn't crash

Of course the server is not setup to power external SATA drives, it has enough power for its own systems, and does include 2 onboard connections for 2 extra Sata drives directly connected to the motherboard, but not past that. I solved this by using external SSD enclosures, that unfortunately didn't provide their own power and required both 5V for the drives and 12V for the fans. As it turns out, I was not able to find external 12V/5V power supplies that had enough amps for 18 SSDs, so I made my own using a separate 40A %v power supply that could be boosted up to account for voltage sag, and a separate 12V power supply to run the fans in the enclosures. In the end, it looks like this:

Replacing the N355 StoneStorm server quad M2 daughterboard due to its weak 3.3V Voltage Regulator and lack of amps

By then my system wouldn't really work anymore, I needed a new daughterboard, so I ordered one from CWWK but that meant a one week+ delay with shipping from china (by then the servers were not available on amazon 2 day shipping anymore). So I discussed this with Gemini, and it gave me enough help to find the right 3.3V Vreg I could order overnight and how to wire it on my failed board:

The main problem was that the new board design was incompatible with the metal plate I had from the old one, so I contacted CWWK to inform them that they should ship both the plate and daughterboard at the same time, and they nicely sent me the correct plate which wasn't absolutely required for things to work, but provided EMI shielding between the motherboard and the PCI/M2 slots:

the old metal plate just didn't work with the new daughterboard (see the voltage regulator hitting the plate)

This page has more details on the daughterboard Vreg emergency repair ( StoneStorm N355 M2 Daughterboard Repair with Gemini )

Review of the rest of the CWWK StoneStorm N355

There are actually very few mini PCs that offer 4 M2 slots at all, so I was very happy google gemini helped me find it.

The storage, the NVME drives I already had, I also had 6 4TB SSDs, so I had to buy 12 more to get 18, and that did cost a bit more than $2000 getting the cheapest 4TB drives I could find on ebay (while avoiding DRAM-less QLC, which are the worst).

Without the metal place EMI shield, it's actually easy to make sure the pins go in the correct holes on the mainboard:

But if you have the metal plate, in theory you can only slide it in the right way, but in my experience, that's dodgy, you're working blind, you don't really have feedback when you push it in, and it's sadly possible to be off by one, which could damage both the motherboard and the daughterboard. This is really the weakest point of this computer, the daughterboard is a pain to remove (lots of screws, including removing the M2 cards), and you never quite know if you're pushing it back in correctly. I did push it in wrong 3 times while working on it and it's only luck that caused no damage when I did that (but the system didn't turn on).

This is what the motherboard looks like, with the single memory slot (so you need to take everything apart to change the memory). You can also notice the 2 white ribbon connectors for 2 sata drives, and the mini pin headers (not compatible with the regular motherboard headers that use bigger pins and bigger pitch).

the M2 looking card in my hand is the connector/bridge for the daugherboard

Now, I filled it every port and header, you can see ther serial header, the 2 Sata ones, and 2 USB2:

The problem is nothing keeps those header plugs in, and they are not big, so they tend to pop out, and guess what, if they do, you have to take the entire PC apart again, including the painful removing of the M2 daughterboard:

so my 'fix' was this, I put paper on top of the headers so that the pressure of the daughterboard would keep them in place

I'll be honest this is not ideal, but with my fix, it's workable, and again that's a lot of IO from a PC that small.

The bios does support serial console on the single onboard serial port (on its own special small header), and after figuring out the non obvious bios options, I got it to work, but unfortunately it does not support serial break used by for sysrq over serial (a probelm for linux)

it really wan't clear what COM0 was, apparently a direct connection to the CPU for debugging

turns out I did have to use COM1 (and ignore COM0)

after that, I could get the bios over serial, which made it easier to feed screens to gemini for help

So there you go, it was a bit tricky to setup, the M2 daughterboard update is greatly recommended if you somehow get the old one. Once I applied my fixes, it was amazingly stable under high load, and high IO, including putting way more M2 load and drives that this PC was ever built, for. Also, sales@cwwkpc.com did a great job with support and getting back to me when I needed help (to swap the metal plate).

Future, 20 Camera Frigate NVR. Will N355 be enough?

But if I need a bigger machine, it may be tricky to find one that has 4 M2 slots, as they think one or two PCI slots are enough, when not only they are not for me, but getting a PCIe card that can handle 18 sata drives is actually quite expensive (about 10x more than my solution, and they often require custom cables instead of just SATA)

Watch this space, I hope to have an update in a couple of months.