I am moving my main server to this small N355 StoneStorm server. It does include 4 NVME slots thanks to a daugtherboard that plugs into this port:
The daughter board looks like this:
and the back looks like this, input for 12V power from the motherboard and Vret to make 3.3V
Then, because I was reckless, I swapped M2 cards while the PC was powered, thinking they were hot swappable like Sata, but they sure are not. Eventually I caused a partial failure of the board. Gemini helped me debug the linux errors from the cards plugged in itand confirmed my guess that the voltage regulator on the board was now partially damaged and gave power, but not enough to be reliable under load.
A new daughterboard is only $20, but that's 2 week shipping from China while my new server would be down. So, it actually knew the daughterboard, it knew where to buy it from and the shipping delay, and it also knew how it was made before I gave it my picture. It confirmed that the best course of action would be to desolder the 100 component that was an inductor, and I could just unsolder it and connect a new 3.3V source to it.
After helping me find a new 5A Vreg on amazon, it looked at the picture of the board, and it confirmed the top pad, is the one feeding 3.3V to the rest of the board. It suggested I use flux to help unsolder the SMD component after I declined the suggestion to use a hot air gun that would be too likely to remove other components, some of which had to be useful:
yes, it looked ugly, but the inductor was removed safely, I then cleaned the flux
Then I happened to have the right female connector in my collection of cables and plugs (happy it does come handy at least a few times), and soldered in the new Vreg:
I had to move it on the other side so it would fit with the case closed
And there you go, I literally fixed the daughterboard and the PC works again! Yeah! I honestly was impressed by Gemini's knowledge of existing boards and its ability to read pictures I sent it, identify the components and confirm test points and how to verify I was adding the new power source in the correct place.
π
2026-06-19 01:01
in Computers, Poland, Poland2023, Sciencemuseums
I went to three vintage computer museums, 2 across the street from one another in Warsaw, and a 3rd one in Poznan. They were all fun to visit but the 2nd one in Warsaw was manned by someone who didn't speak much but Polish, however he had one of the most impressive private collections I've seen, including some expansion boards that allowed improving the old computers like Amstrad CPC with extra resident commands, a newer Basic, and a bunch of games you could run instantly via flash instead of a slow load from floppy, or an even slower load from tape (those were sometimes 10mn or more).
The first one was fun:
But nothing compared to the second one, the highlight by far, hidden in a subway tunnel:
I immediately noticed the expansion board that was used to instantly load games
and I remember playing this game as a kid
this was the magic board which has its own computer and flash, kudos to whoever designed it
and my second computer I spent so much time with
a cute baby version of it
a pristine version of my Amstrad CPC 464 with the correct colors, QWERTY and tape player
I had heard about the atari TT but never seen one! It had a built in hard drive!
I still have that exact spare SIM 9 bit RAM at home :)
and more
the famous Atari ET was featured
Atari 260ST, I had never heard of it! It's a rare machine that pre-dated the 512 STF
Amstrads had to sell as Schneider in Germany
this is another rare one I had never heard of and could accept game cartridges
someone made an updated DOS in 2015, 30 years later!
I had no fun at all ;)
the more boring PCs that killed all our joyous fun computers
an Atari 65XE, before my time. an Atari 800 era 8bit CPU
This was honestly the best private vintage computer museum I've seen in the world. Big props to the engineer who has maintained that collection.
A few days later, In Poznan, there was also an old computer and video game museum. It didn't compare, but fur kids they had enough video games running:
Upgrading a 29 year old linux install and 16 year old server
My main home linux server, called gargamel, was actually first built in 1997, with an early version of debian (slink) that did not yet support apt-get, just to say how old that was. The impressive part is that my current gargamel server (a dual core duo built back in 2008 (Intel i3 2100T dual core CPU), where I built the entire server from scratch, motherboard, case, CPU, RAM and all. Later, I was able to upgrade the CPU to a quad core (Core(TM) i7-3770 CPU @ 3.40GHz), the biggest that would fit in that motherboard, and maxed out the memory to 32GB, which was useful for a few things, including operating over 20 terabyte filesystems: some pictures of gargamel here
Old pictures:
Dual PIIi Xeon with 10x SCSI drives
Then I upgraded it to 10x hot swappable Sata drives which I upgraded over time
Old motherboard with dual CPU before dual core existed, WinTV capture card and sound card
then it was time to upgrade that old board
new version with single dual core I3 2100T (so obsolete today) and later upgraded to a quad core i7-3770
It's impressive that the server has run non stop since 2010, and outside of the CPU and RAM upgrade, maxed to what that motherboard would allow, I only upgraded hard drives and that's it. It had a few failures, pretty much all related to drives, and I think the power supply actually survived the entire time.
That said, it was literally a 32bit debian install (upgraded many times) from 1997, so it's literally going to turn 30 years next year. 30 years without ever re-installing, and just upgrading userland and kernels on a per need basis. I could have done a full userland upgrade to 64bit, but to save myself some upgrade work, I kept most things 32bit while having a 64bit kernel and a few utilities like btrfs-tools in 64bit, as required by some kernel interfaces. So, who here has installed windows (or hell, even linux) and kept it upgraded without ever re-installing for 30 years?
That said, the hardware being more than 15 years old, did a good job, but it's really old and not power efficient anymore. At the same time, after 25+ years of having big disk arrays (I had up to 26 drives spinning early on, read up here, and settled on 2 arrays of 5 drives, or 10 drives running), it was finally time to switch to solid state drives (aka flash drives). Of course, those had been around for quite a while, but I had to get enough of them at a reasonable enough price. It wasn't my original plan, but after some ebay buying, I got another 12 SSDs to supplement the 6 4TB ones I already had, and ended up with 18 4TB drives. Now, I just needed a new server :)
Attempt to upgrade gargamel with an rPi5
I figured I would go for low power server, but I got a bit too optimistic and tried a Raspberry Pi 5, split its M2 PCI bus 4 ways. On paper, I got things to work, but in real life a single lane PCI split 4 ways with a PCI switch, once it's running 3 SATA cards and 18 drives, some cheap DRAM-less ones that can hang quite a while before they ack, ended up causing PCI timeouts and resets that crashed one or more sata cards and ruined the raid arrays I tested.
Now, trying an rPi5 was not entirely crazy as I successfully upgraded my 2 internet facing servers magic and moremagic to rPi5 and while I did encounter some power issues with high performance Samsung M2 PRo SSDs crashing due to lack of M2 slot power (which given that all that power comes from a tiny ribbon cable, is not very surprising), the Pi5s were otherwise pretty stable and replaced some huge power hungry (200 to 400W) obsolete dell servers.
Here are the related posts:
Long story short, I was able to run 2 SSDs and 18 drives on a rPi5, but that was too much PCI traffic to go through a quad bus splitter, it worked until it didn't a bus timeout reset all the ports and crashed the Sata controllers:
So, no, running 2 or 3 Sata controllers and 18 SATA drives on a rPi5 is not reasonable sounding, and indeed does not work.
Moving from 2x 5 drive Sata Raid5s to 18 Sata SSDs
After retiring my SCSI drives (73GB and a few rare 180GB ones), I went to Sata and multiple iterations from 1TB drives, 2TB, 4TB, 6TB and 12TB for the last array. I just upgraded one array at a time when the previous array was getting too full and the drives too old:
I loved those 5 drive triple height hot swappable sata carriers
First, I used a Sata Port Multiplier which had performance issues and obviously split the Sata bandwidth by 5, but at the time it was acceptable although array rebuilds did suck. As I increased the drive sizes from 1TB to 12TB, array rebuilds became more and more unbearable, so I tried to find reasonably priced (less than $500) Sata cards with 10 ports or more, but turns out those were rare, and some like and adaptec card I found, had terrible bios and firmware and insisted in trying to do its crappy RAID which I was never going to use (that was Adaptec Series 7 2274500-R 71605ESinglePCI-Express 3.0 x8 Low Profile SATA / SAS RAID Controller Card). That Adaptec card had 16 ports, awesome, but no real direct access to each port, it was very hard to even query the drive's SMART status due to the hardware raid crap in the middle...
Eventually I found the RocketRAID 2720 which only supported 8 ports, but with another silicon image 3124 which supported another 4 ports, allowed me to run the 10 ports as direct connect. It did require some special expensive splitter cables, but it worked overall and wasn't terribly priced.
For switching to SSDs, I knew switching to NVMEs was not going to be reasonable, first because they cost a fortune, and second because I wouldn't find a way to connect enough of them to a motherboard, so I Went to the Sata SSD route. While they are limited to 600MB/s, with 18 of them, that's still 10.8GB/s of bandwidth or 86Gbit/s, which is more than I'd be able to get from a motherboard anyway, so going to the SATA route was not terrible. I didn't need insane bandwidth, just 60TB of SSD storage at decent enough speed (faster than the hard drives they were replacing)
So, I settled with 3 Sata controllers running at PCIe 3.0 speeds, which around 1GB/s per card, only the speed of 2 SSDs shen I'd plug 6 of them in each card, but that's still 3GB/s or 18Gbit/s of combined bandwidth across 3 PCIe slots, which is plenty.
tried 3 different sata cards
it's a bit crowded, so it's good to label the cables to know where they go
Moving from an rPi5 to a quad M2 lane N355 StoneStorm server
After losing a couple of weeks building and testing that rPi5 solution, I had to admit that it was not going to work, and even if it did, dividing a single PCI lane bandwidth by 4, only to use 9 port sata cards, diving that 4th by 9 again, would negate any performance advantage from NVME drives.
I asked Google Gemini was was the next best low power solution to run all my drives, and it first suggested a N150 system. I quickly found out that N355 as a more recent and more powerful than the older N150. Once I had settled on N355, gemini recommended a few systems that had 4 NVME slots, and one of them was from CWWK ( https://cwwkpc.com ). Their StoneStorm N355 server is amazing in how much it packs in such little space. After picking it and setting it up, I did struggle with working on it and wiring, as things are very cramped and most things are hidden under the quad NVME daughter board.
This is how the server is meant to be used (although I found out later, that the daugherboard would not have enough power to deal with the peak demands of PRO drives):
And instead, I did this (I used 3 cards to spread out the I/O load over 3 PCI lanes, whereas on rPi5 it was a single lane split 4 ways, so that wouldn't work)
spreading the I/O load across 3 M2/PCI sata cards
At this point, I need to note that on a money standpoint, those M2 sata cards are incredibly cheap (around $40 or less) while getting a full size PCIe Sata card that can do 9 drives, costs up to 10 times more, and often includes some crap raid firmware I didn't want. So, on top of being so compact, those cards are amazing value compared to using a bigger server that doesn't offer 4 NVME slots, but actual PCI slots.
one downside is the chips get warm, so I had to add a fan to keep them cool so they didn't crash
Of course the server is not setup to power external SATA drives, it has enough power for its own systems, and does include 2 onboard connections for 2 extra Sata drives directly connected to the motherboard, but not past that. I solved this by using external SSD enclosures, that unfortunately didn't provide their own power and required both 5V for the drives and 12V for the fans. As it turns out, I was not able to find external 12V/5V power supplies that had enough amps for 18 SSDs, so I made my own using a separate 40A %v power supply that could be boosted up to account for voltage sag, and a separate 12V power supply to run the fans in the enclosures. In the end, it looks like this:
Replacing the N355 StoneStorm server quad M2 daughterboard due to its weak 3.3V Voltage Regulator and lack of amps
Despite this, I ended up with my arrays crashing, gemini helped analyse the logs and confirmed that the sata chips were crashing due to voltage sag, PCI/M2 3.3V that came from the M2 daugherboard. Turns out that the voltage regulator on that board was only good for 7A on a good day, and mine partially failed under load (having an NVME PRo drive didn't help as those take extra power)
By then my system wouldn't really work anymore, I needed a new daughterboard, so I ordered one from CWWK but that meant a one week+ delay with shipping from china (by then the servers were not available on amazon 2 day shipping anymore). So I discussed this with Gemini, and it gave me enough help to find the right 3.3V Vreg I could order overnight and how to wire it on my failed board:
`
The board I built only had a 5A 3.3V Vreg, I was not able to find a 10A one, but it survived until limited load while the new board arrived. When I finally got it, I found out that the new board was actually a new version with an upgraded regulator that can sustain closer to 10A at 3.3V, which was exactly what I needed. The new board even had a dual pin power input to receive a bit more 12V power from the motherboard:
The main problem was that the new board design was incompatible with the metal plate I had from the old one, so I contacted CWWK to inform them that they should ship both the plate and daughterboard at the same time, and they nicely sent me the correct plate which wasn't absolutely required for things to work, but provided EMI shielding between the motherboard and the PCI/M2 slots:
the old metal plate just didn't work with the new daughterboard (see the voltage regulator hitting the plate)
As I alluded above, this server offers a lot in a small space, it's actually amazing how much it does offer if you use all of its IO. And at the time the price was incredible, only $350 without the RAM (now the price has unfortunately increased by $200). I didn't f around this time and just maxed out the RAM from the start, 48GB DDR5 for $472, even if RAM pricing had already doubled or tripled by then. Yes, the RAM cost 20% more than the entire PC, and it's probably even worse now (actually the RAM price almost doubled since I bought it!!!)
If you want to know which mini PC exactly, I found this one:
There are actually very few mini PCs that offer 4 M2 slots at all, so I was very happy google gemini helped me find it.
The storage, the NVME drives I already had, I also had 6 4TB SSDs, so I had to buy 12 more to get 18, and that did cost a bit more than $2000 getting the cheapest 4TB drives I could find on ebay (while avoiding DRAM-less QLC, which are the worst).
Without the metal place EMI shield, it's actually easy to make sure the pins go in the correct holes on the mainboard:
But if you have the metal plate, in theory you can only slide it in the right way, but in my experience, that's dodgy, you're working blind, you don't really have feedback when you push it in, and it's sadly possible to be off by one, which could damage both the motherboard and the daughterboard. This is really the weakest point of this computer, the daughterboard is a pain to remove (lots of screws, including removing the M2 cards), and you never quite know if you're pushing it back in correctly. I did push it in wrong 3 times while working on it and it's only luck that caused no damage when I did that (but the system didn't turn on).
This is what the motherboard looks like, with the single memory slot (so you need to take everything apart to change the memory). You can also notice the 2 white ribbon connectors for 2 sata drives, and the mini pin headers (not compatible with the regular motherboard headers that use bigger pins and bigger pitch).
the M2 looking card in my hand is the connector/bridge for the daugherboard
Now, I filled it every port and header, you can see ther serial header, the 2 Sata ones, and 2 USB2:
The problem is nothing keeps those header plugs in, and they are not big, so they tend to pop out, and guess what, if they do, you have to take the entire PC apart again, including the painful removing of the M2 daughterboard:
so my 'fix' was this, I put paper on top of the headers so that the pressure of the daughterboard would keep them in place
I'll be honest this is not ideal, but with my fix, it's workable, and again that's a lot of IO from a PC that small.
The bios does support serial console on the single onboard serial port (on its own special small header), and after figuring out the non obvious bios options, I got it to work, but unfortunately it does not support serial break used by for sysrq over serial (a probelm for linux)
it really wan't clear what COM0 was, apparently a direct connection to the CPU for debugging
turns out I did have to use COM1 (and ignore COM0)
after that, I could get the bios over serial, which made it easier to feed screens to gemini for help
So there you go, it was a bit tricky to setup, the M2 daughterboard update is greatly recommended if you somehow get the old one. Once I applied my fixes, it was amazingly stable under high load, and high IO, including putting way more M2 load and drives that this PC was ever built, for. Also, sales@cwwkpc.com did a great job with support and getting back to me when I needed help (to swap the metal plate).
Future, 20 Camera Frigate NVR. Will N355 be enough?
As a file server and I/O wise, I'm pushing that StoneStorm N355 beyond what it was meant to do, but it is running around 60TB over 18 SSDs without issues. As a file server, it's great. Now, as I write this, I'm setting up frigate NVR with up to 20 cameras and will see how that CPU handles the load, or if I'll need a bigger CPU to do this.
But if I need a bigger machine, it may be tricky to find one that has 4 M2 slots, as they think one or two PCI slots are enough, when not only they are not for me, but getting a PCIe card that can handle 18 sata drives is actually quite expensive (about 10x more than my solution, and they often require custom cables instead of just SATA)
Watch this space, I hope to have an update in a couple of months.
DRAM-less Teamgroup drives are garbage and failed with no warning within months
After a few months in my Raid 6 setup, 2 Teamgroup drives failed badly enough that they started rejecting all SATA commands, and a 3rd one failed enough that it was kicked off the bus, causing a triple failure and collapse of RAID6. There was no warning in SMART, they just died. This is complete garbage, do not buy.
Replacing a 16 year old Sandy Bridge Server running 12 Spinning Rust Drives with something more efficient
My old Intel Sandy Bridge server gargamel built in 2010, initially with a dual core duo, later upgraded to a quad core with hyperthreading, was 16 years old. It was still working, but I had already replaced the drives multiple times from 2TB to 4TB, 6TB, and eventually 12TB drives as the previous drives were getting old and started failing ( My first ridiculous NAS was 2TB, with 26 SCSI SCA Drives in 3 enclosures, circa 2002 ).
I setup that last server with 10 SATA drives in 2 enclosures of 5 drives each. It's been running for over 15 years with a just a few drive upgrades and replacements now at 64TB of spinning rust. Turns out I didn't really need that much but on the last drive upgrade, I went directly from 6TB to 12TB..
The server still works fine, but it's ultimately still running a debian install from 1999 that's been upgraded all these years, including a 32/64bit dual userland without systemd.
But fighting "progress" only goes so far, and my 2nd disk array with 10Y+ old 4TB drives was starting to have more drive failures. Also, I realized that 250W+ of power is a bit more than needed, so I decided to upgrade to an rPi5 with 16GB of RAM and see if I could make a decent linux server out of it.
Considering a rPi5 with 20 SSDs
Here is what I did:
An rPi5 supports PCI, but I got a bit over ambitious with it. I got a 4X M2 slot switch for 2 used 2 old leftover NVMEs I boot from in raid0 (500GB each)
I also bought 2 9 port M2 sata cards which allow for 18 drives.
First I was thinking about having a few SSDs and re-use my 12TB drives in an external enclosure I already have. I also found a USB-3 to 3 aata adapter that I can run the disk enclosure with, using USB3 which is 5Gbit/s instead of going through a PCI sata card..
But in the end, I decided to go without any spinning drives at all and went for a bunch of ebay 4TB SSDs to fill up all 18 slots, yielding 56TB. It was never the plan to have that much, but it's a pain to upgrade the arrays later and it felt more efficient to just fill up the raids with more drives. So I now have
/dev/mapper/dshelf1 30T 5.4T 24T 19% /mnt/btrfs_pool1 => 10x cheap TLC + QLC SSDs in raid6
/dev/mapper/dshelf2 25T 128G 25T 1% /mnt/btrfs_pool2 => 8x more expensive MLC/TLC enterprise drives in raid5
/dev/mapper/dshelf3 447G 6.1M 445G 1% /mnt/btrfs_pool3 => left over space from some QLC drives that are 4.09TB
/dev/mapper/dshelf4 447G 6.1M 445G 1% /mnt/btrfs_pool4 => left over space from MLC drives
The next problem is "how do you power 18 directly connected external drives?". You're going to tell me to just get drive enclosures, but turns out there aren't any or many external drive enclosures for 2.5" drives that offer direct sata connection as well as their own power. You would think it shouldn't be too hard to buy reasonably sized standalone 12V/5V power supplies for sata drives that offer more than 20A fo 5V (even NVME drives can take more than 1A each), but I didn't find any without buying a full bore ATX power supply and deal with it not coming on on its own because it's not connected to a motherboard), so I had to make my own: I took a 40A 5A power supply I laying around for LEDs, joined it with a 12V 7A power supply, and made my own Sata power bus.
From there, I could indeed have 18 drives hang off the sata power plugs ;)
Or do something a bit better and found these nice enclosures. Unfortunately they cost $90 each when they don't even provide their own power, and sadly the built in fans require 12V, so I have to send them dual power just for that otherwise I'd be able to power the entire thing from 5V:
Making all this work on an rPi5
So you're going to tell me that maybe an rPi5 wasn't really meant to have a PCI bus, never mind to run 20 SSDs (18 Sata + 2 M2 NVME), and maybe you'd be right, but I got excited when I got this quad NVME expander board for my Pi5:
I mean it does look pretty and exciting ;)
blinkenlights win ;)
But what I didn't pay enough attention to is that it's still a single lane PCI bus (after all the Pi5 is not exactly a real server board), so what that PCI splitter board does is use a PCI switching chip to create 4 lanes out of 1 by switching PCI packets. This does not create extra bandwidth but just puts more drives on the same single channel bus. I got things to work but unsurprisingly, doing a 10 drive raid6 rebuild was slow, only 50MB/s, which is slower than the speed of a single drive. Sata does support 6GBit/s (and SSDs support around 600MB/s per drive) but all the drives together add up to 10.8GB/s of combined bandwith, or 96Gbit/s, about 15 times what my single lane PCI bus can do :)
So yes, it can work, but it's not fast. For reference, with an unrestricted sata bus, rebuild speed can be up to 600MB/s, which is the limit speed of the drive writing parity. In real life, the more drives you have, the more data is on the sata bus or busses during rebuild, as explained above, so it's of course rare to get the full speed, especially with 10 drives, but for me it was sad to get below 100MbB/s as I was getting more than that with my spinning rust drives.
This what it looks like, by the way:
So given that, my master plan of building a big NAS does not make a lot of sense, so the quad splitter does not make a lot of sense for anything than 4NVME drives that you are ok with running at much lower speed than they can do (a 4 lane NVME drive would run at 1/16th of its speed getting 1/4th of one lane). In the end, for a couple of drives, using the dual splitter GeekPi board to power 2 independent boards, is not such a bad idea and using the real sata Hat, offers real power to the drives (up to 6-7A I think), saving the trouble of having to make your own power supply like I did:
routing the Pi5 Ribbon is a bit tricky and requires longer ribbon cables to read the middle splitter board
SSDs and Prices, using cheaper DRAM-less SSDs and QLC RDAT drives with Raid5/6
OBviously I picked the wrong time to buy a bunch of SSDs. Proper 4TB SSDs run around $700, if not worse, so I went for low grade DRAM-less TLC or QLC drives off Ebay (still around $300 a piece). I figured with RAID6, it would not be so bad, and for one of my 2 arrays, write performance and many rewrites were not a concern. I also found out that the TeamGroup 4TB drives were a mix of TLC, QLC, with 3 different kinds of controllers and some were 4.09TB where others were just 4.00TB. Then I found out about discard/TRIM support and this:
/dev/sdr * Deterministic read ZEROs after TRIM
/dev/sdi * Deterministic read data after TRIM
The better, expensive drives guarantee RZAT, and the cheap ones are RDAT. The RDAT drives cannot support TRIM through raid5 or raid6 because raid requires that drives return 0 after TRIM so that parity works out later, and RDAT drivers do not give that guarantee, linux raid nicely detects that and turns off discard support. This however also means that after deleting data, there no way to mark that flash as free for the drives, you can trim or fstrei. The only sad thing with btrfs is that it does wear leveling of the underlying drives, which means over time all the SSD blocks get used, and there is no way to tell the drives what blocks are free, which is not ideal for QLC drives especially as they are quite slow to rewrite blocks when they don't have plenty of free space.
Knowing that, I made sure to build that array as a write once mostly, which will make the write penalty not as important.
My other array used for backups and lots of rewriting, I made use to use higher grade DRAM TLC Samsung and Micron enterprise drives I had laying around. I still had one drive in that array that didn't support RZAT but with those higher rate drives, not having TRIM was not as bad (they do a better job rewriting and do ok enough with their reserved space).
Stressing the rPi5 and the ASM1184e PCI switch
I then learned a bunch of the the limitations of PCI port switches like ASM1184e. Once I started using mine seriously, got a bunch of weird errors and disconnects until Gemini found that it's a known issue with them overheating under load. I just put an RC plane video chip radiator on the chip and now the radiator is hot and the chip seems to work reliably.
Then I found out that my cheap teamgroup 4TB DRAM-less drives (the real TLC DRAM ones are now hovering between 6 to $700 a pop for 4TB) are fine, until they stall during a big copy/btrfs scrub or whatever.
When they stall, they eventually time out the PCI bus, which behind the quad PCI switche, causes the rPI to reset everything, and in the end this caused enough PCI mayhem that the sata cards were reset and 3 of the teamgroup drives crashed and failed to write what they had to a point that they were corrupted enough for linux to not be able to use their partitions anymore. Yes, a single drive stall caused a PCI timeout long enough to crash/reset the SATA controllers, which apparently managed to get the cheap teamgroup drives to corrupt the partition table blocks and have the blocks be unmapped and unreadable and unwritable:
nvme nvme0: controller is down; will reset: CSTS=0x3, PCI_STATUS=0x10
[57247.067230] ata8.00: exception Emask 0x0 SAct 0x0 SErr 0x400001 action 0x6 frozen
[57247.076133] ata8: SError: { RecovData Handshk }
[57247.081246] ata8.00: failed command: READ DMA
[57247.086014] ata8.00: cmd c8/00:08:c8:03:5b/00:00:00:00:00/e1 tag 2 dma 4096 in
[57247.086014] res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[57247.101469] ata8.00: status: { DRDY }
[57247.105822] ata8: hard resetting link
[57247.153797] nvme nvme0: 3/0/0 default/read/poll queues
[57247.587051] ata8: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[57247.630423] ata8.00: supports DRM functions and may not be fully accessible
[57247.750869] ata8.00: supports DRM functions and may not be fully accessible
[57247.807025] ata8.00: configured for UDMA/133
[57247.811957] sd 7:0:0:0: [sdh] tag#2 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=32s
[57247.822653] sd 7:0:0:0: [sdh] tag#2 Sense Key : 0xb [current]
[57247.829121] sd 7:0:0:0: [sdh] tag#2 ASC=0x0 ASCQ=0x0
[57247.835477] sd 7:0:0:0: [sdh] tag#2 CDB: opcode=0x88 88 00 00 00 00 00 01 5b 03 c8 00 00 00 08 00 00
[57247.845243] I/O error, dev sdh, sector 22741960 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[57247.855511] ata8: EH complete
[57247.872535] ata8.00: Enabling discard_zeroes_data
[60367.285453] ata9.00: exception Emask 0x10 SAct 0x0 SErr 0x400100 action 0x6 frozen
[60367.293666] ata9.00: irq_stat 0x08000000, interface fatal error
[60367.300313] ata9: SError: { UnrecovData Handshk }
[60367.306530] ata9.00: failed command: WRITE DMA EXT
[60367.311966] ata9.00: cmd 35/00:00:78:8c:f7/00:05:1e:00:00/e0 tag 9 dma 655360 out
[60367.311966] res 50/00:00:ff:03:f7/00:00:1e:00:00/e0 Emask 0x10 (ATA bus error)
[60367.328871] ata9.00: status: { DRDY }
[60367.333036] ata9: hard resetting link
[60367.805496] ata9: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[60367.863205] ata9.00: configured for UDMA/133
[60367.868064] ata9: EH complete
[60397.357520] nvme nvme0: controller is down; will reset: CSTS=0x3, PCI_STATUS=0x10
[60397.453616] nvme nvme0: 3/0/0 default/read/poll queues
[60398.929509] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x400001 action 0x6 frozen
[60398.959616] ata1: SError: { RecovData Handshk }
[60398.966761] ata1.00: failed command: READ DMA
[60398.972859] ata1.00: cmd c8/00:08:78:b9:4a/00:00:00:00:00/e2 tag 22 dma 4096 in
[60398.972859] res 40/00:00:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[60398.990825] ata1.00: status: { DRDY }
[60398.995717] ata1: hard resetting link
[60399.473455] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[60399.541525] ata1.00: configured for UDMA/133
[60399.546657] sd 0:0:0:0: [sda] tag#22 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=32s
[60399.557577] sd 0:0:0:0: [sda] tag#22 Sense Key : 0xb [current]
[60399.564532] sd 0:0:0:0: [sda] tag#22 ASC=0x0 ASCQ=0x0
[60399.570665] sd 0:0:0:0: [sda] tag#22 CDB: opcode=0x88 88 00 00 00 00 00 02 4a b9 78 00 00 00 08 00 00
[60399.580758] I/O error, dev sda, sector 38451576 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[60399.590585] ata1: EH complete
[60399.640204] ata1.00: Enabling discard_zeroes_data
[72688.943036] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x400001 action 0x6 frozen
[72688.951084] ata1: SError: { RecovData Handshk }
[72688.956422] ata1.00: failed command: WRITE DMA
[72688.961594] ata1.00: cmd ca/00:20:00:ac:82/00:00:00:00:00/e5 tag 14 dma 16384 out
[72688.961594] res 40/00:00:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[72688.977731] ata1.00: status: { DRDY }
[72688.981969] ata1: hard resetting link
[72688.986211] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x400001 action 0x6 frozen
[72688.994663] ata2: SError: { RecovData Handshk }
[72688.999881] ata2.00: failed command: WRITE DMA
[72689.005000] ata2.00: cmd ca/00:20:c0:b0:82/00:00:00:00:00/e5 tag 19 dma 16384 out
[72689.005000] res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[72689.022962] ata2.00: status: { DRDY }
[72689.027430] ata2: hard resetting link
[72689.499039] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[72689.506396] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[72689.611777] ata1.00: configured for UDMA/133
[72689.616890] ata1: EH complete
[72689.723181] ata2.00: configured for UDMA/133
[72689.728156] ata2: EH complete
[72689.865333] ata1.00: Enabling discard_zeroes_data
[72689.871277] ata2.00: Enabling discard_zeroes_data
[73227.538624] nvme nvme1: controller is down; will reset: CSTS=0x3, PCI_STATUS=0x10
[73227.640436] nvme nvme1: D3 entry latency set to 8 seconds
[73227.658550] nvme nvme1: 1/0/0 default/read/poll queues
[86766.334170] nvme nvme0: controller is down; will reset: CSTS=0x3, PCI_STATUS=0x10
[86766.442187] nvme nvme0: 3/0/0 default/read/poll queues
[86766.863105] ata6.00: exception Emask 0x0 SAct 0x0 SErr 0x400001 action 0x6 frozen
[86766.877356] ata6: SError: { RecovData Handshk }
[86766.884232] ata6.00: failed command: WRITE DMA
[86766.891103] ata6.00: cmd ca/00:80:18:95:b5/00:00:00:00:00/e6 tag 20 dma 65536 out
[86766.891103] res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[86766.908556] ata6.00: status: { DRDY }
[86766.914377] ata6: hard resetting link
[86766.919016] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x400001 action 0x6 frozen
[86766.930307] ata2: SError: { RecovData Handshk }
[86766.937937] ata2.00: failed command: READ DMA
[86766.943738] ata2.00: cmd c8/00:38:a0:e6:3b/00:00:00:00:00/e5 tag 4 dma 28672 in
[86766.943738] res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[86766.965459] ata2.00: status: { DRDY }
[86766.970640] ata2: hard resetting link
[86766.976782] ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x400001 action 0x6 frozen
[86766.989369] ata3: SError: { RecovData Handshk }
[86767.001777] ata3.00: failed command: WRITE DMA
[86767.010295] ata3.00: cmd ca/00:80:18:95:b5/00:00:00:00:00/e6 tag 21 dma 65536 out
[86767.010295] res 40/00:00:06:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[86767.060215] ata3.00: status: { DRDY }
[86767.071409] ata3: hard resetting link
[86767.550253] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[86767.563271] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[86767.572715] ata6: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[86767.585598] ata6.00: supports DRM functions and may not be fully accessible
[86767.616959] ata6.00: supports DRM functions and may not be fully accessible
[86767.631980] ata3.00: configured for UDMA/133
[86767.639404] ata3: EH complete
[86767.643354] ata6.00: configured for UDMA/133
[86767.661059] ahci 0001:03:00.0: port does not support device sleep
[86767.663591] ata3.00: Enabling discard_zeroes_data
[86767.676336] ata6: EH complete
[86767.745871] ata2.00: configured for UDMA/133
[86767.754280] ata2: EH complete
[86767.772933] ata2.00: Enabling discard_zeroes_data
[95256.566913] nvme nvme0: controller is down; will reset: CSTS=0x3, PCI_STATUS=0x10
[95256.574928] nvme nvme1: controller is down; will reset: CSTS=0x3, PCI_STATUS=0x10
[95256.679475] nvme nvme1: D3 entry latency set to 8 seconds
[95256.689110] nvme nvme0: 2/0/0 default/read/poll queues
[95256.694718] nvme nvme1: 1/0/0 default/read/poll queues
[95256.697626] I/O error, dev nvme0n1, sector 264208 op 0x1:(WRITE) flags 0x29800 phys_seg 1 prio class 2
[95256.712397] I/O error, dev nvme0n1, sector 264208 op 0x1:(WRITE) flags 0x29800 phys_seg 1 prio class 2
[95256.722258] md: super_written gets error=-5
[95256.727133] md/raid1:md0: Disk failure on nvme0n1p2, disabling device.
[95256.727133] md/raid1:md0: Operation continuing on 1 devices.
[95256.742401] I/O error, dev nvme0n1, sector 77334752 op 0x1:(WRITE) flags 0x4000800 phys_seg 1 prio class 2
[95256.753375] BTRFS error (device nvme0n1p3): bdev /dev/nvme0n1p3 errs: wr 1, rd 1, flush 0, corrupt 0, gen 0
[95256.764177] I/O error, dev nvme0n1, sector 77335776 op 0x1:(WRITE) flags 0x4000800 phys_seg 1 prio class 2
[95256.774805] BTRFS error (device nvme0n1p3): bdev /dev/nvme0n1p3 errs: wr 2, rd 1, flush 0, corrupt 0, gen 0
[97602.825969] ata6.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[97602.833948] ata6.00: failed command: WRITE DMA EXT
[97602.839911] ata6.00: cmd 35/00:00:78:5a:4c/00:04:09:00:00/e0 tag 22 dma 524288 out
[97602.839911] res 40/00:01:06:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[97602.858583] ata6.00: status: { DRDY }
[97602.863617] ata6: hard resetting link
[97603.337938] ata6: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[97603.346750] ata6.00: supports DRM functions and may not be fully accessible
[97603.370306] ata6.00: supports DRM functions and may not be fully accessible
[97603.430476] ata6.00: configured for UDMA/133
[97603.445466] ahci 0001:03:00.0: port does not support device sleep
[97603.452251] ata6: EH complete
[97637.643844] BTRFS warning (device dm-1): csum failed root 263 ino 3692950 off 386400256 csum 0xd04e5f48 expected csum 0x6b9afaa1 mirror 1
[97637.657936] BTRFS error (device dm-1): bdev /dev/mapper/dshelf2 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[97638.110104] BTRFS warning (device dm-1): csum failed root 263 ino 3692950 off 386400256 csum 0xd04e5f48 expected csum 0x6b9afaa1 mirror 1
[97638.123856] BTRFS error (device dm-1): bdev /dev/mapper/dshelf2 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[97662.159091] BTRFS warning (device dm-1): csum failed root 263 ino 3692950 off 386400256 csum 0xd04e5f48 expected csum 0x6b9afaa1 mirror 1
[97662.173941] BTRFS error (device dm-1): bdev /dev/mapper/dshelf2 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
[97662.906008] BTRFS warning (device dm-1): csum failed root 263 ino 3692950 off 386400256 csum 0xd04e5f48 expected csum 0x6b9afaa1 mirror 1
[97662.920993] BTRFS error (device dm-1): bdev /dev/mapper/dshelf2 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
Recovering unusable DRAM-less Teamgroup drives
By then it was impossible to read or write to the 3 Teamgroup drives that failed, so I had to blkdiscard (TRIM) the entire 3 crashed drives (out of 7) to restart with 0's everywhere (which included full data loss of course), and start over.
Gemini gave me linux kernel sata and PCI options to make it less likely for this to happen again, but it also warned me it very much can happen again and DRAM less drives should never be behind a PCI switch.
DRAM-less Teamgroup drives are garbage and failed with no warning within months
After a few months in my Raid 6 setup, 2 Teamgroup drives failed badly enough that they started rejecting all SATA commands, and a 3rd one failed enough that it was kicked off the bus, causing a triple failure and collapse of RAID6. There was no warning in SMART, they just died. This is complete garbage, do not buy
Conclusion
At the same time, it became painfully obvious that the rPi5 has single lane PCI, and all those PCI switches are adding more channels while dividing the single lane bandwidth, making things slower and slower, was a bit of fool's errand.
By then, I had to admit defeat and since I wanted to run frigate for my cameras anyway, Gemini suggested I get an N355 based server which has an H264 and H265 ASIC for all those video streams (while rPi5 would have to do it in software), and at least 4 PCI lanes, which is much better (it's still 4x single lane M2 NVME, but at least 4 times faster and without a PCI switch to confuse things and cause full hangs if a single sata drive is freezing while writing its data)
Moved this to an N355 PC (without PCI switch and 4 real NVME slots) and it worked
I randomly happened to be a the CHM to drop off some donations, and noticed a room I didn't remember and found out they now had a fully working IBM 1402 (actually they had two).
My 2 main computers have been called magic and moremagic since the late 90's. Most people do not know wy, so here is the story I read back in the 90's, reposted from http://www.catb.org/jargon/html/magic-story.html
A Story About 'Magic'
Some years ago, I (GLS) was snooping around in the cabinets that housed the MIT AI Lab's PDP-10, and noticed a little switch glued to the frame of one cabinet. It was obviously a homebrew job, added by one of the lab's hardware hackers (no one knows who).
You don't touch an unknown switch on a computer without knowing what it does, because you might crash the computer. The switch was labeled in a most unhelpful way. It had two positions, and scrawled in pencil on the metal switch body were the words 'magic' and 'more magic'. The switch was in the 'more magic' position.
I called another hacker over to look at it. He had never seen the switch before either. Closer examination revealed that the switch had only one wire running to it! The other end of the wire did disappear into the maze of wires inside the computer, but it's a basic fact of electricity that a switch can't do anything unless there are two wires connected to it. This switch had a wire connected on one side and no wire on its other side.
It was clear that this switch was someone's idea of a silly joke. Convinced by our reasoning that the switch was inoperative, we flipped it. The computer instantly crashed.
Imagine our utter astonishment. We wrote it off as coincidence, but nevertheless restored the switch to the 'more magic' position before reviving the computer.
A year later, I told this story to yet another hacker, David Moon as I recall. He clearly doubted my sanity, or suspected me of a supernatural belief in the power of this switch, or perhaps thought I was fooling him with a bogus saga. To prove it to him, I showed him the very switch, still glued to the cabinet frame with only one wire connected to it, still in the 'more magic' position. We scrutinized the switch and its lone connection, and found that the other end of the wire, though connected to the computer wiring, was connected to a ground pin. That clearly made the switch doubly useless: not only was it electrically nonoperative, but it was connected to a place that couldn't affect anything anyway. So we flipped the switch.
The computer promptly crashed.
This time we ran for Richard Greenblatt, a long-time MIT hacker, who was close at hand. He had never noticed the switch before, either. He inspected it, concluded it was useless, got some diagonal cutters and diked it out. We then revived the computer and it has run fine ever since.
We still don't know how the switch crashed the machine. There is a theory that some circuit near the ground pin was marginal, and flipping the switch changed the electrical capacitance enough to upset the circuit as millionth-of-a-second pulses went through it. But we'll never know for sure; all we can really say is that the switch was magic.
I still have that switch in my basement. Maybe I'm silly, but I usually keep it set on 'more magic'.
1994: Another explanation of this story has since been offered. Note that the switch body was metal. Suppose that the non-connected side of the switch was connected to the switch body (usually the body is connected to a separate earth lug, but there are exceptions). The body is connected to the computer case, which is, presumably, grounded. Now the circuit ground within the machine isn't necessarily at the same potential as the case ground, so flipping the switch connected the circuit ground to the case ground, causing a voltage drop/jump which reset the machine. This was probably discovered by someone who found out the hard way that there was a potential difference between the two, and who then wired in the switch as a joke.