Marc's Public Blog

Table of Content for linuxha:

2026/04/13: Attempt to replace my 60TB Spinning Rust NAS with a Raspberry Pi 5 with Quad PCI Splitter and 18 2.5 Sata SSDs TLC and DRAM-less TeamGroup QLC drives: Unreliable single Lane PCI bus Under Load (single page)
2026/02/14: TCL 98QM8K vs Samsung QN90D and Denon AVR3808 (single page)
2025/07/07: Using Raspberry Pi5 as a Server With Raid1, Btrfs, and Multiple NVME M2 or Sata Drives (single page)
2025/07/07: Using a Raspberry Pi 5 (Rpi5) as a Server With Btrfs, Raid1, Serial Console and Dual NVME/SD Card Recovery Boot (single page)
2025/01/17: House Lights with Pixelblaze on ESP32 monitored with Raspberry Pi (single page)
2023/03/09: Solar Monitoring Bit Flip Rabbithole (single page)
2014/07/23: Making a super cheap water/flood sensor (single page)
2013/12/09: Keeping Track of How Much Your Roomba is Really Working (single page)
2013/11/16: Mythtv + Denon-AVR-3808CI 2 Way Communication (single page)
2013/11/10: Reviewing IP Webcams for Linux and Zoneminder:Dlink DCS900, Ubnt Aircam, Foscam FI8904W FI8910W, FFI9820W, FI9821W, Wansview NCB541W, and Zavio F3210 (single page)
2013/01/30: 2 way talking to a Lockstate LS-DB-500R lock because the Lockstate wifi Remotelock is no good for me (single page)
2011/11/11: Recent Power Monitoring and Misterhouse Talks (single page)
2011/08/03: Weather Monitoring with Oregon Scientific WMR968 (single page)
2011/07/24: Hacking an external antenna onto an X10 CM19a, and adding misterhouse support (single page)
2011/03/11: 1-wire Saved Our Freezer Food (single page)
2010/08/13: Fine grained house-wide power monitoring with Brultech ECM1240, ecmread.py (with net metering support), and graphing with cacti (single page)
2010/08/06: Temperature, moisture, humidity, and UV monitoring and graphing with 1wire devices, owfs, and cacti (single page)
2010/08/06: Booster Fans and Heating/Cooling with Outside Air with Misterhouse (single page)
2010/06/25: Rfxcom, W800, and MR26a on misterhouse, X10Sec Garage and Mailbox Setup (single page)
2010/06/18: Powering UPSes with Marine or AGM batteries (single page)
2010/06/17: Dealing with many USB to Serial Port Converters on linux and device naming (single page)
2010/03/11: One Year Of Solar Panels (single page)
2010/02/14: Daily Power and Solar Stats (single page)
2009/12/23: Temperature monitoring and graphing with 1wire devices, digitemp, misterhouse, and cacti (single page)
2009/11/08: Brand One Powermeter, Solar and Power Monitoring with Cacti and Real Time PG&E Time of Use price calculation (single page)
2009/08/04: Work on HAI Omnistat code for misterhouse (single page)
2009/05/27: Mythtv + Denon-AVR3808 Hack (single page)
2009/03/16: Insteon Setup and my own Blender Defender (single page)
2004/03/25: TV and remotes with linux (single page)

More pages: April 2026 February 2026 July 2025 January 2025 March 2023 July 2014 December 2013 November 2013 January 2013 November 2011 August 2011 July 2011 March 2011 August 2010 June 2010 March 2010 February 2010 December 2009 November 2009 August 2009 May 2009 March 2009 March 2004

2026/04/13 Attempt to replace my 60TB Spinning Rust NAS with a Raspberry Pi 5 with Quad PCI Splitter and 18 2.5 Sata SSDs TLC and DRAM-less TeamGroup QLC drives: Unreliable single Lane PCI bus Under Load

π 2026-04-13 01:01 in Linux, Linuxha

Replacing a 16 year old Sandy Bridge Server running 12 Spinning Rust Drives with something more efficient

My old Intel Sandy Bridge server gargamel built in 2010, initially with a dual core duo, later upgraded to a quad core with hyperthreading, was 16 years old. It was still working, but I had already replaced the drives multiple times from 2TB to 4TB, 6TB, and eventually 12TB drives as the previous drives were getting old and started failing ( My first ridiculous NAS was 2TB, with 26 SCSI SCA Drives in 3 enclosures, circa 2002 ).

I setup that last server with 10 SATA drives in 2 enclosures of 5 drives each. It's been running for over 15 years with a just a few drive upgrades and replacements now at 64TB of spinning rust. Turns out I didn't really need that much but on the last drive upgrade, I went directly from 6TB to 12TB..

The server still works fine, but it's ultimately still running a debian install from 1999 that's been upgraded all these years, including a 32/64bit dual userland without systemd. But fighting "progress" only goes so far, and my 2nd disk array with 10Y+ old 4TB drives was starting to have more drive failures. Also, I realized that 250W+ of power is a bit more than needed, so I decided to upgrade to an rPi5 with 16GB of RAM and see if I could make a decent linux server out of it.

Considering a rPi5 with 20 SSDs

Here is what I did:

An rPi5 supports PCI, but I got a bit over ambitious with it. I got a 4X M2 slot switch for 2 used 2 old leftover NVMEs I boot from in raid0 (500GB each)

I also bought 2 9 port M2 sata cards which allow for 18 drives.

First I was thinking about having a few SSDs and re-use my 12TB drives in an external enclosure I already have. I also found a USB-3 to 3 aata adapter that I can run the disk enclosure with, using USB3 which is 5Gbit/s instead of going through a PCI sata card..

But in the end, I decided to go without any spinning drives at all and went for a bunch of ebay 4TB SSDs to fill up all 18 slots, yielding 56TB. It was never the plan to have that much, but it's a pain to upgrade the arrays later and it felt more efficient to just fill up the raids with more drives. So I now have

/dev/mapper/dshelf1   30T  5.4T   24T  19% /mnt/btrfs_pool1 => 10x cheap TLC + QLC SSDs in raid6
/dev/mapper/dshelf2   25T  128G   25T   1% /mnt/btrfs_pool2 => 8x more expensive MLC/TLC enterprise drives in raid5
/dev/mapper/dshelf3  447G  6.1M  445G   1% /mnt/btrfs_pool3 => left over space from some QLC drives that are 4.09TB
/dev/mapper/dshelf4  447G  6.1M  445G   1% /mnt/btrfs_pool4 => left over space from MLC drives

The next problem is "how do you power 18 directly connected external drives?". You're going to tell me to just get drive enclosures, but turns out there aren't any or many external drive enclosures for 2.5" drives that offer direct sata connection as well as their own power. You would think it shouldn't be too hard to buy reasonably sized standalone 12V/5V power supplies for sata drives that offer more than 20A fo 5V (even NVME drives can take more than 1A each), but I didn't find any without buying a full bore ATX power supply and deal with it not coming on on its own because it's not connected to a motherboard), so I had to make my own: I took a 40A 5A power supply I laying around for LEDs, joined it with a 12V 7A power supply, and made my own Sata power bus.

From there, I could indeed have 18 drives hang off the sata power plugs ;)

Or do something a bit better and found these nice enclosures. Unfortunately they cost $90 each when they don't even provide their own power, and sadly the built in fans require 12V, so I have to send them dual power just for that otherwise I'd be able to power the entire thing from 5V:

Making all this work on an rPi5

So you're going to tell me that maybe an rPi5 wasn't really meant to have a PCI bus, never mind to run 20 SSDs (18 Sata + 2 M2 NVME), and maybe you'd be right, but I got excited when I got this quad NVME expander board for my Pi5:

I mean it does look pretty and exciting ;)

But what I didn't pay enough attention to is that it's still a single lane PCI bus (after all the Pi5 is not exactly a real server board), so what that PCI splitter board does is use a PCI switching chip to create 4 lanes out of 1 by switching PCI packets. This does not create extra bandwidth but just puts more drives on the same single channel bus. I got things to work but unsurprisingly, doing a 10 drive raid6 rebuild was slow, only 50MB/s, which is slower than the speed of a single drive. Sata does support 6GBit/s (and SSDs support around 600MB/s per drive) but all the drives together add up to 10.8GB/s of combined bandwith, or 96Gbit/s, about 15 times what my single lane PCI bus can do :)

So yes, it can work, but it's not fast

So given that, my master plan of building a big NAS does not make a lot of sense, so the quad splitter does not make a lot of sense and in the end, for a few drives, using the dual splitter GeekPi board to power 2 independent boards, is not such a bad idea and using the real sata Hat, offers real power to the drives (up to 6-7A I think), saving the trouble of having to make your own power supply like I did:

routing the Pi5 Ribbon is a bit tricky and requires longer ribbon cables to read the middle splitter board

TODO raid rebuild

SSDs and Prices, using cheaper DRAM-less SSDs and QLC RDAT drives with Raid5/6

OBviously I picked the wrong time to buy a bunch of SSDs. Proper 4TB SSDs run around $700, if not worse, so I went for low grade DRAM-less TLC or QLC drives off Ebay (still around $300 a piece). I figured with RAID6, it would not be so bad, and for one of my 2 arrays, write performance and many rewrites were not a concern. I also found out that the TeamGroup 4TB drives were a mix of TLC, QLC, with 3 different kinds of controllers and some were 4.09TB where others were just 4.00TB. Then I found out about discard/TRIM support and this:

/dev/sdr * Deterministic read ZEROs after TRIM /dev/sdi * Deterministic read data after TRIM

The better, expensive drives guarantee RZAT, and the cheap ones are RDAT. The RDAT drives cannot support TRIM through raid5 or raid6 because raid requires that drives return 0 after TRIM so that parity works out later, and RDAT drivers do not give that guarantee, linux raid nicely detects that and turns off discard support. This however also means that after deleting data, there no way to mark that flash as free for the drives, you can trim or fstrei. The only sad thing with btrfs is that it does wear leveling of the underlying drives, which means over time all the SSD blocks get used, and there is no way to tell the drives what blocks are free, which is not ideal for QLC drives especially as they are quite slow to rewrite blocks when they don't have plenty of free space.
Knowing that, I made sure to build that array as a write once mostly, which will make the write penalty not as important.
My other array used for backups and lots of rewriting, I made use to use higher grade DRAM TLC Samsung and Micron enterprise drives I had laying around. I still had one drive in that array that didn't support RZAT but with those higher rate drives, not having TRIM was not as bad (they do a better job rewriting and do ok enough with their reserved space).

Stressing the rPi5 and the ASM1184e PCI switch

I then learned a bunch of the the limitations of PCI port switches like ASM1184e. Once I started using mine seriously, got a bunch of weird errors and disconnects until Gemini found that it's a known issue with them overheating under load. I just put an RC plane video chip radiator on the chip and now the radiator is hot and the chip seems to work reliably.

Then I found out that my cheap teamgroup 4TB DRAM-less drives (the real TLC DRAM ones are now hovering between 6 to $700 a pop for 4TB ï��) are fine, until they stall during a big copy/btrfs scrub or whatever.
When they stall, they eventually time out the PCI bus, which behind the quad PCI switche, causes the rPI to reset everything, and in the end this caused enough PCI mayhem that the sata cards were reset and 3 of the teamgroup drives crashed and failed to write what they had to a point that they were corrupted enough for linux to not be able to use their partitions anymore. Yes, a single drive stall caused a PCI timeout long enough to crash/reset the SATA controllers, which apparently managed to get the cheap teamgroup drives to corrupt the partition table blocks and have the blocks be unmapped and unreadable and unwritable:


nvme nvme0: controller is down; will reset: CSTS=0x3, PCI_STATUS=0x10
[57247.067230] ata8.00: exception Emask 0x0 SAct 0x0 SErr 0x400001 action 0x6 frozen
[57247.076133] ata8: SError: { RecovData Handshk }
[57247.081246] ata8.00: failed command: READ DMA
[57247.086014] ata8.00: cmd c8/00:08:c8:03:5b/00:00:00:00:00/e1 tag 2 dma 4096 in
[57247.086014]          res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[57247.101469] ata8.00: status: { DRDY }
[57247.105822] ata8: hard resetting link
[57247.153797] nvme nvme0: 3/0/0 default/read/poll queues
[57247.587051] ata8: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[57247.630423] ata8.00: supports DRM functions and may not be fully accessible
[57247.750869] ata8.00: supports DRM functions and may not be fully accessible
[57247.807025] ata8.00: configured for UDMA/133
[57247.811957] sd 7:0:0:0: [sdh] tag#2 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=32s
[57247.822653] sd 7:0:0:0: [sdh] tag#2 Sense Key : 0xb [current] 
[57247.829121] sd 7:0:0:0: [sdh] tag#2 ASC=0x0 ASCQ=0x0 
[57247.835477] sd 7:0:0:0: [sdh] tag#2 CDB: opcode=0x88 88 00 00 00 00 00 01 5b 03 c8 00 00 00 08 00 00
[57247.845243] I/O error, dev sdh, sector 22741960 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[57247.855511] ata8: EH complete
[57247.872535] ata8.00: Enabling discard_zeroes_data
[60367.285453] ata9.00: exception Emask 0x10 SAct 0x0 SErr 0x400100 action 0x6 frozen
[60367.293666] ata9.00: irq_stat 0x08000000, interface fatal error
[60367.300313] ata9: SError: { UnrecovData Handshk }
[60367.306530] ata9.00: failed command: WRITE DMA EXT
[60367.311966] ata9.00: cmd 35/00:00:78:8c:f7/00:05:1e:00:00/e0 tag 9 dma 655360 out
[60367.311966]          res 50/00:00:ff:03:f7/00:00:1e:00:00/e0 Emask 0x10 (ATA bus error)
[60367.328871] ata9.00: status: { DRDY }
[60367.333036] ata9: hard resetting link
[60367.805496] ata9: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[60367.863205] ata9.00: configured for UDMA/133
[60367.868064] ata9: EH complete
[60397.357520] nvme nvme0: controller is down; will reset: CSTS=0x3, PCI_STATUS=0x10
[60397.453616] nvme nvme0: 3/0/0 default/read/poll queues                                                 
[60398.929509] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x400001 action 0x6 frozen                      
[60398.959616] ata1: SError: { RecovData Handshk }                                                        
[60398.966761] ata1.00: failed command: READ DMA                                                          
[60398.972859] ata1.00: cmd c8/00:08:78:b9:4a/00:00:00:00:00/e2 tag 22 dma 4096 in                        
[60398.972859]          res 40/00:00:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[60398.990825] ata1.00: status: { DRDY }
[60398.995717] ata1: hard resetting link
[60399.473455] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[60399.541525] ata1.00: configured for UDMA/133
[60399.546657] sd 0:0:0:0: [sda] tag#22 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=32s
[60399.557577] sd 0:0:0:0: [sda] tag#22 Sense Key : 0xb [current] 
[60399.564532] sd 0:0:0:0: [sda] tag#22 ASC=0x0 ASCQ=0x0 
[60399.570665] sd 0:0:0:0: [sda] tag#22 CDB: opcode=0x88 88 00 00 00 00 00 02 4a b9 78 00 00 00 08 00 00
[60399.580758] I/O error, dev sda, sector 38451576 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[60399.590585] ata1: EH complete
[60399.640204] ata1.00: Enabling discard_zeroes_data
[72688.943036] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x400001 action 0x6 frozen
[72688.951084] ata1: SError: { RecovData Handshk }
[72688.956422] ata1.00: failed command: WRITE DMA
[72688.961594] ata1.00: cmd ca/00:20:00:ac:82/00:00:00:00:00/e5 tag 14 dma 16384 out
[72688.961594]          res 40/00:00:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[72688.977731] ata1.00: status: { DRDY }                                                                  
[72688.981969] ata1: hard resetting link                                                                  
[72688.986211] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x400001 action 0x6 frozen                      
[72688.994663] ata2: SError: { RecovData Handshk }                                                        
[72688.999881] ata2.00: failed command: WRITE DMA                                                         
[72689.005000] ata2.00: cmd ca/00:20:c0:b0:82/00:00:00:00:00/e5 tag 19 dma 16384 out                      
[72689.005000]          res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)                       
[72689.022962] ata2.00: status: { DRDY }                                                                  
[72689.027430] ata2: hard resetting link
[72689.499039] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[72689.506396] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[72689.611777] ata1.00: configured for UDMA/133
[72689.616890] ata1: EH complete
[72689.723181] ata2.00: configured for UDMA/133
[72689.728156] ata2: EH complete
[72689.865333] ata1.00: Enabling discard_zeroes_data
[72689.871277] ata2.00: Enabling discard_zeroes_data
[73227.538624] nvme nvme1: controller is down; will reset: CSTS=0x3, PCI_STATUS=0x10
[73227.640436] nvme nvme1: D3 entry latency set to 8 seconds
[73227.658550] nvme nvme1: 1/0/0 default/read/poll queues
[86766.334170] nvme nvme0: controller is down; will reset: CSTS=0x3, PCI_STATUS=0x10
[86766.442187] nvme nvme0: 3/0/0 default/read/poll queues
[86766.863105] ata6.00: exception Emask 0x0 SAct 0x0 SErr 0x400001 action 0x6 frozen
[86766.877356] ata6: SError: { RecovData Handshk }
[86766.884232] ata6.00: failed command: WRITE DMA
[86766.891103] ata6.00: cmd ca/00:80:18:95:b5/00:00:00:00:00/e6 tag 20 dma 65536 out
[86766.891103]          res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[86766.908556] ata6.00: status: { DRDY }
[86766.914377] ata6: hard resetting link
[86766.919016] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x400001 action 0x6 frozen
[86766.930307] ata2: SError: { RecovData Handshk }
[86766.937937] ata2.00: failed command: READ DMA
[86766.943738] ata2.00: cmd c8/00:38:a0:e6:3b/00:00:00:00:00/e5 tag 4 dma 28672 in
[86766.943738]          res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[86766.965459] ata2.00: status: { DRDY }
[86766.970640] ata2: hard resetting link
[86766.976782] ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x400001 action 0x6 frozen
[86766.989369] ata3: SError: { RecovData Handshk }
[86767.001777] ata3.00: failed command: WRITE DMA
[86767.010295] ata3.00: cmd ca/00:80:18:95:b5/00:00:00:00:00/e6 tag 21 dma 65536 out
[86767.010295]          res 40/00:00:06:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)                       
[86767.060215] ata3.00: status: { DRDY }                                                                  
[86767.071409] ata3: hard resetting link                                                                  
[86767.550253] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)                                     
[86767.563271] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)                                     
[86767.572715] ata6: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[86767.585598] ata6.00: supports DRM functions and may not be fully accessible
[86767.616959] ata6.00: supports DRM functions and may not be fully accessible
[86767.631980] ata3.00: configured for UDMA/133
[86767.639404] ata3: EH complete
[86767.643354] ata6.00: configured for UDMA/133
[86767.661059] ahci 0001:03:00.0: port does not support device sleep
[86767.663591] ata3.00: Enabling discard_zeroes_data
[86767.676336] ata6: EH complete
[86767.745871] ata2.00: configured for UDMA/133
[86767.754280] ata2: EH complete
[86767.772933] ata2.00: Enabling discard_zeroes_data
[95256.566913] nvme nvme0: controller is down; will reset: CSTS=0x3, PCI_STATUS=0x10
[95256.574928] nvme nvme1: controller is down; will reset: CSTS=0x3, PCI_STATUS=0x10
[95256.679475] nvme nvme1: D3 entry latency set to 8 seconds
[95256.689110] nvme nvme0: 2/0/0 default/read/poll queues
[95256.694718] nvme nvme1: 1/0/0 default/read/poll queues
[95256.697626] I/O error, dev nvme0n1, sector 264208 op 0x1:(WRITE) flags 0x29800 phys_seg 1 prio class 2
[95256.712397] I/O error, dev nvme0n1, sector 264208 op 0x1:(WRITE) flags 0x29800 phys_seg 1 prio class 2
[95256.722258] md: super_written gets error=-5
[95256.727133] md/raid1:md0: Disk failure on nvme0n1p2, disabling device.
[95256.727133] md/raid1:md0: Operation continuing on 1 devices.
[95256.742401] I/O error, dev nvme0n1, sector 77334752 op 0x1:(WRITE) flags 0x4000800 phys_seg 1 prio class 2
[95256.753375] BTRFS error (device nvme0n1p3): bdev /dev/nvme0n1p3 errs: wr 1, rd 1, flush 0, corrupt 0, gen 0
[95256.764177] I/O error, dev nvme0n1, sector 77335776 op 0x1:(WRITE) flags 0x4000800 phys_seg 1 prio class 2
[95256.774805] BTRFS error (device nvme0n1p3): bdev /dev/nvme0n1p3 errs: wr 2, rd 1, flush 0, corrupt 0, gen 0
[97602.825969] ata6.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[97602.833948] ata6.00: failed command: WRITE DMA EXT
[97602.839911] ata6.00: cmd 35/00:00:78:5a:4c/00:04:09:00:00/e0 tag 22 dma 524288 out
[97602.839911]          res 40/00:01:06:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[97602.858583] ata6.00: status: { DRDY }
[97602.863617] ata6: hard resetting link
[97603.337938] ata6: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[97603.346750] ata6.00: supports DRM functions and may not be fully accessible
[97603.370306] ata6.00: supports DRM functions and may not be fully accessible
[97603.430476] ata6.00: configured for UDMA/133
[97603.445466] ahci 0001:03:00.0: port does not support device sleep
[97603.452251] ata6: EH complete
[97637.643844] BTRFS warning (device dm-1): csum failed root 263 ino 3692950 off 386400256 csum 0xd04e5f48 expected csum 0x6b9afaa1 mirror 1
[97637.657936] BTRFS error (device dm-1): bdev /dev/mapper/dshelf2 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[97638.110104] BTRFS warning (device dm-1): csum failed root 263 ino 3692950 off 386400256 csum 0xd04e5f48 expected csum 0x6b9afaa1 mirror 1
[97638.123856] BTRFS error (device dm-1): bdev /dev/mapper/dshelf2 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[97662.159091] BTRFS warning (device dm-1): csum failed root 263 ino 3692950 off 386400256 csum 0xd04e5f48 expected csum 0x6b9afaa1 mirror 1
[97662.173941] BTRFS error (device dm-1): bdev /dev/mapper/dshelf2 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
[97662.906008] BTRFS warning (device dm-1): csum failed root 263 ino 3692950 off 386400256 csum 0xd04e5f48 expected csum 0x6b9afaa1 mirror 1
[97662.920993] BTRFS error (device dm-1): bdev /dev/mapper/dshelf2 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0

Recovering unusable DRAM-less Teamgroup drives

By then it was impossible to read or write to the 3 Teamgroup drives that failed, si I had to blkdiscard (TRIM) the entire 3 crashed drives (out of 7) to restart with 0's everywhere (which included full data loss of course), and start over.
Gemini gave me linux kernel sata and PCI options to make it less likely for this to happen again, but it also warned me it very much can happen again and DRAM less drives should never be behind a PCI switch.

At the same time, it became painfully obvious that the rPi5 has single lane PCI, and all those PCI switches are adding more channels while dividing the single lane bandwidth, making things slower and slower, was a bit of fool's errand.
By then, I had to admit defeat and since I wanted to run frigate for my cameras anyway, Gemini suggested I get an N355 based server which has an H264 and H265 ASIC for all those video streams (while rPi5 would have to do it in software), and at least 4 PCI lanes, which is much better (it's still 4x single lane M2 NVME, but at least 4 times faster and without a PCI switch to confuse things and cause full hangs if a single sata drive is freezing while writing its data)

See more images for Attempt to replace my 60TB Spinning Rust NAS with a Raspberry Pi 5 with Quad PCI Splitter and 18 2.5 Sata SSDs TLC and DRAM-less TeamGroup QLC drives: Unreliable single Lane PCI bus Under Load

2026/02/14 TCL 98QM8K vs Samsung QN90D and Denon AVR3808

π 2026-02-14 01:01 in Linuxha, Public

I started with a Samsung QN90D which talked perfectly to my Denon AVR3808CI AV receiver. It was able to talk HDMI CEC to it, it knew how to send volume commands directly to the receiver while sending sound over optical fiber given that the receiver only supports HDMI 1.3 CEC but not HDMI 2.0 with eARC (audio return channel).

Unfortuantely the samsung had some ridiculous OS that kept forcing me into making and using a Samsung account for just about anything, including watching youtube, it was terrible. So, I decided to switch to a TCL instead. The TV panel is great, the Google TV OS on it is decent, but:

they are using a 32 bit ARM CPU in their top of the line TV. It will stop working in 12 years due to time_t year 2038 overflow because they saved a few dollars on the CPU (android will never support a 64bit time_t migration in 32bit userland because everyone will have or should have switched to 64bit CPUs)

the TV refused to use HDMI-CEC as soon as I turned optical output for sound. That same HDMI-CEC did work otherwise

Their GoogleTV also helpfully removed infrared support in the remote, so it cannot send IR codes to the receiver even though the remote and googleTV support it.

Then I spent 2.5 months with their tech support team. Worst experience ever. They asked me for my feedback, this is what I sent them. Not impressed TCL, this is a new level of amateur hour I had never seen in a company that size. Actually I also forgot to mention that after my TV never getting any update whatsoever, and support saying I should have a later version that wasn't pushing, they sent me one by Email, with a link to their intranet, requiring a microsoft live.com account, and eventually telling me I'm not a TCL intranet user so I can't download it. To this day, they have no idea how to fix it or how to send a file to any customer.

First, I want to point out that I'm an engineer who has worked with android more than 15 years and filed over 1000 android bugs at google. I have also worked with many tech companies, found bugs in their products and helped them address them.
I'm sorry to say, but this has been the worst support experience of any major company I've ever dealt with.
1-2 days after I bought the TV, now 2.5 months ago I immediately called you to report the biggest issue with your software, namely that when using an older AV receiver that supports HDMI 1.3 with CEC, your TV is able to talk CEC to it, but as soon as I switch the TV sound to optical out, it refuses to send CEC commands over the HDMI cable coming from the receiver. This works perfectly with the Samsung TV I had, and fails with your because your engineers not only didn't test this, but did not even understand or envision that this exists and that it should work.
The first agent, after being confused about how HDMI CEC works, said she would file a bug or a report and someone would get back to me. After a month, no one ever did, and no report was filed, it was just entirely lost.
Now, what a normal company would have done is filed a bug, sent it to their engineering team for review, and they would decide whether they want to support this use case.
With you, nothing went anywhere, no one got back to me when they said they would, and it took over 10 phone calls and followups from me to get some traction, often wasting 30 to 60 minutes with agents that simply do not understand that eARC and HDMI CEC are separate and that you can have one without the other. They kept repeating it's impossible to send volume commands over an optical cable, which is totally true, and would not understand that HDMI CEC is separate, is what the volume commands are sent over, and that it works totally fine with both your TV and my denon AVR 3808 receiver, until I turn optical output on and then you disable it.
I spent over 5H on the phone with mostly agents that were undertrained, were unable to access my ticket in your own system (I should add it took you over 1.5 months before you even filed this ticket after I called multiple times asking why I had no ticket, no Email and no status). Why are your support agents not even able to access the ticket system, what madness is this?
Emails sent in reply to the ticket went 100% unanswered, so I would have to call to get an agent to ask me questions for 10 to 30 minutes before they would have to call someone else to access the ticket system they can't see, and get what I typed there.
I also tried to file 2 more bugs, which I think were both ignored, one of them being that Google TV, the HDMI dongle buy from google supports sending IR codes from the remote to the receiver to change volume directly. Once I re-paired my googletv remote with your TV, your TV wiped my remote, removed the IR codes that were working, and you also removed the part of googleTV that supports re-adding IR code to my remote. This is another example of already working functionality that you have removed, and no support agent ever acknowledged this and maybe filing a bug to consider adding it back, would be a good idea.

Jow that you have wasted 2.5 months of my time before maybe finally filing one bug out of 3 internally, too late for me to return the TV and too long for me to keep waiting for this to ever work, so now I'm going to lose $1000 to $2000 to replace my top of line AV receiver with a newer one that supports eARC because of your unwillingness to support HDMI CEC without eARC or have ever considered how many AV receivers are in the field that do not support eARC and yet support HDMI CEC

2025/07/07 Using Raspberry Pi5 as a Server With Raid1, Btrfs, and Multiple NVME M2 or Sata Drives

π 2025-07-07 01:01 in Btrfs, Linux, Linuxha

I was looking at replacing some ancient servers with lower power rPi5s after figuring out that they could finally work with proper NVME andSata drives so you can have a real server setup on reliable storage, including raid1, btrfs backups and all that good stuff.

what a Pi5 looks like

>>> Go Here for How to Setup Serial Boot And All The Config Files <<<

This is what I found with some research:

https://www.amazon.com/dp/B0B5RJHYFD or https://www.amazon.com/dp/B0D8BCWHPT are M2 M key SATA cards. They're really cool, you get 6 SATA slots from a single M2 slot (without power, though). But make sure you buy the class (M version), not B+M (2 notches). As I found out the hard way, B+M M2 does plug into M key slots, but are really only compatible with B key slots (details explained here: https://www.partitionwizard.com/partitionmanager/m2-m-vs-bm.html )

https://www.amazon.com/dp/B0D9D2W8MF very cool 4x M2 M-key slot expansion board, compatible with the M2 NGFF Sata Cards mentioned above. The one thing to note is that it does not provide 5V power anywhere, which is unfortunate if you want to then plug SATA drives (flash 2.5" sata recommended for power reasons).

https://www.amazon.com/dp/B0D3LP9KBH is a more compact 2 M2 NVME expansion board. It also could support SATA drives if you use the Sata M-key PMP mentioned above

https://www.amazon.com/dp/B0DQV74TDJ GeeekPi S021 SATA 3.0x2 for Raspberry Pi 5, with Active Cooler and 12V 6A Power Supply is probably the best if you only need to plug 1 or 2 SATA drives that require actual power. This board does provide power for actual spinning rust drives should you want that.

https://www.amazon.com/dp/B0D2VTL9G5 GeeekPi Dual FPC PCIe HAT for Raspberry Pi 5, B12 HAT 1 to 2 PCIe Interface allows you to use 2 of the boards above should you wish although routing cables is going to be a bit rough

https://www.amazon.com/dp/B0D5CRSC64 GeeekPi Quad FPC PCIe HAT for Raspberry Pi 5, B14 HAT 1 to 4 PCIe Interface is the same but with 4 outputs. The seller shows how to use this for single NVME boards, and honestly you probably don't want to do this when you can buy the dual or quad NVME board all in one.

While the PCIe expander cards are cool, and I have successfully managed to make a quad NVME board work at the same time as the SATA board listed below, in real life you'd likely want to use the dual or quad NVME board and find 5V power for your SATA drive. This is fine as long as you have a lower power drive (laptop flash) as you can steal 5V from the GPIO pins, but if you need more amps for a real drive or multiple flash sata drives, then you want to look into a real external 5V power supply like the one provided in the GeeekPi S021 SATA 3.0x2 for Raspberry Pi 5, with Active Cooler and 12V 6A Power Supply mentioned above.

Realistically you may also be perfectly happy with a USB 3 to 2.5" SATA adapter since it can still give you 3GB/s for flash drives

This is what it looks like:

you can easily have 24 sata drives with the 4X NVME board and 6 sata port NVME M2 cards

yes, I got excited when I got both SATA and NVME on a Raspberry Pi 5 ;)

this PCIe doubler can be used to chain other cards (not needed for me)

if you want, you can use the dedicated SATA card plugged directly into the Pi or PCIe doubler

be careful not to get a SATA PMP M2 (B+M with too notches) as those won't work. The other is a PCI sata card

when using M2 Sata cards, you can steal 5V power from GPIO

For my 2nd server, I just used a dual NVME HAT and SATA card with 6 ports

different vendors of SATA cards of either 5 or 6 ports

the PCIe 2 or 4 port expanders allow lots of stacking, but the example they give with single NVME M2 adapters, are useless since I bought a 4 slot NVME HAT

>>> Go Here for How to Setup Serial Boot And All The Config Files <<<

You can also see that video on how to turn a Pi5 into a high performance NAS:

See more images for Using Raspberry Pi5 as a Server With Raid1, Btrfs, and Multiple NVME M2 or Sata Drives

2025/07/07 Using a Raspberry Pi 5 (Rpi5) as a Server With Btrfs, Raid1, Serial Console and Dual NVME/SD Card Recovery Boot

π 2025-07-07 01:01 in Btrfs, Linux, Linuxha

I've been looking at using a Pi5s to replace some old power hungry intel servers I have that are ultimately close to 20 years old. I know there are plenty of random small SBC servers out there, but I didn't want to get locked in some vendor solution, and I figured if I could avoid spending $500+ for custom servers from a company that may not be around in 10 years, I should.
The big thing I wanted was a very low power, high efficiency server with a BMC, which doesn't quite define the Rasberry Pi 5, but since I had been working with rPi3 and rPi4 for my LED Outfit, I was curious to see if I could reasonably use a rPi5 as a headless server.

>>> Go Here for How to Setup NVME and Sata on a Pi5 <<<

I figured I would give rPi5s a try, especially as they now natively supported PCIe cards including M2 NVME storage, and with the right adapter card, PCIe SATA (bypassing USB). Most importantly, they now have some reasonably capable flashable firmware that allows booting from NVME (or USB storage), bypassing the sdcard entirely, which we all know is not a great proposition for a reliable server since sdcards are very much known to die, even if you buy the more fancy ones.
My other requirements were to be able to:

use raid1 (or raid5) for the root filesystem

use btrfs, I rely on snapshots and btrfs send/receive backups too much to want ext4

serial console for debugging if networking fails

serial console for debugging if boot fails (sadly they are not the same)

remote power reboot of some kind

Serial and Sysrq

All of these, I had without issues in the current servers I use, either with a BMC which effectively is its own computer with its own IP to monitor and power cycle the main computer, or a bios with serial support, which thankfully they've had for 25 years or more, and some remote power cycle via PDU.

I quickly found out that rPi5 would more or less do all of these, except the boot time serial support. The boot firmware does support outputting to the built in debug serial port on the board, ttyAMA10, if you forgive the somewhat unobtanium 3 pin plug they felt compelled to use and that was honestly hard to procure until I found USB serial adapters that included that wretched 3 pin plug. I mean, was it so bad to re-use the GPIO spaced pins used for GPIO?
As I found out later, it deos not seem that rPi5 really meant for users to be using the serial port like you would on a real server, which is disappointing (developers told me that really everyone uses HDMI or HDMI KVMs, never mind that KVMs cost more per port than the rPi itself, sigh...). I was also pretty annoyed because I have lots of similar looking 3 pin plugs from my RC planes used for I2C on ardupilot and they are literally off by a portion of 1mm. The lack of standardization and pointless spread of incompatible connectors is utterly ridiculous.

this adapter works, offers ttyUSB0 and the right Rpi5 compatible plug

this other adapter kind of works, but offers ttyACM0 instead which is not compatible with break/sysrq

see the small UART plug between HDMI0 and HDMI1, sad they didn't just use normal pins with the same spacing as GPIO ones

In summary: /dev/ttyACM USB adapters do not support sending BREAK (ALT+F in minicom) and therefore are not suitable since you will not be able to send sysrq commands for reboot. Make sure you adapter provides /dev/ttyUSB0 and that BREAK works.

So, once you have serial working, you make sure sysrq works and you're set, correct? Well, not really:

I didn't find any good way to remotely power cycle the Pi via a GPIO without using an externally controllable relay and splicing a USB-C power cable. So make sure to have panic=15 or equivalent in your /boot/cmdline.txt to make sure the kernel will never just hang.

never use halt -p since you will not be able to recover (I don't know if you can wire power switch to other pi to actuate it without a relay). Similarly, if you send sysrq-o, it will power down the rPi, and you'll be out of luck after that.

sysrq is absolutely required for reboots if things are bad, even a mere systemd shutdown bug, which of course I encountered while installing watchdog, but it only works over serial with your USB-serial adapter supports it. FTDI compatible ones do, the ttyACM0 did not.

setup hardware watchdog while you're at it (the watchdog package)

rPi5 Early Boot Configuration and rescue booting over serial

Rpi5 doesn't have a PC style bios that boots an MBR (thankfully), or EFI which honestly is way too complex and unnecessary, so they made their own boot firmware that is a able to enumerate devices somewhat (at least the first one of each type, sdcard, USB storage, NVME, SATA), and can be told to boot that.

How do you tell it to boot a specific device? The answer is it depends, the boot menu is only available on HDMI (not serial) and only if the eeprom cannot boot the default device, or takes too long to do so. Think of it as lilo like if you remember those days: you configure it from an already working system, and if your system does not boot, you cannot select another boot option unless it was already pre-configured (if you get really hosed, there is some rescue way to reflash a firmware from an sdcard).

The limitations are:

your boot options can only have a single linux kernel per device

you cannot change the linux boot command line without rebooting from another OS to edit cmddline.txt

you cannot change which device to boot from, over serial, same as previous

Those were all pretty serious limitations for me: how can I fix a non booting linux system if I can't do any of the above over serial (doing it over HDMI is a non starter for me since it's for remote use in a colo or a closet, and I have 0 interest in using an IP KVM to digitize and send HDMI output over the internet when clearly this should just work over serial on a slow connection and displayed in any terminal including one on my phone.

Now, the above bug mentions doing network booting. It's true that if you set that up, you can simply change the filesystem that is being seen remotely, but I'm not a fan of network filesystems as far as root filesystem is involved, and now you need a 2nd server to be the network filesystem for the first one, so if you use a Pi for that 2nd server, how do you remotely manage that one?

This is where I filed this bug: RFE rPi5: Need Boot Media Selection Menu On Build in Serial (ttyAMA10) and found out that I was the first person asking for this/needing this. This confirmed that rPi5 isn't yet being used in datacenters or remote servers for deployments that need high reliability, including:

editing/fixing the linux boot command line of a non bootable system

reverting to an older kernel if the new one doesn't boot, including its initramfs

booting from another device if the boot device fails to boot (drive/device that went bad)

While I hope the above RFE will be addressed at some point, rescue booting without a KVM was a 100% requirement for me, so I found an acceptable workaround which is to basically use a 2nd Pi to monitor the first one and be a backup server if the first Pi dies or becomes entirely unbootable, and use a GPIO trick that the bootloader does allow to boot from a different device.

The booting firmware is configured with rpi-eeprom-config --edit, which saves your settings into a file that is read at the next boot, updates the firmware, causes a 2nd reboot and then boots with your new settings. This is what I used for my setup (please note you should not trust ChatGPT or Gemini to give you correct values for this as they will literally invent some that do not exist):

aragorn:~# rpi-eeprom-config [all] # These two are needed for early boot and bootloader over serial CONSOLE=UART BOOT_UART=1 # Avoid this: # 0.96 RPi: BOOTSYS release VERSION:69471177 DATE: 2025/05/08 TIME: 15:13:17 # 0.96 BOOTMODE: 0x06 partition 63 build-ts BUILD_TIMESTAMP=1746713597 serial 2aef62b3 boardrev e04171 stc 965833 # 0.97 AON_RESET: 00000003 PM_RSTS 00001575 # 0.97 POWER_OFF_ON_HALT: 1 WAIT_FOR_POWER_BUTTON 0 power-on-reset 0 # 0.98 RP1_BOOT chip ID: 0x20001927 # 0.99 Halt: power_off: 1 POWER_OFF_ON_HALT=0

# tries other non bootable before nvme to bring menu on HDMI # read right to left, 4: USB, 6: NVME, 1: sdcard, f: restatt #BOOT_ORDER=0xf164 # boot nvme first and if not, sdcard BOOT_ORDER=0xf16 # Sometimes needed by NVME drives to show up in time for boot PCIE_PROBE=1

# You should only use GPIO 22 to 27 as they are pull down # by default [gpio26=1] # this would boot sdcard directly: #BOOT_ORDER=0xf1 # boot other absent devices first force menu before choosing sdcard # read right to left, 4: USB, 1: sdcard, 6: NVME, f: restart BOOT_ORDER=0xf614

What does this mean? If GPIO26 is taken high (and it is pulled down by default), it will use the 2nd BOOT_ORDER to boot. That 2nd one is designed to boot from a non existent usb storage, which because of timeout will cause the boot menu to show up on HDMI should you at a local console. Then this times out, and continues with sdcard boot which will have a default rescue linux install I can use to fix the main NVME install that isn't booting anymore.
Just like lilo as opposed to grub, you cannot edit the linux boot command line if it is bad (it's saved in /boot/cmdline.txt which is really /boot/firmware/cmdline.txt, which is saved on the vfat partition that the firmware is able to open and read from a bit like a poor man's EFI partition.
config.txt similarly includes lots of options used by the booting firmware, including which initramfs, if any, to give to the booting kernel. Note that only one booting kernel is allowed by boot device, so you have no way to rollback kernels or fix anything without some rescue media, which in my case is the sdcard boot if my NVME boot stops working. I'll show my relevant config below.


aragorn:~# cat /boot/cmdline.txt 
root=/dev/md0 rootflags=subvol=root rootwait net.ifnames=0 logo.nologo console=tty1 console=ttyAMA10,115200 panic=15 rd.shell

aragorn:~# grep -1 -i initramfs /boot/config.txt 
# >>>  kernel and initramfs come from from /boot/firmware/, not /boot/ <<<
# initramfs must be copied manually
# -rwxr-xr-x 1 root root 22792837 Jul  5 06:08 initramfs_2712
# -rwxr-xr-x 1 root root  9962173 Jul 11 05:46 kernel_2712.img
initramfs initramfs_2712 followkernel

# This should help if you have multiple kernels
auto_initramfs=1

aragorn:~# tail -5 /boot/config.txt 
dtparam=watchdog=on

# This is suggested by PCI expander, but seems unncessary
#dtparam=picex1
#dtparam=picex1_gen=3

Ok, so let's recap. The above allows booting on NVME by default (I made the NVME bootable device by copying a working bootable sdcard onto an NVME after plugging it in a working rPi), but if you toggle the correct GPIO, it will boot from sdcard instead of NVME allowing a rescue boot over serial. The only thing to be careful of is you cannot shut down the device (halt or otherwise) as you have no way to power it back up that I know about (without an external relay to toggle power or the power switch).

This is what it looks like. Only pay attention to the black/white/red servo cable with red and white swapped, just like you would for RX/TX on a serial connection:

Now, you're going to ask me about the blue and yellow probes, and that was because I found out a vexing bug I was not able to explain: when reading the GPIO output on pin 19, if it's set to 1 (asking for rescue boot from the other pi), the voltage dips from 3.3V to 1.5V which is an invalid value and causes the other pi not to detect that you want it to boot in rescue mode.
You will want to use rpi-boot-rescue (excerpt below) or write your own, but basically I found out that gpioget damages the output bit when you read it, despite the correct push/pull values. I'll give the meat of the script here:

export GPIOSND=19
export GPIORCV=26

if [ "$1"  "cycle" || "$1"  "--cycle" ]]; then
    $0 --set
    sleep 45
    $0 --unset
elif [ "$1"  "get" || "$1"  "--get" ]]; then
    echo "Boot rescue bit $GPIOSND: $(gpioget gpiochip0 $GPIOSND) (received by boot pin $GPIORCV)"
    # I do not understand why readin gthe value messes up the output votlage if output it 1, but it does
    echo "Please reset pin value as reading it can change the output voltage from 3.3V to 1.5V"
elif [ "$1"  "set" || "$1"  "--set" || "$1"  "rescue" || "$1"  "-y" || "$1" = "--yes"  ]]; then
    echo "Enable GPIO $GPIOSND to enable >>>RESCUE<<< boot on connected device (received on $GPIORCV)"
    rescue=1
    gpioset -B pull-up -D push-pull gpiochip0 $GPIOSND=$rescue || echo "apt install gpiod"
    echo "Don't forget to reset to normal after your debug boot"
else
    echo "Ground GPIO $GPIOSND to enable normal boot on connected device (received on $GPIORCV)"
    rescue=0
    gpioset -B pull-down -D push-pull gpiochip0 $GPIOSND=$rescue || echo "apt install gpiod"
fi

good 'rescue' output

becomes bad voltage when you use gpioget

this is what gpioget does

After figuring out this issue with bad voltages that prevented my rescue boot from working, now it does and I can use Pi #2 to tell Pi #1 to boot into sdcard for rescue purposes.

TODO: try pinctrl instead of gpiod (I'm told it may not have that issue where reading the pin affects the voltage).

If you think this is unnecessary or overkill, keep in mind that the first dietpi upgrade I did failed to make a new initramfs (which isn't supported by them), I stupidly rebooted, and then the initramfs did not match the new kernel so the kernel modules in the initramfs refused to load. As a result, the root filesystem could not mount and the system was completely unusable and indeed impossible to rescue without the rescue sdcard boot I had just made.

Now, you're going to tell me you're not going to use raid1 or btrfs and you don't need initramfs. Sure, but you're still one mistake away from an unbootable system, be it fstab, kernel command line, bad kernel or kernel module, or simply your boot device going bad, or your root filesystem getting damaged and becoming unbootable. I will agree that none of those happen often on an average day, so for your little home project, you could decide not to care, but if you're actually wanting to use a rPi5 as a real server, serving real users, and without always having local access to it, what I did above will make a lot more sense to you.
As a bonus the 2nd pi that monitors the first one can be used as a backup server to take over (or both can run at the same time if you have some service that can work with load balancing). In my case, both Pis can monitor one another.

Boot Configuration

I'll just share what mine looks like if that helps, yours will be different

aragorn:~# cat /etc/fstab tmpfs /tmp tmpfs size=8095M,noatime,lazytime,nodev,nosuid,mode=1777

LABEL=btrfs_boot / btrfs nofail,subvol=root 0 0 LABEL=btrfs_pool1 /var/local/space btrfs defaults,subvol=pool1 0 0 LABEL=btrfs_pool2 /var/local/space2 btrfs defaults,subvol=pool2 0 0

LABEL=btrfs_boot /mnt/btrfs_boot btrfs noatime,lazytime,rw LABEL=btrfs_pool1 /mnt/btrfs_pool1 btrfs noatime,lazytime,rw,noauto,x-systemd.automount LABEL=btrfs_pool2 /mnt/btrfs_pool2 btrfs noatime,lazytime,rw,noauto,x-systemd.automount UUID=84958e33-a82c-4864-83b9-f483529a3bed /mnt/sdcard ext4 noatime,lazytime,rw,noauto,x-systemd.automount

UUID=7E8C-86F8 /boot/firmware vfat noatime,lazytime,rw 0 2 UUID=F6DA-B64E /mnt/sdcard_boot vfat noatime,lazytime,rw,noauto,x-systemd.automount UUID=7E8C-86F8 /mnt/firmware1 vfat noatime,lazytime,rw,noauto,x-systemd.automount UUID=4122-81AA /mnt/firmware2 vfat noatime,lazytime,rw,noauto,x-systemd.automount aragorn:~# cat /proc/partitions major minor #blocks name 259 0 500107608 nvme0n1 259 1 131072 nvme0n1p1 259 2 31034880 nvme0n1p2 259 3 468940120 nvme0n1p3 259 4 488386584 nvme1n1 259 5 131072 nvme1n1p1 259 6 31034880 nvme1n1p2 259 7 457219096 nvme1n1p3 179 0 31166976 mmcblk0 179 1 131072 mmcblk0p1 179 2 31034880 mmcblk0p2 8 0 58615704 sda 9 0 31017472 md0 aragorn:~# cat /proc/mdstat Personalities : [raid1] md0 : active raid1 nvme0n1p2[0] nvme1n1p2[1] 31017472 blocks super 1.2 [2/2] [UU]

Btrfs, Initramfs, Dracut

If you don't care about having raid1 for reliability, and don't care about btrfs and its snapshots that can be used to go back in time and erase mistakes, as well as make frequent and lightweight backups using btrfs send, then you don't care about initramfs and you can skip this section.

But if you do, read on... The first thing to know about initramfs is that it works, but it's not supported by dietpi and may not fully by supported by some other rPi distros, by that I mean that installing new kernels would automatically make an initramfs, copy it in the right place, and make sure it gets loaded along with that new kernel. This means if any kernel is updated, you must make a new initramfs, make sure it ends up in /boot/firmware and that it is correctly referenced in /boot/firmware/config.txt . If you are lucky it might all happen automatically when you install a new kernel (as it normally does on regular linux distributions), but on dietpi it did not and gave me an unbootable system due to mismatch of modules version in initrd. Feedback I got after writing this page is that it's supposed to "just work" on the stock RPI OS, which is good news.

aragorn:~# grep -1 -i initramfs /boot/config.txt

# >>> kernel and initramfs come from from /boot/firmware/, not /boot/ <<< # initramfs must be copied manually # -rwxr-xr-x 1 root root 22792837 Jul 5 06:08 initramfs_2712 # -rwxr-xr-x 1 root root 9962173 Jul 11 05:46 kernel_2712.img initramfs initramfs_2712 followkernel

# This should help if you have multiple kernels auto_initramfs=1

Now, because you need panic=15 or equivalent so that your kernel does not just hang and never reboot if something goes bad, this unfortunately also tells initramfs to reboot if it can't mount the root filesystem instead of giving you a debug shell. You can tell it to give you one with break=premount, but you cannot add kernel command line options at boot time on a pi, so I thought I would switch to dracut instead which with "rd.shell" as a boot option will indeed give you a shell to debug and fix things if needed, without paying attention to panic=15

Bad, initramfs will not give you a shell if it can't mount, unless you give it break=premount but you cannot edit the command line at runtime

Gave up waiting for root file system device.  Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
 - Missing modules (cat /proc/modules; ls /dev)
ALERT!  /dev/md0 does not exist.  Dropping to a shell!
Halting automatically due to panic= boot argument

root=/dev/md0 rootflags=subvol=root rootwait net.ifnames=0 logo.nologo console=tty1 console=ttyAMA10,115200 rd.shell panic=15
[   34.842920] reboot: System halted

Good (use dracut instead of initramfs):

[  197.820143] dracut-initqueue[337]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fmd0.sh: "
if ! grep -q After=remote-fs-pre.target /run/systemd/generator/systemd-cryptsetup@*.service 2>/dev/null; then
         Starting dracut-emergency.ï��ce - Dracut Emergency Shell...
[  197.848080] dracut-initqueue[337]:     [ -e "/dev/md0" ]
[  197.848136] dracut-initqueue[337]: fi"
[  197.848171] dracut-initqueue[337]: Warning: dracut-initqueue: starting timeout scripts
[  197.848205] dracut-initqueue[337]: Warning: Could not boot.
Warning: "/dev/md0" does not exist

Generating "/run/initramfs/rdsosreport.txt"

Entering emerPress Enter for maintenance
(or press Control-D to continue): 
#   <-- here you can debug and fix your system

This is what I do to generate my dracut initramfs, but note that dietpi dracut does not know to copy the initramfs to /boot/firmware, so you will have to do so, or nothing will work:

aragorn:~# cat /var/local/scr/rpi-dracut 
#!/bin/bash

#dracut --install "bash find ls" --regenerate-all -p --force
dracut --install "bash find ls grep more vi vim vim.nox" --regenerate-all -p --force --no-hostonly
cat <<EOF

Add rd.break=premount to force a debug shell (which must be done via rpi-boot-rescue 
and the rescue partition).

man dracut.kernel and dracut for more details

vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
You must copy generated initrd to /boot/firmware and check path in /boot/config.txt
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
EOF

There you go, hope this helps. Again, the initrd bit is not needed for most users unless you use a raid root filesystem, btrfs, or other more exotic configurations like this.

>>> Go Here for How to Setup NVME and Sata on a Pi5 <<<

See more images for Using a Raspberry Pi 5 (Rpi5) as a Server With Btrfs, Raid1, Serial Console and Dual NVME/SD Card Recovery Boot

2025/01/17 House Lights with Pixelblaze on ESP32 monitored with Raspberry Pi

π 2025-01-17 01:01 in Arduino, Electronics, Linuxha

got a few extra colorful yard lights ;)

To debug some early issues I had with the pixelblaze, I soldered a few extra wires to add serial monitoring:

the serial port is temporarily going to a rPi3a which in turn makes it available over an ssh connection

up left if the pixelblaze pro expansion board that gives 8 channels, handles power distribution and converts the 12V to 5V for the pixelblaze itself

For the pixels, I picked rolls of 1000 pixels at 5cm and 10cm pitch from Ray Wu, trusted seller of blinky stuff, and one bonus of not doing this installation a few years earlier is that new pixels have been designed: WS2818.

WS2813 were a improvement for having a backup data line but still running on 5V which would have clear voltage drop issues.

WS2815 are the 12V version, which is good for dealing with voltage drop

WS2818 is yet another improvement over WS2815, still 12V with a backup data line, but more reliable and efficient

The last bit "more efficient" actually worked to my advantage as I was able to make a string with 600 pixels and not have to re-inject power anywhere. I measured line voltage at the end and it was only 5V but the pixels still worked great at the reduced voltage.

The advice I got on the leds are awesome group was to try hot glue, and it was a good idea. Thankfully I had a battery powered hot glue gun, which was a must have. The other important bit was a solder reflow hot air gun, which did have to be plugged in the few times I had to unglue strips to move them after I found a better routing or visual pattern:

And in all, I actually need 3 tools, the 3rd one was a battery operated air can replacement which allowed me to blow cold air on the hot glue points and have them dry in 10 to 15 seconds instead of 1mn. That was a lifesaver:

After laying strips, house looked like this:

After the first 2 days of work, I had 4 strips, around 1500 LEDs:

or on youtube:

But it really got better once I added 2 more strips and upp'ed the count to 2000, which was a lot more visually pleasing:

or on youtube:

This is my current demo list:

Blink fade

Color twinkles

Fireflies

Honeycomb

Matching rainbows

Rainbow melt

Show color shift

Sound blink fade

Pew pew

Sound rays

Spark center

Static random colors

These look interested but are too fast:

Millipede

Pulse 2e

Spin cycle

See more images for House Lights with Pixelblaze on ESP32 monitored with Raspberry Pi

2023/03/09 Solar Monitoring Bit Flip Rabbithole

π 2023-03-09 01:01 in Linux, Linuxha

My own "I'm just going to spend 5mn to fix this" story, that lasted over 6H.

Power outage rebooted one of my energy monitors and it's now counting one of my solar arrays backwards because of an apparent bit flip in the flash (it's an old device)

10 days later I'm finally home long enough to look at this, I "just" need to connect to it via some windows only app, connect via the serial port and reprogram the lost/corrupted settings

Ah, but my only desktop/windows machine left in the house blew a capacitor on its motherboard recently, and I haven't been able to justify the time to fix it (that would be a rabbit hole on its own and I know this), so I can't use that

unfortunately that one device only works with a real serial port, not any USB-serial adapters I've been able to try (and I have 4 different brands/chips), so that prevents use of any of my laptops

oh, wait, I do have a much older thinkpad with a docking station that has a _real_ serial port, let's try this.

ok, it boots, but the windows XP partition gives some "NTLDR missing". I haven't used it in some 15 years, so I wouldn't know

ok, let's boot linux, ubuntu oneiric, oh my. Virtualbox won't work, modules are broken, I try a few of the kernels, a mix of 32bit kernels with himem for the 8GB of RAM, and some x64 kernels with 32bit userspace. None of them want to build working virtualbox modules.

apt-get install is of course broken with a weird dependency chain, not going to go to _that_ rabbithole either (see, I've learned from previous times)

maybe I can make that WXP partition boot again? Ok, I'll skip details on how that ends up in cleaning half my bedroom while looking for old boot/recovery CDs (BartPE, UBCD, etc...), another 1.5h rabbithole

eventually get the boot CDs, they also BSOD during boot, look in bios, change the SATA support from AHCI to compatible, that fixes it

fast forward through some "details", boot WXP recovery console, apply "fixboot" and "fixmbr" which of course removes the install-mbr bootloader I had installed to dual boot WXP and llinux, and was not present on my linux recovery CDs

eventually boot some linux recovery that allows me to decrypt the linux partition (luksopen), chroot into it because it's not the same binary format, and install-mbr -t 54 -v -e 14FA -p 2 -is /dev/sda

eventually get WXP to boot, for some reason the wireless support does not work anymore and can't connect. Ok, use ethernet instead and everything is so old that none fhe web browsers work, because everything is https and the ciphers are all incompatible with older browsers now

retrieve the device windows software from my normal laptop, copy it over, install it, it runs, uses the real serial port on the ultrabay, and connects to my power monitor. I revert the broken settings on the device over serial, and save them.

SUCCESS!

That only took about 6H, why it it dark outside? Oh, it's past midnight. What was I doing before that?

Contact Email

Marc's Public Blog - Linux Home Automation