Marc's Public Blog - Linux Hacking

>>> Go Here for How to Setup NVME and Sata on a Pi5 <<<

I figured I would give rPi5s a try, especially as they now natively supported PCIe cards including M2 NVME storage, and with the right adapter card, PCIe SATA (bypassing USB). Most importantly, they now have some reasonably capable flashable firmware that allows booting from NVME (or USB storage), bypassing the sdcard entirely, which we all know is not a great proposition for a reliable server since sdcards are very much known to die, even if you buy the more fancy ones.
My other requirements were to be able to:

Serial and Sysrq

I quickly found out that rPi5 would more or less do all of these, except the boot time serial support. The boot firmware does support outputting to the built in debug serial port on the board, ttyAMA10, if you forgive the somewhat unobtanium 3 pin plug they felt compelled to use and that was honestly hard to procure until I found USB serial adapters that included that wretched 3 pin plug. I mean, was it so bad to re-use the GPIO spaced pins used for GPIO?
As I found out later, it deos not seem that rPi5 really meant for users to be using the serial port like you would on a real server, which is disappointing (developers told me that really everyone uses HDMI or HDMI KVMs, never mind that KVMs cost more per port than the rPi itself, sigh...). I was also pretty annoyed because I have lots of similar looking 3 pin plugs from my RC planes used for I2C on ardupilot and they are literally off by a portion of 1mm. The lack of standardization and pointless spread of incompatible connectors is utterly ridiculous.

this adapter works, offers ttyUSB0 and the right Rpi5 compatible plug

this other adapter kind of works, but offers ttyACM0 instead which is not compatible with break/sysrq

see the small UART plug between HDMI0 and HDMI1, sad they didn't just use normal pins with the same spacing as GPIO ones

In summary: /dev/ttyACM USB adapters do not support sending BREAK (ALT+F in minicom) and therefore are not suitable since you will not be able to send sysrq commands for reboot. Make sure you adapter provides /dev/ttyUSB0 and that BREAK works.

So, once you have serial working, you make sure sysrq works and you're set, correct? Well, not really:

rPi5 Early Boot Configuration and rescue booting over serial

How do you tell it to boot a specific device? The answer is it depends, the boot menu is only available on HDMI (not serial) and only if the eeprom cannot boot the default device, or takes too long to do so. Think of it as lilo like if you remember those days: you configure it from an already working system, and if your system does not boot, you cannot select another boot option unless it was already pre-configured (if you get really hosed, there is some rescue way to reflash a firmware from an sdcard).

The limitations are:

Those were all pretty serious limitations for me: how can I fix a non booting linux system if I can't do any of the above over serial (doing it over HDMI is a non starter for me since it's for remote use in a colo or a closet, and I have 0 interest in using an IP KVM to digitize and send HDMI output over the internet when clearly this should just work over serial on a slow connection and displayed in any terminal including one on my phone.

Now, the above bug mentions doing network booting. It's true that if you set that up, you can simply change the filesystem that is being seen remotely, but I'm not a fan of network filesystems as far as root filesystem is involved, and now you need a 2nd server to be the network filesystem for the first one, so if you use a Pi for that 2nd server, how do you remotely manage that one?

This is where I filed this bug: RFE rPi5: Need Boot Media Selection Menu On Build in Serial (ttyAMA10) and found out that I was the first person asking for this/needing this. This confirmed that rPi5 isn't yet being used in datacenters or remote servers for deployments that need high reliability, including:

While I hope the above RFE will be addressed at some point, rescue booting without a KVM was a 100% requirement for me, so I found an acceptable workaround which is to basically use a 2nd Pi to monitor the first one and be a backup server if the first Pi dies or becomes entirely unbootable, and use a GPIO trick that the bootloader does allow to boot from a different device.

The booting firmware is configured with rpi-eeprom-config --edit, which saves your settings into a file that is read at the next boot, updates the firmware, causes a 2nd reboot and then boots with your new settings. This is what I used for my setup (please note you should not trust ChatGPT or Gemini to give you correct values for this as they will literally invent some that do not exist):

aragorn:~# rpi-eeprom-config 
[all]
# These two are needed for early boot and bootloader over serial
CONSOLE=UART
BOOT_UART=1
# Avoid this:
# 0.96 RPi: BOOTSYS release VERSION:69471177 DATE: 2025/05/08 TIME: 15:13:17
# 0.96 BOOTMODE: 0x06 partition 63 build-ts BUILD_TIMESTAMP=1746713597 serial 2aef62b3 boardrev e04171 stc 965833
# 0.97 AON_RESET: 00000003 PM_RSTS 00001575
# 0.97 POWER_OFF_ON_HALT: 1 WAIT_FOR_POWER_BUTTON 0 power-on-reset 0
# 0.98 RP1_BOOT chip ID: 0x20001927
# 0.99 Halt: power_off: 1
POWER_OFF_ON_HALT=0

# tries other non bootable before nvme to bring menu on HDMI
# read right to left, 4: USB, 6: NVME, 1: sdcard, f: restatt
#BOOT_ORDER=0xf164
# boot nvme first and if not, sdcard
BOOT_ORDER=0xf16
# Sometimes needed by NVME drives to show up in time for boot
PCIE_PROBE=1

# You should only use GPIO 22 to 27 as they are pull down
# by default
[gpio26=1]
# this would boot sdcard directly:
#BOOT_ORDER=0xf1
# boot other absent devices first force menu before choosing sdcard
# read right to left, 4: USB, 1: sdcard, 6: NVME, f: restart
BOOT_ORDER=0xf614

What does this mean? If GPIO26 is taken high (and it is pulled down by default), it will use the 2nd BOOT_ORDER to boot. That 2nd one is designed to boot from a non existent usb storage, which because of timeout will cause the boot menu to show up on HDMI should you at a local console. Then this times out, and continues with sdcard boot which will have a default rescue linux install I can use to fix the main NVME install that isn't booting anymore.
Just like lilo as opposed to grub, you cannot edit the linux boot command line if it is bad (it's saved in /boot/cmdline.txt which is really /boot/firmware/cmdline.txt, which is saved on the vfat partition that the firmware is able to open and read from a bit like a poor man's EFI partition.
config.txt similarly includes lots of options used by the booting firmware, including which initramfs, if any, to give to the booting kernel. Note that only one booting kernel is allowed by boot device, so you have no way to rollback kernels or fix anything without some rescue media, which in my case is the sdcard boot if my NVME boot stops working. I'll show my relevant config below.

aragorn:~# cat /boot/cmdline.txt 
root=/dev/md0 rootflags=subvol=root rootwait net.ifnames=0 logo.nologo console=tty1 console=ttyAMA10,115200 panic=15 rd.shell

aragorn:~# grep -1 -i initramfs /boot/config.txt 
# >>>  kernel and initramfs come from from /boot/firmware/, not /boot/ <<<
# initramfs must be copied manually
# -rwxr-xr-x 1 root root 22792837 Jul  5 06:08 initramfs_2712
# -rwxr-xr-x 1 root root  9962173 Jul 11 05:46 kernel_2712.img
initramfs initramfs_2712 followkernel

# This should help if you have multiple kernels
auto_initramfs=1

aragorn:~# tail -5 /boot/config.txt 
dtparam=watchdog=on

# This is suggested by PCI expander, but seems unncessary
#dtparam=picex1
#dtparam=picex1_gen=3

Ok, so let's recap. The above allows booting on NVME by default (I made the NVME bootable device by copying a working bootable sdcard onto an NVME after plugging it in a working rPi), but if you toggle the correct GPIO, it will boot from sdcard instead of NVME allowing a rescue boot over serial. The only thing to be careful of is you cannot shut down the device (halt or otherwise) as you have no way to power it back up that I know about (without an external relay to toggle power or the power switch).

This is what it looks like. Only pay attention to the black/white/red servo cable with red and white swapped, just like you would for RX/TX on a serial connection:

Now, you're going to ask me about the blue and yellow probes, and that was because I found out a vexing bug I was not able to explain: when reading the GPIO output on pin 19, if it's set to 1 (asking for rescue boot from the other pi), the voltage dips from 3.3V to 1.5V which is an invalid value and causes the other pi not to detect that you want it to boot in rescue mode.
You will want to use rpi-boot-rescue (excerpt below) or write your own, but basically I found out that gpioget damages the output bit when you read it, despite the correct push/pull values. I'll give the meat of the script here:

export GPIOSND=19
export GPIORCV=26

if [ "$1"  "cycle" || "$1"  "--cycle" ]]; then
    $0 --set
    sleep 45
    $0 --unset
elif [ "$1"  "get" || "$1"  "--get" ]]; then
    echo "Boot rescue bit $GPIOSND: $(gpioget gpiochip0 $GPIOSND) (received by boot pin $GPIORCV)"
    # I do not understand why readin gthe value messes up the output votlage if output it 1, but it does
    echo "Please reset pin value as reading it can change the output voltage from 3.3V to 1.5V"
elif [ "$1"  "set" || "$1"  "--set" || "$1"  "rescue" || "$1"  "-y" || "$1" = "--yes"  ]]; then
    echo "Enable GPIO $GPIOSND to enable >>>RESCUE<<< boot on connected device (received on $GPIORCV)"
    rescue=1
    gpioset -B pull-up -D push-pull gpiochip0 $GPIOSND=$rescue || echo "apt install gpiod"
    echo "Don't forget to reset to normal after your debug boot"
else
    echo "Ground GPIO $GPIOSND to enable normal boot on connected device (received on $GPIORCV)"
    rescue=0
    gpioset -B pull-down -D push-pull gpiochip0 $GPIOSND=$rescue || echo "apt install gpiod"
fi

good 'rescue' output

becomes bad voltage when you use gpioget

After figuring out this issue with bad voltages that prevented my rescue boot from working, now it does and I can use Pi #2 to tell Pi #1 to boot into sdcard for rescue purposes.

TODO: try pinctrl instead of gpiod (I'm told it may not have that issue where reading the pin affects the voltage).

If you think this is unnecessary or overkill, keep in mind that the first dietpi upgrade I did failed to make a new initramfs (which isn't supported by them), I stupidly rebooted, and then the initramfs did not match the new kernel so the kernel modules in the initramfs refused to load. As a result, the root filesystem could not mount and the system was completely unusable and indeed impossible to rescue without the rescue sdcard boot I had just made.

Now, you're going to tell me you're not going to use raid1 or btrfs and you don't need initramfs. Sure, but you're still one mistake away from an unbootable system, be it fstab, kernel command line, bad kernel or kernel module, or simply your boot device going bad, or your root filesystem getting damaged and becoming unbootable. I will agree that none of those happen often on an average day, so for your little home project, you could decide not to care, but if you're actually wanting to use a rPi5 as a real server, serving real users, and without always having local access to it, what I did above will make a lot more sense to you.
As a bonus the 2nd pi that monitors the first one can be used as a backup server to take over (or both can run at the same time if you have some service that can work with load balancing). In my case, both Pis can monitor one another.

Boot Configuration

aragorn:~# cat /etc/fstab 
tmpfs /tmp tmpfs size=8095M,noatime,lazytime,nodev,nosuid,mode=1777 

LABEL=btrfs_boot / btrfs nofail,subvol=root 0 0 
LABEL=btrfs_pool1 /var/local/space  btrfs defaults,subvol=pool1 0 0 
LABEL=btrfs_pool2 /var/local/space2 btrfs defaults,subvol=pool2 0 0 

LABEL=btrfs_boot  /mnt/btrfs_boot  btrfs noatime,lazytime,rw 
LABEL=btrfs_pool1 /mnt/btrfs_pool1 btrfs noatime,lazytime,rw,noauto,x-systemd.automount 
LABEL=btrfs_pool2 /mnt/btrfs_pool2 btrfs noatime,lazytime,rw,noauto,x-systemd.automount 
UUID=84958e33-a82c-4864-83b9-f483529a3bed /mnt/sdcard ext4 noatime,lazytime,rw,noauto,x-systemd.automount 

UUID=7E8C-86F8 /boot/firmware vfat noatime,lazytime,rw 0 2 
UUID=F6DA-B64E /mnt/sdcard_boot vfat noatime,lazytime,rw,noauto,x-systemd.automount 
UUID=7E8C-86F8 /mnt/firmware1 vfat noatime,lazytime,rw,noauto,x-systemd.automount 
UUID=4122-81AA /mnt/firmware2 vfat noatime,lazytime,rw,noauto,x-systemd.automount 
aragorn:~# cat /proc/partitions 
major minor  #blocks  name 
 259        0  500107608 nvme0n1 
 259        1     131072 nvme0n1p1 
 259        2   31034880 nvme0n1p2 
 259        3  468940120 nvme0n1p3 
 259        4  488386584 nvme1n1 
 259        5     131072 nvme1n1p1 
 259        6   31034880 nvme1n1p2 
 259        7  457219096 nvme1n1p3 
 179        0   31166976 mmcblk0 
 179        1     131072 mmcblk0p1 
 179        2   31034880 mmcblk0p2 
   8        0   58615704 sda 
   9        0   31017472 md0 
aragorn:~# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 nvme0n1p2[0] nvme1n1p2[1]
      31017472 blocks super 1.2 [2/2] [UU]

Btrfs, Initramfs, Dracut

But if you do, read on... The first thing to know about initramfs is that it works, but it's not supported by dietpi and may not fully by supported by some other rPi distros, by that I mean that installing new kernels would automatically make an initramfs, copy it in the right place, and make sure it gets loaded along with that new kernel. This means if any kernel is updated, you must make a new initramfs, make sure it ends up in /boot/firmware and that it is correctly referenced in /boot/firmware/config.txt . If you are lucky it might all happen automatically when you install a new kernel (as it normally does on regular linux distributions), but on dietpi it did not and gave me an unbootable system due to mismatch of modules version in initrd. Feedback I got after writing this page is that it's supposed to "just work" on the stock RPI OS, which is good news.

aragorn:~# grep -1 -i initramfs /boot/config.txt 

# >>>  kernel and initramfs come from from /boot/firmware/, not /boot/ <<<
# initramfs must be copied manually
# -rwxr-xr-x 1 root root 22792837 Jul  5 06:08 initramfs_2712
# -rwxr-xr-x 1 root root  9962173 Jul 11 05:46 kernel_2712.img
initramfs initramfs_2712 followkernel

# This should help if you have multiple kernels
auto_initramfs=1

Now, because you need panic=15 or equivalent so that your kernel does not just hang and never reboot if something goes bad, this unfortunately also tells initramfs to reboot if it can't mount the root filesystem instead of giving you a debug shell. You can tell it to give you one with break=premount, but you cannot add kernel command line options at boot time on a pi, so I thought I would switch to dracut instead which with "rd.shell" as a boot option will indeed give you a shell to debug and fix things if needed, without paying attention to panic=15

Bad, initramfs will not give you a shell if it can't mount, unless you give it break=premount but you cannot edit the command line at runtime

Gave up waiting for root file system device.  Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
 - Missing modules (cat /proc/modules; ls /dev)
ALERT!  /dev/md0 does not exist.  Dropping to a shell!
Halting automatically due to panic= boot argument

root=/dev/md0 rootflags=subvol=root rootwait net.ifnames=0 logo.nologo console=tty1 console=ttyAMA10,115200 rd.shell panic=15
[   34.842920] reboot: System halted

Good (use dracut instead of initramfs):

[  197.820143] dracut-initqueue[337]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fmd0.sh: "
if ! grep -q After=remote-fs-pre.target /run/systemd/generator/systemd-cryptsetup@*.service 2>/dev/null; then
         Starting dracut-emergency.�ce - Dracut Emergency Shell...
[  197.848080] dracut-initqueue[337]:     [ -e "/dev/md0" ]
[  197.848136] dracut-initqueue[337]: fi"
[  197.848171] dracut-initqueue[337]: Warning: dracut-initqueue: starting timeout scripts
[  197.848205] dracut-initqueue[337]: Warning: Could not boot.
Warning: "/dev/md0" does not exist

Generating "/run/initramfs/rdsosreport.txt"

Entering emerPress Enter for maintenance
(or press Control-D to continue): 
#   <-- here you can debug and fix your system

This is what I do to generate my dracut initramfs, but note that dietpi dracut does not know to copy the initramfs to /boot/firmware, so you will have to do so, or nothing will work:

aragorn:~# cat /var/local/scr/rpi-dracut 
#!/bin/bash

#dracut --install "bash find ls" --regenerate-all -p --force
dracut --install "bash find ls grep more vi vim vim.nox" --regenerate-all -p --force --no-hostonly
cat <<EOF

Add rd.break=premount to force a debug shell (which must be done via rpi-boot-rescue 
and the rescue partition).

man dracut.kernel and dracut for more details

vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
You must copy generated initrd to /boot/firmware and check path in /boot/config.txt
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
EOF

There you go, hope this helps. Again, the initrd bit is not needed for most users unless you use a raid root filesystem, btrfs, or other more exotic configurations like this.

>>> Go Here for How to Setup NVME and Sata on a Pi5 <<<