I've been looking at using a Pi5s to replace some old power hungry intel servers I have that are ultimately close to 20 years old. I know there are plenty of random small SBC servers out there, but I didn't want to get locked in some vendor solution, and I figured if I could avoid spending $500+ for custom servers from a company that may not be around in 10 years, I should.
The big thing I wanted was a very low power, high efficiency server with a BMC, which doesn't quite define the Rasberry Pi 5, but since I had been working with rPi3 and rPi4 for my LED Outfit, I was curious to see if I could reasonably use a rPi5 as a headless server.
I figured I would give rPi5s a try, especially as they now natively supported PCIe cards including M2 NVME storage, and with the right adapter card, PCIe SATA (bypassing USB). Most importantly, they now have some reasonably capable flashable firmware that allows booting from NVME (or SATA), bypassing the sdcard entirely, which we all know is not a great proposition for a reliable server since sdcards are very much known to die, even if you buy the more fancy ones.
My other requirements were to be able to:
use raid1 (or raid5) for the root filesystem
use btrfs, I rely on snapshots and btrfs send/receive backups too much to want ext4
serial console for debugging if networking fails
serial console for debugging if boot fails (sadly they are not the same)
remote power reboot of some kind
Serial and Sysrq
All of these, I had without issues in the current servers I use, either with a BMC which effectively is its own computer with its own IP to monitor and power cycle the main computer, or a bios with serial support, which thankfully they've had for 25 years or more, and some remote power cycle via PDU.
I quickly found out that rPi5 would more or less do all of these, except the boot time serial support. The boot firmware does support outputting to the built in debug serial port on the board, ttyAMA10, if you forgive the somewhat unoptanium 2 pin plug they felt compelled to use and that was honestly hard to procure until I found USB serial adapters that included that wretched 2 pin plug. Note that I'm pretty annoyed about it, because I have lots of similar looking 2 pin plugs from my RC planes and ardupilot hardware, and they are literally off by a portion of 1mm. The lack of standardization and pointless spread of incompatible connectors is utterly ridiculous.
114*|this adapter works, offers ttyUSB0 and the right Rpi5 compatible plug
112*|this other adapter kind of works, but offers ttyACM0 instead which is not compatible with break/sysrq
In summary: /dev/ttyACM USB adapters do not support sending BREAK (ALT+F in minicom) and therefore not suitable since you will not be able to send sysrq commands for reboot. Make sure you adapter provides /dev/ttyUSB0 and that BREAK works.
So, once you have serial working, you make sure sysrq works and you're set, correct? Well, not really:
I didn't find any good way to remotely power cycle the Pi via a GPIO without using an externally controllable relay and splicing a USB-C power cable. So make sure to have panic=15 or equivalent in your /boot/cmdline.txt to make sure the kernel will never just hang
never use halt -p, cannot recover (I don't know if you can wire power switch to other pi to actuate it without a relay). Simlarly, if you send sysrq-o, it will power down the rPi, and you'll be out of luck after that.
sysrq is absolutely required for reboots if things are bad, even a mere systemd shutdown bug, which of course I encountered while installing watchdog, but it only works over serial with your USB-serial adapter supports it. FTDI comaptible ones do, the ttyACM0 did not.
rPi5 Early Boot Configuration
aragorn:~# tail -10 /boot/config.txt
#sdram_freq_min=400
arm_64bit=1
dtoverlay=disable-wifi
dtparam=watchdog=on
# This is suggested by PCI expander, but seems useless
#dtparam=picex1
#dtparam=picex1_gen=3
aragorn:~# cat /boot/cmdline.txt
root=/dev/md0 rootflags=subvol=root rootwait net.ifnames=0 logo.nologo console=tty1 console=ttyAMA10,115200 panic=15 rd.shell
The early boot is configured like this
aragorn:~# rpi-eeprom-config
[all]
CONSOLE=UART
BOOT_UART=1
# tries other non bootable before nvme to bring menu on HDMI
# read right to left, 4: USB, 6: NVME, 1: sdcard, f: restatt
#BOOT_ORDER=0xf164
# boot nvme first and if not, sdcard
BOOT_ORDER=0xf16
# Sometimes needed by NVME drives to show up in time for boot
PCIE_PROBE=1
[gpio17=1]
# this would boot sdcard directly:
#BOOT_ORDER=0xf1
# boot other absent devices first force menu before choosing sdcard
# read right to left, 4: USB, 1: sdcard, 6: NVME, f: restatt
BOOT_ORDER=0xf614
aragorn:~#
Boot Configuration
aragorn:~# cat /etc/fstab
tmpfs /tmp tmpfs size=8095M,noatime,lazytime,nodev,nosuid,mode=1777
LABEL=btrfs_boot / btrfs nofail,subvol=root 0 0
LABEL=btrfs_pool1 /var/local/space btrfs defaults,subvol=pool1 0 0
LABEL=btrfs_pool2 /var/local/space2 btrfs defaults,subvol=pool2 0 0
LABEL=btrfs_boot /mnt/btrfs_boot btrfs noatime,lazytime,rw
LABEL=btrfs_pool1 /mnt/btrfs_pool1 btrfs noatime,lazytime,rw,noauto,x-systemd.automount
LABEL=btrfs_pool2 /mnt/btrfs_pool2 btrfs noatime,lazytime,rw,noauto,x-systemd.automount
UUID=84958e33-a82c-4864-83b9-f483529a3bed /mnt/sdcard ext4 noatime,lazytime,rw,noauto,x-systemd.automount
UUID=7E8C-86F8 /boot/firmware vfat noatime,lazytime,rw 0 2
UUID=F6DA-B64E /mnt/sdcard_boot vfat noatime,lazytime,rw,noauto,x-systemd.automount
UUID=7E8C-86F8 /mnt/firmware1 vfat noatime,lazytime,rw,noauto,x-systemd.automount
UUID=4122-81AA /mnt/firmware2 vfat noatime,lazytime,rw,noauto,x-systemd.automount
aragorn:~# cat /proc/partitions
major minor #blocks name
259 0 500107608 nvme0n1
259 1 131072 nvme0n1p1
259 2 31034880 nvme0n1p2
259 3 468940120 nvme0n1p3
259 4 488386584 nvme1n1
259 5 131072 nvme1n1p1
259 6 31034880 nvme1n1p2
259 7 457219096 nvme1n1p3
179 0 31166976 mmcblk0
179 1 131072 mmcblk0p1
179 2 31034880 mmcblk0p2
8 0 58615704 sda
9 0 31017472 md0
aragorn:~# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 nvme0n1p2[0] nvme1n1p2[1]
31017472 blocks super 1.2 [2/2] [UU]
Btrfs, Initramfs, Dracut
aragorn:~# cat /var/local/scr/rpi-dracut
#!/bin/bash
#dracut --install "bash find ls" --regenerate-all -p --force
dracut --install "bash find ls grep more vi vim vim.nox" --regenerate-all -p --force --no-hostonly
cat <<EOF
Add rd.break=premount to force a debug shell (which must be done via rpi-boot-rescue
and the rescue partition).
man dracut.kernel and dracut for more details
vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
You must copy generated initrd to /boot/firmware and check path in /boot/config.txt
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
EOF