radagast: QMP monitor and fast NVMe swap

QMP monitor socket

Added 2026-10-15 to /home/john/extdrive/vm/start-radagast-fast.sh: -qmp unix:/var/run/radagast-qmp.sock,server=on,wait=off. Gives a JSON control socket into the running QEMU process for live device hot-plug/unplug, status queries, and HMP passthrough — without a guest restart.

Handshake is required before any other command:

echo '{"execute":"qmp_capabilities"}' | sudo nc -U /var/run/radagast-qmp.sock

Useful commands proven working live on 2026-10-15:

query-status — VM running/paused state.
query-pci / human-monitor-command with info qtree — inspect bus topology and find qdev paths (needed for device_del on devices added without an explicit id=).
human-monitor-command with hostfwd_add tcp::5902-:5902 — add a SLIRP NAT port forward live. Confirmed with an RFB handshake through it. Not persistent — only lasts for the current QEMU process; add to the -netdev line in the start script to survive a restart.
blockdev-add + device_add — hot-add a disk. See "AHCI vs NVMe hotplug" below: this only works for devices on buses that support hotplug.

AHCI vs NVMe hotplug (important distinction)

The machine is i440FX with ACPI PCI hotplug enabled on the root bus (acpi-root-pci-hotplug = true, confirmed via info qtree). But that only covers hotplugging whole PCI functions onto pci.0 — it does not extend to AHCI's inner SATA ports. Attempting device_add ide-hd,bus=ahci0.2,drive=d2 on a running guest fails: "Bus 'ahci0.2' does not support hotplugging".

Working alternative: add an emulated NVMe controller directly on pci.0 instead of a 3rd AHCI disk. FreeBSD's nvme(4) driver picked it up immediately with no guest-side reconfiguration, appearing as nda0. virtio-blk/virtio-scsi were not tried for this — virtio is documented elsewhere as wedging this particular guest kernel solid, so NVMe was preferred anyway.

# host side, live, no restart:
blockdev-add {driver:raw, node-name:d2, file:{driver:file, filename:<img>, aio:native, cache:{direct:true}}}
device_add {driver:nvme, drive:d2, serial:swap0}
# guest sees it as nda0 within seconds, no reboot/rescan needed.

Fast NVMe swap

radagast's normal swap is a 4 GiB ZVOL on zroot (the USB-attached WD disks — confirmed genuine USB 3.0/UASP at 5 Gbps via lsusb -t on pomelo, Bus 004, driver uas; it is NOT USB 2.0, don't blame the link speed for swap slowness). Under heavy parallel ports builds the pageout daemon couldn't keep up — more likely rotational-disk seek latency plus ZFS zvol/COW overhead than raw USB bandwidth.

A 16 GiB sparse image on pomelo's real NVMe (/home/john/extdrive/vm/radagast-swap.img, on nvme0n1p4) was tested as a faster alternative. Test: a small C program (/tmp/whale.c on radagast at the time, not preserved) allocated and touched 20 GiB on the 12 GiB-RAM guest. Result: ~9.3 GiB pushed cleanly onto NVMe swap, load stayed low (~0.2–0.3), no OOM kill — drained back to near-zero on exit. Confirms the fast-swap approach actually fixes the OOM problem seen with large LLVM/Firefox-class builds.

Current status (2026-10-15): fast swap OFF by default, available to hot-add

Decision: leave the NVMe swap detached by default (saves host NVMe bandwidth/cache for no reason when not needed) and only hot-add it when a build is expected to need it. The -drive/-device nvme lines are commented out in start-radagast-fast.sh; the image file itself is left in place.

Guest is currently back on the ZVOL swap: /etc/fstab has /dev/zvol/zroot/swap none swap sw 0 0, and zfs get org.freebsd:swap zroot/swap is on.

To hot-add the fast swap (no restart)

Host: blockdev-add + device_add nvme,drive=d2 via QMP (commands above).
Guest: confirm the device appears (dmesg | grep nvme; usually nda0).
Guest: swapon /dev/nda0; then swapoff /dev/zvol/zroot/swap; then zfs set org.freebsd:swap=off zroot/swap (REQUIRED — the dataset property re-enables the ZVOL at boot regardless of fstab).
Guest: update /etc/fstab to swap the zvol line for /dev/nda0 none swap sw 0 0 if you want it to survive a reboot too (or just leave fstab alone if this is a temporary hot-add only — fstab doesn't matter until next boot).

To remove the fast swap again

Order matters — do not skip step 1. On 2026-10-15 a hot-unplug was done directly via device_del while ~30 MB of pages were still live on the NVMe swap device. The guest hit swap_pager: I/O error - pagein failed and syslogd (which had pages out there) segfaulted and died. The VM itself stayed up and reachable — only the one process was killed — but it was entirely avoidable.

Guest FIRST: swapoff /dev/nda0, then confirm with swapctl -l that it actually drained to empty.
Guest: restore the ZVOL — zfs set org.freebsd:swap=on zroot/swap; swapon /dev/zvol/zroot/swap; fix /etc/fstab back to the zvol line.
Only then, host side: find the device's qom path (qom-list on /machine/peripheral-anon, look for child<nvme>) and device_del it, then blockdev-del the backing node.

If this process crashes again, it is because step 1 was skipped — check ps aux | grep syslogd and restart it (service syslogd restart) if so; otherwise check for any other process that logged pager errors around the same time.

To make the fast swap permanent again (survive a restart)

In start-radagast-fast.sh: uncomment the SWAPIMG= line and the -drive if=none,id=d2,..."$SWAPIMG"... / -device nvme,drive=d2,serial=swap0 lines. In the guest: set /etc/fstab's swap line to /dev/nda0 and zfs set org.freebsd:swap=off zroot/swap. Confirmed working across a full clean shutdown/restart cycle on 2026-10-15 — the guest auto-detected nda0 and swapped onto it with zero manual steps.