Parent host note: hosts/radagast.
Current checkpoint — 2026-09-21: radagast is running 13.4-RELEASE. The next hop to 14.4-RELEASE is building from source with -O1; not installed yet. Build driver /root/hop144.sh, log /root/hop144.log, pid /root/hop144.pid, completion status /root/hop144.status. Build started 05:19 BST with -j12. Preserve source-only upgrade, manual configuration reconciliation, and no zpool upgrade.
Bring radagast to FreeBSD 15.x entirely by compiling world and kernel from source: no binary upgrade or clean reinstall. John's optimization preference confirmed 2026-09-20: use -O1 for future intermediate builds and -O2 for the final FreeBSD 15 world and kernel. The already completed 13.4 build used tree-default flags (normally -O2); installed it as requested rather than rebuilding. Do not use mergemaster: review configuration contents directly and preserve host-specific and moved-root state. Do not run zpool upgrade.
Starting state (found 2026-09-08): kernel already 11.1-STABLE (built Oct 2017 by root@mango) but userland/world still 10.4-STABLE — someone built and installed a newer kernel once but never finished the matching installworld. Base compiler was still clang 3.4.1 (2014-vintage).
Before any of this could start, the VM itself (/home/john/extdrive/vm/start-freebsd-zroot-vm.sh, see hosts/radagast for its origin) needed fixing:
virtio-scsi-pci/virtio-net-pci wedged the guest kernel solid on first real boot — unresponsive even to kill -9 -1 from inside the guest, with zero host-side disk I/O (a hung kernel, not a stuck process). Diagnosed as a virtio version/feature-negotiation mismatch between this ~2013 FreeBSD 10.4 guest driver and modern QEMU's virtio defaults. Fixed by switching the launch script to plain ahci/ide-hd for disks and e1000 for the NIC — both rock-solid in the FreeBSD 10.4 GENERIC kernel, no virtio driver needed. Device names inside the guest changed accordingly: disks da0/da1 → ada0/ada1 (harmless — ZFS finds zroot by pool label not device path); NIC vtnet0 → em0.sysrc ifconfig_em0=DHCP and sysrc synchronous_dhclient=YES in /etc/rc.conf (nothing had configured em0 at all after the NIC rename; a first attempt to set this was lost/never actually persisted, caught and fixed on the next boot)./etc/ssh was completely empty (no sshd_config, no host keys) and sshd_enable defaults to NO in this release — sshd had never actually been configured, despite the binary being present. Fixed by writing a minimal /etc/ssh/sshd_config (PermitRootLogin without-password, PubkeyAuthentication yes, UseDNS no — the last one specifically to avoid a hang: a config-less sshd defaults to UseDNS yes and stalls forever on a reverse-DNS lookup for the NAT gateway address, which looks like a dead connection from the client side, not an auth problem), plus sysrc sshd_enable=YES. john's authorized_keys already had a matching key (john@banana, matches ~/.ssh/id_rsa_radagast on pomelo); root had no ~/.ssh at all, created fresh with the same key. Both ssh -p 2222 -i ~/.ssh/id_rsa_radagast john@localhost and the root@ equivalent now work reliably.zroot/usr/obj created as its own ZFS dataset, mounted at /usr/obj (FreeBSD's default MAKEOBJDIRPREFIX). Point: build artifacts land here and can be zfs destroy -r + recreated fresh between build attempts without touching /usr/src, /usr/src11, or anything else under /usr. Used this reset twice already (see build log below).swap0/swap1 partitions (those belong to the separate obj ZFS pool — see hosts/radagast — explicitly left alone): zfs create -V 4G -o compression=off -o checksum=off -o org.freebsd:swap=on -o primarycache=metadata -o secondarycache=none zroot/swap, added to /etc/fstab, swapon -a./usr/src (10.4-STABLE, an old Subversion checkout — stale, since svn.freebsd.org was retired years ago) and, crucially, /usr/src11 (11.1-STABLE, 4.5G, exact match for the running kernel) were already present on disk from 2017 — no need to fetch anything to do the first hop./usr/bin/cc is clang 3.4.1. Also found /usr/local/bin/clang40 (LLVM/clang 4.0.1) already installed via pkg — the generation FreeBSD 11.1 itself ships as base compiler.FreeBSD's source upgrade path only supports one or two major versions at a time — each release's UPDATING file has one-time, order-dependent steps, most notably the ino64 (64-bit inode/timestamp) transition around the 11.2/12.0 boundary, which needs a specific two-pass build/install sequence when crossing it from anything older. Planned hop sequence, each its own full buildworld → buildkernel → installkernel → reboot to single-user → installworld → mergemaster → reboot cycle:
11.1 (current) → 11.4 → 12.0 (ino64 pass) → 12.4 → 13.0 → 13.4 → 14.x. Later hops will need source fetched fresh (HTTPS release tarballs from download.freebsd.org, not svn/git, to keep to "from source" without needing to bootstrap a VCS client).
Not yet taken (as of this note). Plan: right before the first installkernel/installworld of each hop (buildworld itself only writes into /usr/obj, so it's non-destructive and doesn't need a snapshot first), take:
zfs snapshot zroot@pre-<hop>-world
zfs snapshot zroot/usr@pre-<hop>-worldScoped to just the two datasets an installworld actually writes to (zroot itself for /etc//bin//boot etc., zroot/usr for /usr/bin//usr/lib//usr/local etc.) — not a whole-pool recursive snapshot, since zroot/home, zroot/p, zroot/q, zroot/tmp hold the real family document archive and are never touched by an OS upgrade at all. Rollback path if a hop goes wrong: export from FreeBSD, zpool import -f zroot on the Linux host (pomelo), zfs rollback both datasets — already proven to work cleanly this session for unrelated reasons (recovering from the virtio kernel wedge).
Flags throughout: CFLAGS="-O1 -pipe" COPTFLAGS="-O1 -pipe", -j4 (4 vCPUs assigned to the VM). Deliberately not tuned for runtime performance — these intermediate hops won't be run for long.
CC=/usr/local/bin/clang40 etc., hoping the tree's MK_SYSTEM_COMPILER logic would detect the match and skip building a fresh in-tree compiler (normally the most expensive part of a buildworld). Failed — MK_SYSTEM_COMPILER turns out to be hardcoded no in this tree's share/mk/src.opts.mk, so it doesn't actually short-circuit anything here. Real failure: dozens of declaration conflicts with target of using declaration already in scope errors compiling contrib/llvm/lib/Support/APInt.cpp against /usr/include/c++/v1/cmath.cc (3.4.1) instead — wiped zroot/usr/obj and reran. Identical failure, same lines in the same header. This ruled out compiler choice as the cause — something was wrong with the header itself, independent of which compiler read it./usr/include/c++/v1 had been silently clobbered to a third, internally-inconsistent vintage — neither /usr/src's (10.4) nor /usr/src11's (11.1) own bundled libc++ matched what was installed (124 files in src11's copy vs the installed tree missing/differing on dozens of files, including lacking C++17 hypot overloads present even in the older 11.1 tree — i.e. installed headers predate both candidates). Most likely cause: installing the llvm40 port back in 2017 silently overwrote the base libc++ headers as a side effect. Matched exactly (same error text, same root cause) to a real FreeBSD Forums case ("messed up C++ library", thread 61364) resolved the identical way.mv /usr/include/c++ /usr/include/c++.bak-2026-09-08 (preserved, not deleted), then restored a pristine copy from the matching source tree: cp -a /usr/src11/contrib/libc++/include /usr/include/c++/v1. Wiped zroot/usr/obj again and restarted the build — progressed cleanly past the previous failure point (Support/Mutex.o, Support/NativeFormatting.o, groff utilities all compiling with zero error: lines in the log).buildworld for the 11.1 hop is running (as of 2026-09-08, started ~11:26), detached via daemon(8) so it survives SSH disconnects and doesn't depend on this session at all. Log: /root/buildworld.log inside the guest, pidfile /root/buildworld.pid. Not yet done: finish this buildworld, then buildkernel, snapshots, installkernel/installworld/mergemaster for 11.1, then repeat the whole cycle for each subsequent hop (11.4 → 12.0 → … → 14.x).
Update 2026-09-08 14:57: buildworld for the 11.1 hop completed successfully — World build completed on Tue Sep 8 14:57:14 BST 2026 in the log, ~3.5 hours (started 11:26). Only non-fatal output: two groff macro warnings building share/doc/usd/contents/contents.ms (old documentation source, unrelated to the compiled system itself). Next: buildkernel, then the snapshot-and-install step described above.
No local checkout existed for 11.4 (only 10.4 and 11.1 were on disk from 2017). Fetched fresh: 11.4-RELEASE's src.txz had already been pulled off the main download.freebsd.org mirror (EOL release) — found on archive.freebsd.org/old-releases/ instead. Downloaded on pomelo (modern TLS, avoids repeating the guest's-own-TLS-is-too-old problem hit with the Windows VM earlier this session), transferred in via scp -O (legacy SCP protocol — our minimal sshd_config has no Subsystem sftp line, so modern scp's default SFTP transfer fails with "subsystem request failed"). Extracted to /usr/src12 (1.4G). One snag: the release tarball's internal path is usr/src/... (two directory levels), so tar --strip-components=1 left everything nested under /usr/src12/src/ on the first attempt — needed --strip-components=2. Confirmed: REVISION="11.4" BRANCH="RELEASE".
Ran the rest of the 11.4 hop unattended overnight with a hard deadline (disks needed free by 06:00, host hibernated by 06:15 for a trip). buildworld (-j4, from /usr/src12) FAILED with a real compile/link error at 01:31:07 — not a deadline timeout, a genuine build failure. Exact cause not yet diagnosed (log is on radagast's own disk, only readable by booting the VM, deliberately not done before the disk-removal deadline). Nothing on zroot is at risk — buildworld only writes into disposable zroot/usr/obj; the system remains on its last good state (matched, bootable 11.1-STABLE from the previous hop). The 8-vCPU restart, buildkernel, the pre-install snapshot, and everything after were never reached. Live status snapshot at note _ (scratch, will be overwritten). Next: boot the VM once the disks are back, check /root/buildworld114.log for the actual error — check first whether it's a repeat of the libc++ header-corruption class of bug already fixed once for the 11.1 hop before assuming something new.
Booted the VM again and read /root/buildworld114.log (32MB) as the previous note said to. It was NOT a repeat of the libc++ header corruption, and not a code problem at all — it was memory exhaustion. The only error: lines are at 175018–175019: c++: error: unable to execute command: Killed followed by linker command failed due to signal. Confirmed in /var/log/messages: pid 91552 (ld), uid 0, was killed: out of swap space at 03:05:11, then a cascade of c++, cc and make kills at 03:06:30. The LLVM link step is the peak-memory point of a buildworld, and 4GB RAM + 4GB swap with -j4 was not enough.
Fix applied the same evening: the VM is now started with -m 8192 (was 4096) and -cpu host, via start-radagast-fast.sh — a variant of the stock launch script that also forces usermode NAT rather than bridging, because enp1s0 on pomelo now carries the NAS network (192.168.99.1/24 plus a dnsmasq DHCP server for the NAS) and the stock script would hand that interface to a bridge and cut the NAS off. Guest confirmed seeing 8.0 GiB and 8 CPUs. The 11.4 buildworld has not been re-run yet; when it is, expect the LLVM link step to be the thing to watch, and consider dropping to -j2 for that phase if 8GB still proves tight.
Standing instruction from John (2026-09-16): mergemaster may well be unusable on these hops. If it errors, hangs, or needs interaction, do not fight it — skip it, leave the existing /etc untouched, and leave configuration reconciliation to John to do by hand later. This is consistent with the decision taken on the 11.1 hop to defer full /etc reconciliation to the final target release: nothing is functionally broken by leaving /etc as it is, and the box will not run any intermediate release for long. Leaving it alone also guarantees sshd_config and master.passwd survive, so remote access cannot be lost as a side effect of an upgrade step.
Correction to the "Flags throughout" line recorded for the 11.1 hop. That note says the builds used CFLAGS="-O1 -pipe" COPTFLAGS="-O1 -pipe" on the make command line. Do not do this. A variable assigned on the make command line takes precedence over every CFLAGS+= append in the Makefiles at rule-expansion time, so the per-directory flags are silently discarded. The 11.4 buildworld died at 23:31 in lib/libgcc_eh with fatal error: 'unwind.h' file not found, because the append CFLAGS+= -I${SRCTOP}/contrib/llvm-project/libunwind/include never reached the compiler.
Demonstrated directly in /usr/src12/lib/libgcc_eh with make -n Unwind-sjlj.o: without the override the generated command is cc -fpic ... -O2 -pipe -I.../libunwind/include -I. -D_LIBUNWIND_IS_NATIVE_ONLY -MD -std=gnu99 -fstack-protector-strong ...; with CFLAGS="-O1 -pipe" added it collapses to cc -fpic -fvisibility=hidden -DVISIBILITY_HIDDEN -O1 -pipe -c ... and nothing else — character-for-character the command that failed. Note that make -V CFLAGS is not a valid way to test this: it shows the appends applying and gives a false negative. Use make -n <target>.
Why it never bit before: 11.1's lib/libgcc_eh used GNU libgcc and needed no include path, so the stripped flags caused no visible failure. 11.4 switched to LLVM libunwind, which does. The wider point is that everything built under the override also lost -std=gnu99, -fstack-protector-strong, -MD dependency tracking and per-directory -D defines — so the 11.1 world currently installed on this box was itself built with truncated flags, and is worth rebuilding cleanly at some point even though it runs. Fix applied: /root/hop114.sh now runs plain make -j8 buildworld with the tree defaults (-O2 -pipe). Object tree wiped before the restart — this is the one case where a wipe is right, because the objects are wrong rather than incomplete and make would not rebuild them (their mtimes are newer than the sources). Failed log kept at /root/hop114-failed-cflags.log.
The whole hop ran clean once the CFLAGS problem above was fixed. uname -r, freebsd-version -k and freebsd-version -u all now report 11.4-RELEASE, kernel and world built the same night from /usr/src12. Base compiler moved from clang 3.4.1 (2014) to clang 10.0.0. Pool healthy throughout; no OOM kills this time (peak swap use across the entire build was 8.9 MB, against 16 GB available).
-j8 with 8 GB RAM: buildworld 23:44→01:31 (1h47m), buildkernel 01:31→01:37 (6m), installkernel ~1m, installworld ~4m. For comparison the -j4/4 GB attempt on 2026-09-09 had not finished buildworld after 7 hours. CPU was the limiting factor throughout — 99% user, 0% idle, and the USB disks measured 0.00% utilisation, so the USB 2.0 link does not constrain compilation at all (it only ever constrained swap).zroot@pre-11.4-world / zroot/usr@pre-11.4-world taken before installkernel, and zroot@post-11.4-hop / zroot/usr@post-11.4-hop after. The 11.1 pair is still present too.mergemaster deliberately NOT run, per John's standing instruction above. /etc is untouched — sshd_config still the 218-byte file from 2026-09-08. installworld raised no missing-user or missing-group errors, so nothing is functionally waiting on it./etc/fstab was not enough — the dataset property org.freebsd:swap=on makes FreeBSD enable it at boot regardless, and it reappeared alongside the NVMe device. Since FreeBSD interleaves across swap devices, the slow USB-backed ZVOL would have dragged the fast one down. Fixed with zfs set org.freebsd:swap=off zroot/swap. Set it back to on when reverting to ZVOL swap later.The /usr/obj mount-shadowing bug — diagnosed and FIXED 2026-09-17. Symptom: at boot zroot/usr/obj mounted before zroot/usr, so /usr landed on top and the whole object tree became invisible (the directory looked empty, and df /usr/obj misleadingly reported zroot/usr). It recurred on every boot, not once as the 11.1 hop's note implied, and it is dangerous: a build started in that state writes into zroot/usr instead of the obj dataset. Root cause: every dataset in this pool carries the pool-default mountpoint (zroot/usr → /zroot/usr, source default), and FreeBSD resolves such inherited mountpoints against the root dataset's actual mount point — which the kernel sets to / via vfs.root.mountfrom — so /usr, /home, /p, /q and /tmp all land correctly. zroot/usr/obj was the sole dataset with an explicit local mountpoint=/usr/obj, which opts out of that inheritance; zfs mount -a orders by mountpoint path, and /usr/obj sorts before /zroot/usr, so obj went first.
The fix is one command: zfs inherit mountpoint zroot/usr/obj. That drops the local override so obj inherits from zroot/usr, making its path a genuine child of its parent's, which both fixes the mount ordering and keeps it consistent with every other dataset in the pool. zroot itself was deliberately not touched — no change to the boot path, so no risk of an unbootable system. Verified over two consecutive reboots: mount now shows zroot/usr on /usr followed by zroot/usr/obj on /usr/obj, and du -sh /usr/obj reports the real 8.3G rather than 108K. Reverting, if ever needed, is zfs set mountpoint=/usr/obj zroot/usr/obj.
Standing rule for this and future ZFS layouts (John, 2026-09-17): keep a flat zroot as bootfs — boot environments are not wanted, and ZFS snapshots already give the rollback that matters. The rule that makes a flat layout safe is the direct lesson of this bug: never give a child dataset a local mountpoint property. Let them all inherit. With a flat bootfs the kernel mounts zroot at / and inherited children resolve against that real mount, so they land correctly regardless of what zfs get displays. A local override opts out of that and can mount before its parent. If a dataset genuinely needs to live somewhere outside its parent's path, that is a sign it should be a child of a different dataset instead.
Method — worth reusing. Done with radagast shut down and the pool imported on pomelo (Linux, OpenZFS 2.2.2) rather than live on FreeBSD, chosen because there is no usable console for radagast: its VNC is on 127.0.0.1:5905 but pomelo has only vncviewer (needs a human at the X session), no vncdo or vncsnapshot. A mountpoint change that broke the boot could not have been repaired remotely, so the recovery environment was made the working environment from the start. Sequence: zpool import -f -N -R /mnt/radagast zroot (-N = import without mounting, so nothing lands at a path that matters; -f is needed because FreeBSD does not export its root pool on shutdown), inspect, change, zpool export, boot and verify.
zpool upgrade on this pool. FreeBSD 11.4's ZFS predates most of what OpenZFS 2.2 offers, and both FreeBSD and Linux nag about it. Feature flags were recorded before and after the round trip and are identical — the same 12 enabled/active, nothing new. Linux merely lists 27 additional features it knows about, all disabled; listing is not enabling.zfs mount -a mounted obj before the root dataset itself, visible in /proc/self/mountinfo as a mount whose parent ID is the host's root filesystem rather than zroot's mount. Once covered, the dataset cannot be unmounted by path (zfs unmount says "no such pool or dataset", umount says "no mount point specified") and zpool export — even -f — fails. Recovery is to unmount the covering mounts by hand, deepest first, until the orphan becomes reachable again.zroot/usr/obj for a path where that dataset was completely hidden. Use du -sh against zfs get referenced instead: 108K visible versus 8.26G referenced is what exposed the shadowing.Next hop: 11.4 → 12.0, which is the ino64 transition and needs the specific two-pass sequence in 12.0's UPDATING — read that first, it is the most order-dependent step in the whole path. Source is not on disk: /usr/src is 10.4, /usr/src11 is 11.1, /usr/src12 is (confusingly) 11.4. Fetch 12.0's src.txz on pomelo rather than in the guest (the guest's TLS is too old for modern mirrors), transfer with scp -O (the minimal sshd_config has no Subsystem sftp), and extract with tar --strip-components=2, not 1. Consider renaming the trees to match their contents before this gets worse.
NVMe swap reverted 2026-09-17. Since the hop showed swap was essentially unused (8.9 MB peak of 16 GiB), John had it put back onto the ZFS disk. Guest /etc/fstab restored to the ZVOL, org.freebsd:swap=on set again on zroot/swap, the ada2 disk dropped from the launch script and radagast-swap.img deleted — pomelo went from 101 GiB free to 117 GiB. Verified after restart: guest sees only ada0/ada1, swap is the 4 GiB ZVOL, pool healthy, still 11.4-RELEASE on both kernel and userland. The lesson worth carrying into the 12.0 hop: the OOM kill was fixed by doubling RAM, not by faster swap — if 12.0 struggles, add RAM first (pomelo has 28 GiB total) before rebuilding the swap image.
The ino64 transition is done and it went cleanly. uname -r, freebsd-version -k and freebsd-version -u all report 12.0-RELEASE; base compiler is now clang 6.0.1. buildworld took 41 minutes and buildkernel 6 more, at -j12 with 12 vCPUs and 12 GB RAM — first attempt, no retries, zero Error code lines, and swap was never touched at all (4096M free throughout). For comparison the 11.4 hop took 1h47m at -j8/8GB. Pool healthy throughout.
THE THING THAT WOULD HAVE BROKEN IT: missing users and groups. 12.0's installworld installs files owned by ntpd:ntpd, and the live system had no such user. It was also missing _ypldap and the video group — fallout from mergemaster being skipped on the 11.1 and 11.4 hops, so /etc had drifted further behind with each one. Without them installworld fails partway through, which is the single worst outcome available: a half-installed world. The canonical procedure catches this with mergemaster -Fp before installworld — a narrow step that only touches master.passwd and group, quite unlike the full /etc reconciliation John asked to skip.
How to check this before every future hop — compare the target source tree's account files against the live system, and against the previous tree, before building anything:
# users/groups the new world wants but the live system lacks
comm -13 <(awk -F: '{print $1}' /etc/master.passwd | sort) \
<(awk -F: '{print $1}' /usr/src-NEW/etc/master.passwd | sort)
comm -13 <(awk -F: '{print $1}' /etc/group | sort) \
<(awk -F: '{print $1}' /usr/src-NEW/etc/group | sort)
# add any that are missing, using the IDs the source expects, e.g.:
grep -E '^(ntpd|_ypldap):' /usr/src-NEW/etc/master.passwd # read the real IDs
pw groupadd ntpd -g 123
pw useradd ntpd -u 123 -g 123 -d /var/db/ntp -s /usr/sbin/nologin -c 'NTP Daemon'/usr/src-12.0, named for what it contains, breaking the pattern that left /usr/src12 holding 11.4. Do the same for the next hop (/usr/src-12.4). Fetched as src.txz from archive.freebsd.org/old-releases/amd64/12.0-RELEASE/ (12.0 is EOL, so not on the main mirror), SHA256-verified against that directory's MANIFEST file — note the checksum lives in MANIFEST, not CHECKSUM.SHA256, which 404s there — and re-verified after transfer onto the guest. Extract with --strip-components=2, not 1.rm -rf /usr/obj/* — this wipe is the documented procedure, not a shortcut); the kernel must have COMPAT_FREEBSD11 (12.0 GENERIC does, verified); and a reboot between installkernel and installworld is mandatory, because the system deliberately runs the new kernel against the old world in between. That intermediate state was observed working exactly as intended: 12.0 kernel, 11.4 userland, network and ZFS fine.mergemaster -Fi and make delete-old. Per John's standing instruction /etc is left alone — sshd_config is still the 218-byte file from 2026-09-08, so remote access could not be lost as a side effect. delete-old removes obsolete base files and can break installed ports, so it is not worth the risk on an intermediate release.pre-12.0-start (before anything), pre-12.0-build (after source and account prep), pre-12.0-world (before installkernel), post-12.0-hop (after). Both zroot and zroot/usr at each point, and the 11.4 pairs are still present.Next: 12.0 → 12.4. The hard part of the path is now behind — ino64 was the one-way transition, and 12.x to 12.x is an ordinary hop. Fetch src.txz for 12.4-RELEASE (also EOL, so archive.freebsd.org/old-releases/) into /usr/src-12.4, verify against MANIFEST, run the account-diff check above, then the same build/install cycle. After that 13.0 → 13.4 → 14.x. Read each release's UPDATING first regardless: 13.0 switches the default to OpenZFS, which is worth understanding before it touches this pool.
13.0 replaces the legacy Illumos-derived ZFS in base with OpenZFS (the unified codebase shared with Linux). From 13.0's UPDATING, entry 20200824: "OpenZFS support has been integrated. Do not upgrade root pools until the loader is updated to support zstd. Furthermore, we caution against 'zpool upgrade'… The change should be transparent unless you want to use new features." The hop itself is therefore not the risk; what you do afterwards is.
DO NOT RUN zpool upgrade UNTIL THE WHOLE PATH TO 14.x IS FINISHED AND PROVEN. This is the important one. zroot currently has 19 features; OpenZFS offers 40+. Upgrading enables the new ones and an older FreeBSD can then no longer import the pool. The entire safety net on these hops is "roll back the snapshot and boot the previous release" — zpool upgrade silently destroys that, and nothing warns you at the time. There is no benefit to doing it early: the new features are opt-in and nothing in the upgrade path needs them. It would not break the pomelo rescue route (OpenZFS 2.2.2 there is newer than anything 13.0 ships), but it ends the ability to go backwards on the guest itself.
zroot is the root pool. The 13.0 loader arrives with installworld (the loader lives in src/stand, part of world), so a normal hop satisfies it. The rule reduces to: never zpool upgrade before the new loader is installed and the machine has been confirmed to boot — which the point above makes moot anyway.zpool import -a in rc.d, gated by zfs_enable=YES. Checked on 2026-09-17: /etc/rc.d/zfs here is dated Dec 31 2016 and was not refreshed by the 12.0 installworld — an in-place installworld does not touch /etc, which is mergemaster's job and is being skipped by choice. This does not matter today because radagast has exactly one pool and it is the root pool, imported by the loader rather than by rc.d. It would matter if a second pool were ever imported — for instance the separate obj pool on those disks' partitions — which would then silently fail to mount at boot until /etc/rc.d/zfs is refreshed.zpool.cache moves from /boot to /etc to match the OpenZFS default. Not a problem: ours is at /boot/zfs/zpool.cache and is being maintained (updated 2026-09-17 18:45), and UPDATING records an explicit fallback to /boot for mountroot.buildworld then buildkernel anyway, this costs nothing to honour.[ ] Agreed with John 2026-09-17, deferred to the 13.0 hop. Measured that day: all 169 scripts in /etc/rc.d carry a stable/10 $FreeBSD$ tag, and /etc/rc.subr — the library every one of them sources — is also stable/10 (2015), 2 KB smaller than 12.0's. So the entire service-startup layer is FreeBSD 10.x while the world is 12.0: two major versions of drift, the accumulated cost of skipping mergemaster on three hops. The system boots and runs fine, so this is not urgent, but it is real.
rc.subr is the dangerous middle ground, and it is on the boot path. Refresh /etc/rc.d/* and /etc/rc.subr together, from the same source tree, as one deliberate operation./etc. John's actual configuration — rc.conf, master.passwd, sshd_config, fstab — stays untouched as always. Confirm before doing it, but that is the reasoning.rc.d/zfs differs from the installed one only by a $FreeBSD$ tag and one desc= line. The script actually wanted, rc.d/zpool, does not exist in 12.0 at all — it is new in 13.0, along with rc.d/zfsbe for boot environments.for cachefile in /etc/zfs/zpool.cache /boot/zfs/zpool.cache; do [ -r $cachefile ] && zpool import -c $cachefile -a -N && break; done. So /etc is tried first and /boot second — no reordering needed, an existing cache at /boot is simply found on the second pass. Sharp edge: the loop breaks on the first cachefile that exists and imports successfully, so a stale or empty /etc/zfs/zpool.cache would import nothing, return 0, break, and silently mask the good one at /boot. The fallback is safe only while /etc/zfs/zpool.cache is absent, not merely wrong.default, which resolves to whatever the running ZFS expects — /boot/zfs/ under legacy ZFS, /etc/zfs/ under OpenZFS — so it adapts by itself at the transition. Pinning it now only creates work later.rc.d/zfs. 13.0's rc.d/zfs only does zfs mount -va and sharing; the import lives in the separate rc.d/zpool. Also, UPDATING's phrase "a fallback to /boot has been added for mountroot" refers to kernel-side root-pool logic, which is a different mechanism from the userland loop above and has not been verified in source.Current checkpoint: radagast is still running 12.0-RELEASE and the 12.4 hop has started. Fetched src.txz and MANIFEST for 12.4-RELEASE from archive.freebsd.org/old-releases/amd64/12.4-RELEASE/ on pomelo, verified SHA256 8b016b370ba9ba3ad5595e2fae72e2f1415f0984a73cbbb63d1affa2d71fa8f9 against MANIFEST on pomelo and again inside the guest, then extracted to /usr/src-12.4 with tar --strip-components=2. The tree identifies as REVISION="12.4" BRANCH="RELEASE".
tests. Added tests::977:977 and tests::977: using the IDs from /usr/src-12.4/etc/master.passwd and /usr/src-12.4/etc/group. No other users or groups were missing.zroot@pre-12.4-build and zroot/usr@pre-12.4-build./root/hop124.sh. It wipes only the old previous-hop /usr/obj/usr tree once, then runs plain make -j12 buildworld from /usr/src-12.4 with no command-line CFLAGS/COPTFLAGS override, followed by make -j12 buildkernel KERNCONF=GENERIC. It retries at lower -j only for memory-looking failures.daemon at 2026-09-19 00:02 BST. PID file: /root/hop124.pid; driver log: /root/hop124-driver.log; build log: /root/hop124.log. buildworld -j12 began at 00:04 BST and the first log sample showed full tree-managed compiler flags, so the previous CFLAGS-stripping failure mode is not recurring.Mid-build status 2026-09-19 00:28 BST: buildworld -j12 is still running cleanly. It passed the initial bootstrap/tool phases, built and installed temporary Clang pieces, moved through libc/OpenSSL/Heimdal/ZFS/ncurses/PAM, and is now in the full LLVM library pass under the 12.4 sysroot. Swap remains unused at 0B; /usr/obj is about 5.4G. No error or OOM signature has appeared in /root/hop124.log.
Mid-build status 2026-09-19 00:58 BST: buildworld -j12 is still running cleanly and has moved past the large LLVM/Clang/LLDB static libraries into base-system libraries, stand, usr.bin and usr.sbin programs. A visible zstreamdump format warning was non-fatal and the build continued. Swap remains unused at 0B; /usr/obj has grown to about 10G. No error or OOM signature has appeared in /root/hop124.log.
The 12.0 → 12.4 hop is complete. buildworld -j12 started at 00:04:03 BST and completed first try at 01:04:17 BST. buildkernel -j12 KERNCONF=GENERIC completed at 01:09:09 BST. No failure signatures appeared in /root/hop124.log or /root/hop124-driver.log; pool stayed healthy; swap remained unused at 0B throughout.
zroot@pre-12.4-build / zroot/usr@pre-12.4-build before building, zroot@pre-12.4-world / zroot/usr@pre-12.4-world before installing, and zroot@post-12.4-hop / zroot/usr@post-12.4-hop after final verification.installkernel completed at 01:10:41 BST; rebooted into 12.4 kernel with 12.0 userland; installworld completed at 01:17:09 BST. Full interactive mergemaster and delete-old were deliberately skipped for this intermediate hop.installworld, new SSH connections started closing with recv_rexec_state: parse config: incomplete message, consistent with the old 12.0 sshd master trying to re-exec the newly installed 12.4 sshd. Stopped QEMU from pomelo, imported zroot with zpool import -f -N -R /mnt/radagast zroot without running zpool upgrade, and replaced /etc/ssh/sshd_config offline with a known minimal config. Tested it on pomelo using sshd -t -f and again with radagast's mounted RSA/ECDSA/ED25519 host keys; both tests returned 0. Exported the pool cleanly and restarted radagast.uname -r / freebsd-version -k / freebsd-version -u all report 12.4-RELEASE; SSH works again on host port 2222; sshd is running in the guest; zpool status -x reports all pools healthy; zroot/usr and zroot/usr/obj are mounted correctly; swap remains the 4G ZVOL with 0B used.Standing rule for future hops: keep a working second sshd path before any reboot. Before rebooting after installkernel or installworld, start or restart a known-maintained sshd instance, test it with a real client connection, and preferably use that tested path to issue the reboot. This catches sshd re-exec/config/key problems before the VM is unreachable. If the QEMU host forward only exposes one guest port, update the launcher to expose a second host port for a second guest sshd before continuing to the 13.0 hop.
2026-09-19 second SSH path setup: radagast now has a separate fallback sshd using /etc/ssh/sshd_config_2223, started by /root/start-sshd2223.sh from /etc/rc.local. The QEMU launcher /home/john/extdrive/vm/start-radagast-fast.sh forwards host localhost:2223 to guest 2223 in addition to the normal host localhost:2222 to guest 22. Added local SSH alias radagast-fallback in /home/john/.ssh/config. Verified after restarting the VM: ssh radagast reports 12.4-RELEASE and healthy pools; ssh radagast-fallback reaches the separate sshd on guest port 2223 and also reports 12.4-RELEASE.
Current checkpoint: radagast is running 12.4-RELEASE and the 13.0 hop has started. Verified both SSH paths first: ssh radagast on host port 2222 and ssh radagast-fallback on host port 2223 both work and report 12.4-RELEASE. Pool is healthy; zroot/usr and zroot/usr/obj are mounted; swap is the 4 GiB ZVOL and was unused at start.
src.txz and MANIFEST for 13.0-RELEASE from archive.freebsd.org/old-releases/amd64/13.0-RELEASE/ on pomelo. SHA256 verified on pomelo and again inside radagast: 66566be2956fd6af2c5e265ec65b62207a10576a548dd2969c944fd019316a04. Extracted to /usr/src-13.0 with tar --strip-components=2; tree identifies as REVISION="13.0" BRANCH="RELEASE".UPDATING entries relevant to this hop. Key points carried forward: OpenZFS is integrated; do not run zpool upgrade. The ZFS cache file moved from /boot to /etc with a mountroot fallback; non-root pool auto-import moved to an explicit rc script; NO_CLEAN builds are unsafe across this transition. This build uses a fresh source path/object path, so it is a clean build./usr/src-13.0/etc/master.passwd or /usr/src-13.0/etc/group are missing from the live system, and there are no new account names relative to /usr/src-12.4./usr/src-13.0/libexec/rc. The files to refresh together later, per the existing 13.0 TODO, are /usr/src-13.0/libexec/rc/rc.subr and the rc.d scripts under /usr/src-13.0/libexec/rc/rc.d/, notably zpool, zfs and zfsbe. Do not cherry-pick a single script against the old rc.subr.zroot@pre-13.0-build and zroot/usr@pre-13.0-build./root/hop130.sh. It runs plain make -j12 buildworld from /usr/src-13.0 followed by make -j12 buildkernel KERNCONF=GENERIC with no command-line CFLAGS/COPTFLAGS override. Logs: /root/hop130.log and /root/hop130-driver.log; pid file: /root/hop130.pid. Started at 2026-09-19 12:58 BST; initial sample shows a fresh object directory at /usr/obj/usr/src-13.0/amd64.amd64 and tree-managed compiler flags.Mid-build status 2026-09-19 13:04 BST: buildworld -j12 is still running cleanly. It is in the large LLVM library pass, now around IR/ProfileData targets after moving through CodeGen. No error/OOM signatures in /root/hop130.log; swap remains 0B used; pool remains healthy; fresh 13.0 object tree is about 750M.
Mid-build status 2026-09-19 13:10 BST: buildworld -j12 is still running cleanly in stage 3 cross tools. It has moved through LLVM transforms into Clang AST/Analysis and now CodeGen objects. No error/OOM signatures in /root/hop130.log; swap remains 0B used; fresh 13.0 object tree is about 1.3G.
Mid-build status 2026-09-19 13:15 BST: buildworld -j12 has passed stage 3 cross tools, recorded build metadata, built includes, and is now in stage 4.2 building libraries. No error/OOM signatures in /root/hop130.log; swap remains 0B used; fresh 13.0 object tree is about 3.1G.
Mid-build status 2026-09-19 13:20 BST: buildworld -j12 is still running cleanly in stage 4.2 building libraries. It has moved through OpenSSL/libcrypto, private ssh/libinstall, clang runtime pieces, libarchive/libunbound/ATF, and is now back in the large LLVM library pass around Analysis/CodeGen/GlobalISel. No error/OOM signatures in /root/hop130.log; swap remains 0B used; fresh 13.0 object tree is about 4.9G.
Mid-build status 2026-09-19 13:26 BST: buildworld -j12 is still running cleanly in stage 4.2 building libraries. It has moved through LLVM DebugInfo, MC/parser, AArch64 and ARM target backends, and is now around the PowerPC target backend. No error/OOM signatures in /root/hop130.log; swap remains 0B used; fresh 13.0 object tree is about 5.1G.
Mid-build status 2026-09-19 13:30 BST: buildworld -j12 is still running cleanly in stage 4.2 building libraries. It has moved through LLVM X86 backend, transforms/instrumentation, Clang ARCMigrate, and is now around Clang AST libraries. No error/OOM signatures in /root/hop130.log; swap remains 0B used; fresh 13.0 object tree is about 5.7G.
Mid-build status 2026-09-19 13:35 BST: buildworld -j12 is still running cleanly in stage 4.2 building libraries. It has moved through Clang Driver, Lex/Parse/Rewrite/Sema, and is now in Clang StaticAnalyzer libraries. No error/OOM signatures in /root/hop130.log; swap remains 0B used; fresh 13.0 object tree is about 6.0G.
Mid-build status 2026-09-19 13:40 BST: buildworld -j12 is still running cleanly in stage 4.2 building libraries. It has moved through Clang StaticAnalyzer and into LLDB libraries, currently around LLDB language/runtime/object/process plugins including the FreeBSD/POSIX process code. No error/OOM signatures in /root/hop130.log; swap remains 0B used; fresh 13.0 object tree is about 6.5G.
Mid-build status 2026-09-19 13:42 BST: buildworld -j12 has passed stage 4.2 building libraries and is now in stage 4.3 building lib32 shim libraries. No error/OOM signatures in /root/hop130.log; swap remains 0B used; fresh 13.0 object tree is about 6.8G.
Mid-build status 2026-09-19 13:45 BST: buildworld -j12 is still running cleanly in stage 4.3 building lib32 shim libraries. It has moved through libc/msun/ncurses/kerberos/libthr, zstd/crypto, heimdal/ssh, clang sanitizer runtimes, and kerberos install pieces. No error/OOM signatures in /root/hop130.log; swap remains 0B used; fresh 13.0 object tree is about 7.9G.
Mid-build status 2026-09-19 13:47 BST: buildworld -j12 has passed stage 4.3 building lib32 shim libraries and is now in stage 4.4 building everything. No error/OOM signatures in /root/hop130.log; swap remains 0B used; fresh 13.0 object tree is about 8.7G.
Mid-build status 2026-09-19 13:51 BST: buildworld -j12 is still running cleanly in stage 4.4 building everything. The latest sample is in base libraries/tests around googletest/libarchive/atf. No real error/OOM signatures in /root/hop130.log; the visible Undefined symbols lines are bootloader self-check commands, not failures. Swap remains 0B used; fresh 13.0 object tree is about 11G.
Phase change 2026-09-19 13:53 BST: buildworld -j12 completed successfully at 13:53:04 BST after 3262 seconds. The driver recorded buildworld done and immediately started make -j12 buildkernel KERNCONF=GENERIC at 13:53:06 BST. Kernel build is now in stage 3.1 building everything. No real error/OOM signatures; swap remains 0B used; object tree is about 13G.
Build complete 2026-09-19 13:57 BST: buildkernel -j12 KERNCONF=GENERIC completed successfully at 13:57:13 BST after 246 seconds. The whole detached driver recorded hop130 done at 13:57:13 BST. No real error/OOM signatures appeared; pool stayed healthy in prior checks; swap remained 0B used; final object tree is about 15G. Next step is the install phase: retest both SSH paths, take pre-13.0 install snapshots, run installkernel from /usr/src-13.0 and reboot through the tested fallback path.
Install phase 2026-09-19 13:58 BST: pre-install snapshots zroot@pre-13.0-world and zroot/usr@pre-13.0-world were taken, then make installkernel KERNCONF=GENERIC from /usr/src-13.0 completed at 13:58:18 BST. The old 12.4 kernel was preserved as /boot/kernel.old and the 13.0 GENERIC kernel/modules were installed under /boot/kernel. Next action is reboot via the already-tested fallback sshd path and verify 13.0 kernel with 12.4 world before running installworld.
Post-kernel reboot 2026-09-19 14:00 BST: reboot through radagast-fallback succeeded. Both normal SSH and fallback SSH returned. System is in the expected mixed state: uname -r and freebsd-version -k report 13.0-RELEASE while freebsd-version -u still reports 12.4-RELEASE. Pool is healthy; zroot/usr and zroot/usr/obj are mounted; swap remains 0B used; guest sshd listeners are up on 22 and 2223. Next action is installworld from /usr/src-13.0.
Installworld 2026-09-19 14:05 BST: make installworld from /usr/src-13.0 completed with status 0 at 14:05:56 BST. The known regcomp@FBSD_1.6 rerun case did not occur; the install log scan found no matching error/fatal/failure lines. Immediately afterward both SSH paths still worked, and uname -r / freebsd-version -k / freebsd-version -u all reported 13.0-RELEASE. Pool remained healthy; swap remained 0B used. Next action is final reboot through fallback and post-reboot verification.
The 12.4 → 13.0 hop is complete. buildworld -j12 completed first try at 13:53:04 BST; buildkernel -j12 KERNCONF=GENERIC completed at 13:57:13 BST; installkernel completed at 13:58:18 BST; reboot to the 13.0 kernel succeeded; installworld completed with status 0 at 14:05:56 BST; final reboot succeeded. Final verification: uname -r / freebsd-version -k / freebsd-version -u all report 13.0-RELEASE. Normal SSH and fallback SSH both work; guest sshd listeners are up on 22 and 2223. zpool status -x reports all pools healthy; zroot/usr and zroot/usr/obj are mounted; swap remains 0B used.
zroot@pre-13.0-build / zroot/usr@pre-13.0-build before building; zroot@pre-13.0-world / zroot/usr@pre-13.0-world before installing; zroot@post-13.0-hop / zroot/usr@post-13.0-hop after final reboot verification.zpool upgrade was run. The root pool remains backward-compatible for rollback purposes.mergemaster and delete-old were skipped for this intermediate hop, consistent with the prior standing instruction. The separate fallback sshd on guest port 2223 remains in place and tested.Next: decide whether to do the previously documented 13.0 /etc/rc.d + /etc/rc.subr refresh before moving on. The system boots and the root pool imports without it, but the installed rc framework is still old because /etc reconciliation has intentionally been skipped. If deferring that refresh, the next release hop is 13.0 → 13.4 from source, with the same rules: verify both SSH paths first, read UPDATING for 13.4, take snapshots, build/install from source, and do not run zpool upgrade.
Refreshed /etc/rc.subr and the full /etc/rc.d script set together from /usr/src-13.0/libexec/rc after the 13.0 hop. Before changing files, took snapshot zroot@pre-13.0-rc-refresh and copied the previous files to /root/rc-backup-20260919-rc13. Verified after reboot through fallback SSH: normal SSH and fallback SSH both work; uname -r / freebsd-version -k / freebsd-version -u all report 13.0-RELEASE; pool is healthy; zroot/usr and zroot/usr/obj are mounted; service -e now shows /etc/rc.d/zpool before zfsbe and zfs; guest sshd listeners are up on 22 and 2223.
Standing instruction from John 2026-09-19: do not use mergemaster for /etc reconciliation. Review the files directly and decide what to do from the actual content. Remember that this is not the OS's original location: preserve host-specific and relocation-specific state unless there is a clear base-system reason to change it. Treat files covering identity, disk/pool layout, network, SSH access, accounts, rc/local boot glue, and moved-root assumptions as authoritative local configuration, not as noise to overwrite.
Did not use mergemaster. Manually compared live /etc to /usr/src-13.0/etc after the 13.0 hop and after the rc refresh. Important context: this OS is not in its original location, so files describing host identity, disk/pool layout, network/SSH access, accounts, resolver state, and boot glue were treated as intentional local state unless proven otherwise.
/etc/rc.conf, /etc/fstab, /etc/rc.local, /etc/hosts, /etc/resolv.conf, /etc/ssh/sshd_config, and /etc/ssh/sshd_config_2223. These encode the moved VM's current boot/network/SSH reality.master.passwd differs from stock mostly because of real users, root shell/password state, package-era service accounts, and moved-location accounts such as john-m. No 13.0 base users are missing. Decision: preserve.group had no missing 13.0 base groups, but it contained exact duplicate package-era groups: _tss, avahi, cups, and messagebus. Backed up to /etc/group.bak-20260919-manual-etc and removed only the duplicate repeated lines. Verified no base groups missing afterward./etc/shells differs from stock by listing /usr/local/bin/bash and /usr/local/bin/rbash. Several local users use bash as their shell, so this is intentional and was preserved./etc/mail/mailer.conf differs only by an older tag/comment and an extra send-mail alias pointing at base sendmail. Harmless compatibility alias; preserved.uname -r / freebsd-version -k / freebsd-version -u all still report 13.0-RELEASE; pool is healthy; normal SSH and fallback SSH both work; id john succeeds over both host-forwarded SSH paths.Still no mergemaster. After the first manual /etc pass, did a second pass over stock base templates that were still FreeBSD 10-era but had no local semantic changes. Took snapshot zroot@pre-13.0-etc-stock-refresh and backed the old files up under /root/etc-backup-20260919-etc13-stock. Refreshed from the 13.0 source tree: /etc/defaults/rc.conf, /etc/defaults/periodic.conf, /etc/devd.conf, /etc/newsyslog.conf, /etc/syslog.conf, /etc/login.conf, /etc/nsswitch.conf, /etc/crontab, /etc/apmd.conf, /etc/devfs.conf, /etc/inetd.conf, /etc/freebsd-update.conf, /etc/ntp.conf, and /etc/dhclient.conf. Rebuilt /etc/login.conf.db with cap_mkdb /etc/login.conf.
/etc/defaults/rc.conf so the 13.0 rc scripts see the 13.0 default variables; refreshed /etc/devd.conf for 13.0 hotplug/service actions; refreshed /etc/login.conf for 13.0 login class defaults; refreshed /etc/nsswitch.conf to add the stock netgroup line; refreshed /etc/crontab so cron uses the modern base PATH including /usr/local/sbin and /usr/local/bin./etc/ntp.conf: 13.0 ntpd tried to bind IPv6 wildcard :: and exited on this IPv4-only VM. Added ntpd_flags="-4" to /etc/rc.conf. Also overrode stale 13.0 leapfile source with current IANA URL https://data.iana.org/time-zones/data/leap-seconds.list and fetched a current file expiring 2027-06-28. Backups: /etc/rc.conf.bak-20260919-ntpd4 and /etc/rc.conf.bak-20260919-ntp-iana./usr/local/etc/rc.d but are not enabled by /etc/rc.conf or service -e. They were left alone because they are package/local state, not base /etc drift. The only enabled local service is apache24.rcorder had no errors; newsyslog -n returned 0; syslogd reloaded; devd restarted; sshd -t passed for both /etc/ssh/sshd_config and /etc/ssh/sshd_config_2223; ntpd started successfully with IPv4 only.Post-refresh reboot verification passed. Rebooted through radagast-fallback; after boot both normal SSH and fallback SSH worked. uname -r / freebsd-version -k / freebsd-version -u all report 13.0-RELEASE; zpool status -x reports all pools healthy; zroot/usr and zroot/usr/obj are mounted; swap remains 0B used; ntpd, cron, devd, sshd and apache24 are running. Took post-change snapshot zroot@post-13.0-etc-refresh.
Current checkpoint: radagast is running 13.0-RELEASE and the 13.4 hop has started. Verified both SSH paths first: normal radagast and fallback radagast-fallback both work and report 13.0-RELEASE. Pool is healthy; zroot/usr and zroot/usr/obj are mounted; swap is the 4 GiB ZVOL and was unused at start.
src.txz and MANIFEST for 13.4-RELEASE from archive.freebsd.org/old-releases/amd64/13.4-RELEASE/ on pomelo. SHA256 verified on pomelo and again inside radagast: fa3613b372a51ed36e11f33b08c181ffe047df70ec0e89c70f413b1009ea5c97. Extracted to /usr/src-13.4 with tar --strip-components=2; tree identifies as REVISION="13.4" BRANCH="RELEASE".UPDATING entries. Key carried-forward point for this hop: the 2024-04-19 LLVM/libc++ update says make delete-old matters after install because obsolete libc++ headers can cause C++ compilation errors. Do not run it blindly during build/install, but handle it manually after 13.4 is booted and verified.realtime::47:, idletime::48:, and u2f:*:116:. Backed up /etc/group to /etc/group.bak-pre-13.4 and added those lines; no source-required groups are missing afterward.zroot@pre-13.4-build and zroot/usr@pre-13.4-build./root/hop134.sh. It runs plain make -j12 buildworld from /usr/src-13.4 followed by make -j12 buildkernel KERNCONF=GENERIC with no command-line CFLAGS/COPTFLAGS override. Logs: /root/hop134.log and /root/hop134-driver.log; pid file: /root/hop134.pid. Started at 2026-09-19 14:27 BST; initial sample shows a fresh object directory at /usr/obj/usr/src-13.4/amd64.amd64 and tree-managed compiler flags.Mid-build status 2026-09-19 14:29 BST: buildworld -j12 is running cleanly in stage 3 cross tools, currently compiling LLVM analysis objects. No error/OOM signatures in /root/hop134.log; swap remains 0B used; fresh 13.4 object tree is about 692M.
Used the build from /usr/src-13.4 completed on 2026-09-19 at 15:48 BST (/root/hop134.log). Took zroot@pre-13.4-world and zroot/usr@pre-13.4-world, then installed GENERIC successfully (/root/installkernel134.log; completion 20:50 BST). Reboot verified 13.4 kernel with 13.0 userland. make installworld completed with exit status 0 (/root/installworld134.log). No matching fatal-error/Stop-in/Error-code lines were found in that log.
During installworld, new connections to both SSH ports failed because the old running sshd could not pass re-exec state to the newly installed binary (console: recv_rexec_state: parse config: incomplete message). The original installation connection survived and reported successful completion. Sent Ctrl-Alt-Delete through the local QEMU VNC console for a clean reboot; both SSH paths then recovered. The initial installkernel kldxref warnings for two iwlwifi firmware modules disappeared when rerunning the newly installed 13.4 kldxref.
Manual /etc reconciliation: backed up /etc under /root/etc-backup-20260920-134/etc. /root/refresh-etc134.sh validates stock-file identity before replacement. Refreshed the base rc framework, rc.d scripts and defaults/rc.conf; the remaining old rc framework files were byte-identical to the original /usr/src stock versions. Refreshed the previously reviewed stock templates (periodic defaults, devd, newsyslog, syslog, login, nsswitch, crontab, apmd, devfs, inetd, freebsd-update, ntp and dhclient) after verifying they still matched 13.0 stock. Rebuilt login.conf.db. Preserved and byte-compared rc.conf, fstab, rc.local, hosts, SSH configs, master.passwd, group, shells and mailer.conf. Resolver configuration was preserved; DHCP remains responsible for it. No mergemaster used.
LLVM/libc++ cleanup: obtained the 13.4 obsolete-file list in /root/old-files134.list. Of 243 obsolete libc++ header candidates, 177 were present as files. Checked that each was absent from the current libc++ source headers, then moved those files into /root/obsolete-libcxx-20260920-134/ (moved.list records them). Post-cleanup audit found no remaining candidate files. Did not blindly run delete-old or delete-old-libs. FreeBSD clang 18.1.6 compiled and ran /root/cxx134-check.cc with C++20, -O1 and pthreads, exercising containers, algorithms, async/threading and exception handling.
Final reboot verification at about 21:02 BST: freebsd-version -kru reports 13.4-RELEASE for all three; normal radagast SSH and radagast-fallback work, listening on guest ports 22 and 2223. All pools healthy; /, /home, /p, /q, /tmp, /usr and /usr/obj mounted correctly. ntpd, cron, devd, sshd and apache24 running; NTP has a selected synchronized peer; swap usage 0B. sshd configuration checks, rcorder, newsyslog dry run and Apache syntax checks passed. Apache responds with an HTTP authentication challenge on its existing port 8888 (not port 80). Existing Apache ServerName warnings and the previously missing sendmail configuration remain outside this hop. Post-hop snapshots: zroot@post-13.4-hop and zroot/usr@post-13.4-hop. No zpool upgrade; family-data datasets untouched.
Downloaded 14.4-RELEASE src.txz and MANIFEST from https://download.freebsd.org/releases/amd64/14.4-RELEASE/; verified SHA256 on pomelo and radagast: 1b337069331dbf0791dc43129900377da256ba1ec0ef94e0d3f6e6232de84fe2. Extracted /usr/src-14.4. Both SSH paths and healthy ZFS verified. Snapshots zroot@pre-14.4-build and zroot/usr@pre-14.4-build. Driver builds world then GENERIC kernel with explicit CFLAGS='-O1 -pipe' and COPTFLAGS='-O1 -pipe'; live compile commands confirm -O1. No missing base users or changed base groups found. UPDATING reviewed: OpenSSL 3 transition, OPIE removal (old PAM entries found and need targeted removal), DMA default mailer (preserve local mailer configuration), major-version installworld should run in single-user mode; VM console is secure and accessible via local VNC. Installation and final validation remain pending.
Pre-install preparation 2026-09-21: backed up PAM to /root/pam-backup-pre144/pam.d and removed only active pam_opie.so/pam_opieaccess.so authentication lines from sshd, system, other, ftp, ftpd and telnetd. Fresh root SSH logins on both paths pass. Prepared /root/refresh-etc144.sh (stock-file preservation preflight passes), including 14.4 split devd configuration snippets, and /root/installworld144-single.sh for console single-user installation after the new kernel is built and installed. Scripts are prepared but not executed. VM console helper on pomelo: /home/john/tmp/radagast-console.py; local VNC 127.0.0.1:5905. Build continues; no installkernel/installworld yet.