Incident: /usr/local deleted via symlink — lessons

On 2026-10-05, rm -rf /usr/local.new deleted the entire installed ports tree because /usr/local was a symlink to /usr/local.new. This was a radagast-specific upgrade artifact (ports were built into /usr/local.new during the openssl cascade, and the symlink was the final switchover step — never replaced with a real directory). Required full rebuild of all ML packages.

Root cause 1 — not checking for symlinks before rm -rf

Always resolve the physical path before a destructive operation on a directory. Two checks that would have caught this:

ls -la /usr/ # shows: local -> /usr/local.new
cd /usr/local && pwd -P # prints: /usr/local.new — not /usr/local

The pwd -P check is the canonical quick test. If pwd -P and pwd differ, you are inside a symlink and rm -rf on the target deletes the real data.

Root cause 2 — tautological inode audit

The audit script checked: "does /usr/local/x have the same inode as /usr/local.new/x?" When /usr/local is a symlink to /usr/local.new, both paths resolve to the same file — the inodes always match. The audit was vacuously true and gave false confidence.

A correct audit must check for symlinks FIRST:

# Before any inode comparison: if [ -L /usr/local ]; then echo "STOP: /usr/local is a symlink to $(readlink /usr/local)" exit 1 fi

Root cause 3 — snapshot after, not before

The ZFS snapshot was taken AFTER rm -rf completed — preserving the broken state, not the working one. Snapshots must be taken BEFORE any destructive or hard-to-reverse operation. The rule:

Snapshot → Verify → Act, never Act → Snapshot.

Root cause 4 — unfinished upgrade artifact

/usr/local -> /usr/local.new was a symlink left over from the ports-tree rebuild during the openssl cascade. The correct final step of that upgrade was to replace the symlink with a real directory (rename or bind-mount). A dangling upgrade artifact disguised as a normal directory was a latent bomb.

After any ports-tree migration that uses a staging directory: verify ls -la /usr/ shows local as a real directory, not a symlink, before considering the migration done.

Checklist for future rm -rf on large directories

Recovery

155 staged ports reinstalled immediately from work/stage/ directories. Python ML packages (pytorch, scipy, numpy, pyarrow, llvmlite, scikit-*, h5py, opencv-python-headless, xgboost, librosa) required full recompile — rebuild16.sh. Total rebuild time: several hours.