[x] Ensure new-host provisioning playbook writes ~/.netrc correctly for WebDAV. (done — launch_instance.yml, skeleton tag)
libcurl requires the keyword login (not user) for the username field in ~/.netrc. cadaver accepts user, so mixed setups silently fail to authenticate against WebDAV servers. The provisioning playbook must generate:
Not:
[ ] Remove unattended-upgrades from existing hosts and prevent it on new deploys. John does not want unexpected package changes on working EC2 hosts. Ansible/provisioning must disable and remove unattended-upgrades and must ensure new deployments do not install or enable it. Required state: APT::Periodic::Unattended-Upgrade "0"; or no unattended-upgrades config, apt-daily-upgrade.timer disabled/masked or absent, and package unattended-upgrades removed/purged. Apply to kelp and gravlax immediately; update Ansible on pomelo later and test on a disposable host.
[ ] Add an rc.local guard against unattended-upgrades. Because the package/timers may sneak back in via image defaults or dependencies, managed hosts should run a boot-time check from /etc/rc.local. The check must disable/mask unattended-upgrades and apt-daily-upgrade.timer/apt-daily-upgrade.service if present. If unattended-upgrades is installed, enabled, or active, the guard must add a clear warning block to /etc/motd so login sessions show that the host violated the no-unattended-upgrades policy. Add this to Ansible for future deploys and test that it catches a deliberately re-enabled timer.
2026-07-16 note: applied the no-unattended-upgrades policy manually to kelp and gravlax. Added an rc.local-start guard under /etc/rcs/no-unattended-upgrades on both hosts. The guard disables/masks unattended-upgrades units if they reappear, writes a warning block into /etc/motd only when a violation is detected, and logs to /var/log/no-unattended-upgrades-guard.log. If Ansible can provide a safe authenticated notes-write mechanism at boot, extend the guard to also report violations to notes; otherwise keep notes reporting as an Ansible/operator task rather than embedding secrets in boot scripts.
Also ensure the file has mode 0600 — libcurl refuses to read a world-readable netrc.
Investigate whether Ansible or provisioning caused broad system ownership drift on the local host. Symptom discovered while exporting OwnTracks KML: default ssh failed before connection with Bad owner or permissions on /etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf; sudo also failed because /etc/sudo.conf is owned by uid 65534.
Observed ownership before attempted repair: /, /etc, /etc/sudo.conf, /etc/ssh, /etc/ssh/ssh_config.d, /usr, /usr/lib, /usr/lib/systemd, /usr/lib/systemd/ssh_config.d, and /usr/lib/systemd/ssh_config.d/20-systemd-ssh-proxy.conf were all nobody:nogroup (uid/gid 65534). Modes were mostly normal (755 directories, 644 files); the issue is ownership.
This does not look like a narrow SSH configuration issue. It affects broad package-owned system paths under /, /etc, and /usr, so likely causes include a bad recursive chown, a bad restore/rsync/tar extraction, or filesystem/user-namespace ownership mapping. Pomelo was unreachable over SSH during the check, so the primary ~/ansible repo could not be inspected yet.
[x] RESOLVED 2026-09-05 at John's request; no further investigation required. Current checks on pomelo, kelp and gravlax found correct critical ownership and working SSH/sudo. Added a read-only ownership audit before privileged provisioning; live audits, disposable-file tests and syntax checks passed. No ownership repair was performed. The original July environment and historical root cause were not established; preceding incident details and the draft repair are retained as historical context, not outstanding actions. Findings: docs/system-ownership-incident.md. COMMITTED 2026-09-17 as f52bec1 on branch m and pushed.
[x] Raise Apache worker capacity on gravlax and manage it via Ansible. DONE 2026-09-05: deployed roles/apache_capacity via apache_capacity.yml and integrated it into setup_server.yml provisioning. Active MPM is event (Apache 2.4.68). MaxRequestWorkers increased from 150 to 400; ServerLimit from default 16 to 32 (16 active children × 25 threads, plus room for draining generations). KeepAliveTimeout reduced to 2 seconds; ProxyTimeout set to 120 seconds of upstream inactivity. Added local-only status reporting and a systemd timer recording worker counters every minute in journald, with pressure warnings (diagnostic logging, not external alerts). Apache configtest passed and full restart succeeded; a repeat Ansible deployment made zero changes. Notes returned HTTP 200; WebDAV and public/private/misc MCP returned expected HTTP 401 authentication challenges; external server-status returned HTTP 403. Existing live virtual hosts were preserved. COMMITTED 2026-09-17 as 5c5879c (role) and e25ba49 (setup_server integration) on branch m, and pushed. [ ] ROOT-CAUSE FOLLOW-UP: the 2026-08-05 outage filled the shared Apache scoreboard, affecting WebDAV, notes/MCP and private store. Historical Apache error logs have rotated away; journal confirms a midnight graceful reload and a full restart at 06:07 UTC that day. Current baseline was healthy (1 busy status request, 49 idle workers, no stopping processes or active client connections). Eight CLOSE-WAIT REST backend sockets did not correspond to occupied workers and do not prove a leak. Live MCP proxies already disable backend connection reuse. Long-lived SSE pinning workers or draining generations remains a hypothesis, not an established cause or a fixed leak; active heartbeats can keep streams alive beyond ProxyTimeout. Review journalctl -u apache-worker-sample.service and capture local full server-status plus ss -tanp during recurrence to attribute pressure. Investigation, limitations and commands: roles/apache_capacity/README.md.
[x] RESOLVED 2026-09-05 at John’s request; no further investigation required. Current checks on pomelo, kelp and gravlax found correct critical ownership and working SSH/sudo. Added a read-only ownership audit before privileged provisioning; live audits, disposable-file tests and syntax checks passed. No ownership repair was performed. The original July environment and historical root cause were not established; preceding incident details and the draft repair are retained as historical context, not outstanding actions. Findings: docs/system-ownership-incident.md. Changes remain local/uncommitted.
[x] Raise Apache worker capacity on gravlax and manage it via Ansible. DONE 2026-09-05: deployed roles/apache_capacity via apache_capacity.yml and integrated it into setup_server.yml provisioning. Active MPM is event (Apache 2.4.68). MaxRequestWorkers increased from 150 to 400; ServerLimit from default 16 to 32 (16 active children × 25 threads, plus room for draining generations). KeepAliveTimeout reduced to 2 seconds; ProxyTimeout set to 120 seconds of upstream inactivity. Added local-only status reporting and a systemd timer recording worker counters every minute in journald, with pressure warnings (diagnostic logging, not external alerts). Apache configtest passed and full restart succeeded; a repeat Ansible deployment made zero changes. Notes returned HTTP 200; WebDAV and public/private/misc MCP returned expected HTTP 401 authentication challenges; external server-status returned HTTP 403. Existing live virtual hosts were preserved. Changes remain uncommitted in /home/john/ansible. [ ] ROOT-CAUSE FOLLOW-UP: the 2026-08-05 outage filled the shared Apache scoreboard, affecting WebDAV, notes/MCP and private store. Historical Apache error logs have rotated away; journal confirms a midnight graceful reload and a full restart at 06:07 UTC that day. Current baseline was healthy (1 busy status request, 49 idle workers, no stopping processes or active client connections). Eight CLOSE-WAIT REST backend sockets did not correspond to occupied workers and do not prove a leak. Live MCP proxies already disable backend connection reuse. Long-lived SSE pinning workers or draining generations remains a hypothesis, not an established cause or a fixed leak; active heartbeats can keep streams alive beyond ProxyTimeout. Review journalctl -u apache-worker-sample.service and capture local full server-status plus ss -tanp during recurrence to attribute pressure. Investigation, limitations and commands: roles/apache_capacity/README.md.
[ ] Deploy John's personal SSH key instead of generating a new admin key each run. The provisioning playbook currently generates a fresh admin keypair per run; because each ephemeral host has a different hostname, stale admin@<hostname> entries accumulate in authorized_keys on gravlax. Deploy John's personal public key instead.
[x] Version-control the delayrepay toolkit + ~/aws scripts. DONE (local) 2026-09-02 — TWO local git repos on kelp (not pomelo — 2026-09-17 correction; they are at kelp:~/delayrepay and kelp:~/aws), both branch main, remotes set to git@github.com:john-critchley/<name>.git: (1) ~/delayrepay now at ee82bb1; .gitignore excludes keys/certs, dataset/, evidence-*.tar.gz, .claude/, __pycache__/, scratch images. (2) ~/aws now at 790ecd3 — tracks only awslaunch.py + README + .gitignore; DEFAULT-DENY .gitignore ignores everything and whitelists just those, so browser-secrets/ (GWR key4.db+logins.json), viewer-keys/ (private keys), viewer-csrs/, host-certs/, stunnel/ are all excluded. [ ] STILL OPEN — PUSH TO GITHUB (both private). Verified 2026-09-17: neither GitHub repo exists yet (git ls-remote returns "Repository not found" — note that is a 404, not an auth failure, so kelp's SSH key works fine and only repo CREATION is blocked). The recorded blocker is wrong: this note said to do it from pomelo because pomelo holds the GitHub PAT, but pomelo's ~/.netrc has no api.github.com entry. So the PAT must be found or a new one minted before either repo can be created; after that git push -u origin main from kelp will work as-is. This is now the last place in the estate where work exists in only one copy.
See ansible/aws-browser-host — plan to make kelp a provisioning controller and add a browser-host role/tag (Firefox+VNC+stunnel) for on-demand Delay Repay hosts.
2026-08-28: the browser_host role + provision_browser_host.yml (role-based refactor + stunnel viewer PKI + the WebDAV parent-dir MKCOL fix) were committed and pushed to GitHub as branch browser-host (commit 4a4132f, 5 commits ahead of origin/m). See delayrepay/provisioning. Two decisions remain:
[ ] Merge browser-host into m (or open a PR)? 2026-09-02: John's decision — LEAVE ON THE BRANCH for now (no merge/PR yet). The branch is pushed and in sync with origin/browser-host. Note the local working tree also has superseded cruft from the 2026-08-29 kombu session — setup_browser_host_enhanced.yml (a monolithic playbook fully superseded by the browser_host role, which additionally does the stunnel mTLS PKI + secure GWR-login injection it lacks), setup_browser_host.yml.backup, and a stale hosts.ini kombu entry (host terminated) — all safe to discard when the branch is next landed. At merge time also decide whether to retire the old monolithic setup_browser_host.yml.
[x] Version-control the delayrepay toolkit + ~/aws scripts. DONE (local) 2026-09-02 — TWO local git repos, both branch main, remotes set to git@github.com:john-critchley/<name>.git, ready to push: (1) ~/delayrepay commit 93a896c (19 files); .gitignore excludes keys/certs, dataset/, evidence-*.tar.gz, .claude/, __pycache__/, scratch images. (2) ~/aws commit 99a7cd3 — tracks only awslaunch.py + README + .gitignore; DEFAULT-DENY .gitignore ignores everything and whitelists just those, so browser-secrets/ (GWR key4.db+logins.json), viewer-keys/ (private keys), viewer-csrs/, host-certs/, stunnel/ are all excluded. [ ] STILL OPEN — PUSH TO GITHUB (both private): DEFERRED 2026-09-02 until pomelo is powered on (it holds the GitHub PAT; kelp/gravlax have none). When pomelo is up: add machine api.github.com login john-critchley password <PAT> to ~/.netrc, then create both private repos via the API and git push -u origin main each. SSH-to-GitHub already works from kelp; the PAT is only needed to CREATE the repos. (Repos are private, so the claim YAML CTR needs no sanitising.)
[x] Create a GLOBAL todo index that links every other todo note. DONE 2026-09-02 — created TODO (top-level master index), linked from CONTENTS and added to TRIGGERS. It links ansible/todo, envoy/todo, gdata-server/todo (+sub-pages), popit3/todo, notes-browser/todo, hibernate-on-approach/todo, ssl/certhub-todo, john/actions, todo/buy-gym-sandals, and the Future roadmap. [ ] STILL OPEN (stretch): a small tool that scans all notes for [ ] markers and regenerates the index automatically so it cannot drift.
[ ] browser_host role fixes found during the first live claim (2026-09-02, host morel). (1) VNC dies on first boot — tigervncserver@:1 exits status=1 with ‘Could not migrate ~/.vnc to ~/.config/tigervnc’ because ~/.config doesn’t exist yet at boot; the host is then undrivable. Fix: have the role create ~/.config (or write the VNC config straight to ~/.config/tigervnc). (2) Firefox login autofill — the run must verify/re-inject the key4.db + logins.json pair with Firefox CLOSED, then (re)launch, or the saved GWR credential doesn’t autofill (seen: logins.json missing from the live profile). (3) minor: the superseded setup_browser_host_enhanced.yml has a wrong www.delayrepay.gwr.com start URL (doesn’t resolve; role’s start_url is correct). Detail in delayrepay/lessons.
[ ] browser_host role — same fixes STILL open; all recurred on host enoki (2026-09-03, return-leg claim) plus more gaps. The VNC ~/.config/tigervnc bug and the missing Firefox login both bit again. Additional this run: (4) the role creates NO Firefox profile at all — had to firefox -CreateProfile "gwr /home/admin/.mozilla/firefox/gwr" (needs DISPLAY=:1), scp key4.db+logins.json into it, then launch firefox -P gwr <url>; the role should create the profile AND inject the credential pair so login autofills on boot. (5) evidence images aren't staged — scp the ticket JPEG(s) to /home/admin/reclaim/ from WebDAV before the run. (6) maximise the Firefox window (wmctrl -i -r <id> -b add,maximized_vert,maximized_horz) for clean full-res driving. NET GOAL: the role boots a CLAIM-READY host (VNC up, gwr profile + creds, images staged, window maximised) so no manual bring-up. Also keepalive.py updated (proactive pointer-move nudge + saves the modal to WebDAV when seen) but the reactive ‘Stay signed in’ click still does NOT extend the session — needs a real-modal capture to calibrate the true button. Detail: delayrepay/lessons.
[ ] Investigate why ~/later/30_forget_host_keys.sh --yes hangs. 2026-09-03: after 20_terminate_instance.sh cleanly terminated enoki, running 30 (--yes) stalled and had to be interrupted. Most likely culprit: run sudo sed -i "/\b$NAME\b/d" /etc/hosts — sudo with no cached credential blocks on a password prompt in the non-interactive harness shell (no tty), hanging forever. Other candidates: a WebDAV curl -X DELETE on delayrepay/certs/<name>/ (collection delete) with no timeout, or ssh-keygen -R iterating. FIX: make the script non-blocking — use sudo -n (skip + warn if it can't), add curl --max-time, and/or wrap steps in timeout. The termination itself (step 20) worked; 30 is only local/WebDAV housekeeping, so enoki was already gone. The known_hosts / /etc/hosts / WebDAV-cert cleanup for enoki was NOT completed and should be finished manually or after the fix.
• Fix awslaunch.py SG ingress handling (found 2026-09-09): on a reused/running host, awslaunch.py does NOT add kelp's current public egress IP to the awslaunch-ssh SG port-22 allowlist — it had accumulated six stale old-kelp /32s and omitted the live one (63.182.255.21/32), so provision_browser_host.yml timed out at Wait for SSH (241s) against healthy host porcini. Manual fix: curl -s https://checkip.amazonaws.com → aws ec2 authorize-security-group-ingress ... --cidr <ip>/32, then revoked the six stale ranges. Proper fix: awslaunch should detect kelp's real egress IP, add that /32, and prune /32s it previously added (tag/describe its own rules, or use a managed prefix list) so the allowlist can't drift. Port 15901 stays world-open (mTLS) and is unaffected.
• Decouple host teardown from the single mutable instance.env (found 2026-09-09): the ~/later/ teardown scripts (20_terminate_instance.sh, 30_forget_host_keys.sh) source one shared ~/later/instance.env, which provision_browser_host.yml OVERWRITES on every run. So you cannot safely provision a new host while an old one still needs teardown — provisioning the new host first would repoint the teardown at the new host. This forces an unwanted serialize (tear down old, THEN provision new); hit while replacing porcini with chanterelle. Fix: make host state per-host and target teardown by identity, not by whatever the last provision wrote. E.g. provision writes ~/later/instance-<name>.env (keep a small live-host registry); teardown scripts take a --name <host> (or --id <instance-id>) argument and load that host's env, defaulting to the most-recent only when unambiguous. Then ‘provision new’ and ‘terminate old’ can run concurrently (and it also enables running >1 browser host at once). Until then, teardown-first is the safe order.
• Fix the GWR-login Firefox-profile injection (found 2026-09-09): the browser_host role injects key4.db+logins.json into a discovered profile, but the running firefox-esr launches a DIFFERENT profile (the install-default .default-esr, not the Default=1 .default), so the saved GWR login never autofills — reproduced on porcini AND chanterelle. Fix: inject into (or launch firefox with --profile pointing at) the exact profile firefox-esr uses; verify about:logins shows delayrepay.gwr.com after provision. Also ensure the role's Firefox launch runs INSIDE the XFCE session (with DBUS_SESSION_BUS_ADDRESS) so the window actually renders. Related still-open role bug: VNC dies on first boot (tigervncserver@:1 inactive) — needs a restart-after-first-boot in the role.
• Stable name→IP resolution for the ephemeral browser host across machines (raised 2026-09-09): each provision gives the host a new public IP, and every machine that talks to it must learn the new mapping — kelp (driving + /etc/hosts), and the Mac/pomelo viewers (whose stunnel connect = currently hardcodes the IP, so it must be edited per host). This is the main viewer-reconnect friction. Want a single stable name (e.g. gwr-browser.critchley.biz) that resolves to the current host on ALL machines, so stunnel/viewer/known_hosts configs reference the NAME and never change host-to-host. Options: (a) push /etc/hosts to each machine on provision — simple but needs sudo/remote access to every viewer, brittle; (b) dynamic DNS — update an A record (via the existing ZoneEdit account) to the new IP each provision; all machines resolve it automatically within TTL, no per-machine edits (probably lowest friction); (c) a resolver extension (NSS module / dnsmasq / local stub) that resolves a special name by reading delayrepay/current-host.env from WebDAV or a tiny lookup endpoint — most flexible, most work; (d) pair with a fixed CN (and optionally a reused Elastic IP) so stunnel checkHost and the host key also stay static. Recommendation: (b) dynamic DNS + (d) fixed CN, so the viewer stunnel.conf becomes constant. Related: identity-based / stable-endpoint principle.
2026-09-09 — implemented, PENDING next-provision verification (no host stood up): (1) Firefox-login injection — refined root cause: it was NOT (only) a profile mismatch; the mint task killed firefox by launcher-PID only, so a surviving process rewrote logins.json on exit and wiped the just-injected login. Fix in role browser_host/tasks/main.yml (branch browser-host): kill ALL firefox before inject via pkill -f '[f]irefox', launch pinned to --profile, and a fail-fast assert if no profile was minted. (2) VNC first boot — create ~/.config+~/.cache before starting tigervnc, plus a Restart=on-failure systemd drop-in. (3) awslaunch SG — ensure_security_group now runs on the REUSE path too (was launch-only), and prunes stale tcp/22 /32s. Also (toolkit, delayrepay repo): drive.py text (one-pass OCR→text), and runner assert_text/pause actions + offline --dry-run, with a new self-verifying spec claim/claim-2026-09-09.yaml. All compile/parse/dry-run clean; none committed.
Commit sums (2026-09-09, before → after): delayrepay b5e7fa0 → ee82bb1 (branch main); ansible 5dfb9a0 → 38b2e6f (branch browser-host); aws 99a7cd3 → 790ecd3 (branch main). Only the intended files were staged — the ansible repo's pre-existing hosts.ini change and untracked setup_* files were left uncommitted.
VALIDATED on a fresh provision (maitake): the Firefox-login fix now uses firefox-esr -CreateProfile (no browsing session → nothing to shut down → the injected login can't be clobbered), replacing the mint-then-kill dance (which either let a still-exiting Firefox rewrite logins.json, or self-killed via pkill -f firefox matching its own argv, rc -15). Confirmed on maitake: autofill populated john.gw@critchley.biz + password with zero manual fixup, VNC survived first boot, failed=0. Commit (ansible, browser-host): 38b2e6f → e1d6462.
Audit found work existing in exactly one place. kelp's /home/john/git/ansible (branch browser-host) held two unpushed commits — including e1d6462, the Firefox-login fix validated on maitake — and kelp's ~/delayrepay and ~/aws repos had never been pushed at all. Meanwhile pomelo's /home/john/ansible sat on branch m, 11 commits behind browser-host, carrying a large uncommitted pile. gravlax's ~/ansible turned out not to be a git repo at all — just a stray directory copy; ignore it.
5dfb9a0..e1d6462. No longer single-copy.252b0ab..280466b, eight commits in dependency order (gitignore, ownership audit, apache_capacity role, patched at binaries, multi-SAN vhosts, setup_server wiring, launch_instance, inventory+misc). John confirmed the "leave it until I'm totally happy" state had simply gone stale: "it has been a bit so remove this text from the notes and go for it. I think it is all good by now."apache_capacity vs browser_host).setup_browser_host_enhanced.yml and setup_browser_host.yml.backup — now gitignored rather than deleted, so they stay on disk but cannot be committed by accident. .claude/ and transcript.en.vtt gitignored too. secrets.yml was already ignored and was never touched.How it was verified — reusable. A checker (ansible_integrity.py) walks a checkout and reports broken internal dependencies: unparseable YAML, copy/template src pointing at files that do not exist, roles referenced but absent, missing include/import targets, and notified handlers that nothing defines. It was validated against deliberately injected breakage before being trusted — delete a referenced file, remove a referenced role, corrupt a playbook's YAML, and confirm each is caught. Then every one of the eight commits was extracted with git archive and checked individually, not just the final tree: all reported exactly the same single finding as the pre-existing baseline, so no commit introduced a broken dependency. ansible-playbook --syntax-check was run as an independent second opinion and agreed.
[ ] REAL BUG FOUND, NOT YET FIXED: setup_controller.yml does not parse. Line 108 reads - name: Note: SSH private key must be copied manually — an unquoted YAML scalar containing a colon-space, so YAML reads it as a nested mapping and refuses. ansible-playbook --syntax-check setup_controller.yml fails with "Colons in unquoted values must be followed by a non-space character". This playbook cannot run at all, and the fault predates all of today's work — it is present on both m and browser-host. Fix is one line: quote the name. Left alone deliberately because it was outside the scope of reconciling the branches, and fixing it would have changed the verification baseline mid-flight.
[ ] gdata_api_pass is still CHANGEME, so its htpasswd task never runs. Found 2026-09-17 by characterising secrets.yml structurally (without reading values). setup_server.yml guards the "Create gdata API htpasswd file" task with when: gdata_api_pass != 'CHANGEME', and the variable is still the literal placeholder — so /etc/apache2/.htpasswd-gdata is silently never created on any provisioned host. This is worse than a failure, because the run reports success. Either set a real value or, if the gdata API no longer needs its own htpasswd, remove the task and the variable so the gap is not carried forward to kelp.
• Check: setup_server.yml wrongly deploys gdata-server onto kelp (2026-09-21): setup_server.yml (tags gdata/mcp) installs and runs a LOCAL gdata notes stack on kelp — units gdata-mcp-server.service (public, ports 8020/8023) + gdata-server.service, syncing code to gdata_server_dest=/home/john/py/gdata-server. John does NOT want a gdata instance on kelp (the authoritative notes server is gravlax; kelp only needs the stunnel client tunnels on 8021/8024 to reach it). Removed manually from kelp 2026-09-21 (stopped+disabled, unit files + ~/py/gdata-server deleted; backup ~/backups/gdata-server-kelp-removed-20260921.tar.gz). To fix: scope the gdata/mcp tasks to gravlax only (host-conditional) so a kelp reprovision won't re-add them. Also review the odd gdata_server_src == gdata_server_dest (~/py/gdata-server → itself) self-sync. (Kelp is persistent right now, so the manual removal will hold until a reprovision.)