[x] Ensure new-host provisioning playbook writes ~/.netrc correctly for WebDAV. (done — launch_instance.yml, skeleton tag)
libcurl requires the keyword login (not user) for the username field in ~/.netrc. cadaver accepts user, so mixed setups silently fail to authenticate against WebDAV servers. The provisioning playbook must generate:
Not:
[ ] Remove unattended-upgrades from existing hosts and prevent it on new deploys. John does not want unexpected package changes on working EC2 hosts. Ansible/provisioning must disable and remove unattended-upgrades and must ensure new deployments do not install or enable it. Required state: APT::Periodic::Unattended-Upgrade "0"; or no unattended-upgrades config, apt-daily-upgrade.timer disabled/masked or absent, and package unattended-upgrades removed/purged. Apply to kelp and gravlax immediately; update Ansible on pomelo later and test on a disposable host.
[ ] Add an rc.local guard against unattended-upgrades. Because the package/timers may sneak back in via image defaults or dependencies, managed hosts should run a boot-time check from /etc/rc.local. The check must disable/mask unattended-upgrades and apt-daily-upgrade.timer/apt-daily-upgrade.service if present. If unattended-upgrades is installed, enabled, or active, the guard must add a clear warning block to /etc/motd so login sessions show that the host violated the no-unattended-upgrades policy. Add this to Ansible for future deploys and test that it catches a deliberately re-enabled timer.
2026-07-16 note: applied the no-unattended-upgrades policy manually to kelp and gravlax. Added an rc.local-start guard under /etc/rcs/no-unattended-upgrades on both hosts. The guard disables/masks unattended-upgrades units if they reappear, writes a warning block into /etc/motd only when a violation is detected, and logs to /var/log/no-unattended-upgrades-guard.log. If Ansible can provide a safe authenticated notes-write mechanism at boot, extend the guard to also report violations to notes; otherwise keep notes reporting as an Ansible/operator task rather than embedding secrets in boot scripts.
Also ensure the file has mode 0600 — libcurl refuses to read a world-readable netrc.
Investigate whether Ansible or provisioning caused broad system ownership drift on the local host. Symptom discovered while exporting OwnTracks KML: default ssh failed before connection with Bad owner or permissions on /etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf; sudo also failed because /etc/sudo.conf is owned by uid 65534.
Observed ownership before attempted repair: /, /etc, /etc/sudo.conf, /etc/ssh, /etc/ssh/ssh_config.d, /usr, /usr/lib, /usr/lib/systemd, /usr/lib/systemd/ssh_config.d, and /usr/lib/systemd/ssh_config.d/20-systemd-ssh-proxy.conf were all nobody:nogroup (uid/gid 65534). Modes were mostly normal (755 directories, 644 files); the issue is ownership.
This does not look like a narrow SSH configuration issue. It affects broad package-owned system paths under /, /etc, and /usr, so likely causes include a bad recursive chown, a bad restore/rsync/tar extraction, or filesystem/user-namespace ownership mapping. Pomelo was unreachable over SSH during the check, so the primary ~/ansible repo could not be inspected yet.
Attempted repair from this environment on 2026-07-09 failed with Operation not permitted for each chown; this shell is uid=1000(admin), not root, and sudo is broken by the ownership problem. No ownership changes were applied.
# Minimal root-shell repair proposed after confirming sudo/ssh paths were also nobody:nogroup.
# Run from a real root shell on the affected host, e.g. after `sudo -i` from a working login.
chown root:root / /etc /usr /usr/lib /usr/lib/systemd /usr/lib/systemd/ssh_config.d
chown root:root /etc/sudo.conf /etc/sudoers /etc/sudoers.d /usr/bin/sudo
chown root:root /etc/ssh /etc/ssh/ssh_config /etc/ssh/ssh_config.d
chown root:root /usr/lib/systemd/ssh_config.d/20-systemd-ssh-proxy.conf
chown -h root:root /etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf
chmod 755 / /etc /usr /usr/lib /usr/lib/systemd /usr/lib/systemd/ssh_config.d
chmod 644 /etc/sudo.conf /etc/ssh/ssh_config /usr/lib/systemd/ssh_config.d/20-systemd-ssh-proxy.conf
chmod 440 /etc/sudoers
chmod 750 /etc/sudoers.d
chmod 4755 /usr/bin/sudo
chmod 755 /etc/ssh /etc/ssh/ssh_config.d
# Verify after repair:
stat -c '%u:%g %U:%G %a %n' / /etc /etc/sudo.conf /etc/sudoers /etc/sudoers.d /usr/bin/sudo /etc/ssh /etc/ssh/ssh_config /etc/ssh/ssh_config.d /etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf /usr /usr/lib /usr/lib/systemd /usr/lib/systemd/ssh_config.d /usr/lib/systemd/ssh_config.d/20-systemd-ssh-proxy.conf
ssh -G localhost >/dev/null
sudo -n true && echo sudo-okWhen on pomelo again: inspect ~/ansible for any task that can recursively chown /, /etc, /usr, package trees, extracted root filesystems, or mounted volumes. Check Ansible logs on pomelo (/var/log/ansible.log) around the provisioning time. Then reproduce on a disposable host before running against this system again.
[ ] Raise Apache worker capacity on gravlax (MPM MaxRequestWorkers / ServerLimit — the "scoreboard"). 2026-08-05: webdav.critchley.biz stopped responding from the Mac — external TCP connect to :443 timed out (http_code=000), and the Apache scoreboard was full (all worker slots in use). Because this same Apache also fronts the notes/MCP servers and the private store, a full scoreboard stalls everything behind it, not just WebDAV. Actions: (1) raise MaxRequestWorkers/ServerLimit for the active MPM (check event vs prefork) in /etc/apache2/mods-available/mpm_*.conf; (2) investigate WHY workers are exhausted — likely long-lived/leaked connections (MCP SSE/streamable-HTTP keepalives, stunnel-proxied sockets, or slow WebDAV clients holding workers) rather than genuine load; (3) manage the setting via Ansible so it survives reprovision.