Hard-won lessons from building the GWR Delay Repay automation (driving Firefox on an ephemeral AWS host over VNC, from delayrepay). Grouped by area. Each is problem → cause → fix.
The browser host / controller are 2-vCPU (t4g.large / kelp). Full-frame tesseract LSTM OCR on a 1920×1080 frame is 10–30s per call; PSM 6 is ~3.7s; the legacy engine (--oem 0) is not installed. That is far too slow to sit in a click loop, and it (not tesseract per se) is what lost the first manual run to the session timeout.
The key architecture (John's idea): below OCR sits a template-patch match. Cache a small patch of each control at its expected coordinate; at runtime matchTemplate it in a small window there.
Filling the MUI journey form was the single hardest UI interaction and took several fixes:
2026-09-02: the first FULLY end-to-end Delay Repay claim was driven through the AWS browser host (login → journey → delay → ticket → price/reference → photo upload → review) and submitted — claim ref GWR-1210-786-241 (26 Aug, Bristol Parkway → Paddington, Advance Single, £109.50, 15–29 min band). The decisive factor was keepalive.py running: an earlier manual drive with NO keepalive hit the few-minute session timeout and was silently logged out mid-claim, losing everything (the ‘11692’ being typed landed in the re-shown login box). With keepalive the second run completed at leisure while a human reviewed.
Permission-prompt fix (image-analysis loop): the harness stored a SEPARATE exact-match allow entry for every unique magick … -crop WxH+X+Y …, so each new crop was an unseen command and re-prompted. Fix: WILDCARD rules Bash(magick )/Bash(identify )/Bash(convert ) in project .claude/settings.json, AND lead the command with the bare token (a cd … &&/timeout NN prefix breaks prefix-matching). VNC capture / webdrive driver run as leading python3 <abspath> … (covered by the existing Bash(python3 )).
TOWARD MINIMUM INVOLVEMENT (2026-09-03 roadmap). The end-to-end run this session needed the operator for: connecting a viewer, several ad-hoc coordinate fixes, the Review approval, and the reCAPTCHA. Only the last two are irreducible. To close the gap: (1) make the browser_host role boot a CLAIM-READY host — VNC up (seed ~/.config/tigervnc), Firefox 'gwr' profile created with key4.db+logins.json injected, evidence JPEGs staged in ~/reclaim/, window maximised — so no manual bring-up; see [[ansible/todo]]. (2) Fix keepalive to PROACTIVELY nudge (done in keepalive.py: periodic pointer-move; still needs a real-modal capture to also fix the reactive click) so unattended runs survive. (3) Codify the full form flow in the runner incl. the two branches learned this session — the keyboard date-picker nav and the 'Manually enter your journey' path — and drive all buttons by verified full-res coords / crop+compute, never scaled-eyeball. (4) Client connect scripts now read delayrepay/current-host.env for the live IP (no hard-coded IP) and can run tunnel-only (no auto-viewer, which avoided the macOS Screen-Sharing password prompt); the proper long-term is a provisioning DNS A-record so the viewer connects by name. Net: a future claim should be one command + one Review 'OK' + one CAPTCHA.
Checked other evenings for delays not yet claimed by joining GPS (locations) to the WTT timetable tables — see delay-review-2026-09-03 for method and findings. Headline: 23 July 2026 return leg (1B32, Paddington→Swansea) looks like a genuine ~48 min delay into Bristol Parkway, corroborated by a ~25 min stationary GPS trace at the Paddington platform — not yet claimed, needs the ticket detail. A few other evenings sit near the 15 min threshold but with lower-confidence headcode matching.
awslaunch SG drift — stale SSH allowlist (2026-09-09, porcini). Re-provisioning a still-running host failed at Wait for SSH even though the instance was healthy (ok/ok/running). Cause: the awslaunch-ssh security group's tcp/22 ingress listed six old kelp IPs but not kelp's current egress IP — kelp is ephemeral, so its public IP changes each session, and awslaunch.py appends stale addresses rather than reconciling to the live one. Diagnosis: compare curl -s https://checkip.amazonaws.com against the SG's port-22 CIDRs; instance health via describe-instance-status. Fix applied: authorize the current /32, revoke the six stale ones (port 15901 world-open under mTLS is fine). Root-cause fix tracked in ansible/todo. Lesson: when SSH to a provisioned host times out (not refused) after a kelp change, suspect the SG source-IP before the host.
Session 2026-09-09 (chanterelle) — four lessons. (1) reCAPTCHA needs a human at Submit: the claim drives automatically right up to the final Submit claim, but reCAPTCHA escalates scripted Submit clicks to an interactive image challenge — a human must solve it and click Submit. Design the runner to pause and hand off at Submit. (2) GWR-login injection lands in the wrong Firefox profile: the provision injects key4.db+logins.json but the running firefox-esr uses a DIFFERENT profile (e.g. xqwdkjf4.default-esr) than the one injected, so autofill fails (seen on BOTH porcini and chanterelle). Workaround: with Firefox stopped, copy ~/aws/browser-secrets/{key4.db,logins.json} into the profile firefox-esr actually launches, then relaunch INSIDE the XFCE session (export the session's DBUS_SESSION_BUS_ADDRESS from an xfce4-panel/xfsettingsd proc, else Firefox starts but renders no window). Real fix in ansible/todo. (3) VNC-dies-on-first-boot recurred on the fresh host (tigervncserver@:1 inactive) — sudo systemctl restart tigervncserver@:1 fixes it; still unfixed in the role. (4) Collection receipts are photographed + uploaded to WebDAV reclaim/ within ~15 min either side of the ticket photos (same-day), so fetch ticket + receipt together by timestamp.
The original gwr-view script cached only ONE host's cert at a time (generic filenames ca.pem/$H.crt/$H.key, keyed by a single .host marker) — enrolling a new host overwrote the previous one's cert, and only one stunnel section could exist at a time. Replaced (on pomelo) by ~/.gwr-viewer/gwr-enroll-host.sh, which gives each host its OWN directory (~/.gwr-viewer/hosts/<host>/) and its own named stunnel.conf section ([<host>-vnc]), so multiple hosts can have live tunnels from the same stunnel process at once. It backs up stunnel.conf before rewriting it, picks a free local port automatically, and prints the vncviewer command at the end rather than launching it.
Correction: only the .key is private. An earlier session treated the CA (server.pem) and the per-viewer cert ($H.crt) with the same caution as the private key when moving/handling them — unnecessary. Certs are public by design; only $H.key needs deleting from WebDAV after fetch and chmod 600 locally.