The code that drives Firefox over VNC, in /home/john/delayrepay/. Built on ~/git/vnc_control/vnclib.py (low-level RFB client). Parent: delayrepay; lessons: delayrepay/lessons.
Cheapest first, per the lessons:
Replaces ad-hoc python3 -c one-liners: drive.py shot NAME | find TEXT | click-text TEXT [--index N] | type TEXT | key CHORDS | scroll up|down N | click X Y. Default target 127.0.0.1:5902.
Status as run end-to-end at least once (see delayrepay/lessons for the sessions that proved each step): login (patch) → date picker, incl. the keyboard nav and the 'Manually enter your journey' branch (ocr_only) → From/To (focus-into-body + Down/Enter) → pick-train / delay band / ticket type (patch/OCR clicks) → ticket price, reference, photo upload (GTK dialog) → Confirm → Review, which GWR auto-completes from a saved payment card. The only step that cannot be scripted is the final Submit: GWR's reCAPTCHA escalates a scripted click to an interactive image challenge, so a human must solve it and click Submit — see 'Current gaps' below.
claim/runner.py executes a YAML step list (claim/claim-2026-08-26.yaml); the flow is DATA, so a different journey/ticket is a different data: block. Each click_text step is a tiered click_element keyed by a stable per-step id. Actions:
What works end-to-end: login (patch) → date (ocr_only) → From/To (focus-into-body + Down/Enter). What is NOT scripted yet: pick-train, delay band and ticket type are patch/OCR clicks that mostly work; the TAIL — ticket price, reference, photo UPLOAD (GTK dialog; tickets are on the host ~/reclaim/), payment (choose the saved HSBC card), review, SUBMIT — is not yet automated.
label/harness.py: freeze (snapshot current OCR → dataset/golden.json), check (regression vs golden, 16/16 within 14px), build (dataset/labels.jsonl [+ overlays]), patches (crop reference patches from the golden anchors). label/labels.yaml is the curated vocabulary. This both trains the patch fast-path and guards locate.py against regressions.
Background watcher: region-OCRs a central band every ~25s and clicks 'Stay signed in' when the timeout modal appears, so an unattended session survives. Run alongside any live session (run_claim.sh starts it).
/home/john/delayrepay/: webdrive/, drive.py, upload_shots.py, keepalive.py, run_claim.sh, claim.conf, claim/ (runner.py + *.yaml), label/ (harness.py + labels.yaml), dataset/ (golden.json, labels.jsonl, patches/, patch_misses.jsonl), client/gwr-view, README.md.
• reCAPTCHA at Submit is an irreducible human step — by design, not a bug to fix. Minimum involvement per claim is therefore one Review approval plus one CAPTCHA solve.
• keepalive's reactive click is unconfirmed — the proactive idle-reset nudge (periodic pointer-move) works, but clicking the OCR-found 'Stay signed in' button has not been confirmed to actually extend the session; a real-modal capture is needed to calibrate the true clickable target. Detail: delayrepay/lessons.
• Host bring-up / role gaps (VNC-on-first-boot, Firefox profile/login injection, staging evidence images, window maximising) are tracked as infrastructure work in ansible/todo, not here — that note has the current status of each fix, some already validated.