The code that drives Firefox over VNC, in /home/john/delayrepay/. Built on ~/git/vnc_control/vnclib.py (low-level RFB client). Parent: delayrepay; lessons: delayrepay/lessons.
Cheapest first, per the lessons:
Replaces ad-hoc python3 -c one-liners: drive.py shot NAME | find TEXT | click-text TEXT [--index N] | type TEXT | key CHORDS | scroll up|down N | click X Y. Default target 127.0.0.1:5902.
claim/runner.py executes a YAML step list (claim/claim-2026-08-26.yaml); the flow is DATA, so a different journey/ticket is a different data: block. Each click_text step is a tiered click_element keyed by a stable per-step id. Actions:
What works end-to-end: login (patch) → date (ocr_only) → From/To (focus-into-body + Down/Enter). What is NOT scripted yet: pick-train, delay band and ticket type are patch/OCR clicks that mostly work; the TAIL — ticket price, reference, photo UPLOAD (GTK dialog; tickets are on the host ~/reclaim/), payment (choose the saved HSBC card), review, SUBMIT — is not yet automated.
label/harness.py: freeze (snapshot current OCR → dataset/golden.json), check (regression vs golden, 16/16 within 14px), build (dataset/labels.jsonl [+ overlays]), patches (crop reference patches from the golden anchors). label/labels.yaml is the curated vocabulary. This both trains the patch fast-path and guards locate.py against regressions.
Background watcher: region-OCRs a central band every ~25s and clicks 'Stay signed in' when the timeout modal appears, so an unattended session survives. Run alongside any live session (run_claim.sh starts it).
/home/john/delayrepay/: webdrive/, drive.py, upload_shots.py, keepalive.py, run_claim.sh, claim.conf, claim/ (runner.py + *.yaml), label/ (harness.py + labels.yaml), dataset/ (golden.json, labels.jsonl, patches/, patch_misses.jsonl), client/gwr-view, README.md.