General principles for an LLM agent operating remote infrastructure — driving a GUI over VNC/RFB and standing up / tearing down disposable hosts. Abstracted from the delayrepay browser-automation work; project specifics live in delayrepay/lessons. Part of PROGRAMMING_RULES.
Reading and driving a screen
- To READ text off a screen, do ONE OCR→text pass and read the text — do not ingest full-resolution screenshots as vision (expensive in tokens), and do not OCR the same frame once per keyword (N passes over one image). Reserve vision screenshots for layout/visual verification (is the field filled? did the page advance?).
- OCR misses light text on coloured buttons (white “Log in”/“Confirm”/“Submit” on teal). Locate those controls by an adjacent OCR-findable anchor, or by cropping to pixel coordinates — not by text search.
- Anti-bot challenges (reCAPTCHA) are a hard stop for full automation: scripted clicks get escalated to an interactive challenge. Drive up to the challenge, then hand off to a human to solve it and do the final submit. Do not try to defeat captchas — build the human-in-the-loop pause in deliberately.
Ephemeral / disposable hosts
- Honour the single-use design: do not reuse a host meant to be disposable. State drift accumulates (stale browser profiles/locks, stale allowlist entries, half-dead sessions) and debugging the drift costs more than a clean re-provision. If a reused host misbehaves, provision fresh before spending long untangling it.
- Do not key teardown to a single mutable “current” state file that the provisioner overwrites — target teardown by entity identity (name/id) so “provision new” and “tear down old” can run concurrently, and you never destroy the wrong instance.
- Allowlists keyed to an ephemeral endpoint’s IP go stale AND accumulate (each new incarnation adds a rule; the old ones linger). Reconcile to the live value and prune the dead ones — don’t just append.
Diagnostics & authentication planes
- Connection symptom → cause: a TCP timeout means firewall / security-group / wrong source-IP; connection refused means the service is down or not listening; a 401 that re-prompts means the credential is being rejected (a transport/credential distinction that saves you looking in the wrong place).
- Distinguish authentication planes: a working web/portal session (cookie) does NOT imply that API/feed credentials (HTTP Basic / token) work. A data feed can 401 while the same account logs into the website fine — usually the account isn’t activated/provisioned for the feed, or the feed wants a different credential. Verify the actual auth mechanism from the docs (Basic vs bearer token vs OAuth) before assuming.