How Claude drove Firefox on the VNC desktop to fill a GWR Delay Repay claim (26 Aug 2026 journey). Companion to vnc_control (the RFB screenshot toolkit) and the claim data note train-delay-refund-2026-08-26. The big win over vnc_control's RFB approach: drive input with xdotool directly on display :1 (reliable, synchronous) and capture with import -window root.
Built this session. Solves the core problem: keystrokes must land in Firefox, never in the operator's own terminal, regardless of which workspace the human is viewing. EVERY action first activates Firefox BY NAME, then acts, then (by default) returns the view to the Claude Konsole so the human sees the terminal between steps.
| Command | What it does |
|---|---|
| ffdrive activate | windowactivate --sync the Firefox window by name; leaves it focused |
| ffdrive goto <url> | activate, Ctrl+L, clear, type url, Return |
| ffdrive type <text> | activate, xdotool type (30ms/char so Firefox never drops leading chars) |
| ffdrive key <chord...> | activate, xdotool key (e.g. ctrl+Home, Page_Down, Escape) |
| ffdrive click <x> <y> | activate, mousemove --sync + click at 1920x1080 coords |
| ffdrive shot [name] | activate, import -window root to ~/tmp/<name>.png, return to terminal |
| ffdrive max | recover from stray window tiling: windowmove 0 0 + windowsize to display geometry |
Global flag --stay: do NOT return to the terminal after the action (use for chained steps mid-form; end the chain with a plain shot to hand the view back). activate always stays.
Focus by name is the whole point: even if the human has switched to the Claude workspace to approve a prompt, the next ffdrive call re-grabs Firefox before typing, so input can't leak into the Claude session. It also uses windowactivate --sync which switches XFCE workspace to wherever Firefox is.
Allowlisted in delayrepay/.claude/settings.local.json (permissions.allow: Bash(ffdrive ...)) so the driver commands never prompt — that removal of prompts is what lets the human stay on the Firefox workspace instead of switching back to approve each Bash call.
identify); xdotool click coords are in that same space 1:1. The Read tool downscales SOME screenshots for display — do not infer scale from how big it looks.convert shot.png -crop WxH+X+Y +repage crop.png. A crop at offset (X,Y) maps 1:1 back to real coords: real = (X+px, Y+py). Cropping also forces the Read tool to show it larger.1. Quick Find trap. Typing into a field that is NOT actually focused makes Firefox eat the keystrokes as find-as-you-type: '/' opens Quick Find, a leading letter/digit opens find. Symptom: a typed URL 'https://delayrepay.gwr.com/en/login' collapsed to a Google search for 'en/login' (only the tail after the last '/' survived), and '26/08/2026' put '8/2026' in the Find bar. Fix: ALWAYS confirm the caret is in the target field (screenshot shows a cursor / green outline) before typing. Escape closes the Find bar.
2. MUI date picker — the overlay-click trap (cost the most time). The 'Travel date' field is NOT text-editable (typing triggers Quick Find). You must click calendar-day cells. BUT the open calendar popup OVERLAYS the 'From' field below it — so 'clicking From' while the calendar was still open actually clicked the calendar cell at those coordinates (the 'Thu, 3rd-week' = 13 cell), silently setting the date to 13/8/2026 over and over. Root cause of a long 'the date keeps reverting to 13' detour. Fixes: (a) make sure the calendar is CLOSED before clicking any field beneath it; (b) better, drive the date by keyboard — the field's own help (the keyboard icon) lists: Alt+Down open, arrows move (Left/Right = ±1 day, Up/Down = ±1 week, PgUp/PgDn = ±1 month, Home/End = first/last of month), Enter select current, Escape close.
3. Station autocomplete (From/To). Click field, Ctrl+A, Delete to clear, type the station, wait ~1s for the dropdown, then click the matching option (e.g. 'LONDON PADDINGTON [PAD]'). Do not just leave typed text — it must be picked from the list.
4. Time field ('Leaving at'). Offers a 15-minute dropdown but also accepts free-typed times like 06:59 (example text says '09:30 or 14:55'). Type it directly.
5. Window randomly re-tiled. Mid-session the Firefox window shrank to a quarter (top-left), breaking every coordinate. Detect with xdotool getwindowgeometry, fix with ffdrive max. (A screenshot that suddenly renders 'small with grey margins' = real window resize, not zoom — zoom would not shrink the tab bar/chrome.)
6. Reference-field validation. 'Ticket number, collection or booking reference' rejected the full serial (11692-11693-7417-1836-22-06-X10) with 'Must be a 5-digit number, collection or booking reference'. The 5-digit coupon number (11692) was accepted.
John drops phone photos into WebDAV https://webdav.critchley.biz/reclaim/ (creds in ~/.netrc, machine webdav.critchley.biz, login john; access via python webdav4 as in john/system-config). This morning's two files: IMG_1488.HEIC = the ticket, IMG_1489.HEIC = the collection receipt.
magick IMG_1488.HEIC IMG_1488.png (ImageMagick has the HEIC/libheic delegate here). pillow_heif is NOT installed.magick IMG_1488.png -crop 2720x1980+150+1480 +repage crop.png; magick crop.png -quality 90 -strip ticket.jpg). Result: ~/tmp/ticket-26aug-bpw-pad.jpg.Clicking 'Upload a ticket' opens a separate GTK window titled 'File Upload' (becomes the active window). Do NOT use ffdrive here — its activate-by-name would raise the MAIN Firefox window and defocus the dialog. Instead send keys straight to the active dialog with plain xdotool:
DISPLAY=:1 xdotool key --clearmodifiers ctrl+l # GTK location bar
DISPLAY=:1 xdotool type --clearmodifiers /home/john/tmp/ticket-26aug-bpw-pad.jpg
DISPLAY=:1 xdotool key Return # selects the fileInside the GTK dialog '/' is safe (it is not Firefox web content). Verify the active window with xdotool getactivewindow getwindowname (should read 'File Upload').
Nothing submitted. Filled: journey (Bristol Parkway→London Paddington, 26/8/2026, 06:59–08:12 train), delay band 15-29 min, ticket Paper/Single, Price £109.50, reference 11692. The 'File Upload' dialog was open, about to receive the path to ~/tmp/ticket-26aug-bpw-pad.jpg. Still to do: complete the upload, Compensation step, then Review + final submit (get John's OK before submitting — it's a financial claim). Expected payout ~25% of £109.50 ≈ £27.38.
The delayrepay session times out and logs out, and the in-progress claim is NOT saved. The next attempt starts from scratch at step 1 — re-enter the whole journey, delay band, ticket type/duration, price, reference, and re-upload the ticket. The ONLY thing that carries over is the saved login credentials (the portal login is remembered), so the manual login step should be quick or automatic next time. So plan any resume as a full re-run, not a continuation. Keep the cropped ticket at ~/tmp/ticket-26aug-bpw-pad.jpg (or re-fetch the HEICs from webdav reclaim/ and re-crop) ready to re-upload. Also expect to redo the one-off environment fixes: ffdrive is already installed and allowlisted, but re-run ffdrive max if the window is mis-tiled and re-check the Firefox window id/name at the start.