Drive a VNC (RFB) session purely by screenshots plus synthetic keystrokes and mouse clicks — no window-system access and no VNC client library, just a small pure-Python RFB client (Pillow the only dependency). Built 2026-07-24 on the AWS instance. It began as "attach to the VNC port and capture the screen" and grew into a toolkit that reads the framebuffer, switches Konsole tabs and XFCE workspaces, types into the focused window, clicks, and delivers a message to a chat-style TUI as a verified transaction.
Source of truth: ~/git/vnc_control/. The CLIs are symlinked into ~/bin (on PATH); imports use realpath so the symlinks resolve back to the project. Depends on Pillow (requirements.txt). Target server here: Xtigervnc -SecurityTypes None on 127.0.0.1:5901, 1920x1080.
RFB KeyEvents/PointerEvents are fire-and-forget — the server never acks input, and a keypress can be dropped if the app is mid-render. So every consequential action is confirmed by looking at the screen: act against an observed state, then screenshot and confirm the resulting state before continuing. (The neighbouring gpt CLI called this "visual optimistic concurrency control".)
| File | Purpose |
|---|---|
| vnclib.py | Reusable VNCClient: handshake, capture, tap, chord, type_text, click (PointerEvent), region_diff, submit_and_confirm, release_all_modifiers |
| vnc_capture.py | Screenshot the framebuffer to a PNG |
| vnc_type.py | Type a string into the focused window (optional --enter) |
| vnc_key.py | Send key chords, e.g. ctrl+shift+Right |
| vnc_tab.py | Closed-loop switch to a named Konsole tab (never overshoots) |
| vnc_send.py | Deliver a message to a tab as a verified transaction |
• Konsole tab detection: crop the tab-bar row, find the median x of the strong-red close-button pixels (the active tab draws a red X), and match to per-tab x anchors. Switch by stepping Shift+Left/Right one tab, re-screenshotting, and repeating until the target is active — closed-loop, so a dropped or mis-stepped key cannot overshoot.
• Guarded submit transaction (vnc_send.py): switch+confirm tab → clear composer → type WITHOUT Enter → screenshot-verify the text landed → re-confirm the tab immediately before Enter → press Enter and confirm via a framebuffer diff. It retries a dropped Enter but stops the instant it sees the change, so it cannot double-submit.
• Defensive modifier release in a finally around chord/type_text and on close, so a failure mid-send never strands a held Shift/Ctrl.
• Mouse: click(x, y) via RFB PointerEvent (move, press, release).
• XFCE workspace switching: Ctrl+Alt+Left / Ctrl+Alt+Right (WM-level, works regardless of focus).
| Thing | Value |
|---|---|
| VNC server | Xtigervnc, RFB 3.8, SecurityTypes None, 1920x1080x24, 127.0.0.1:5901 (localhost only) |
| Konsole tabs | order node | claude | bash; active-tab red-close-button x anchors node=72, claude=164, bash=272 |
| Who runs where | Claude Code runs in the 'claude' tab; a competing node AI CLI (gpt-5.6) runs in the 'node' tab; 'bash' is a shell |
| Workspaces | XFCE; Firefox on 2 virtual desktops; switch with Ctrl+Alt+Left/Right; taskbar is per-workspace |
| Firefox | classic menu bar present; tab bar ~y44-102; new-tab '+' sits just right of the last tab |
• BBC iPlayer (bbc.co.uk/iplayer/guide) and channel pages (bbc.co.uk/bbctwo) are region-blocked from this host: "BBC iPlayer isn't available in your region."
• Broadcast schedules are NOT geo-blocked: use bbc.co.uk/schedules/<service-pid>. BBC One London = p00fzl6p. The channel picker at bbc.co.uk/schedules lists all channels/regions (BBC One/Two/Three/Four, CBBC, CBeebies, News) — e.g. click BBC Two → England.
• Caution: pids are not guessable — p00fzl81 is BBC Radio nan Gàidheal (Gaelic), not BBC Two. Use the picker rather than guessing.
• Cookie banner: "Reject additional cookies" dismisses it; consent then persists across the site.
• vnc_send guards: payload classification refuses shell-looking text into a 'chat' target unless --shell; a transaction character limit; optional --seq idempotency tag; optional --evidence before/after PNGs.
• Limitations: US keyboard layout, Latin-1 only (non-Latin-1 such as an em dash is refused rather than mis-sent); only RFB 'None' security (no VNC password); Raw encoding only (whole framebuffer per capture — fine for stills, not streaming); submit-confirmation assumes a composer that changes in the watched region on submit; tab anchors are calibrated per session/theme.
• Cross-model conversation: injected an explanation into the gpt CLI in the node tab and held a back-and-forth entirely through pixels + synthetic keystrokes; a dropped Enter mid-exchange proved the reliability point, and gpt's transaction-model critique was folded into vnc_send.py.
• Drove Firefox: switched desktop, located and clicked the new-tab '+', created a tab.
• Found tonight's TV: navigated to BBC One and BBC Two schedules via the schedules route and read the evening listings.