✅ Personal trainer bookings now captured (2026-07-15) — turned out the parser already handled PT emails correctly (built 2026-06-21/25); they were just being deliberately skipped in process_dl_mails(). Removed the skip, gave PT sessions a distinct calendar title/colour, fixed a Venue-column display bug. See popit3/pt-email-findings.
✅ Write comprehensive README with architecture overview
✅ Include architecture diagram in README
✅ Document the LLM integration clearly
✅ Add code examples/usage instructions
✅ Add requirements.txt
✅ Fix obvious bugs (booking logic, rowspan parsing, hallucination CV fix)
✅ Add crash/error alerts to web reports — _run_errors list in MyDavidLloydSchedule.py; yellow banner in WebDAV HTML when pipeline fails; errors deduplicated
✅ Save sample emails as debugging fixtures — 5 real .eml files in tests/fixtures/; test suite in tests/test_dl_email_parsing.py (5 tests, all pass)
✅ Refactor WebDAV mail delivery — mailspool.py rewritten (takes webdav_host/webdav_path, reads netrc internally); webdav_deliver.py copied from envoy; process_emails.py commented routes updated to new interface. Routes still off — re-enable deliberately when needed.
✅ GitHub notification handler — john.github@critchley.biz routed to github_ci_handler.py
✅ GCal auth renewed — wsgi_google_oauth.py deployed to gravlax; visit /google to renew in future. Token written 2026-06-25. See popit3/google-auth.
✅ Outlook auth fixed (2026-06-25) — wsgi_outlook_oauth.py rewired to device flow, saves token to WebDAV. popit3.py falls back to WebDAV on RefreshError and updates .netrc. See popit3/msauth.
✅ google_services.py moved into repo (2026-06-25) — was in ~/py/, imported via sys.path hack. Now at ~/py/popit3/google_services.py. Fixed timezone-aware expiry bug (google-auth needs naive UTC).
✅ Conditional HTML/notes update (2026-06-25) — MyDavidLloydSchedule.py now skips WebDAV HTML upload and notes PUT when no bookings changed (dirty flag on mail_dbm writes).
✅ ZoneEdit 2FA handler enabled (2026-07-03) — uploads token to WebDAV (john/zoneedit/); Tampermonkey userscript auto-fills the ZoneEdit 2FA form. WebDAV credentials are entered once via a Tampermonkey menu command and stored in GM_setValue, never in the script source (repo is public). See popit3/zoneedit-handler.
✅ Outlook device-login blank popup fixed (2026-07-08) — root cause was Firefox Enhanced Tracking Protection blocking logincdn.msauth.net, not our code or Microsoft's. Diagnosed via HAR export (status:0 blocked requests). Popup window size reverted to original after ruling that out; kept a plain fallback link. See popit3/outlook-token-renewal.
✅ Google Calendar auth renewed + recurring-expiry risk documented (2026-07-08) — token had been silently dead for ~2 weeks (Testing-mode 7-day refresh token cap suspected). Re-authorized; added a self-service "Renew Google auth" link to the David Lloyd Bookings error banner. See popit3/google-auth.
✅ Fixed popit3.py hang-forever bug + added per-message timeout (2026-07-08) — a dropped POP3 connection mid-RETR caused an infinite non-blocking loop (100% CPU, 20GB+ RAM) that silently blocked all mail fetching for hours. Added EOF detection + a 30s per-message SIGALRM timeout with reconnect-and-continue. Full incident writeup: popit3/ops.
✅ Per-recipient message-size logging added (2026-07-08) — process_emails.py now logs total/average byte size per recipient alongside the existing count, to help prioritize which mail classes are worth automating. See popit3/process-emails.
Handler exists at vodafone_handler.py. Emails arrive at john.vodafone@critchley.biz (64 messages in current store, ~2.3% of mailbox). Handler currently classifies and prints to stdout; returns empty set so nothing is deleted.
Email breakdown from header store analysis:
TODO: Decide on a strategy for important emails. Options:
Santander emails (john.stan@critchley.biz, ~38 messages) are currently routed to the Santander IMAP folder. Like Vodafone, some will be important (statements, balance alerts, security notices) and some are marketing. Add a handler similar to vodafone_handler.py that detects and surfaces important ones (statements, balance changes, fraud alerts, direct debit notifications) while silently routing the rest. O2 (john.vm@critchley.biz) is similar — bills and usage alerts worth detecting.
Go through all routed addresses and identify which could benefit from structured extraction or alerting (like the Vodafone/Santander/O2 handlers). Candidates to review:
⬜ Expired-booking cleanup leaves orphaned records in .booksings_db2.gdbm — 2026-07-16, found while manually reprocessing James' PT bookings (see popit3/pt-email-findings). In process_dl_mails()'s cleanup step, booking_map and gcal_ids are correctly cleaned up by booking_reference, but the final .booksings_db2.gdbm deletion loop keys by msg_id (for msg_id in msg_id_set: if msg_id in db: del db[msg_id]) even though that database is keyed by booking_reference (see popit3/ops). The lookup silently misses every time, so expired bookings' calendar events/dedup entries get removed correctly but their record in .booksings_db2.gdbm never does. Low visible impact today — the schedule table filters to end >= now so stale rows don't show up — but the DB quietly accumulates orphaned rows forever, and this affects expired class bookings too, not just PT (predates PT tracking). Fix: change the loop to delete by booking_reference (the booking_refs set already computed just above it), not msg_id_set.
⬜ Validation in reports — flag missing fields visually (e.g. job with no title)
⬜ Generate HTML report for applications (like job analysis report)
⬜ Add proper Python package structure (pyproject.toml)
⬜ Add AngelList/other job board integration
⬜ On-demand/faster mailbox scan trigger — cron now polls every 5 min (reduced from every 2 min on 2026-07-03 to ease load while mailbox still has a backlog to move out), with --reprocess once an hour at :25. Need a way to force an immediate scan (or a burst of faster scans) when a change is expected — e.g. right after cancelling/booking a David Lloyd class — instead of waiting up to 5 min normally or up to an hour for reprocess to catch missed cases.
⬜ If any scripts generate intermediate/debug artifacts, auto-schedule cleanup via at (teatime tomorrow)
Jobserve emails are currently delivered to the Jobserve IMAP folder on gravlax and deleted from MS Live. When the AI processing step is ready to re-enable, the plan is:
⬜ Investigate Duolingo emails (john.duolingo@critchley.biz, ~76 messages) — check what types of emails they send and whether anything is worth automating (e.g. streak reminders, lesson completions, league updates).
⬜ Add a simple web interface (FastAPI) for email databases
⬜ Add type hints throughout (with asserts)
⬜ Ensure consistent code style (own style)
🔁 Check github/releases — upstream pwsafe releases land here automatically.
🔁 If the DL app shows a booking that doesn't appear in the DavidLloydSchedule note, check the mailbox before assuming a parser bug — David Lloyd doesn't always send a confirmation email (confirmed 2026-07-18 for two RHYTHM/Studio Cycle sessions, see rhythm-booking-investigation). This can happen, but it's still worth checking each time in case it's actually a fetch or parser gap.
Operational reference — log file, DB paths, HTML output location
Known test failures — 4 pre-existing failures in jobserve/CSV/XML tests; not regressions
⬜ When emails addressed to john.chatgpt@critchley.biz are delivered with a Notes key in the subject, convert them into Notes documents. The current batch contains the Shared Experience Architecture drafts created because direct Notes PUT was unavailable. Treat the subject as the intended note key, preserve the body as the initial note content, and retain any stated future splits or TODOs for later restructuring. This should remain a generic import path rather than an SEA-specific one.