Key file locations and operational details for the popit3 pipeline.
~/.mail/.booksings_db2.gdbm — DL booking records (booking_ref → JSON). NOT ~/.booksings_db2.gdbm — a stale duplicate with that name exists directly under ~/, untouched since 2025-09-16; ignore it, it's dead.
~/.mail/.booking_map2.gdbm — dedup tracker (booking_ref → [[msg_id, uidl], ...]). Same caveat as above — the correct copy lives under ~/.mail/, not ~/.
~/.mail/.dl_gcal_ids.gdbm — GCal event IDs (booking_ref → event_id). Filename and path both corrected 2026-07-18 — was previously documented as ~/.gcal_ids2.gdbm, which never existed.
~/.email3.mail.gdbm — raw email storage (uidl → bytes)
~/.email3.meta.gdbm — email metadata
WebDAV path: john/DavidLloydSchedule.html — uploaded on every run, includes error banner when pipeline has failures
Notes document: DavidLloydSchedule — JSONHTL table of upcoming bookings (no error banner here; banner is WebDAV HTML only)
Symptom: cron runs kept completing with RC 0 every 5 minutes, but a single popit3.py process from an earlier run was still alive hours later, pinned at ~97–100% CPU and consuming 20GB+ RAM. No new mail was being fetched for any account during this time — every subsequent cron invocation hit the locked mail gdbm, printed a one-line ERROR (see below), and silently skipped fetching, so nothing else in the logs looked obviously broken.
Silent symptom in out.log: ERROR: [Errno 11] Resource temporarily unavailable: '/home/john/.email3.mail.gdbm'. This is normally benign (see "Locked database" above) — the difference here was that it appeared on every single run for hours, not just once.
Diagnosis:1. ps aux | grep popit3 — found the stuck PID, noticed 100% CPU / huge %MEM and an ELAPSED time far longer than a normal run (~2–5s).2. cat /proc/<pid>/status — confirmed VmRSS in the tens of GB, state R (actively running, not blocked).3. sudo py-spy dump --pid <pid> --locals (py-spy at ~/.local/bin/py-spy) — showed the exact Python stack + local variables without killing the process. This pinpointed it: stuck inside popit3.Pop3TLS._readline → _read_multiline → fetch_message_bytes fetching one specific msgnum via RETR.
Ruling out "huge attachment": wrote a small standalone script reusing popit3.'s connection/auth helpers to run LIST <msgnum> and a full bare LIST (all messages) — both single-shot, non-multiline-body commands the server answers instantly, so they cannot hang the way RETR did. The stuck message was only ~88KB; the largest message in the whole mailbox (784 messages) was ~1MB. Size was a red herring — the script was deleted afterward since it was only needed for that one diagnosis.
Root cause: _readline() read one byte at a time via self.file.read(1) and treated an empty read (b'') as normal end-of-buffer, silently returning whatever had been accumulated so far. But a closed/reset socket returns b'' immediately and forever, without blocking — it does not raise or block. So _read_multiline() never saw the .\r\n terminator, kept looping, and turned into a non-blocking, CPU-spinning infinite loop appending empty lines forever. That explains both symptoms at once: 97-100% CPU (spinning, not blocked on I/O) and unbounded RAM growth (an ever-growing Python list).
Fix (2026-07-08, committed d155d89 in popit3):• _readline() now raises ConnectionError on EOF instead of returning silently.• Each RETR is wrapped in a 30s SIGALRM timeout (fetch_message_bytes_with_timeout, FETCH_TIMEOUT = 30 in popit3.py). On timeout or ConnectionError, that one message is skipped (left unfetched, so it is retried on the next run), the POP3 connection is closed and reopened (the stream is desynced after an aborted RETR — a partial multiline response would corrupt any further reads on the same connection), and the loop continues with the next message instead of hanging or aborting the whole run.
Emergency response procedure for a stuck popit3.py (generalizable if this recurs):1. Diagnose first with py-spy (above) rather than assuming — confirms it is actually stuck vs. just slow.2. Disable cron while investigating: crontab -l then comment out the three popit3.py lines (05 23 reprocess, 25 6-22 reprocess, the 5-minute regular run) and reapply with crontab <file>. NOTE: crontab could not read a file under /tmp in this environment — write the edited crontab to a path under $HOME instead.3. Kill the process: SIGTERM first (kill <pid>), then SIGKILL (kill -9 <pid>) if it does not die — a process blocked deep in a C-level socket read often ignores SIGTERM.4. Verify the gdbm lock is released: try opening ~/.email3.mail.gdbm read-only in a one-off Python snippet; it should succeed without EAGAIN.5. Re-enable cron once the underlying bug is fixed.