A single module that owns all email persistence for popit3. Downstream handlers (jobserve, DL, etc.) receive (msg_id, email.message) pairs and do domain-specific work. This module stores mail and keeps it findable. It does not know about job scoring or gym schedules.
Message-ID is the primary key. Deduplication and idempotent reprocessing are automatic. Secondary indices live in the same GDBM file — no separate database. Deletion cleans up all index entries. Indices are supplementary caches; a reindex() method can rebuild from primary records after any inconsistency.
Three key prefixes coexist in one file, distinguished by their first character. No collision is possible by construction.
<message-id@host> — primary record (starts and ends with angle bracket). Value: JSON dict.
Header: normalized_value — secondary index (e.g. Subject: live wire). Value: JSON list of Message-IDs.
UIDL: 12345 — reverse UIDL lookup. Value: JSON string (single Message-ID).
_uidls — JSON list of all UIDLs ever downloaded. Used by the POP3 download loop to skip already-seen messages without scanning all primary records.
Stored at key <message-id>. Fields:
uidl — POP3 UIDL string, for server-side deletion.
size — raw byte count.
raw — full raw email decoded as latin-1 (same encoding as ~/.email3.mail.gdbm). The existing _norm() BOM-stripping in process_emails.py applies unchanged.
headers — dict of all RFC 2047-decoded headers (Subject, From, To, Date, Message-ID, In-Reply-To, References, List-Id, etc.).
date_iso — parsed datetime as ISO 8601 string, UTC. Derived from the Date header.
from_addr — single parsed addr-spec from From header, fully lowercased.
to_addrs — list of parsed addr-specs from To and Cc headers, fully lowercased.
indexed_at — ISO 8601 timestamp of when the record was written.
Each index key maps to a JSON list of Message-IDs. When a message is added the relevant index lists are updated; when deleted they are cleaned up. If a list becomes empty the key is deleted.
Key format: Subject: live wire
Normalization: lowercase; strip leading reply/forward prefixes (Re:, Fwd:, FW:, AW:, RE:) repeatedly until none remain; collapse internal whitespace.
The same normalization is applied at query time. A caller searching for Re: Live Wire or live wire or Fwd: Re: Live Wire all produce the same normalized key and retrieve the same results. There is no separate raw-subject index; the query side handles any prefix stripping transparently.
Key format: From: joe@example.com
Address processing: strip display name (everything outside angle brackets), strip comments (everything inside parentheses), then apply RFC 5322 getaddresses() parsing. Fully lowercase the result. One From header → one index entry. Store original in primary record headers dict; store parsed addr-spec in from_addr.
Key format: To: jane@example.com
Same address processing as From. A single To header with multiple recipients produces one index entry per address. Cc addresses are indexed under the same To: prefix. Bcc is not present in delivered mail and is not indexed.
Key format: Date: 2026-06-26
Normalization: parse the RFC 2822 date string, convert to UTC, format as YYYY-MM-DD. Multiple messages on the same calendar day share a single index key. Exact time comparison is done via date_iso in the primary record.
Key format: In-Reply-To: <parent-msg-id@host>
The value in this header is already a Message-ID so no normalization beyond stripping surrounding whitespace is needed. The index maps a parent Message-ID to the list of its direct replies — forward links. Combined with the In-Reply-To field stored in each primary record (backward link), both directions of a thread can be traversed. This index is the basis for walk_thread().
Key format: References: <ancestor-msg-id@host> (one entry per Message-ID in the References header value)
A single References header contains a space-separated list of Message-IDs, one per ancestor in the thread chain from oldest to newest. Each is indexed separately. This allows finding all descendants of a given ancestor, not just direct replies — useful for deep thread reconstruction when some intermediate messages were never downloaded or have been deleted.
Key format: List-Id: debian-announce.lists.debian.org
Normalization: lowercase; strip surrounding angle brackets (the <id.host> form is common). Enables bulk retrieval or deletion of all mail from a given list, and is useful for routing decisions (newsletter vs. direct mail) without inspecting the body.
Given any Message-ID in a conversation, walk_thread() returns all reachable Message-IDs in the thread, sorted chronologically by date_iso.
Algorithm:
1. Attempt to find the thread root via the References header of the starting message — the first entry in References is the oldest ancestor. If that Message-ID is in the store, use it as root.
2. If References is absent or its root is not in the store, walk In-Reply-To links backward from the starting message until a message with no In-Reply-To in its primary record is reached, or the referenced Message-ID is not in the store. That is the earliest reachable ancestor.
3. BFS forward from the root: call find('In-Reply-To', root_id) to get direct replies, then recurse for each reply. Collect all visited Message-IDs.
4. The starting message is always included in the result even if ancestors were not found.
The result is best-effort: messages deleted from the store, or never downloaded, create gaps. The walk does not error on gaps — it simply stops at the boundary of what is available.
Delete path (delete(msg_id)):
1. Read the primary record (to know which index keys to update).
2. For each indexed header in the record: compute normalized key, load current list, remove this Message-ID, write back (or delete key if list is now empty).
3. Delete the UIDL: N reverse-lookup key.
4. Remove the UIDL from the _uidls list.
5. Delete the primary record.
This is not atomic. If the process dies between steps 2 and 5 the primary record remains but index entries have been cleaned. That is the safer failure direction — a query misses one result rather than returning a dangling reference. Call reindex() to restore full consistency.
reindex() drops all index keys (any key not starting with < or _uidls or UIDL:), then scans every primary record and rebuilds all index entries from scratch. Use after a crash, after adding a new indexed header, or during development.
The existing UIDL-keyed store at ~/.email3.mail.gdbm can seed the new store via migrate_from_uidl_store(source_path):
1. Open source in read mode.
2. For each UIDL key: decode value to get raw email bytes (stored as latin-1 string, reconstruct bytes with the same encoding). Parse with email.message_from_bytes() applying _norm().
3. Extract Message-ID header. If absent or malformed, generate a synthetic one from UIDL (e.g. <synthetic-uidl-12345@local>).
4. Call store.add(uidl=key, raw_bytes=...). Already-present Message-IDs are skipped.
5. The _uidls list is populated as a side effect of each add() call.
After migration, the old store can be kept as a backup or retired.
The download loop replaces direct POP3-to-gdbm with a call to the store. Rough flow:
1. Open store. POP3 LIST → UIDL list.
2. For each server UIDL not in store.contains_uidl(): download raw bytes, call store.add(uidl, raw).
3. Dispatch: iterate store.message_ids(), call store.get(msg_id), reconstruct email.message from the raw field, route by To address as now.
4. Handlers call store.delete(msg_id) when done, instead of returning a UIDL set.
Handler signatures change from process_x_mails([(uidl, msg)]) to process_x_mails([(msg_id, msg)]). The uid=int(uid) assumption in newparser_jobserve.py is removed; everything else is unchanged.
popit3/message-store/api — class and function reference. popit3/database — existing gdata wrapper and DB files.