Class and function breakdown for message_store.py. See popit3/message-store for the design rationale.
DEFAULT_DB_PATH = '~/.email3_msgid.gdbm' — default path for the new Message-ID keyed store.
DEFAULT_INDEXED_HEADERS = ['Subject', 'From', 'To', 'Cc', 'Date', 'In-Reply-To', 'References', 'List-Id'] — headers indexed on every add() call unless overridden.
Wraps a single GDBM file. Uses gdata.gdata for JSON serialisation, consistent with the rest of popit3. Not a context manager itself; opens and closes the underlying gdata on each public method call to keep lock windows short. For batch operations (e.g. download loop), callers may open a single gdata.gdata context and pass it in via an optional parameter — see __init__ note below.
gdbm_path — path to the GDBM file (expanded with os.path.expanduser). Defaults to DEFAULT_DB_PATH.
indexed_headers — list of header names to index. Defaults to DEFAULT_INDEXED_HEADERS. Stored as an instance attribute; reindex() uses this list.
Parse raw_bytes as an email message (applying _norm() for BOM stripping). Extract Message-ID; normalise to ensure surrounding angle brackets. Write primary record. Update all index entries. Register UIDL.
Returns the Message-ID string.
Idempotent: if the Message-ID is already present the UIDL is updated if changed and the method returns immediately without reindexing. Logs a warning if the same UIDL maps to a different Message-ID (should not happen in practice).
Raises ValueError if no Message-ID header can be found or synthesised.
Return True if this UIDL is in the _uidls list. Used by the POP3 download loop as a fast pre-check before attempting to retrieve a message. O(n) on the UIDL list; acceptable for typical mailbox sizes.
Return the primary record dict for the given Message-ID. Raises KeyError if not present.
Convenience wrapper: calls get(), decodes the raw field from latin-1 back to bytes, applies _norm(), and returns an email.message.Message object ready for handler use.
Read primary record. Remove this Message-ID from every index list it appears in (using the stored header values to compute the same keys used at write time). Delete UIDL: N key. Remove UIDL from _uidls list. Delete primary record.
No-op if Message-ID is not present. Does not raise.
Look up the secondary index for header and value. Applies the same normalisation used at write time before constructing the key.
Returns a list of Message-ID strings (may be empty). Never raises.
Examples:
store.find('Subject', 'Re: live wire') — strips Re: prefix, looks up Subject: live wire, returns Message-IDs of all messages in that thread.
store.find('From', 'Joe Bloggs <joe@example.com>') — strips display name, looks up From: joe@example.com.
store.find('Date', '2026-06-26') — looks up Date: 2026-06-26 directly.
store.find('In-Reply-To', '<parent@host>') — returns Message-IDs of all direct replies to <parent@host>.
Return all reachable Message-IDs in the same conversation as msg_id, sorted by date_iso ascending. The starting message is always included.
Step 1 — find root: read primary record of msg_id. If it has a References header, split on whitespace and take the first Message-ID. If that Message-ID is in the store, use it as root.
Step 2 — fallback: if References is absent or root not in store, follow In-Reply-To links backward (get(rec['headers']['In-Reply-To']) repeatedly) until a message has no In-Reply-To or its parent is not in the store. That is the earliest reachable ancestor; use as root.
Step 3 — BFS forward from root: call find('In-Reply-To', current_id) to get direct replies; add each to the visit queue. Collect all visited Message-IDs. Guard against cycles with a visited set.
Returns sorted list. Gaps (deleted or never-downloaded messages) are silent — the walk does not error, it just cannot traverse beyond the gap.
Yield all primary record keys — i.e. all GDBM keys that start with <. Used to drive the dispatch loop in process_emails.py and by reindex().
Rebuild all secondary indices from scratch.
1. Delete every key that is not a primary record (not starting with <) and not _uidls or UIDL: *.
2. Iterate message_ids(). For each primary record, call _index_add() for every header in self.indexed_headers.
Safe to run at any time. Locks the file for the duration — avoid running while the download loop is active.
Return the uidl field from the primary record, or None if not present.
Look up UIDL: {uidl} key. Returns the Message-ID string or None.
Seed this store from an existing UIDL-keyed ~/.email3.mail.gdbm.
Open source_path via gdata.gdata in read mode. For each key (UIDL): get value dict, extract mail field (latin-1 string), encode back to bytes with latin-1, call self.add(uidl=key, raw_bytes=...). Already-present Message-IDs are skipped (idempotent).
Prints a summary: added, skipped, errors.
Strip UTF-8 BOM (\xef\xbb\xbf) from the start of raw_bytes if present. Copied from the existing pattern in process_emails.py.
Parse an address header value into a list of normalised addr-spec strings.
1. Strip comments: remove everything inside () including nested parens (regex or simple state machine).
2. Call email.utils.getaddresses([header_value]) to handle comma-separated addresses, display names, and quoted strings.
3. For each (display_name, addr_spec) pair: if addr_spec is non-empty, lowercase the entire string and append to result.
Returns empty list on failure rather than raising.
Lowercase the value. Repeatedly strip leading whitespace and any of the prefixes: re:, fwd:, fw:, aw: (case-insensitive, hence lowercase first) until none remain. Collapse runs of internal whitespace to single spaces. Strip leading/trailing whitespace.
Called both on write (to compute the index key) and on query (in find()). Identical logic ensures the two always agree.
Parse RFC 2822 date string via email.utils.parsedate_to_datetime(). Convert to UTC. Format as YYYY-MM-DD. Returns None on parse failure (the caller then skips Date indexing for that message).
Split the References header value on whitespace. Return a list of the distinct Message-ID strings found (each should start and end with angle brackets; skip any token that does not).
Return f'{header}: {normalized_value}'. Single place that defines the key format; both write and read paths call this.
Normalise raw_value for the given header (dispatch to the appropriate normaliser). For headers that produce multiple values (To, Cc, References), iterate and add msg_id to each index list. Load existing list from db, append if not already present, write back.
Mirror of _index_add. Normalise, compute key(s), load list, remove msg_id, write back or delete key if list is now empty.
Used internally by walk_thread(). Returns the Message-ID of the earliest reachable ancestor of msg_id in the store. See walk_thread algorithm steps 1 and 2.
popit3/message-store — overall design. popit3/database — gdata wrapper.