Thread Chain Implementation

Design note: envoy/thread-chain-design

Branch: thread-chain — not yet merged to m

What was added

New code in orchestrator.py

• symbol: _MID_PATTERN description: re.compile for RFC 2822 <...@...> message IDs
• symbol: _parse_email_date() description: parse Date header to sortable datetime
• symbol: EmailCache description: dict keyed by normalised Message-ID (no angle brackets); put/get/remove/update_folder
• symbol: _fetch_headers_only() description: IMAP BODY.PEEK[HEADER.FIELDS] fetch, returns dict with body=None
• symbol: fetch_thread_chain() description: builds pool: trigger + ancestors sorted newest-first; returns (pool, not_available)

fetch_thread_chain logic

1. Seed cache with trigger (full body already fetched)
2. Parse References header for ancestor Message-IDs (cap at 15). Fall back to In-Reply-To if absent.
3. Scan bundle note JSON for Message-IDs (for continuation runs where bundle_key is known)
4. For each candidate: headers-only IMAP search across INBOX → Done → Sent → other folders
5. Upgrade In-Reply-To (direct parent) to full body — fetch and update cache in-place
6. Sort ancestors by date newest-first
7. Return [trigger] + sorted_ancestors, and a set of not_available Message-IDs

call_llm changes

• Added thread_not_available: Optional[set] parameter
• All '=== SECTION ===' markers replaced with '〶 Section name' on its own line
• Emails presented as '〶 Email #1', '〶 Thread parent #2', '〶 Thread ancestor #3', etc.
• Headers-only entries show a note: [Headers only — use add_emails with Quick-ID #N to fetch full body]
• Not-found ancestors shown in '〶 Thread ancestors not found' block with #NotAvailable label

main() changes

• EmailCache and thread pool built BEFORE the iteration loop
• For fresh emails: no bundle note scan (bundle_key not known yet)
• For continuation emails: bundle note loaded for Message-ID scan
• call_llm receives thread_pool instead of [email_msg]
• execute_actions receives thread_pool + gathered_emails

Gather phase — Quick-ID upgrade-in-place

When LLM requests add_emails with a Quick-ID like #3:

1. Build _qid_map from thread_pool + gathered_emails
2. If the Quick-ID refers to a headers-only pool entry: fetch full body from IMAP, em.update() in-place, update cache
3. If body already present: report 'already have full body'
4. Remaining refs (real Message-IDs or unknown Quick-IDs) go to fetch_emails_by_id as before

Key design decisions

• Trigger is always #1 regardless of date — predictable for LLM
• Thread pool is built once before the while loop — not rebuilt per iteration
• Cache stores live dicts — upgrade-in-place mutates the pool entry (no re-indexing needed)
• bundle note scan only on continuation runs — first triage has no bundle_key yet
• Not-available ancestors shown to LLM but not given Quick-IDs
• imap_client.py unchanged — headers fetch done via client.connection.fetch() directly (same pattern as search_emails_imap)

Files changed

• file: orchestrator.py lines_added: ~230 lines_removed: ~60
• file: imap_client.py change: none

Semantic validation — diag_thread_chain.py

Script: diag_thread_chain.py — inject a synthetic 3-email thread, run fetch_thread_chain, render exactly what the LLM receives, optionally send to ask for validation.

Usage: python3 diag_thread_chain.py --inject --ask

First run results

• item: ['All 3 emails found, 0 not available.']
• item: ['Email #2 (direct parent, In-Reply-To) correctly upgraded to full body.']
• item: ['Email #3 (oldest ancestor) correctly left as headers-only.']
• item: ['Thread ordering correct: #1 trigger (newest) → #2 direct parent → #3 oldest.']
• item: ['LLM semantic check: ', {'em': 'context is clear and actionable; thread structure immediately understandable.'}]

Bug found and fixed during diagnostic

• item: [{'code': 'parse_email_headers'}, ': MIME-encoded subject/from/to not decoded (e.g. em-dash rendered as ', {'code': '=?utf-8?b?4oCU?='}, '). Fixed by adding ', {'code': '_decode_header_value()'}, ' using ', {'code': 'email.header.decode_header'}, ' + ', {'code': 'make_header'}, '.']

LLM design feedback (not bugs)

• item: ['Numbering counterintuitive (#1 is newest). Expected — by design, trigger is always #1.']
• item: ['Suggest labelling Email #1 as "Current inbound email". Reasonable; could add in future.']

Scenario test suite — diag_thread_chain.py --scenario

8 named scenarios covering edge cases. Run with: python3 diag_thread_chain.py --scenario all

Results (all 8 PASS)

• item: [{'strong': 'baseline'}, ': 3-email A→B→C, all present. pool=3, not_available=0. PASS.']
• item: [{'strong': 'missing-ancestor'}, ': C references A+B, only B injected; A absent. pool=2, not_available={A}. PASS.']
• item: [{'strong': 'first-message'}, ': No In-Reply-To or References. pool=1, not_available=0. PASS.']
• item: [{'strong': 'references-only'}, ': References→A, no In-Reply-To. pool=2, no direct-parent label. PASS.']
• item: [{'strong': 'broken-irt'}, ': In-Reply-To→X (unknown), References→A. Initially FAILED — X never searched. Fixed (see below). PASS after fix.']
• item: [{'strong': 'upgrade-sim'}, ': Simulate add_emails upgrade of #3 headers-only→full body in-place. Quick-IDs stable, pool size unchanged. PASS.']
• item: [{'strong': 'folder-restore'}, ': INBOX selected before fetch_thread_chain; INBOX restored after. PASS.']
• item: [{'strong': 'duplicate-mids'}, ': References contains same MID twice. seen-set deduplication prevents double fetch; pool=2. PASS.']

Bug found and fixed: broken-irt

fetch_thread_chain only used In-Reply-To as a fallback when References was empty. If In-Reply-To pointed to a MID absent from References (malformed email), that MID was never searched and never appeared in not_available.

Fix: always add In-Reply-To to candidates if not already in seen. For normal emails, IRT is already in References so seen-set prevents duplicate. For broken emails, IRT is searched and, if absent, lands in not_available.