PopIt3 Architecture

PopIt3 uses a modular pipeline architecture:

Data Flow

Outlook.com (POP3) → popit3.py (sync+store) → GDBM DB → process_emails.py (routing) → Specialized handlers → GDBM DB → Report generators → WebDAV

Components

1. Email Sync: popit3.py

• Connects to Outlook.com via POP3 + TLS + OAuth2• Uses MSAL library to refresh tokens from ~/.netrc• Stores emails in GDBM keyed by UIDL (unique ID)• Format: {UIDL: {size: int, mail: str}} where mail is raw email decoded latin-1• Detects new messages with UIDL and LIST commands• Timeout: 30 seconds

2. Database: gdata.py + GDBM

• Wrapper around GNU Database Manager (gdbm)• Stores key-value pairs where values are JSON documents• Two modes: read-only 'r' or create-if-not-exist 'c'• Thread-safe at file level; managed by context manager (with statement)• Files: ~/.email3.mail.gdbm (raw emails), ~/.jobserve.gdbm (analyzed jobs)

3. Email Routing: process_emails.py

• Main processor dispatcher• Routes based on To: header• Integrates specialized handlers via imports• Creates WebDAV clients with retry logic and exponential backoff• Authentication from netrc

4. Job Analysis Handler: newparser_jobserve.py

• Classifies emails: alert, suggestion, application, or unclassified• Uses js_alert_parser to extract: title, company, location, salary, skills, URL, ref, posted date• Calls OpenAI GPT-4o-mini for scoring (0-10 scale)• Prompt includes CV context from CV_PATH environment variable• Stores in ~/.jobserve.gdbm with structure: {job_type, parsed_job, score, score_reason, ...}• Deduplicates via Message-ID

5. Gym Schedule Handler: MyDavidLloydSchedule.py

• Extracts David Lloyd booking emails• Parses booking details from HTML email body• Maintains calendar in GDBM• Exports timetable.html for web viewing

6. Report Generator: job_analysis_report.py

• Reads from ~/.jobserve.gdbm• Generates three tables: scored jobs, applications, unclassified emails• HTML with sorting, color coding by score (green=high, red=low)• Deploys to webdav.critchley.biz via HTTP PUT• Auto-cleanup: deletes jobs >14 days old, applications >28 days old

Storage Models

Raw Emails: GDBM key=UIDL, value={size, mail:str}Job Records: GDBM key=Message-ID, value=JSON with classification markersClassification: job_type field (alert|suggestion|application) or unclassified:{} marker

Authentication

• OAuth2: Stored in ~/.netrc with account='MSAL:<client_id>', password=refresh_token• WebDAV: Pulled from ~/.netrc for webdav.critchley.biz• OpenAI: Stored in ~/.openai (key file)

Deployment

• Server: Linux box (details in popit3/deployment)• Cron jobs: Update reports hourly/daily• Web: Reports on https://www.critchley.biz/JobAnalysis and /DavidLloyd/timetable.html