• 2026-09-18 beta test: Misc pg_query role/default-grant fixes DEPLOYED; tool-description cleanup still required. Original findings: ChatGPT exposed only sql and an old read-only description; owntracks_rw was rejected; owntracks_we and owntracks_ro lacked access to a newly admin-created public.mcptest table until explicitly granted. Fixed by renaming the existing role to owntracks_rw (preserving grants), updating socket authentication and the login enum, and configuring default table and sequence privileges for objects created as owntracks_adm in public. Existing mcptest grants were repaired. Deployed/restarted on gravlax. This refreshed ChatGPT session successfully used owntracks_ro, owntracks_rw and owntracks_adm for SELECT, DDL, grants and DML, including identity-sequence access; role separation works. Remaining tool-description requirement: the pg_query description must contain only a pointer to location-db/usage. Do not duplicate permissions, parameters, schema, tables, columns, limits, spatial instructions, examples or other operational information in the tool description; those facts belong in the note and duplicated copies become stale—especially the currently exposed claims that the tool is SELECT-only and capped at 500 rows. The input schema must still expose the actual sql and login parameters, but descriptive documentation should remain solely in the note.
• Para bullet items not on separate lines: In the notes viewer, bullet items within a single para block (separate strings in the array) are rendered run-on rather than on separate lines. Visible on e.g. README/runnable in the Execution Model and Features sections. Fix belongs in the viewer rendering of para arrays.
• 2026-08-26: misc email_send delivery path — RESOLVED. misc-server/email-tools-defects previously and incorrectly implied the live fix was WebDAV writes straight into gravlax's /mail/envoy/new/ (which would have meant test sends could land directly in Envoy's real inbox). Confirmed the actual, deployed mechanism is local Postfix SMTP on port 25, as misc-server/email-tools always documented — the WebDAV approach was an abandoned intermediate attempt. Corrected email-tools-defects to match; the local mail.critchley.biz mailbox used by the misc email tools remains a separate test mailbox, not Envoy's real gravlax inbox (see envoy/mail-delivery-loop).
• ~~misc-server tools index vs. live connector — check email_list_folders~~: FALSE ALARM 2026-09-05. Logged 2026-09-05 after a first tool_search on the misc connector only surfaced 5 of the 6 documented misc-server tools. Re-checked with a more targeted search and direct call: email_list_folders is present and working (returned Archive, Done, Sent, INBOX). No docs/code drift — the miss was a tool_search query issue, not a missing tool.
• Source control for live gdata-server: ~/py/gdata-server on gravlax is still not a git checkout (deployed by scp/rsync of individual files, not a clone). Partially resolved 2026-07-08: discovered the local pomelo clone (~/py/gdata-server) was 4 commits behind origin/m -- a multi-store --name/private-instance feature had been pushed and deployed straight to gravlax but never pulled back locally. Rebased local onto origin/m, pushed, and confirmed gravlax's live gdata_mcp_server.py is now byte-identical to git HEAD. Remaining gap: gravlax itself is still not a git clone, so this local-to-remote drift can recur -- see the deployment-verification note in gdata-server/mcp-server for the check-before-rsync procedure to catch it early, until gravlax is made an actual clone.
• Notes written in isolation: Notes have been created without updating parent pages or adding cross-links. Need a proactive convention: whenever a note is created or significantly updated, immediately check and update parent/related notes. See README/orphaning.
• Full-corpus JSONHTL lint scan, findings piped to this todo: Run the existing gdata-server/linter (lint_jsonhtl.py --all) across every note in the store, not just ad-hoc single documents. Motivating example: TFG/note-for-James carried a non-standard nested meta: {...} object duplicating top-level metadata fields (found 2026-07-27, fixed by flattening to the standard schema) — exactly the kind of "any other top-level key is flagged as unrecognised" case the linter's Document Level checks already cover, but nobody had run it against this note. Two parts: (1) do a one-off --all run now and fix what it finds; (2) extend the linter's existing "Linter output as a note" future-improvement item so that instead of (or as well as) a standalone linter/report, each run appends any newly-found irregularities as dated entries directly to this todo list, so known-error classes get tracked and actioned rather than only reported. Candidate cadence: same cron/Envoy pass as the periodic reconciliation item above, or a standalone scheduled job.
• CertHub private notes still in old markdown-in-para style: certhub/spec/api and certhub/jks-design still use markdown bold/- bullet text inside para blocks instead of proper list/table blocks, the same pattern fixed 2026-07-31 in certhub/spec (markdown table → proper table block) and location-db/gazetteer (Source values list → table). Candidates for the next full-corpus lint pass above; certhub/improvements, certhub/next-steps and certhub/portal-cert-todo not yet checked.
• JSONHTL_SCHEMA / linter drift behind JSONHTL_SPEC (svg, image, details blocks): JSONHTL_SPEC added three block types — image, details, svg — on 2026-07-17, but JSONHTL_SCHEMA (updated 2026-07-03) and gdata-server/linter's (updated 2026-07-06) "known block types" list still only cover para, heading, codeblock, list, table. Found 2026-07-27 while checking whether the linter and spec are kept in sync. Since the schema's block definitions use additionalProperties: false on a closed oneOf, any document using svg/image/details would currently fail schema validation despite being spec-legal — the reverse of the usual drift direction (spec moved ahead of schema, not schema stricter than spec by design). Fix: add block_svg/block_image/block_details definitions to JSONHTL_SCHEMA and register the three types in gdata-server/linter's Blocks section, per the existing Sync Rule between the two notes.
• Periodic reconciliation via Envoy: Cron job sends a short mail to Envoy saying something like: "Scan the notes system and look for inconsistencies — stale status, missing links, orphaned pages, outdated todos. Correct what you can directly; send a mail to ask about anything that needs a decision." No bespoke scanner needed; Envoy already has notes access and can reason about what it finds.
• Re-evaluate read-on-trigger cadence (August 2026): The TRIGGERS note currently uses a freeform cadence — agents act on a trigger whenever they recognise it, with no fixed checkpoints. Assess in August whether this is working in practice (across Claude, Codex, Envoy) or whether fixed checkpoints (e.g. read-the-map before-execute and before-commit, tied to the SCP phases) give more reliable rule-reading. Decide then whether to keep freeform or switch.
• Bootstrap connector minimalism: Keep the MCP connector bootstrap intentionally tiny. It should direct unfamiliar agents to read README first and let the Notes cascade (README → TRIGGERS → workflow → project notes) determine behaviour. This allows conventions to evolve by editing notes rather than redeploying the connector.
• Review STATE_OF_PLAY scope: STATE_OF_PLAY is titled as storage architecture but includes an Optical RX Status section. Move that section to a domain-specific note or rename/split the note if it is intentionally broader.
• MCP interface improvements: Findings from hands-on MCP tests are recorded in gdata-server/todo/mcp-interface-improvements.
• JSONHTL documentation improvements: Assessment and recommended doc/linter improvements are recorded in gdata-server/todo/jsonhtl-documentation-improvements.
• ChatGPT notes review follow-ups: Additional MCP response-shape, documentation-drift, todo-schema, and workflow-split suggestions are recorded in gdata-server/todo/chatgpt-notes-review-2026-06-29.
• Return block IDs from get by default: Structured edits via batch or patch require block IDs, yet they are only returned when include_block_ids=True is passed. Since the docs explicitly recommend ID-based over index-based ops, IDs should be returned by default (or always) to eliminate the extra round-trip before every edit.
• Section-aware / structured edits: Most index corruption comes from agents reasoning about a flat block list when they mean to operate on a section (a heading plus its bullets). An op set that addresses sections directly — move_section, append_to_section(heading) — would let edits like "add this link under Papers" be expressed as intent rather than as fragile positional block ops. Lower priority than dry_run and move_block but the same theme: raise the abstraction so the common edit is also the safe one.
→ Now specced: proposals/section-editing (nested section blocks + heading/CSS-selector editing — the section.* op family). Phase 0 (allow + render the section block, non-breaking) shipped 2026-08-04. Related: proposals/verb-taxonomy.
• Scan for broken/dangling links: No process currently checks that note-to-note links (and MCP keys referenced in links) actually resolve. Need a pass — manual or scripted — over the keyspace to find dangling links (e.g. links left pointing at deleted transient notes such as tmp/newport-rhythm-2026-07-18, deleted 2026-07-20). Related to the backlink index item above but distinct: this is about validating existing links, not indexing them.
• Append-only log subsystem: Provide a log.md equivalent (per Karpathy's LLM-WIKI.md) using KV-store structures rather than a single flat note — day-bucketed chunks (log/d/YYYY-MM-DD) with prev pointers forming a linked list, a small log/head pointer, and an optional month/root index tree. A dedicated log.append(op, title, refs, note) endpoint would encapsulate head-read → bucket-append → index-update. Full design and rationale in proposals/log-structure. Minimal first step: just day-buckets + head pointer.
• Backlink index: No way to find which notes link to a given key. A backlinks <key> query (even read-only, no auto-update) would make rename maintenance and impact analysis tractable. Currently requires manually reading all candidate parent notes.
• Link graph — visualisation AND compiled index: Two related deliverables from one link-graph over the keyspace (nodes = notes, edges = links). (a) Visual graph for navigation and spotting orphans — a panel in gdata-browser or a standalone HTML artifact (the Obsidian graph-view idea). (b) Compiled graph index — a generated GRAPH_REPORT-style note (cf. the Graphify + Obsidian + Claude Code pattern, June 2026) that an agent reads as a single lookup instead of fanning out across many gets to reconstruct relationships. It would surface hub notes (high in-degree — the ones many notes depend on), cluster related notes, and flag orphans and dangling links. The token-efficiency claim in the wild is large (the Graphify writeup cites up to 70x fewer reads by replacing grep-fan-out with one graph query); the same logic applies here — reading one compiled index beats re-deriving the link structure each session. Rebuild on change (the notes analogue of Graphify's post-commit git hook is an Envoy/cron pass or a server-side recompute when any note's links change). This is also the natural consumer of the backlink index item above — backlinks are the in-edges; the graph report is the whole edge set compiled. Ties to the graph-visualisation gap identified in the Karpathy/Obsidian comparison session and to LLM-WIKI.md (compile-don't-re-derive).
• Private notes instance: Add a second notes_web.py / gdata-server instance on a separate port (e.g. 8022) serving a private key namespace. Should share the same OAuth auth layer as the existing instance (port 8021). Key design questions: separate database file vs. key prefix separation within the same DB; how the notes browser selects which instance to connect to.
• Version-change summary field: The README pattern if you already know version N, skip this relies on the reader knowing what changed. A short changed string field per document (e.g. "changed": "added megadoc trap section") would let an LLM decide whether to re-read without doing so, making the version-skip optimisation actually work.
• Pretty visualisation / graph view: Add a visual graph of note links (inspired by Obsidian's graph view). Nodes = notes, edges = links. Useful for navigation and spotting orphans. Could be a panel in gdata-browser or a standalone HTML artifact.
• Runnable sheet input fields fillable via control API: Currently sheet.inputs.set only works on input fields declared via input_prompts in the codeblock spec. Input fields that appear at runtime via input() calls cannot be pre-filled or submitted through the control socket. Both cases should be settable via the API so that runnable sheets can be driven programmatically without manual interaction.
• Proposal: implement table.fill_na: Replace null (or empty) cells in a table with a supplied fill value, optionally restricted to a named column. Signature: table.fill_na(block, value, column=None). Was listed in the supported-ops error string but never implemented and has never been called (zero hits in server log as of 2026-06-26). Low priority; add when there is a real use case.
• Regex/pattern filtering for keys: Add optional server-side filters to the notes keys operation so callers can discover a relevant subset without retrieving and filtering the entire keyspace. Prefer explicit prefix and pattern (regular expression) parameters; define whether regex matching is full-key or search, validate invalid patterns with a clear error, and retain current unfiltered behaviour by default. Useful for project-scoped discovery such as ^certhub/. A filtered discovery response should include a compact navigation hint: repeated use to find the same family of notes is evidence that its parent/index/hub should gain better hyperlinks. An unfiltered keys call is a stronger navigation red flag: return a non-blocking hint to start from CONTENTS/project links, or improve those links, rather than treating a full keyspace scan as normal navigation. Do not claim that specific links are missing; the server cannot infer that.
• Add dry_run support (raised priority): A dry_run: true option for patch and batch returning the resulting document without persisting it. Promoted from nice-to-have after a concrete incident on 2026-06-28: a multi-op CONTENTS batch mixing deletes and positional inserts corrupted the index (links in wrong sections, an unrelated entry deleted, a duplicated item) and needed a full-document put to repair. dry_run would have surfaced the bad result before it was written — it is the strongest single guard against blind structural edits, since the agent cannot see the rendered document. Pair with returning the post-edit block list so the caller can diff.
• Reduce fragile JSON/string reparsing: Only unwrap JSON-encoded strings for block, fields, or ops when the incoming value is actually a string. Prefer native JSON objects from MCP callers. This should help avoid apostrophe/double-decoding problems.
• Make batch reliable from ChatGPT: batch is the documented safest path for multi-step edits, but it failed from ChatGPT at dispatch/resource resolution rather than with a Notes-level error. Fixing this should be high priority because it enables atomic edits with if_rev and block IDs.
• Prefer IDs by default for edit workflows: Since safe patching requires stable block IDs, either return block IDs from all get calls by default or add a dedicated get_for_edit(key) convenience endpoint that always returns {document, rev, block_ids}.
• Concept index (composition layer): A second navigational layer alongside the link graph — an inverse concept → notes map built by matching-pursuit decomposition of note embeddings at ingest, so retrieval is plain lookup rather than query-time similarity. Same compile-don't-re-derive logic as the link-graph item above, on the composition axis rather than the link axis. Pre-design rationale, mechanism (TBDs), honest costs, and the paired hierarchical-CONTENTS restructuring in ideas/concept-factorisation.
• Verb API implementation plan: Ordered build sequence (reorder → dry_run → MOVE/COPY → ETag → metadata ops), with the core-once/three-interfaces constraint, the gravlax clone/deploy reality, per-phase tests, and a definition of done. Starting note for the Claude Code session: gdata-server/todo/verb-api-implementation.
• Multiple same-anchor insert_after ops reverse-order (low priority): within one batch, N insert_after ops all targeting the same block_id insert in reverse of array order (each new block goes immediately after the anchor, pushing the previous one down). Documented as a quirk in README/mcp-note-editing and hit again 2026-07-22. Deliberately low priority: it's fully self-correcting — a single reorder fixes it and the reversal is obvious the moment you outline. The wasteful failure mode is the recovery, not the quirk: escalating to a whole-document put to fix one mis-placed block (also 2026-07-22). If ever tackled at source, the clean fix is to let insert_after accept an ordered list of blocks and insert them in array order. Meanwhile the mitigation is a workflow rule, now in README/agent (“Use targeted edits when safe” → recovery discipline): insert bottom-up or via reorder, and never escalate to put for a one-block slip.
• Verb taxonomy (overarching): Keep get/put as basic whole-document KV; move all structured ops (patch/batch/move/diff/dry_run) onto PATCH and POST /{key}, and map notes operations onto HTTP extension methods (MOVE for key rename, COPY, PROPFIND/PROPPATCH for metadata, If-Match/ETag for concurrency in place of if_rev). Many items below (dry_run, move, richer returns, metadata reads, prefer-IDs) are facets of this single design — see proposals/verb-taxonomy.
• Clarify patch as single-op convenience and batch as preferred multi-op path: Document patch for simple one-shot edits and batch for coordinated edits, especially when metadata needs updating in the same logical change.
• Document storage format inconsistency with include_block_ids flag: When notes are created via put() with simple JSON value (not JSONHTL structure), subsequent reads with include_block_ids=true sometimes fail or return the document as a JSON string rather than a proper document object. Encountered 2026-08-31 when creating notes via MCP put tool; workaround was to recreate the notes with explicit JSONHTL structure. Root cause unclear: may be related to how values are serialized on write or deserialized on read when block IDs are requested. Affects agent workflows that create multiple linked notes in sequence (e.g., project indexes with child notes). Recommend: (1) ensure put() always normalizes input to JSONHTL structure before storage, (2) verify include_block_ids deserialization path, (3) add a test case for get(..., include_block_ids=true) on freshly-created notes.
• Add simple convenience operations: Consider agent-friendly helpers such as append_para, append_heading, or auto_update_meta. Example: append a paragraph while automatically incrementing version and setting updated to today. This avoids forcing callers to construct full JSONHTL for common append-only notes. The log subsystem is the canonical use case motivating an atomic append_entry op.
• Plain get(key) response issue in ChatGPT: get(key) without include_block_ids succeeds but returns an empty resource in the ChatGPT connector, while get(key, include_block_ids=true) returns the full document. Investigate the plain-get response rendering/resource path. This may be a connector rendering issue rather than a storage issue.
• Clarify patch as single-op convenience and batch as preferred multi-op path: Document patch for simple one-shot edits and batch for coordinated edits, especially when metadata needs updating in the same logical change.
• 2026-06-28: reorder op implemented across HTTP (POST+PATCH), MCP tool, and CLI. Strict validation (422 on missing/unknown/duplicate IDs), if_rev optimistic concurrency. 18 HTTP + 11 MCP tests. Deployed to gravlax.
• 2026-06-28: Logged a structural-edit incident and acted on it. A multi-op CONTENTS batch mixing deletes with positional inserts corrupted the index; repaired via full put. Root-cause analysis in troubleshooting (sequential-application trap). Actions taken: raised dry_run priority, added move_block and section-aware edit requests above, and recorded the working rule to prefer full put for reorganisations.
• ~~MCP SSE -32602 on reconnect~~: FIXED 2026-06-28. Root cause: SSE client caches session state and skips initialize after server restart. Fix: changed ~/.claude.json "type": "sse" → "type": "http" (Streamable HTTP, stateless). See note gdata-server/session-2026-06-28.
• 2026-05-28: Generalized MEMORY for all LLM assistants, added MEMORY/claude and MEMORY/codex, refreshed the historical Envoy status wording, split README/tools into a tooling index plus README/command-line-tools and README/mcp-tools, normalized CLI codeblocks, updated current gdata-server deployment summary, and cleaned up the duplicate Future heading.
• 2026-07-07: HTML renderer metadata list formatting fixed: updated notes_web.py in /home/john/git/gdata-server so list metadata such as tags renders as comma-separated text instead of Python repr syntax. Deployed manually with rsync -av /home/john/git/gdata-server/notes_web.py john@gravlax.critchley.biz:~/py/gdata-server/ and restarted gdata-mcp-server.service. Verified README/mcp-note-editing renders tags documentation, tools, mcp, notes.
• ~~notes load should not silently store ops-shaped input as a document~~: FIXED 2026-07-08. PUT (and load/write, which are PUT under the hood) now rejects a JSON array shaped like an ops/patch payload (a list of objects with an 'op' key) instead of silently storing it verbatim. Other non-dict JSON (lists, strings, numbers) is still accepted -- only the specific ops-shaped corruption pattern is blocked. Fixed in both gdata_server.py (handle_PUT_request) and gdata_mcp_server.py (db_put, backs the MCP put/patch/batch tools). Deployed to gravlax and verified live.
• ~~Batch/patch apostrophe encoding bug~~: FIXED 2026-07-08. _parse_json_robust (used to tolerate \' where a plain apostrophe was meant) had a redundant duplicate condition and silently re-raised the pre-repair error on double failure, hiding what the repair attempt actually produced. Simplified the check and added a combined error message showing both the original and post-repair parse errors. Same fix applied to both gdata_server.py and gdata_mcp_server.py; test suite (196 tests) passes; deployed to gravlax.
• 2026-07-10: outline / block-head read op implemented and deployed: added plain-text block previews plus stable IDs via REST POST /{key} {"op":"outline"} and MCP outline tool. Added tests in test_patch_api.py and test_mcp_tools.py; verified live on gravlax. During deployment, drift check caught live-only 2026-07-08 gdata_mcp_server.py fixes missing from the local clone and merged them before rsync; source-control reconciliation remains a process issue.
• 2026-07-27: "HTML renderer leaks raw meta object" — root cause found, not a renderer bug: investigated by comparing TFG/note-for-James (broken footer) against writing/failure-does-not-increase-authority-linkedin (renders fine). The linkedin note uses the standard flat top-level metadata schema (title/version/created/updated/tags alongside content, per README/metadata). The James note instead had a non-standard nested meta: {...} object duplicating title/created/tags alongside separate top-level version/updated fields — the renderer doesn't recognise the nested meta key so dumps it raw, and the duplicated version/updated produced the second footer line. Fixed by flattening TFG/note-for-James to the standard schema via put (rev r16). Was a data problem specific to that one note, not a systemic renderer flaw; no server-side fix needed.
• ~~Cell name label low contrast in runnable sheet~~: FIXED 2026-07-27. Set an explicit dark foreground colour (wx.Colour(30, 40, 70)) on self.name_label in sheet_ui.py so the cell name renders clearly against the light blue-grey (#ECF0F7) header strip.
• 2026-07-28: Per-key HTML view implemented (Claude Code): notes_web.py content-negotiates on Accept at /notes/{key} (proxied publicly as https://www.critchley.biz/notes/{key}), serving rendered HTML by default plus JSON/YAML/Markdown alternates. Full design in gdata-server/http-formats. Confirmed live for fitness/bodypump-attendance (SVG chart and table render correctly). Closes the gap raised 2026-07-27.
• 2026-08-04: Source control for live gdata-server — RESOLVED. ~/py/gdata-server on gravlax is now a real git clone: git init in place → fetch → reset origin/m, then checked out all tracked files to HEAD (2 stale runtime files refreshed, 129 missing populated). Operational/host-local files (secrets, .bak-, backups/, gdata_module/, patches/, resetpw) are in .git/info/exclude; the repo's .gitignore covers .gdbm/.log/__pycache__/tmp/. Working tree clean, tracks origin/m, git pull deploys, and gdata_mcp_server.py/notes_web.py/gdata_oauth.py hash-match HEAD. GitHub SSH wired in ~/.ssh/config (repo-scoped deploy key generated, pending registration; working account key in use meanwhile). Verified live: both services active, import gdata OK, public /notes/ 200, private gate 302/200. Closes the deploy-drift class and unblocks the Envoy execution-sandbox prerequisite.
• 2026-08-04: JSONHTL_SCHEMA / linter drift (svg, image, details) — RESOLVED. Added block_svg/block_image/block_details to JSONHTL_SCHEMA (v4) and registered them in gdata-server/linter (v5); spec-legal docs using those blocks now validate (verified live via jsonschema against the served schema). The same pass added the nested section block (Phase 0 of proposals/section-editing) to spec/schema/linter and both renderers.
✅ Shipped 2026-08-05 — codeblock canonicalisation (visible, not silent). Root cause of the certhub/dap note rendering empty code on the web: codeblocks stored with language/text were accepted on write, tolerated by desktop/markdown renderers, and only failed on notes_web. Fix: jsonhtl_canon.py shared reader (all renderers), db_put normalises + self-heals, REST GET/PUT emit 203 Non-Authoritative Information + Warning: 299 + X-GData-Warnings for non-canonical docs (canonical stays 200), MCP surfaces warnings in-band. Deployed to gravlax (both services) with no outage; whole corpus migrated to canonical (public 0/501, private 0/92). Design + as-built: proposals/codeblock-canonicalisation. Follow-ups there: shared block constructors, explicit schema version, render-check (empty-from-non-empty = error), and the stale ~/dbcerts/output/fix_cert_pwds_note.json template (Codex-owned).