Karpathy's LLM-WIKI.md paper identifies log.md as a core primitive this notes system lacks: an append-only chronological record of every operation (ingest/query/lint), each entry prefixed ## [YYYY-MM-DD] op | title and parseable with standard Unix tools. This note proposes how to provide that capability, exploiting the fact that the backing store is a GDBM key-value store rather than a filesystem.
The literal Karpathy design is a single growing markdown file. In a flat filesystem that is sensible — appends are cheap (open in append mode) and grep slices it. Reproduced naively as a single JSONHTL note it inherits none of those properties and acquires new costs:
• Append = full rewrite. There is no append-to-document operation that avoids reading and rewriting the whole note. append_block on the server still rewrites the stored value and regenerates all block IDs (see gdata-server put semantics). A log that grows to thousands of entries means every new entry rewrites the entire history.
• Unbounded growth in one value. JSONHTL notes are loaded whole. A single ever-growing log note becomes the exact megadoc the workflow note warns against — every read pulls the entire history into context.
• No slicing. "What happened last week" or "all lint runs" requires loading everything and filtering client-side.
• Concurrency. Two agents (Claude, Codex, Envoy cron) appending at once collide on if_rev and one must re-read the whole document and retry.
The insight worth keeping from log.md is the semantics (append-only, chronological, op-tagged, machine-parseable), not the single-file representation. A KV store lets us keep the semantics while choosing a better physical structure.
A key-value store with hierarchical string keys can express most classical structures by encoding the structure in the keyspace and in pointer fields inside values. Relevant options for a log:
• Flat list under one key — the naive note. Rejected above.
• Time-bucketed keys — one note per period, e.g. log/2026/06/28. The keyspace itself is the index; the bucket boundary caps the size of any value that must be rewritten on append.
• Linked list of chunks — each node holds N entries plus a prev pointer to the previous chunk's key. A head pointer names the newest chunk. Appends touch only the head; history is immutable once a chunk fills.
• Append-only entries + index — every entry is its own key (log/entry/{ulid}) and a separate index note (or per-day index) lists entry keys in order. Maximal granularity; the index is the only thing that grows per append, and even that can be bucketed.
• Tree / B-tree-like — interior notes point to child notes by key (year → month → day → entries). Navigation is O(depth) reads. This is really the bucketed scheme with an explicit multi-level index on top.
These are not mutually exclusive — the recommended design combines bucketing (to bound rewrite cost) with a thin tree index (to make navigation cheap) and ULID-keyed entries (to make individual entries addressable and concurrency-friendly).
Three kinds of key:
Key pattern: log/d/YYYY-MM-DD — one note per day. Each note is an append-only list of entries for that day. A day rarely exceeds a few dozen entries, so rewrite-on-append cost is bounded and small regardless of total history size. Each entry is a structured para, not free text, so it is parseable without regex:
{
"title": "Log 2026-06-28",
"kind": "log-bucket",
"date": "2026-06-28",
"version": 4,
"prev": "log/d/2026-06-27",
"content": [
{"heading": {"level": 1, "text": "Log — 2026-06-28"}},
{"logentry": {
"ts": "2026-06-28T14:02:11Z",
"op": "ingest",
"agent": "claude",
"title": "Karpathy LLM-WIKI paper",
"refs": ["papers/karpathy-llm-wiki"],
"note": "Created paper note; linked from CONTENTS, gdata-server, SCP."
}},
{"logentry": {
"ts": "2026-06-28T15:20:04Z",
"op": "lint",
"agent": "envoy",
"title": "orphan scan",
"refs": [],
"note": "No orphans found. 3 stale todos flagged."
}}
]
}The prev field makes the day-buckets a linked list: from any day you can walk backwards in O(1) reads per day without consulting an index. logentry is a new convention block type — unknown to base renderers, which per the JSONHTL spec rule 2 silently ignore it, but fully structured for machine consumption. A renderer or convention doc can teach the viewer to format it as [ts] op | title.
Key patterns log/index/YYYY/MM (month index → list of day-bucket keys present, with one-line roll-ups) and log/index (root → list of months, newest first). This is the B-tree-like layer: to find a date range you read root → month → day buckets, never scanning the whole log. The root index is small and bounded (one entry per month), so it is cheap to keep current.
Key log/head holds the key of the current (newest) day bucket and the timestamp of the last entry. Writers read log/head first to know where to append. This is the single small note that every append must touch, so it is the concurrency hot-spot — kept deliberately tiny so if_rev retries are cheap.
• Append entry: read log/head; if its bucket is today, append_block a logentry to that day bucket; else create today's bucket with prev = old head, update log/head, and add today to the month index. Worst case three small writes; common case one append_block to a small note.
• Read recent: read log/head → day bucket; follow prev as far back as needed. No full scan.
• Read range: root index → month index → the specific day buckets. Bounded reads.
• Filter by op/agent: currently client-side within the loaded buckets. If this becomes common, add secondary index notes (log/by-op/lint → list of entry refs), mirroring the backlink-index idea already on the todo list.
Day-bucket granularity plus a tiny head pointer minimises contention. The if_rev optimistic-concurrency token already supported by batch handles the rare same-day collision: on 409, re-read the (small) day bucket and retry the append. Because buckets are per-day not per-history, a retry rewrites at most one day, not the whole log. This directly answers the megadoc/rewrite objection.
The design works with current primitives but would benefit from two enhancements already requested on the gdata-server todo:
• An append_para / append_entry convenience op that appends a block and bumps version/updated atomically, so callers need not construct a full document or do read-modify-write. A log is the canonical append-only use case motivating that feature.
• A true server-side atomic append (append without rewriting/reassigning all block IDs) would remove the last rewrite cost. Worth evaluating against the GDBM layer; even without it, day-bucketing keeps rewrites bounded.
A dedicated log.append(op, title, refs, note) endpoint that encapsulates the head-pointer-read → bucket-append → index-update sequence would make logging a single call and keep all three key kinds consistent. This is the cleanest long-term form.
• LLM-WIKI.md — source of the log.md primitive this generalises.
• Structured Context Protocol — the log is the natural place to record phase transitions and commits; an SCP commit step could write a log entry automatically.
• gdata-server/todo — backlink index, tag retrieval, and append convenience ops are all complementary; the secondary-index pattern here reuses the backlink idea.
• Envoy's periodic reconciliation cron is a natural lint writer — its scan results become op: lint log entries, giving a durable history of what was checked and when.
Do not build the whole tree at once. Minimum viable log: implement just the day-bucket pattern (log/d/YYYY-MM-DD with prev) plus log/head. That alone delivers append-only, chronological, walkable history with bounded rewrite cost. Add the month/root index only when walking prev across many days becomes the bottleneck; add secondary op/agent indexes only when filtered queries become common. Each layer is additive and backward-compatible.
• Application: Dreaming vs. the Notes System, Envoy, and MCP (2026-07-18) — concrete forward use case: a revert-capable logentry type for dreaming-driven note edits, carrying enough prior block/document state to undo the change rather than just describing it. This is a specific instance of the op-tagged entry shape above; the note field (or a new prior_state field) would need to hold reconstructable content, not just a human-readable summary, for this use case specifically.