Expanded the experiment to every key in a public-store snapshot: 713 keys, 712 non-empty notes evaluated, one empty note explicitly listed. The original 65/72 candidates were too restrictive. A much larger vocabulary improves topic specificity on several examples, but simply adding words and phrase fragments does not make the glosses reliably faithful. Later matching-pursuit selections can introduce a wrong language, host, season or activity. Use these as experimental navigation labels, not substitutes for the notes.
Final vocabulary sizes: 11,344 raw word candidates (including original phrase controls), 16,341 with phrases; 10,042 and 14,554 after a documented navigation-fragment filter. All six dictionaries were tested with plain matching pursuit and a direct top-three cosine baseline.
25-note qualitative spot review
A searchable, standalone comparison browser is saved at /home/john/py/gloss/artifacts/corpus-2026-09-18/explorer.html. It contains all six dictionaries, both methods, weights, format/chunking flags and captured source text. No network is needed to inspect it. comparison.csv has one row per key; results.json retains complete numerical output. Notes pages above show the original 65 and filtered expanded comparisons for every key.
Candidate extraction was mechanical and corpus-wide. Preserve word forms: no stemming, lemmatization, singular/plural merging or synonym merging. Match casing for vocabulary collection, while preserving original 72 candidate spellings. Candidate words must be at least three characters, contain no digits, and not be function words from the scikit-learn English stop list plus contraction additions. Include observed dictionary words even at document frequency one, other technical words at frequency two, and terms from keys/titles/tags. /usr/share/dict/british-english supplies an attestation check, not a list of all words to embed.
Add the top 5,000 candidate bigrams/trigrams by document frequency, each seen in at least two notes, with function words disallowed at their boundaries. They come from prose, not code blocks, and do not cross block or punctuation boundaries. Preserve function words inside phrases. Exclude the three Gloss write-ups from vocabulary learning. Three mined phrases already occur in the seed dictionary and are not duplicated, leaving 4,997 additions. The vocabulary is learned from the corpus being described: this is transductive coverage, not a held-out generalization test.
Raw expansion exposed a predictable failure: note slugs and navigation wording became high-scoring atoms. Examples included jsonhtl-documentation-improvements and gdata-server todo chatgpt-notes-review. Therefore a post-run diagnostic retained the original72 controls, then admitted only alphabetic words separated by spaces, excluded README/TODO, and excluded two/three-word spans crossing a slash in a source key. This is not a linguistically principled canonical vocabulary; it also removes some legitimate hyphenated technical terms. Both raw and filtered results are retained. No re-embedding was needed for this filter.
Residual boilerplate remains: open questions weak, review and handover, and generic implementation notes can still be selected. Dictionary size is only one variable; phrase quality and the relationship between residual directions and intelligible concepts remain unresolved. This pass uses corpus-attested vocabulary, not a broad external language lexicon, so unseen synonyms can still be missing.
Primary metrics below cover 709 notes, excluding the three Gloss experiment write-ups to avoid self-reference. All 712 non-empty keys still have output. Exact-text deduplication changes nothing: all 712 texts are distinct, though 105 Envoy drain logs are structurally repetitive.
| Dictionary | Atoms | Mean first cosine | Mean residual norm | Below 0.5 | Distinct selected labels |
|---|---|---|---|---|---|
| original65 | 65 | 0.3018 | 0.9305 | 0 | 64 |
| original72 | 72 | 0.3055 | 0.9293 | 0 | 71 |
| corpus_words | 11344 | 0.4565 | 0.8327 | 1 | 729 |
| words_and_phrases | 16341 | 0.5221 | 0.7833 | 1 | 963 |
| clean_words | 10042 | 0.4162 | 0.8616 | 1 | 730 |
| clean_words_and_phrases | 14554 | 0.5088 | 0.7950 | 1 | 925 |
Larger dictionaries have more opportunities to match and therefore can improve geometric fit without improving semantic truth. The raw phrase dictionary fits better than the filtered one even though it contains conspicuous navigation debris. These numbers must not be presented as accuracy or percentage of meaning recovered.
Removing the 105 Envoy drain logs leaves 604 notes. Mean residual is 0.9281 for the original65 and 0.8058 for the filtered expanded dictionary. Thus the geometric improvement is not solely an artefact of repeated operational logs. The primary filtered run uses 925 distinct selected labels from 14,554 available atoms.
Only tmp/envoy.json crosses the 0.5 stopping threshold: its entire source text is the single word envoy, which matches the envoy atom exactly. No substantive multi-concept note reaches the threshold in three steps. No atom repeats within a three-step result. OMP is not tested here; this remains plain, positive-cosine matching pursuit.
| Example | Finding |
|---|---|
| gdata-server/notes/design | note/API/software design → gdata notes/notes client/python module: more specific and useful. |
| ask/jobs/schema | JSON/software design/versioning → json schema/cover letters/docx generation: a substantial topical improvement. |
| Z80-sudoku-challenge/code/deterministic-fill-loop | programming/matching pursuit/fastest → deterministic fill loop/address of cell/bit candidate mask. |
| README/orphaning | avoiding orphaned notes/metadata and links/design documentation captures the purpose more directly and retains avoidance in a phrase. |
| PROGRAMMING_RULES/exceptions | exception handling/perl python/general principles introduces Perl without source support; the direct cosine baseline is more faithful. |
| TFG/exercise-demo-protocol | functional exercise/review and handover/sample demonstrations is worse than the original exercise/demonstration/coaching. |
| poems/first-to-Kathy | summer poem is a false seasonal specific in a poem of enduring affection. |
| hosts/gravlax | freebsd host is a wrong-host/OS association despite useful WebDAV and gravlax/kelp labels. |
Direct cosine is often repetitive—exception handling/error handling/exceptions—but can be more faithful than pursuit. Subtraction trades redundancy for diversity and sometimes that diversity is wrong. This run does not establish an overall winner. A better next experiment would use a frozen, reviewed vocabulary and blind comparison of direct cosine, MP and OMP, with explicit tests for unsupported specifics and polarity. Do not just keep growing the phrase list.
Snapshot captured 2026-09-18T15:30:41.837946+00:00 via POST / with op=dump at the public notes REST endpoint. Private Notes was not accessed. One empty key, scratch/maildir-test-summaries, is represented as empty instead of being assigned invented labels. Eleven long notes were split into contiguous, non-overlapping chunks of at most 6,000 cl100k_base tokens. Each chunk was embedded; the note vector is their token-count-weighted average, normalized. All other note vectors and all atoms are normalized as well. Chunk averaging is an approximation and can flatten multi-topic content; chunked rows are flagged. No note was truncated, and exact token reconstruction was verified for every input.
JSONHTL content wording is retained; metadata is used for vocabulary mining but not injected into ordinary note embeddings. Legacy body fields and non-JSONHTL records use an audited textual fallback. SVG text/title/description labels and captions are included; drawing commands and CSS are excluded. Seven malformed JSON-string records are retained verbatim, including their serialization noise, and flagged; no source note was repaired. The original nine note input hashes are unchanged.
Malformed JSON-string keys: README/documentation-in-notes; ideas/jpeg-obscura/v2-multi-region-implementation; location-db/public-timetable; misc-server; misc-server/email-tools; misc-server/tools; proposals/codeblock-canonicalisation.
Model: text-embedding-3-small, 1,536 dimensions. Content-addressed SQLite embedding cache; 81 original inputs reused initially. The expanded run consumed 738,371 new input tokens in 50 successful requests, including a small rerun after excluding SVG stylesheet text. Estimated cost at $0.02/million is $0.01477; this is a rate-based estimate, not billing verification. All subsequent dictionary filtering and comparisons are offline.
Embedding API reference: OpenAI model documentation.
Eleven automated tests pass, including batched-versus-independent pursuit, per-note stopping, legacy-field preservation, logged malformed-input fallback, SVG extraction and the original projection tests. Dataset checks verify every snapshot key is accounted for, token reconstruction, current input/vocabulary hashes, finite nonzero vectors, and per-step projection energy reduction. Package versions and all numeric results are saved in the artifacts; the API cache makes offline reruns independent of future model-alias changes.
cd /home/john/py/gloss
python -m pip install --user --break-system-packages -r requirements.txt
# Offline reproduction from the completed cache:
python -m unittest -v
python corpus.py analyse
python corpus_export.py
# New corpus snapshot, separate directory:
python corpus.py snapshot --directory artifacts/new-corpus
python corpus.py prepare --directory artifacts/new-corpus
python corpus.py embed --directory artifacts/new-corpus
python corpus.py filter --directory artifacts/new-corpus
python corpus.py analyse --directory artifacts/new-corpus
python corpus_export.py --directory artifacts/new-corpusFiles under /home/john/py/gloss/artifacts/corpus-2026-09-18/: snapshot.json; inputs.json; vocabulary.json; embeddings.sqlite; run.json; matrices.npz; results.json; summary.json; comparison.csv; explorer.html; notes/ (exact published JSONHTL); and preparation.log (format warnings). Install packages into ~/.local using --user --break-system-packages; do not create .venv. The exploratory .venv was removed during this session.
Parent: Gloss; earlier experiment: Nine-note prototype.