Operational lessons from working sessions. Read this when starting work on infrastructure, playbooks, or tooling — these are things that went wrong and cost time.
Companion note: README/diagnosis — for when something that should be working isn't. This note holds specific lessons (mostly “X behaves unexpectedly”); that one holds the general method for telling a system that reports health from a system that actually works, and the rule not to restart before you can explain.
Non-critical side-effect tasks in playbooks (note registry updates, logging, awsctl reloads) must have ignore_errors: yes. A failure in one of these must not abort the play and block essential follow-on tasks.
What went wrong: In launch_instance.yml, the "Update hosts note" task failed (HTTP 404 — the hosts note did not exist yet). This aborted the play before the known_hosts tasks ran. The newly provisioned host could not be reached by subsequent playbooks because its SSH key was never added to ~/.ssh/known_hosts.
Rule: If a task's failure does not make the host unusable, add ignore_errors: yes. Essential tasks (disk format, package install, SSH config, key registration) must still fail loudly.
If simplejson is installed, requests uses it instead of Python's built-in json. The built-in silently serialises float('nan') as the bare token NaN (invalid JSON but no exception). simplejson is strict and raises ValueError: Out of range float values are not JSON compliant.
Root cause pattern: a pandas DataFrame built from records where some rows are missing a field (e.g. a court-booking email with no coach field) has NaN for that column. df.to_dict('records') produces dicts where the key exists with value NaN. Calling .get('coach', '') returns NaN — not the default — because the key is present. Those NaNs then fail JSON serialisation.
Fix: sanitise before serialising. In the row-building step use: def _safe(v): return '' if isinstance(v, float) and v != v else v. No import needed — NaN is the only float not equal to itself (IEEE 754).
When a document is stored double-encoded (the GDBM value is a JSON string literal wrapping the document JSON, rather than the JSON object itself), any call to patch fails with "document is not a JSON object". The 'op' is a required property error reported in earlier sessions was a red herring from a different code path; the root cause is always the double-encoding.
This happens when put is called with a string value (e.g. the MCP client serialises the document to a JSON string before passing it) and the server stores json.dumps(string) rather than json.dumps(parsed_dict). The current MCP put handler fixes this via _parse_json_robust: if value is a string it is parsed as JSON before storage. Existing double-encoded docs must be repaired manually: read via CLI, write a corrected JSON file, PUT via curl.
Fix command: curl -s -X PUT http://127.0.0.1:8021/<key> -H 'Content-Type: application/json' -d @/tmp/fixed.json. All patch operations work correctly on properly encoded documents — patch does not need to be avoided.
MSAL raises API does not accept frozenset({'openid', 'profile', 'offline_access'}) value as user-provided scopes if you include offline_access, openid, or profile in the scopes list passed to initiate_device_flow or similar calls. MSAL adds these automatically.
Only pass the resource-specific scope, e.g.:
SCOPES = ['https://outlook.office.com/POP.AccessAsUser.All']
app.initiate_device_flow(scopes=SCOPES) # MSAL adds offline_access internallyThe google-auth library compares token expiry using _helpers.utcnow() which returns a naive (timezone-unaware) datetime. If the stored expiry string is timezone-aware (e.g. 2026-06-25T13:10:49+00:00), constructing Credentials(expiry=expiry) causes TypeError: can't compare offset-naive and offset-aware datetimes when .expired is accessed.
Fix: strip tzinfo after converting to UTC before passing to Credentials:
if expiry.tzinfo is not None:
expiry = expiry.astimezone(timezone.utc).replace(tzinfo=None)SetEnv directives in Apache <Location> blocks populate the per-request WSGI environ dict passed to the application(environ, start_response) function. They do NOT appear in os.environ. Any WSGI helper function that reads config must receive and use the WSGI environ, not os.environ.get().
def _cfg(wsgi_environ, key, default=None):
return wsgi_environ.get(key) or os.environ.get(key) or defaultUse ~/tmp/ on remote hosts, or ./tmp/ relative to the project directory locally. Never /tmp/ (including any agent-default scratch/temp directory that lives under it).
What went wrong: /tmp triggers a permission prompt on every use in some agent harnesses, and is a shared system directory on remote hosts (gravlax uses ~/tmp/gdata-server-github for its own git clone, for this exact reason). Confirmed again 2026-07-08: crontab <file> failed with "No such file or directory" for a path under an agent scratchpad living in /tmp, but worked immediately once copied to ~/. This had already been learned once before but was re-derived from scratch in a different project/session because it was not recorded here.
Building a notes patch/append_block JSON payload inline as a bash single- or double-quoted string breaks down once the content has apostrophes (contractions like "isn't", "gravlax's") or backticks: the classic '"'"' single-quote-escaping trick is easy to get wrong (one attempt failed with Error parsing arguments: Expecting ',' delimiter), and backticks inside a double-quoted bash string get shell-executed as command substitution, silently corrupting the content instead of erroring.
Fix: for any block with more than one or two apostrophes/backticks, write the JSON to a file first (with the Write tool, or notes load -d <key> <file> for a whole document) rather than building it inline in bash. See README/mcp-note-editing for the general safe-edit workflow.
When a batch/patch call replaces the wrong block_ids — e.g. mis-mapping which block held which content and swapping two blocks' text — the fix is to use the reorder op to move the existing blocks back into correct position, not a full-document put. A full rewrite regenerates every block ID and discards the document's edit history for no reason, when the actual fault was just wrong positions/content for a couple of blocks.
Same fault class, different trigger: a stacked insert_after on the same anchor block (two or more inserts targeting one block_id in the same or successive calls) lands the new blocks in reversed order. All content exists, just mis-positioned — so the fix is the same reorder remedy, not a full put. Recurred 2026-07-18 on videos/lamis-mukta-dreaming-memory: fixed via full put reconstruction when a reorder with the correct block-ID permutation would have sufficed. For the general-purpose pattern that avoids this class of mistake altogether, see the delete → append → reorder recipe in gdata-server/troubleshooting.
What went wrong: while editing location-db/hibernate-on-approach (2026-07-14), a batch call used block IDs from a stale mental model of the document (assumed positions rather than re-checking the actual get(include_block_ids=True) output carefully), and ended up swapping the content of two unrelated blocks. Fixed by discarding the document and reissuing a full put — functionally correct, but heavier than necessary and loses block-ID continuity. A reorder (or a second, carefully-checked batch swapping the two blocks back) would have been the right-sized fix.
When comparing a GPS reading against gazetteer/survey points, compute the actual distance to each nearby point first and lead with the closest match. Do not default to impressionistic "between A and B" framing — if the numbers only support one point, say so plainly; don't hedge towards a second point that is actually much further away.
What went wrong: given a reading ~2.5 m from a named net-line fix and ~78 m from a named lounge fix, described it as "between" the two, implying comparable proximity. Corrected once distances were actually checked. See gazetteer/david-lloyd-tennis-bubble for the survey this arose from.
Same fault class as the entry above, but notable for frequency and irony: in one session, editing fitness/david-lloyd-clubs and README/mcp-note-editing, the wrong-block-ID mistake happened four separate times — the last three while writing up the first one as a lesson. Each time the cause was identical: a block_id chosen from a hand-matched or remembered mapping between block IDs and content, rather than freshly cross-checked line-by-line against the actual get(include_block_ids=True) output at the moment of writing.
• Incident 1: fixing the Bishops Cleeve "not yet open" wording on fitness/david-lloyd-clubs, a batch replace_block targeted the wrong block_id and replaced the entire table with a plain paragraph.
• Incidents 2 and 3: while adding the lesson entry to README/mcp-note-editing itself, two consecutive insert_after calls each landed the new paragraph in the wrong section (first under "Safe Single-Block Changes", then under "Verification Checklist", rather than "Common Failure Modes") — both times based on a block-ID position remembered from a slightly earlier read rather than checked against it directly.
• Incident 4: adding a "Confirmed Working" section to ideas/table-editing-robustness, an insert_before targeted the "Status" heading's block ID but landed the new heading between "Status" and its own paragraph, orphaning both from each other.
All four were caught by reading back immediately afterwards and fixed with a full-document put (the same over-correction flagged in the entry above — a targeted reorder or corrected patch would usually have sufficed, but by incident 3 the priority was stopping the bleeding, not minimalism). ['Takeaway']: knowing about this failure mode does not prevent it — re-checking the block ID against a genuinely fresh read, at the moment of writing, is the only thing that reliably does. Read-back-after-every-write remains non-negotiable regardless of how careful the read-before felt. See Table Editing Robustness for the same session's positive finding: table_op set_cell, scoped by column name and row index rather than a block ID, cannot misfire this way — prefer it over replace_block for table cells whenever it applies.
A matching checksum for a command-line script does not prove its behaviour if it loads sibling modules or libraries from a different directory. Package and deploy one canonical runnable tree, preserving its relative layout; include a manifest/checksum for every supplied dependency; and verify the actual loaded path/module versions before the live test. An entry-point checksum plus a smoke run can otherwise validate a mixed old/new runtime.
Copying 106 files to the WebDAV share via the /z davfs2 mount (rsync -a ... /z/kathy/) exited 0 with no errors. A check by curl against the URL found 105 of the 106 were not on the server — they were still sitting in davfs2's local cache (298MB at the time). They only arrived when the mount was unmounted, which flushed them. The exit code was meaningless, and re-reading through the same mount would have looked correct too, because reads are served from that same cache. Two further hazards John flagged: a large backlog of deferred writes can exhaust Apache's scoreboard/workers on the server when it does flush, and an unmount or host loss before the flush discards the data entirely. Rule: never use /z (or any davfs mount) in a script or anywhere a write is expected to have landed — use curl/webdav4/requests against the URL with ~/.netrc, and verify with a HEAD. Interactive read-only use of the mount is fine. General rule recorded in memory/john; trigger line in TRIGGERS. A warning did exist beforehand in memory/envoy/webdav but said only "the mount is flaky" with no mechanism, was scoped to one subfolder, and sat in a project note — so it read as superstition and was overridden. A prohibition without its mechanism does not survive contact with an agent that has a reason to doubt it.
On hosts provisioned with two users sharing one uid (e.g. admin and john both at 1000 on a delayrepay browser host), ssh resolves ~/.ssh (known_hosts, default identity file) from the FIRST /etc/passwd entry for that uid, not $HOME. Host keys must be written to the first user's ~/.ssh/known_hosts to take effect, and ssh offers that user's default key, not the other user's. Any script or playbook managing known_hosts or keys for a shared-uid pair must write BOTH users' ~/.ssh directories.
ssh-keyscan's output includes a # host SSH-2.0-... banner line on stdout, so stdout_lines[0] is the banner, not the host key. Select the line containing the keytype (e.g. ssh-ed25519) rather than indexing [0], or known_hosts entries end up malformed.
The Ansible known_hosts module errors with "does not match hashed host field" against a known_hosts file that already contains hashed entries. ssh-keygen -R to remove the stale entry, followed by a plain append of the plaintext line (called via shell), is more robust than the module in that situation.
Tag-filtering a playbook run (--tags X) skips any untagged play or task, including one with a side effect a later play assumes happened — e.g. a play 1 host launch + add_host that populates the in-memory inventory group play 2 targets. Play 2 then has no matching hosts and the run appears to succeed while doing nothing. Tag the prerequisite tasks always, or run the full (idempotent) playbook instead of a tag subset.
A WebDAV MKCOL request only creates the immediate collection — MKCOL a/b/c/ fails (409) if a/b/ doesn't already exist. A first-time publish into a nested WebDAV path needs to MKCOL each parent level in turn, not just the final one.
If .claude/settings.json grants one exact-match allow entry per unique command line, a command whose arguments vary every call (e.g. magick ... -crop WxH+X+Y ... with different crop coordinates each time) is an unseen command on every call and re-prompts regardless of how many times it was already approved. Fix: use wildcard rules (Bash(magick ), Bash(identify ), Bash(convert *)) and make sure the command starts with the bare token the rule matches — a leading cd ... && or timeout NN prefix breaks prefix-matching and defeats the wildcard.
Before removing a QMP hot-plugged block device (device_del + blockdev-del) that the guest OS is using as swap, run swapoff on it inside the guest FIRST and confirm it drained (swapctl -l / equivalent) before touching anything host-side. QMP will happily rip out a device that's still backing live guest I/O — it does not check.
What went wrong: on radagast (FreeBSD VM), device_del removed a hot-plugged NVMe swap device while it still had ~30 MB of pages swapped out to it. The kernel hit swap_pager: I/O error - pagein failed and syslogd — which had pages on that device — segfaulted and died. The VM stayed up and reachable; only the one process was collateral damage, caught by noticing it was no longer running and restarting it. Full procedure (including the separate AHCI-vs-NVMe hotplug gotcha this arose alongside): radagast/fast-swap-and-qmp.
An earlier working assumption characterised radagast's zroot disks (sda/sdb on pomelo, USB-attached) as "slow USB 2.0" when explaining swap/I-O slowness. Checked with lsusb -t on pomelo: the dock is on Bus 004, an XHCI SuperSpeed port at 5000M (5 Gbps), using the uas (USB-Attached SCSI) driver — genuine USB 3.0/3.1 Gen1, not USB 2.0. The real bottleneck for swap slowness was more likely rotational-disk seek latency plus ZFS zvol/COW overhead. Rule: check lsusb -t before attributing I/O slowness to USB generation — don't carry forward an unverified assumption about link speed between sessions.
For a FreeBSD port with a very large dependency tree (LibreOffice: ~100+ deps), don't run make install clean as one command. The clean target fires per-dependency immediately after each one installs, so once the main target itself is actually done and working, the process keeps running for a while just serially cleaning every dependency's work directory one at a time — time spent not knowing the real target is ready, and slower overall than batching the cleanup.
What went wrong / how to apply: caught live on 2026-10-15 watching radagast's LibreOffice build: pkg info libreoffice showed it already installed and working minutes before the script's own completion marker fired, because it was still working through dependency cleanup. For any port with a dependency tree large enough to take more than a few minutes (LibreOffice, Firefox, Thunderbird-class — anything pulling in its own JDK/Qt/MySQL-sized sub-builds), run make install first, verify with pkg info <port> as soon as the log looks done, then make clean as a separate trailing step (or skip it if disk space isn't a concern). For small/normal ports the combined install clean is fine and conventional.
FreeBSD root passwd lookups can expose password hashes (2026-10-17). Do not print getent passwd output when running as root on FreeBSD: unlike the usual Linux passwd view, it can return the account's password verifier. Use id -u/id -g for UID/GID checks, and retrieve only specifically needed nonsecret fields for other account metadata. This was discovered during Radagast reverse-tunnel setup; the exposure was recorded privately and John was told to change the affected account password. No hash is reproduced here.