Operational lessons from working sessions. Read this when starting work on infrastructure, playbooks, or tooling — these are things that went wrong and cost time.
Non-critical side-effect tasks in playbooks (note registry updates, logging, awsctl reloads) must have ignore_errors: yes. A failure in one of these must not abort the play and block essential follow-on tasks.
What went wrong: In launch_instance.yml, the "Update hosts note" task failed (HTTP 404 — the hosts note did not exist yet). This aborted the play before the known_hosts tasks ran. The newly provisioned host could not be reached by subsequent playbooks because its SSH key was never added to ~/.ssh/known_hosts.
Rule: If a task's failure does not make the host unusable, add ignore_errors: yes. Essential tasks (disk format, package install, SSH config, key registration) must still fail loudly.
If simplejson is installed, requests uses it instead of Python's built-in json. The built-in silently serialises float('nan') as the bare token NaN (invalid JSON but no exception). simplejson is strict and raises ValueError: Out of range float values are not JSON compliant.
Root cause pattern: a pandas DataFrame built from records where some rows are missing a field (e.g. a court-booking email with no coach field) has NaN for that column. df.to_dict('records') produces dicts where the key exists with value NaN. Calling .get('coach', '') returns NaN — not the default — because the key is present. Those NaNs then fail JSON serialisation.
Fix: sanitise before serialising. In the row-building step use: def _safe(v): return '' if isinstance(v, float) and v != v else v. No import needed — NaN is the only float not equal to itself (IEEE 754).
When a document is stored double-encoded (the GDBM value is a JSON string literal wrapping the document JSON, rather than the JSON object itself), any call to patch fails with "document is not a JSON object". The 'op' is a required property error reported in earlier sessions was a red herring from a different code path; the root cause is always the double-encoding.
This happens when put is called with a string value (e.g. the MCP client serialises the document to a JSON string before passing it) and the server stores json.dumps(string) rather than json.dumps(parsed_dict). The current MCP put handler fixes this via _parse_json_robust: if value is a string it is parsed as JSON before storage. Existing double-encoded docs must be repaired manually: read via CLI, write a corrected JSON file, PUT via curl.
Fix command: curl -s -X PUT http://127.0.0.1:8021/<key> -H 'Content-Type: application/json' -d @/tmp/fixed.json. All patch operations work correctly on properly encoded documents — patch does not need to be avoided.
MSAL raises API does not accept frozenset({'openid', 'profile', 'offline_access'}) value as user-provided scopes if you include offline_access, openid, or profile in the scopes list passed to initiate_device_flow or similar calls. MSAL adds these automatically.
Only pass the resource-specific scope, e.g.:
SCOPES = ['https://outlook.office.com/POP.AccessAsUser.All']
app.initiate_device_flow(scopes=SCOPES) # MSAL adds offline_access internallyThe google-auth library compares token expiry using _helpers.utcnow() which returns a naive (timezone-unaware) datetime. If the stored expiry string is timezone-aware (e.g. 2026-06-25T13:10:49+00:00), constructing Credentials(expiry=expiry) causes TypeError: can't compare offset-naive and offset-aware datetimes when .expired is accessed.
Fix: strip tzinfo after converting to UTC before passing to Credentials:
if expiry.tzinfo is not None:
expiry = expiry.astimezone(timezone.utc).replace(tzinfo=None)SetEnv directives in Apache <Location> blocks populate the per-request WSGI environ dict passed to the application(environ, start_response) function. They do NOT appear in os.environ. Any WSGI helper function that reads config must receive and use the WSGI environ, not os.environ.get().
def _cfg(wsgi_environ, key, default=None):
return wsgi_environ.get(key) or os.environ.get(key) or defaultUse ~/tmp/ on remote hosts, or ./tmp/ relative to the project directory locally. Never /tmp/ (including any agent-default scratch/temp directory that lives under it).
What went wrong: /tmp triggers a permission prompt on every use in some agent harnesses, and is a shared system directory on remote hosts (gravlax uses ~/tmp/gdata-server-github for its own git clone, for this exact reason). Confirmed again 2026-07-08: crontab <file> failed with "No such file or directory" for a path under an agent scratchpad living in /tmp, but worked immediately once copied to ~/. This had already been learned once before but was re-derived from scratch in a different project/session because it was not recorded here.
Building a notes patch/append_block JSON payload inline as a bash single- or double-quoted string breaks down once the content has apostrophes (contractions like "isn't", "gravlax's") or backticks: the classic '"'"' single-quote-escaping trick is easy to get wrong (one attempt failed with Error parsing arguments: Expecting ',' delimiter), and backticks inside a double-quoted bash string get shell-executed as command substitution, silently corrupting the content instead of erroring.
Fix: for any block with more than one or two apostrophes/backticks, write the JSON to a file first (with the Write tool, or notes load -d <key> <file> for a whole document) rather than building it inline in bash. See README/mcp-note-editing for the general safe-edit workflow.
When a batch/patch call replaces the wrong block_ids — e.g. mis-mapping which block held which content and swapping two blocks' text — the fix is to use the reorder op to move the existing blocks back into correct position, not a full-document put. A full rewrite regenerates every block ID and discards the document's edit history for no reason, when the actual fault was just wrong positions/content for a couple of blocks.
Same fault class, different trigger: a stacked insert_after on the same anchor block (two or more inserts targeting one block_id in the same or successive calls) lands the new blocks in reversed order. All content exists, just mis-positioned — so the fix is the same reorder remedy, not a full put. Recurred 2026-07-18 on videos/lamis-mukta-dreaming-memory: fixed via full put reconstruction when a reorder with the correct block-ID permutation would have sufficed. For the general-purpose pattern that avoids this class of mistake altogether, see the delete → append → reorder recipe in gdata-server/troubleshooting.
What went wrong: while editing location-db/hibernate-on-approach (2026-07-14), a batch call used block IDs from a stale mental model of the document (assumed positions rather than re-checking the actual get(include_block_ids=True) output carefully), and ended up swapping the content of two unrelated blocks. Fixed by discarding the document and reissuing a full put — functionally correct, but heavier than necessary and loses block-ID continuity. A reorder (or a second, carefully-checked batch swapping the two blocks back) would have been the right-sized fix.
When comparing a GPS reading against gazetteer/survey points, compute the actual distance to each nearby point first and lead with the closest match. Do not default to impressionistic "between A and B" framing — if the numbers only support one point, say so plainly; don't hedge towards a second point that is actually much further away.
What went wrong: given a reading ~2.5 m from a named net-line fix and ~78 m from a named lounge fix, described it as "between" the two, implying comparable proximity. Corrected once distances were actually checked. See gazetteer/david-lloyd-tennis-bubble for the survey this arose from.
Same fault class as the entry above, but notable for frequency and irony: in one session, editing fitness/david-lloyd-clubs and README/mcp-note-editing, the wrong-block-ID mistake happened four separate times — the last three while writing up the first one as a lesson. Each time the cause was identical: a block_id chosen from a hand-matched or remembered mapping between block IDs and content, rather than freshly cross-checked line-by-line against the actual get(include_block_ids=True) output at the moment of writing.
• Incident 1: fixing the Bishops Cleeve "not yet open" wording on fitness/david-lloyd-clubs, a batch replace_block targeted the wrong block_id and replaced the entire table with a plain paragraph.
• Incidents 2 and 3: while adding the lesson entry to README/mcp-note-editing itself, two consecutive insert_after calls each landed the new paragraph in the wrong section (first under "Safe Single-Block Changes", then under "Verification Checklist", rather than "Common Failure Modes") — both times based on a block-ID position remembered from a slightly earlier read rather than checked against it directly.
• Incident 4: adding a "Confirmed Working" section to ideas/table-editing-robustness, an insert_before targeted the "Status" heading's block ID but landed the new heading between "Status" and its own paragraph, orphaning both from each other.
All four were caught by reading back immediately afterwards and fixed with a full-document put (the same over-correction flagged in the entry above — a targeted reorder or corrected patch would usually have sufficed, but by incident 3 the priority was stopping the bleeding, not minimalism). ['Takeaway']: knowing about this failure mode does not prevent it — re-checking the block ID against a genuinely fresh read, at the moment of writing, is the only thing that reliably does. Read-back-after-every-write remains non-negotiable regardless of how careful the read-before felt. See Table Editing Robustness for the same session's positive finding: table_op set_cell, scoped by column name and row index rather than a block ID, cannot misfire this way — prefer it over replace_block for table cells whenever it applies.
A matching checksum for a command-line script does not prove its behaviour if it loads sibling modules or libraries from a different directory. Package and deploy one canonical runnable tree, preserving its relative layout; include a manifest/checksum for every supplied dependency; and verify the actual loaded path/module versions before the live test. An entry-point checksum plus a smoke run can otherwise validate a mixed old/new runtime.
Copying 106 files to the WebDAV share via the /z davfs2 mount (rsync -a ... /z/kathy/) exited 0 with no errors. A check by curl against the URL found 105 of the 106 were not on the server — they were still sitting in davfs2's local cache (298MB at the time). They only arrived when the mount was unmounted, which flushed them. The exit code was meaningless, and re-reading through the same mount would have looked correct too, because reads are served from that same cache. Two further hazards John flagged: a large backlog of deferred writes can exhaust Apache's scoreboard/workers on the server when it does flush, and an unmount or host loss before the flush discards the data entirely. Rule: never use /z (or any davfs mount) in a script or anywhere a write is expected to have landed — use curl/webdav4/requests against the URL with ~/.netrc, and verify with a HEAD. Interactive read-only use of the mount is fine. General rule recorded in memory/john; trigger line in TRIGGERS. A warning did exist beforehand in memory/envoy/webdav but said only "the mount is flaky" with no mechanism, was scoped to one subfolder, and sat in a project note — so it read as superstition and was overridden. A prohibition without its mechanism does not survive contact with an agent that has a reason to doubt it.