Operational lessons from working sessions. Read this when starting work on infrastructure, playbooks, or tooling — these are things that went wrong and cost time.
Non-critical side-effect tasks in playbooks (note registry updates, logging, awsctl reloads) must have ignore_errors: yes. A failure in one of these must not abort the play and block essential follow-on tasks.
What went wrong: In launch_instance.yml, the "Update hosts note" task failed (HTTP 404 — the hosts note did not exist yet). This aborted the play before the known_hosts tasks ran. The newly provisioned host could not be reached by subsequent playbooks because its SSH key was never added to ~/.ssh/known_hosts.
Rule: If a task's failure does not make the host unusable, add ignore_errors: yes. Essential tasks (disk format, package install, SSH config, key registration) must still fail loudly.
If simplejson is installed, requests uses it instead of Python's built-in json. The built-in silently serialises float('nan') as the bare token NaN (invalid JSON but no exception). simplejson is strict and raises ValueError: Out of range float values are not JSON compliant.
Root cause pattern: a pandas DataFrame built from records where some rows are missing a field (e.g. a court-booking email with no coach field) has NaN for that column. df.to_dict('records') produces dicts where the key exists with value NaN. Calling .get('coach', '') returns NaN — not the default — because the key is present. Those NaNs then fail JSON serialisation.
Fix: sanitise before serialising. In the row-building step use: def _safe(v): return '' if isinstance(v, float) and v != v else v. No import needed — NaN is the only float not equal to itself (IEEE 754).
When a document is stored double-encoded (the GDBM value is a JSON string literal wrapping the document JSON, rather than the JSON object itself), any call to patch fails with "document is not a JSON object". The 'op' is a required property error reported in earlier sessions was a red herring from a different code path; the root cause is always the double-encoding.
This happens when put is called with a string value (e.g. the MCP client serialises the document to a JSON string before passing it) and the server stores json.dumps(string) rather than json.dumps(parsed_dict). The current MCP put handler fixes this via _parse_json_robust: if value is a string it is parsed as JSON before storage. Existing double-encoded docs must be repaired manually: read via CLI, write a corrected JSON file, PUT via curl.
Fix command: curl -s -X PUT http://127.0.0.1:8021/<key> -H 'Content-Type: application/json' -d @/tmp/fixed.json. All patch operations work correctly on properly encoded documents — patch does not need to be avoided.
MSAL raises API does not accept frozenset({'openid', 'profile', 'offline_access'}) value as user-provided scopes if you include offline_access, openid, or profile in the scopes list passed to initiate_device_flow or similar calls. MSAL adds these automatically.
Only pass the resource-specific scope, e.g.:
SCOPES = ['https://outlook.office.com/POP.AccessAsUser.All']
app.initiate_device_flow(scopes=SCOPES) # MSAL adds offline_access internallyThe google-auth library compares token expiry using _helpers.utcnow() which returns a naive (timezone-unaware) datetime. If the stored expiry string is timezone-aware (e.g. 2026-06-25T13:10:49+00:00), constructing Credentials(expiry=expiry) causes TypeError: can't compare offset-naive and offset-aware datetimes when .expired is accessed.
Fix: strip tzinfo after converting to UTC before passing to Credentials:
if expiry.tzinfo is not None:
expiry = expiry.astimezone(timezone.utc).replace(tzinfo=None)SetEnv directives in Apache <Location> blocks populate the per-request WSGI environ dict passed to the application(environ, start_response) function. They do NOT appear in os.environ. Any WSGI helper function that reads config must receive and use the WSGI environ, not os.environ.get().
def _cfg(wsgi_environ, key, default=None):
return wsgi_environ.get(key) or os.environ.get(key) or defaultUse ~/tmp/ on remote hosts, or ./tmp/ relative to the project directory locally. Never /tmp/ (including any agent-default scratch/temp directory that lives under it).
What went wrong: /tmp triggers a permission prompt on every use in some agent harnesses, and is a shared system directory on remote hosts (gravlax uses ~/tmp/gdata-server-github for its own git clone, for this exact reason). Confirmed again 2026-07-08: crontab <file> failed with "No such file or directory" for a path under an agent scratchpad living in /tmp, but worked immediately once copied to ~/. This had already been learned once before but was re-derived from scratch in a different project/session because it was not recorded here.
Building a notes patch/append_block JSON payload inline as a bash single- or double-quoted string breaks down once the content has apostrophes (contractions like "isn't", "gravlax's") or backticks: the classic '"'"' single-quote-escaping trick is easy to get wrong (one attempt failed with Error parsing arguments: Expecting ',' delimiter), and backticks inside a double-quoted bash string get shell-executed as command substitution, silently corrupting the content instead of erroring.
Fix: for any block with more than one or two apostrophes/backticks, write the JSON to a file first (with the Write tool, or notes load -d <key> <file> for a whole document) rather than building it inline in bash. See README/mcp-note-editing for the general safe-edit workflow.
When a batch/patch call replaces the wrong block_ids — e.g. mis-mapping which block held which content and swapping two blocks' text — the fix is to use the reorder op to move the existing blocks back into correct position, not a full-document put. A full rewrite regenerates every block ID and discards the document's edit history for no reason, when the actual fault was just wrong positions/content for a couple of blocks.
Same fault class, different trigger: a stacked insert_after on the same anchor block (two or more inserts targeting one block_id in the same or successive calls) lands the new blocks in reversed order. All content exists, just mis-positioned — so the fix is the same reorder remedy, not a full put. Recurred 2026-07-18 on videos/lamis-mukta-dreaming-memory: fixed via full put reconstruction when a reorder with the correct block-ID permutation would have sufficed. For the general-purpose pattern that avoids this class of mistake altogether, see the delete → append → reorder recipe in gdata-server/troubleshooting.
What went wrong: while editing location-db/hibernate-on-approach (2026-07-14), a batch call used block IDs from a stale mental model of the document (assumed positions rather than re-checking the actual get(include_block_ids=True) output carefully), and ended up swapping the content of two unrelated blocks. Fixed by discarding the document and reissuing a full put — functionally correct, but heavier than necessary and loses block-ID continuity. A reorder (or a second, carefully-checked batch swapping the two blocks back) would have been the right-sized fix.
When comparing a GPS reading against gazetteer/survey points, compute the actual distance to each nearby point first and lead with the closest match. Do not default to impressionistic "between A and B" framing — if the numbers only support one point, say so plainly; don't hedge towards a second point that is actually much further away.
What went wrong: given a reading ~2.5 m from a named net-line fix and ~78 m from a named lounge fix, described it as "between" the two, implying comparable proximity. Corrected once distances were actually checked. See gazetteer/david-lloyd-tennis-bubble for the survey this arose from.