Design remote integration tests so interruption or failure leaves a deterministic recovery path. The recovery mechanism is part of the test, not an afterthought.
Recovery invariants
- Complete and verify the backup before the first mutation. A system that has been changed must always have a usable pre-change snapshot.
- Keep the raw, authoritative pre-change inventory. Derived or filtered inventories are disposable and should be regenerated from the raw record when comparison logic changes.
- Run drift checks before restoration mutates anything. A false-positive refusal is safer than overwriting state whose origin is uncertain.
- Restore through same-filesystem temporary files followed by atomic rename where practical, so observers see either the old complete file or the new complete file.
- Mark restoration complete only after comparing the restored state with the authoritative baseline.
- Keep cleanup separate and gate it on the restoration-complete marker. Do not delete recovery material merely because a test command ended.
- Make teardown rerunnable: a partial teardown should be able to converge safely when run again.
Concurrency and scheduling
- Use a host-level lock that covers all test-cycle run identifiers. Per-run state directories alone do not stop two test cycles from treating each other's candidate state as a baseline.
- A test-cycle lock does not stop normal schedulers or services. Choose a quiet scheduling window, temporarily coordinate the scheduler through an approved process, or design the test to tolerate concurrent legitimate changes.
- Before mutation, record scheduled jobs and check for relevant processes already running. Recheck immediately before the critical phase if the preflight can become stale.
Evidence and tooling
- Record source revision, working-tree state, file checksums, ownership, permissions and timestamps needed to prove what was tested and what was restored.
- Parse structured or tab-separated manifests by field. Do not assume a pathname occupies the remainder of a line.
- Checksum packages at each transfer boundary, including the final downloaded artifact; browsers and transfer tools may cache or rename downloads.
- Provide a checksum-verified helper-refresh path for correcting recovery logic without replacing the captured payload or starting a new run.
- Package evidence separately from backups and working copies so secrets, keys and other sensitive payloads cannot be exported accidentally.
- Have an independent read-only reviewer challenge the recovery invariants and identify live-state assumptions before a canary run.
Shell and cross-platform traps
- With Bash nounset mode, avoid assigning a local variable from another variable declared in the same local statement; split the declarations so the first assignment is complete before expansion.
- When Git Bash invokes Windows executables, disabling path conversion protects remote Unix paths but also prevents conversion of local drive-style paths. Prefer a relative local filename from its directory, or handle local and remote path conversion separately.
Operator interface
- Prompts should state the actual action and accept a simple confirmation phrase. Do not make an operator reproduce internal state tokens unless that token materially protects the action.
- Always provide quick read-only status commands showing the active run, markers, logs, backup verification and current drift result.
- Treat an interrupted controller as expected. The target-side state and recovery instructions must be sufficient without the original controller process.