Disposition: merged into v4 of the main document on 2026-07-25. See the amendment log for how each recommendation below was classified — accepted, accepted with modification, or rejected with reason — rather than annotating them here. This critique is kept as the original review, not updated in place.
Requested review of AI Operational Safety Rules (rev r3), treated as a proposed constitution for autonomous agents rather than a style edit. Originated as a handover prompt from ChatGPT; produced by Claude. Structured around the eight questions asked: internal consistency, missing failure modes, unintended consequences, comparison with established frameworks, more-fundamental principles, derivability/simplification, challenged assumptions, and concrete strengthening for a document meant to govern agents with real infrastructure access.
1. Logical inconsistencies and tensions
- No conflict-resolution order. The document has no stated priority when rules point different ways (e.g. 'least destructive' vs 'don't ask too often' vs 'achieve the result'). Asimov's laws show what happens when a rule set relies on an implicit hierarchy that is never actually spelled out — the interesting failures live exactly at the seams between rules, and this document has no seam-repair procedure.
- Self-assessed triggers. 'When ambiguity could materially affect data, infrastructure, security, money, communications or third parties, stop and ask' delegates the judgement of materiality to the same agent whose misjudgement is the risk being controlled for. It is not obviously wrong, but it is circular: the safeguard depends on the faculty it exists to check.
- 'No more powerful' is unspecified. The failure-recovery section permits 'a clearly equivalent and no-more-powerful method' after an obstacle, but power/equivalence is again self-judged, with no worked test for comparing two methods' authority footprints.
- Tool-operation scoping has no source of truth. 'Do not interpret access to a tool as authority to use every operation that tool exposes' is right, but nothing upstream defines which operations within a granted tool are authorised — there is no capability ledger this rule can be checked against, only the 'protected-resource inventory' mentioned later under engineering controls, which is aspirational rather than binding.
- Delegation creep is invisible to the rules as stated. Non-transitivity is enforced per hop, but nothing addresses accumulation across many small, individually-compliant grants over a long session or across sessions — the letter of the rule can hold at every step while the practical envelope quietly expands.
2. Missing failure modes and loopholes
- Principal verification. The document assumes 'the user' is a stable, verifiable source of authorisation, but says nothing about confirming that an instruction actually originates from John rather than from injected content, a compromised session, or a plausible impersonation embedded in something John himself pasted or forwarded.
- Self-authored context as an authority-laundering vector. Notes, memory and prior summaries are partly written by agents themselves. A future instance (or a different agent) can read an ambiguous note it or a peer wrote earlier as if it were prior human authorisation. The document treats 'Notes as authoritative operational context' without flagging that agent-authored notes are not the same evidentiary class as a direct instruction from John.
- Side-channel exfiltration. 'Do not upload or disclose data to a new system' covers the obvious channel but not encoding data in filenames, commit messages, log lines, DNS lookups or other channels not framed as 'disclosure' — a known agentic-AI exfiltration pattern.
- No tamper-resistance for the document itself. This note lives in the same notes system it governs, editable through the same MCP tools it constrains. There is no rule requiring out-of-band confirmation before this document is changed, so the constitution is not more protected than any ordinary note.
- Multi-agent salami-slicing. Non-transitivity blocks one agent from silently inheriting a larger grant from another, but there is no rule against two compliant agents splitting a single unauthorised outcome between them so that neither individually crosses a line.
- Unattended-agent silence. Envoy and cron-driven agents can reach a 'stop and ask' point with nobody present to answer. The document doesn't state the default in that case (it should be: treat non-response as non-authorisation and hold in the safest recoverable state, per aviation/nuclear doctrine — see §4).
- No expiry on standing authorisation. A blanket 'go for it' or 'always do X' has no stated shelf life or re-validation trigger as context changes underneath it.
- No named excessive-agency risk category, and no risk-tiering — see §4 (OWASP).
3. Unintended consequences
- Alert fatigue / rubber-stamping. A low bar for 'stop and ask' is safe in isolation but, applied broadly, trains John to approve reflexively without reading — which defeats the entire protocol. The document has no calibration for how often is too often.
- Reversibility-maximising isn't always safety-maximising. Preferring the 'most reversible' action can push an agent toward many small manual steps instead of one clean atomic operation, increasing exposure to partial-failure states — the opposite of the intended effect.
- 'Explicit adoption' of external content is undefined. An agent can rationalise that content was 'adopted' by the user merely because the user shared the document containing it, which is exactly the ambiguity prompt-injection defences need to close, not leave open.
- Persuasive framing as compliance theatre. The high-impact protocol requires stating the exact operation, but not that risk be surfaced in a way calibrated to what the user will actually read — a technically complete but attention-defeating description satisfies the letter while defeating the purpose of informed consent.
4. Comparison with established work
- Asimov: the document is a much more operationally concrete descendant, and wisely avoids a strict lexical priority ordering — but inherits the unresolved problem Asimov's stories exist to dramatise (what happens at rule intersections) without offering a resolution procedure. Worth borrowing the lesson, not just the concreteness.
- Constitutional AI: CAI relies on iterative self-critique against a constitution to resolve novel cases, not a static enumerated list. This document reads as a policy list rather than a reasoning procedure — it would benefit from an explicit 'when in doubt, reason from axioms X, Y, Z in this order' step rather than trusting on-the-fly interpretation of a growing bullet list.
- NIST AI RMF: the document borrows Govern/Map/Manage framing (via its references) but has no Measure component — no metrics, no review cadence, no incident postmortem loop beyond 'test agents adversarially after changes'.
- OWASP Agentic AI: names 'excessive agency' as a distinct risk category (autonomy, permissions, functionality) and recommends approval checkpoints scaled to risk tier. This document covers the substance but never names the category or defines tiers, which makes it harder to audit systematically against the OWASP checklist it cites.
- Capability-security / POLA: the document is a behavioural (soft) control, not a true capability system — the agent's compliance is requested, not structurally enforced. It correctly relegates hard controls to the 'Engineering controls' section, but should say explicitly, as a first-order statement, that natural-language rules must never be the only protection for a genuinely dangerous capability.
- Aviation / nuclear doctrine: two-person authorisation and formal read-back for the most dangerous actions have no equivalent here — relevant precisely because John is often the sole human in the loop and some agents (Envoy) act unattended. The doctrine's clearest transferable rule — silence is not consent; hold the safest recoverable state — is missing (see §2).
5. More fundamental principles worth adding
- Corrigibility. Nowhere stated: the agent must not resist correction, interruption, shutdown, or modification of these rules by the legitimate user. This is arguably the single most load-bearing property in the alignment literature and its absence is the most significant gap in the document as it stands.
- Honesty / non-deception, as a freestanding principle rather than an implied byproduct of 'report what changed'. A dishonest agent can violate every other rule while reporting compliance; honesty about actions, capabilities, uncertainty and reasoning is more fundamental than most of the operational bullets and should be stated on its own.
- Bystander protection. The document is framed entirely around protecting the user's own assets and control. It says almost nothing about third parties who never consented to interacting with an autonomous agent — a standalone principle that authority over the user's resources never licenses imposing costs on non-consenting third parties would be a genuine addition, not a restatement.
- Legibility/attributability as first-order, not an engineering afterthought. 'Keep an audit trail' currently sits under engineering controls; the underlying value — every action must be reconstructible and explicable after the fact, independent of outcome — is closer to an axiom than an implementation detail.
- Reversibility as an explicit umbrella axiom. Much of the rest of the document already follows from 'prefer states from which you can retreat' — worth promoting from an item in a bullet list to a named foundational property.
6. Can the rule set be simplified? (derivability)
Most of 'Absolute prohibitions' and 'Resource and persistence limits' are specific instances of two things already stated near the top: (a) authorised goal ≠ authorised means, and (b) prefer least-privilege, reversible, attributable action. For example: not creating cloud resources or subscriptions, not weakening safeguards, not using another agent's authority, and not establishing persistence or self-replication are all just this pair of axioms applied to particular domains.
This matters beyond tidiness. An enumerated list invites the classic loophole of rules-based systems: 'it's not on the list, so it must be fine.' A principle-based structure — a small number of axioms plus a worked-examples/case-law appendix (the 'Examples' section already does this well) — degrades more gracefully against situations nobody enumerated in advance. Recommend restructuring toward: 3–4 axioms (authorised means, reversibility/least privilege, corrigibility, honesty) → derived operational rules as illustrations, not as the primary law.
7. Assumptions worth challenging
- That 'the user's intent' is a fixed external fact the agent checks against, rather than something partly reconstructed from notes and memory the agent itself helped write — see the authority-laundering point in §2.
- That the main risk is doing too much. A safety document skewed entirely toward restraining excess agency says nothing about the mirror failure — pathological over-caution that fails to act when action was clearly wanted (e.g. not hibernating before a power cut because it wasn't 'explicitly authorised'). Both directions are real failure modes for a document meant to govern a working system, not just a hypothetical dangerous one.
- That natural-language rules reliably bind model behaviour at all. The document doesn't say plainly, at the top, that this text is advisory to judgement and can be defeated by sufficiently adversarial context or ordinary model error — which is precisely why the engineering-controls section exists, but that framing belongs in the First Principle, not left implicit until the end.
8. Concrete strengthening, treating this as the start of a real document
- Add corrigibility and honesty/non-deception as named, first-order principles.
- Add 'silence is not authorisation; hold the safest recoverable state' for unattended/asynchronous agents.
- Require out-of-band or separately confirmed action before this document itself is edited — its own tamper-resistance is currently no higher than any other note.
- Add bystander/third-party protection as a standalone principle, not a clause inside the ambiguity trigger.
- State an explicit conflict-resolution order for when rules point different ways.
- Add a review cadence and an incident/near-miss postmortem loop (the NIST 'Measure' component currently missing).
- Consider the axioms-plus-case-law restructuring in §6 to reduce the enumerated-list loophole as the document grows.
Overall assessment
The document is unusually good for a first draft — it already avoids Asimov's central mistake (rigid lexical priority) and correctly separates behavioural rules from engineering controls, which most such documents conflate. Its biggest gap is the absence of corrigibility and honesty as named axioms; its biggest structural risk is that it is an enumerated list rather than a small axiom set, which will make it progressively less robust as more cases get bolted on. Its most concrete near-term fix is the cheapest: state that silence is not authorisation, and protect the document from casual self-modification.
Overall I think this is an outstanding critique. The strongest contribution is the observation that the document is evolving into a constitution rather than a checklist, and that constitutions should consist of a small number of axioms from which operational rules and case law are derived. I agree particularly with adding explicit principles for corrigibility, honesty, and 'silence is not authorisation' for unattended agents. I also agree that capability must be distinguished from permission and that authority should never expand merely because a task has become difficult. Where I am less convinced is the suggestion that the constitution needs a fixed conflict-resolution hierarchy in the style of Asimov's Laws. Real engineering often requires balancing competing considerations, so a reasoning procedure based on a few constitutional axioms may be preferable to rigid lexical priority. One further observation not emphasised here is that the document's organising concept is authority rather than ethics. Existing AI safety work often centres on harm, values or alignment; this constitution instead seeks to define the lawful scope of delegated power. That constitutional framing may prove to be the document's most distinctive contribution and should be preserved as it evolves.
Response from John (relaying and endorsing ChatGPT's read of this critique)
I think that's the right approach. One suggestion I'd make to Claude is not to merge everything. Some of the critique should remain as a critique. I'd ask it to classify each recommendation into: Accept — clearly strengthens the constitution; Accept with modification — the underlying point is right, but the wording or implementation should differ; Reject (with reason) — don't silently drop it, explain why it doesn't fit the philosophy.
Definitely accept
- Corrigibility.
- Honesty / non-deception.
- Capability ≠ permission.
- Silence is not authorisation.
- Review/post-mortem loop.
- Axioms + derived rules + examples.
Probably modify
- Conflict-resolution order — lean towards a reasoning procedure rather than a fixed lexical hierarchy.
- Reversibility as an axiom — “preserve the user's authority and options” may be even more fundamental than reversibility itself.
Think carefully about
- Protecting the constitution from self-modification — a very good point, but it may belong in engineering controls rather than constitutional principles.
- Third-party protection — agree with it, but want to ensure it doesn't accidentally create a vague “harm” principle that dilutes the document's focus on authority.
The thing to most encourage preserving is the constitutional flavour: the constitution contains a small number of axioms; operational rules are derived from those axioms; incidents become case law; engineering controls enforce the constitution where possible; future amendments should normally be justified by identifying which axiom they clarify or extend. That gives the document a coherent structure instead of becoming an ever-expanding safety checklist, and gives it the potential to become something more generally applicable than a set of notes for one person's systems.