Alignment above human level
Human oversight assumes a human can tell good from bad. Above a certain capability line, supervision has to be designed before the loop starts.
The oversight assumption
Every current safety process rests on the same assumption: a competent human can look at the output and tell whether it is acceptable. Review boards, red teams, audit trails, and sign-off chains are all variations of that check.
The check degrades gradually and then stops working. It degrades because output volume outruns review capacity. It stops working when the reviewer can no longer evaluate the claim without doing the same work the system already did.
Three ways supervision fails
- Volume. Review becomes sampling, sampling becomes theatre.
- Circularity. Reviewers start using model assistance to review model output. The assistance is trained on the same distribution it is judging.
- Opacity. The reasoning is correct and unreadable. You can check the conclusion against reality, but not against intent.
What survives
Three approaches get stronger as capability rises, because they do not depend on a human reading the reasoning:
- Formal guarantees. Properties stated in a language a machine can verify, and checked mechanically rather than reviewed by eye.
- Independent verification. Something outside the system — a proof checker, a physical experiment, a second system built by a different team — that can contradict it cheaply.
- Constraint at the boundary. Limiting what the system can affect in the world, rather than limiting what it can think. Permissions, not persuasion.
Above a certain line, oversight is not a review step. It is an architecture decision made before the loop starts.
The governance gap
Governance systems are built around documents: standards, disclosures, impact assessments. Those are all instruments for making human judgement auditable. When human judgement is no longer the binding check, the instruments stop measuring the thing that matters.
The replacement is unglamorous: verifiable claims, independent infrastructure, and permissions systems that do not require anyone to trust a paragraph of prose.
What this means for anyone building
Design the verification path at the same time as the capability path. A system that can prove what it did is worth more than a system that can describe what it did — and it is the only version that can be governed once it is better than its regulators.
