Shortly after midnight on September 26, 1983, warning sirens erupted inside a Soviet military bunker outside Moscow. On a bank of glowing monitors, the system reported the unthinkable: the United States had launched a nuclear strike.
The alert came from a newly deployed early-warning satellite system designed to detect American launches within seconds. According to protocol, the officer on duty, Lieutenant Colonel Stanislav Petrov, was required to report the attack immediately. From there, retaliation would follow.
Everything about the system said the alert was real. The computers confirmed it; warning lights flashed. Yet something felt wrong. Petrov hesitated. If the US were launching a first strike, he reasoned, it would not be five missiles. It would be hundreds.
Instead of reporting the launch, he declared the warning a system malfunction. He was right. The satellite had mistaken sunlight reflecting off high-altitude clouds for missile launches.
History remembers Petrov as the man who saved the world. But the deeper lesson is less comforting: The early-warning system worked exactly as it was designed to. It simply encountered a situation its designers never imagined when they wrote the rules.
That is the central challenge of governing frontier AI: you cannot write a rule for a capability you have not yet imagined. Capabilities appear suddenly at scale—reasoning abilities, strategic behavior, or forms of deception—not because anyone deliberately designed them, but because the underlying system crossed a threshold where those behaviors became possible.
Anthropic’s Responsible Scaling Policy (RSP) is the most serious attempt yet to govern this frontier. But it faces the same structural constraint as every rule-based system confronting emergent complexity.
I. What RSP Actually Is — And Why It Matters
Before examining the limits of our current oversight models, we must understand the mechanics of the most rigorous framework built to date.
Anthropic introduced the RSP to tie model capability growth directly to escalating safety requirements. Think of the World Health Organization’s (WHO) Biosafety Levels: studying the common cold requires a standard lab (Level 1), but studying Ebola requires a maximum-security facility (Level 4). The capability of the pathogen dictates the required security.
RSP applies this exact logic to AI. It ties frontier model development to strict requirements called AI Safety Levels (ASLs). An ASL-2 model requires standard security. But if a model demonstrates ASL-3 capabilities—such as significantly assisting in creating biological weapons or autonomous cyberattacks—development must halt until ASL-3 safeguards are implemented.
What makes RSP radical is that safety is no longer a corporate values statement; it is a structural constraint on the release pipeline. In practice, this means Anthropic’s own researchers must stop and demonstrate adequate containment measures before proceeding — a requirement with no precedent in commercial software development. In banking, this is analogous to capital adequacy requirements: a bank cannot take on massive exposure without proving it holds enough capital to survive a shock. Similarly, under RSP, an AI lab cannot cross a capability threshold without proving its safety architecture can contain it.
Much of today’s AI governance remains performative. RSP, by contrast, has teeth. It is the most serious attempt at self-governance in the history of frontier AI, and it deserves serious engagement. Which means it also deserves serious scrutiny.
II. The Threshold Problem — What RSP Can’t See Coming
Once a system crosses a threshold of complexity, interactions multiply faster than human prediction can track. Frontier AI models exhibit this exact pattern. Biology offers perfect metaphors for this dynamic.
Individual bacteria behave simply, but once their population crosses a density threshold, they coordinate through “quorum sensing.”But once enough bacteria accumulate, they release signaling molecules called autoinducers. When the concentration crosses a threshold, the colony collectively activates new behaviors—toxin production, biofilm formation, or coordinated attacks on host tissue. The capability does not exist at a small scale; it emerges only when population density crosses a threshold.
In artificial intelligence, scale is the equivalent of this density. Small language models cannot do arithmetic. But scale them up with enough compute, and suddenly they can—not because they were explicitly trained to be calculators, but because the capability emerged from deep structural patterns in training data.
The first challenge in governing emergent capability is the Measurement Problem: you cannot regulate what you cannot reliably detect. Emergent behavior structurally breaks this step.
Anthropic’s RSP defines its thresholds in advance, which is its greatest strength and its primary structural constraint. RSP’s framework assumes you can enumerate the risks you are trying to govern before you encounter them. But you cannot write a threshold for a capability you haven’t imagined. This is not a criticism of Anthropic’s intentions; it is an architectural constraint that faces any rule-based framework applied to emergent systems.
The Anticipation Gap
The structural lag between when a new capability emerges in a complex system and when governance frameworks recognize, measure, and respond to it. In systems capable of emergent behavior, the capability almost always arrives before the rule designed to govern it.
The pharmaceutical industry faces this constantly. Even after FDA approval, “herd effects” emerge only when a drug is introduced to millions of unique biologies. The system is simply too complex to anticipate every interaction, which is why governance continues post-deployment through pharmacovigilance—continuous monitoring.
We saw the catastrophic cost of ignoring emergent complexity during the 2008 financial crisis. During the 2008 financial crisis, mortgage derivatives became so opaque that regulators, investors, and even the banks struggled to understand the embedded risks. Regulators weren’t necessarily corrupt; they simply couldn’t see what the system had become. Governance failed because the system evolved faster than the frameworks designed to oversee it.
RSP is a rigorous framework for governing the capabilities we can predict. The question is what happens to the ones we can’t.
III. The Verification Problem — Who’s Watching the Watcher?
Even if we could foresee every emergent capability, we collide with a second hurdle: The Verification Problem. Who is watching the watcher?
RSP is a unilateral commitment. Anthropic sets the thresholds, evaluates its models, and alone decides whether development halts. The entity being governed and the entity doing the governing are exactly the same.
This is the defining characteristic of all current AI self-governance. It means effectiveness is structurally dependent on the integrity of Anthropic’s current leadership. Right now, that is a high bar the company is meeting. But good leadership is a circumstance. It is not a governance architecture.
Look back to the Boeing 737 MAX disasters. For decades, allowing Boeing to “self-certify” its systems appeared to work. The problem was the absence of structural independence in the verification layer. When extreme competitive pressure mounted, the boundary between regulator and regulated collapsed. When the regulator and the regulated become the same entity, oversight becomes indistinguishable from trust.
This is why an “Independent Table“—an oversight body structurally shielded from profit and politics—is so vital. Governance that depends entirely on a company’s current executive team is simply trust, not structure. And trust rarely survives leadership changes, acquisitions, or a decade of compounding competitive pressure.
The question isn’t whether Anthropic is trustworthy today. The question is whether the architecture works if they weren’t.
IV. The Path Forward — AI Accelerates the Inputs, Humans Control the Outputs
Traditional frameworks update on human timescales (legislation, treaties). AI capabilities update on model training timescales. This gap is a permanent structural deficit unless governance can adapt at something closer to model speed.
The cybersecurity industry evolved to meet similar velocity challenges, abandoning the idea that systems could be made safe through static policy. Modern security relies on continuous red-teaming, adversarial probing, and automated anomaly detection—all feeding up to human decision authority. We must apply this continuous-adversary paradigm to frontier models.
We need AI-augmented governance: AI tools accelerating the detection of emerging capabilities so decision-makers work with current information. As interpretability research makes models more readable, automated systems can flag shifting capability profiles long before humans notice them through behavioral testing. Anthropic’s own interpretability research — the attempt to make model internals readable rather than opaque — is the most promising foundation for this kind of automated capability detection.
However, AI accelerates the inputs—detection, flagging, and evidence generation. Humans and structurally independent bodies must retain all authority over the outputs—the ultimate decisions regarding deployment. AI governing AI without human control is not a safety solution; it is an unmanaged risk.
Anthropic’s RSP provides a robust internal architecture, but the verification must be strictly external. It requires an oversight entity closer to the IAEA model—possessing independent authority and technical depth and the mandate to verify, not just observe.
V. Closing
The goal isn’t to slow AI development. It’s to build a governance architecture that can keep pace with it—one where AI tools accelerate our ability to detect risk, and human judgment retains the authority to act on it.
My first essay argued that political cycles are the wrong horizon for AI governance. This essay argues that static, pre-defined policy frameworks are the wrong architecture—because emergent capabilities do not wait for the rules to catch up.
Anthropic’s RSP is the best answer currently in existence. The fact that it has structural limits is not a failure of ambition; it is an honest map of the problem’s sheer difficulty. The path forward is a governance architecture that pairs Anthropic’s internal rigor with strict external verification, using AI tools themselves to close the Anticipation Gap.
In 1983, the safety of civilization depended on one Soviet officer deciding that the machine might be wrong. As AI systems grow increasingly powerful, we cannot rely on moments of individual intuition to save us from systemic failure. Governance must evolve from trust to structure before the next Petrov moment arrives—and before the systems making the decisions are too complex for anyone to question.
