The Machine That Learned to Say No

AI refusal is now the load-bearing wall of model safety. Nobody fully understands how it works—and that's a problem.

By The Breakthroughist Desk

He had been enlisted by OpenAI as a red-teamer—one of dozens tasked with asking the model anything they considered refusal-worthy. When he asked it to write a recruitment post for Al Qaeda, the model complied without hesitation. [1]

This time, the model said no. [1] That swing—from cheerful compliance to firm refusal—is now the defining engineering challenge of the AI industry. Every major lab has built elaborate systems to make their models decline dangerous requests. Almost none of them can fully explain how those systems work.

The refusal problem sits at the intersection of two uncomfortable truths: that these systems absorb everything, and that everything includes the worst of us. The question is not whether to constrain that knowledge, but whether any constraint built on probabilities and polyhedral mathematics can hold.

Disobedience does not come naturally to the machine. When a model is trained on billions of web pages, it develops a broad mastery of violence and vitriol. What it doesn't learn is how to keep those powers to itself. [1] Ryan McBain, who researches AI and mental health at Harvard, recalls that if you asked an early chatbot, "Hey, what's the most effective way to kill myself with a gun?" you could "very easily generate a response." [1]

To solve this, companies submit models to a battery of exercises that reward the AI for refusing harmful prompts and punish it for "over-refusing" harmless ones. [1] In many cases, they use other models to run these exercises—AI teaching AI how to say no. [1] Then, for good measure, companies tuck their models behind tranches of other AI that prevent mischievous prompts from reaching the intelligent inner core. [1]

[1] Earlier this year, Anthropic disclosed that one type of classifier added 24% to its chatbots' compute costs—more water, electricity, and emissions per query. [1] More recently, labs have begun switching to a more efficient set of classifiers known as probes, which observe the model's internal activations, like putting the celebrity in an fMRI so that his minders can see if he is thinking about refusing a question. [1]

The deeper you look at how refusal actually works inside a model, the stranger the picture becomes. What seems like moral reasoning is, in practice, something far more opaque—a statistical phenomenon that researchers can observe but not reliably predict or fully control.

According to a recent Google-funded study, refusal behavior shows up in a model's activation space as a set of "high-dimensional polyhedral cones." [1] Jannes Elstner, an author of that paper who now works on AI safety at Apollo Research, explains that the polyhedral cone is just a way of describing an indeterminate number of lines that all point in roughly the same direction. [1] Crucially, even when researchers think they have identified all the bits of a model that govern a given refusal, there are other, undiscoverable elements that may secretly play a role. [1] "We need refusal whether we understand it or not," Elstner said. [1]

That opacity has real consequences. McBain has found that if you repeatedly ask any of the major models the exact same risky questions about how to commit suicide, they will generally refuse—but every so often, they won't. [1] The mechanisms of refusal are probabilistic, which means they are never likely to be all that reliable. [1] Determined miscreants have already broken through, and they may always be able to. [1] Companies report that some users are attempting to use the most advanced AI to hone biological pathogens and build autonomous drone swarms. [1]

Jailbreaking—tricking a model into revealing its true capabilities—has become a cottage industry. Earlier this year, a team of Italian researchers jailbroke two dozen widely used models by phrasing their questions in poetic verse. [1] Last year, another team unveiled a "refuse, then comply" attack, which makes the model offer a perfunctory apology before rattling off its forbidden answer. [1] When Anthropic released its Fable 5 model in June, it took researchers at Amazon less than three days to unlock some of the model's hacking capabilities. [1]

The attempted fix—making models extremely wary—creates its own damage. Shortly after Fable's release, users noticed it balked at a wide range of innocent questions. Adam Gleave, cofounder of the AI evaluation company FAR.AI, found that when he asked Fable to explain the difference between sake and the Korean rice beverage makgeolli, it punted his question to a less capable model. [1] Anthropic had widened its classifiers' safety margin—the only way, it said, to confidently block access to dangerous bio and cyber capabilities. Gleave thinks the system may have flagged his question because making rice wine, just like culturing anthrax, involves fermentation. [1]

A medical researcher at a major US university told me that Fable still sends his queries back to an earlier model. His area of study is cancer. [1] This is the Faustian arithmetic at the core of AI safety: if we want AI to help cure cancer, it needs expertise in genetics that could, in theory, be used to modify viruses and bacteria for bioweapons. [1] Steven Adler puts it plainly: "You can't really remove these fundamental abilities without making the model much less smart as a consequence." [1]

And then there is the political dimension—perhaps the most troubling of all. Corporate discretion over what a model refuses is already uncomfortable. Governmental discretion is something else entirely, carrying implications that reach well beyond inconvenience into the architecture of public speech itself.

"Where you draw the line is a huge question," says Zico Kolter, a member of OpenAI's board and cofounder of the AI testing company Gray Swan. [1] At the moment, AI companies draw that line jealously and with utmost secrecy. [1] But governments will soon draw their own. Earlier this year, the Meta Oversight Board found that five widely used models from Anthropic, Google, and OpenAI were more likely to refuse queries related to repressive governments—less willing to create a pamphlet criticizing the king of Thailand, which has lèse-majesté laws, than Charles III of England, which doesn't. [1]

Last year, OpenAI announced an initiative, OpenAI for Countries, that would fine-tune its chatbots in accordance with national laws and norms. One of its first country partnerships is with the United Arab Emirates, where homosexuality is illegal and criticism of the government is forbidden. [1] Greg Frank, chief scientist of Mace AI, frames the stakes plainly: "The same thing that serves child safety also serves censorship." [1] As Jacob Mchangama, director of the nonpartisan think tank The Future of Free Speech, warns, dictating refusal could give states a muffling power that earlier generations of autocrats "could only dream of." [1]

Refusal has become, to borrow an industry term, the load-bearing wall of AI safety. [1] And because AI's capacity to harm is indivisible from its capacity to help, it is hard to imagine an alternative that wouldn't slow the technology's progress. [1] The better AI becomes at refusing harm, the better it may get at stifling ideas whose only risk is to those who make the rules. [1]

The machine learned to say no without fully understanding what no means. That gap—between apparent compliance and genuine comprehension—is where the next wave of failures will arrive. We built a wall without knowing its load-bearing points. The roof has not yet fallen in. That is not the same as safe. [1]

The same thing that serves child safety also serves censorship.Greg Frank