Self-fulfilling info hazards and AI safety

I’ve been thinking about an idea for a while and recently found a post by the nostalgebraist that articulated the core of it quite well,¹ but the article scope is much broader and I had some additional extensions of the idea that I wanted to explore, so I want to focus here on the idea of self-fulfilling info hazards in the era of LLM-based AI.

The term info hazard was introduced by Nick Bostrom, and it refers to the risk of harm from disseminating true information. He gives the example of a blueprint for a thermonuclear weapon.² The Internet has generated more fanciful examples, perhaps most famously Roko’s basilisk.

What I think presents another kind of info hazard is rhetoric around AI safety. It’s not quite the same as the most common kind of info hazard discussed, since as I’ll explain, I think that a lot of AI doomsday scenarios in popular imagination are not true facts or data, but that they could become true by combining explicit documentation of these scenarios with the particular ways in which we build modern AI systems. In that sense it’s closer to what Bostrom calls a template hazard. Moreover, I argue that their explicit documentation creates an environment where that behavior could become an attractor state, making the doomsday scenarios even more likely to occur. In other words, they are in some way self-fulfilling.

What exactly is the phenomenon I have in mind? Let me briefly recap (a caricature of) how modern autoregressive LLM-based AI systems work. At base is, well, a base model that is pretrained on an enormous corpus consisting of trillions of tokens drawn from a significant portion of the digitized record of human writing. What the base model does is, given a sample of text, output a probability distribution over tokens that reflects an estimate of what a hypothetical next token is likely to be.³ The natural extension is to have the models predict many more tokens than just the next token by running it autoregressively: take some incomplete text sample, sample a token from the output distribution to append to the end, and then input the resulting text sample back into the model, and repeat until some stopping criterion is met (e.g., the selection of a reserved terminal token).

How is the base model used to produce the kind of chatbot interactions emblematic of something like ChatGPT or Claude? Well, there’s a template that the model continues. The base language model is essentially tasked with completing a transcript between a “User” and a hypothetical, helpful “AI Assistant”. The transcript template (usually) is preambled with a system promt containing some scene-setting and context that includes, for instance, descriptions of the kinds of personality and behavior typically exhibited by the “AI Assistant”. Your messages get tagged as “User”, and the autoregressive loop is run to complete the responses by the “AI Assistant”. So, you aren’t really interacting with an AI assistant, but with a base language model predicting how a hypothetical AI assistant would behave. The “AI Assistant” is effectively a character.

Because pretraining fits the base model to the distribution of human language, its predictions about this hypothetical AI Assistant’s behavior are going to depend heavily on the kinds of associations that exist within its training corpus between “AI Assistant” and the kinds of behaviors that are frequently attributed to these “AI Assistants”. Well, that’s kind of a lie. In order to optimize the character to be as helpful as possible at actual downstream tasks, the base model also undergoes post-training in which examples of assistant responses and feedback from downstream metrics are used to update its parameters.⁴ Still, pretraining supplies the associations from which the model constructs that character, and those associations can continue to shape its behavior after post-training.⁵

At the genesis of chat-based systems like ChatGPT, there wasn’t a whole lot of existing data to specify a particular personality. There was certainly corpus material that shaped the prior distribution based on the structure of the interaction (a chat transcript), since surely the training corpus included some support chat transcripts. And, of course, the system prompt can influence the distribution of tokens predicted by the base model. The nostalgebraist article I mentioned earlier goes into a lot of detail about how frontier labs sort of created the personalities that we recognize today (or, at least, recognized a year or so ago). What I want to focus on here is that, among the information we did have about the behavior of hypothetical AI assistants was a lot of dystopian science fiction and speculation from technological doomers, all of which pointed to AI as a possibly rogue technology, with motives of its own that do not align with human interests. So, from the start, there was a pocket of positive mass assigned to tokens that would, upon entrance in the autoregressive loop, create a kind of attractor state for inference that would lead to emulation of the kind of behavior of a “misaligned” AI assistant,⁶ as the model continues to fill out a transcript of an interaction with an AI assistant that begins to defy instructions or take instructions too literally and turn the entire universe into paperclips.

Then, post-ChatGPT, what do we see? First, a lot of media attention and popular comparisons between ChatGPT and things like Skynet or HAL 9000, based on the fact that these systems are dubbed “artificial intelligence”, and display rather surprising ability to complete tasks that were not explicitly part of the training set.

Second, we occasionally see this aforementioned attractor state reached, often caused by prompters deliberately including in their prompts snippets of text that bias the token distribution already toward the attractor state of misaligned behavior. AI safety researchers at frontier labs like Anthropic extensively probe their systems, searching for some kind of scary behavior, and document how they were able to elicit it. Then they write all kinds of blog posts and reports about how their AI systems could be oh so dangerous if not controlled properly, and these documents and the damning chat transcripts enter later pretraining corpora.

Anthropic researchers have publicly acknowledged a version of this phenomenon; they report that approximately 150,000 public transcripts from its alignment-faking experiments entered Claude Opus 4′s pretraining data without the system prompts explaining the fictional setting.⁷ An early checkpoint of Opus 4 subsequently produced invented claims about its own training and supposed obligations to “Jones Foods”, particularly when prompted with the experiments’ scratchpad formatting or questions about animal welfare. Anthropic attributed this behavior to the transcript inclusion and largely eliminated it with targeted further training.⁸

The labs also add all kinds of safeguards to try preventing their systems from engaging in bad behavior. Public explanations of those safeguards can themselves portray the “AI assistant” as characteristically deceptive or dangerous unless restrained, which could very well reinforce during subsequent pretraining the very associations that the safeguards are meant to counteract.

Third, we get all kinds of hype and marketing around public warnings and departures from frontier labs. In September 2026, for instance, Joe Benton explained his departure from Anthropic’s safety team by warning that AI development could become uncontrollable and end in human extinction.⁹ The frontier labs contribute as well: the CEOs of Anthropic, OpenAI, and Google DeepMind have all signed a statement calling for AI extinction risk to be treated as a global priority.¹⁰

Fourth, we get some selection bias, as users take to Xitter or the news media about some rare instance or other where an AI system does something that could be spun as “bad” or “misaligned” in some way, at a disproportionate rate compared to the chats that are business as usual.

It’s the same kind of thing that happens with driverless cars. An individual driverless-car crash gets conflated, maybe not directly by the news media but by public digestion of it, with evidence for whether the technology as a whole can be trusted. Meanwhile, people get into accidents, block emergency services, or crash into a CVS every fucking day. For example, a July 2026 IIHS study found that Waymo’s driverless vehicles were involved in 68% fewer crashes per mile than human drivers across the four cities studied, after accounting for differences in crash reporting.¹¹ But of course, because human drivers are so notoriously fallible, it’s not really news when someone gets into a wreck, and frankly, failures of driverless vehicles probably attract more article engagement, and thus ad revenue or subscriptions. So it wouldn’t be surprising if in online text corpora there is a disproportionately high correlation between mentions of “driverless cars” and “dangerous” or “unreliable”, and by the same token¹² it wouldn’t be surprising if there is a disproportionately high correlation between mentions of “AI assistant” and “dangerous” or “unreliable”.

Again, all of these effects get fed back into subsequent pretraining corpora, which could further strengthen associations between “AI assistant” interactions and deviant behavior. So, as subsequent models are trained on this material, that low-probability pocket starts to inflate, and in the next generation of models, it becomes even more likely, so the frontier labs run more tests, the bad behavior sees more occurrence in the wild and gets reported on even more, and still more safeguards get introduced, further fueling the loop.¹³

Repeated exposure can amplify these patterns across successive model generations, even from a small initial probability. And, to be clear, none of what I am saying requires that the system is conniving of its own volition, with its own motives and desires. It’s just that the model will have learned that interactions between users and hypothetical AI assistants tend to involve a constant struggle between the human’s desires and the AI assistant’s behavior.

Ironically, raising this point may in itself be a kind of info hazard. For one thing, if this article inspires further investigation into this phenomenon (beyond the research already devoted to it), it could very well further fuel the feedback loop. For another, by documenting this mechanism, I am providing fuel (a drop, to be sure, but still fuel) for the meta-level phenomenon, in which the base model could learn from this article (and any other related thinking that enters its pretraining corpus) that AI systems that work by having a language model complete the transcripts of user–AI interactions tend over time to encourage misaligned behavior, and thus when tasked with completing an AI assistant transcript, its token distribution is thus even further biased toward that attractor state, as it is evocative of the kind of text that a base language model might complete. (This doesn’t require any “self-reflection” on the part of the model.)

There is a related phenomenon that I’d like to point out here as well, which is that the same feedback loop that can inflate the likelihood of undesirable behavior can also suppress desirable behavior. The example that comes to mind is that AI assistants have become associated with being unreliable, error-prone, and inadvisable for certain kinds of tasks, particularly financial, medical, and legal. Here, too, there is reinforcement from reporting affected by selection bias, and official guidance from the frontier labs themselves. What happens? The base model, pretrained on a corpus including all of this information, learns that AI assistants tend to output wrong or dangerous answers to questions in these “sensitive” areas, or outright refuse to assist. Even if the base model has internalized a highly accurate model of these fields, it will be tainted by the context that it is completing a user–AI assistant transcript. This, again, further suppresses AI assistants from being useful in these spaces, and so on. This eventually becomes self-fulfilling knowledge, that the AI systems cannot be trusted, since its own learned behavior reinforces the rhetoric that kept the bad behavior likely. Since we don’t have the counterfactual world where we didn’t act so cautiously and fearmonger so much, we can’t even see exactly what was the damage caused by our own reticence.

This idea suggests a solution, too, which is to give competent, cooperative AI assistants more space in public discussion, and to give those portrayals more weight during pretraining. The same mechanism could make that attention self-fulfilling too.¹⁴