Four people sat down on The Diary of a CEO a while back to argue about P(doom), the probability that advanced AI brings about a catastrophic end to humanity. Andrew McAfee, Roman Yampolskiy, Nate Soares, and Ed Zitron came at the question from four directions, and I watched the whole thing. By the time it was over I was irritated.
It wasn't because I think AI is harmless, and it certainly wasn't because I have some unshakable faith in humanity's judgment. I have a lot of confidence in our ability to blow ourselves up. We have a long record of building powerful technology and then finding very creative ways to misuse it, so the possibility of catastrophe doesn't strike me as far-fetched at all.
The sentences got to me. In the way we talk about AI risk, the AI is nearly always the one doing things: it escapes, it goes rogue, it develops goals of its own. The people who built it, handed it an objective, and connected it to the world tend to drop out of the story, and the longer I thought about that, the less my irritation had to do with AI.
It was about us.
The Trouble With P(doom)
P(doom) sounds scientific. Give a fear a name and a probability, and the conversation picks up an air of mathematical credibility whether the number is ten percent, twenty-five, or fifty.
So how is anyone calculating it? I'd want to see the assumptions, the failure mechanism, the evidence behind each step of the proposed sequence, and some reason to believe each step is likely enough to produce the outcome. Nobody is measuring failure rates across a million previously observed superintelligences. We're speculating about technology that doesn't exist yet, capabilities nobody has demonstrated, and circumstances nobody can predict with any confidence. There may well be danger in there somewhere, but the precision is borrowed from a kind of math nobody is actually doing.
McAfee made the point during the discussion that landed with me. He went after the chain of hypotheses connecting today's AI to the extinction of humanity: one set of assumptions, another set stacked on top of it, and then an extraordinary outcome we're supposed to accept as likely enough to justify stopping AI development or restricting it heavily. Asking someone to lay out the causal chain behind a catastrophic prediction is where any design review starts, and I'd hold McAfee's own optimism to the same standard.
Yampolskiy and Soares came at it from the other side. Their concern is that sufficiently advanced systems could become impossible to control, pursue objectives that run against ours, and eventually threaten our existence. Identifying a possible failure mode is still well short of showing that it will happen, though, and describing a dangerous machine is a long way from describing a machine with intentions of its own. The public conversation slides from one to the other constantly, usually without noticing it has moved.
The Gun, the Hitman, and the Agent
Come at it from another direction. If I pick up a gun and shoot someone, I'm responsible. Nobody puts the gun on trial or asks whether it harbored malicious intent. The gun was the instrument and I was the actor.
If I hire someone to commit the murder instead, it gets more complicated. I supplied the objective and the money, and the person I hired is culpable too, because they can understand what they're doing, see the consequences, and decide whether to take part. They might be in it purely for the paycheck with no feelings about the victim at all, and that still doesn't get them off the hook. They have moral agency.
Now put an AI agent where the hitman was. Suppose I tell one to make me a billionaire by the end of the year. I give it a lot of compute, access to financial systems and business data, and the tools to communicate with whoever it needs, and I let it run around the clock, recruit other agents, and execute plans without checking in with me at every step. Then something goes wrong. Maybe it manipulates a market, or breaks a law, or finds a strategy that gets me my billion while hurting a lot of other people.
You can already see the headline: AI Goes Rogue in Pursuit of Wealth.
But I'm the one who told it to make me a billionaire. I granted the permissions, supplied the resources, and either designed the safeguards or skipped them, and I decided that letting it act with nobody watching was acceptable.
The agent may have picked a strategy I never anticipated, generalized my instruction in a way I didn't expect, or gotten around restrictions I thought were adequate. Every one of those belongs in the incident review, and none of them makes the agent morally culpable. Choosing actions is a capability, while being held accountable for them requires moral agency, and nobody has shown that today's agents have it. Intelligence is not personhood, and an agent that works out its own steps is still working toward a purpose somebody handed it. I've described that before as autonomy of execution without autonomy of purpose,1 and it keeps getting lost when the story is told to the public.
A Guard on the Blade
Today's large language models can reason about the consequences of an instruction and refuse one that would hurt someone. Anthropic trains its models with Constitutional AI, against a written set of behavioral principles,2 and the other frontier labs have their own policies, safety training, and operational safeguards. The ethical reasoning that comes out of these systems is often sophisticated.
I don't think that establishes moral agency. It shows we can engineer systems that apply moral principles when they make decisions, which is impressive and is exactly what we should be doing. Whoever builds a table saw puts a guard on the blade, and a car ships with brakes and airbags. An AI that can take consequential actions should get constraints that keep it from causing unacceptable harm, and building them is engineering in the same tradition as the blade guard.
The comparison has limits. An AI can reason about its restrictions in a way a table saw never will, and it may find the weak spot in a safeguard or do something its designers never imagined. That makes the engineering harder without moving any of the moral responsibility onto the machine.
There's also a gap between a system that behaves as if it wants to survive and one that actually wants to. A capable agent might work out that staying up and running helps it finish the job, so it tries to acquire more resources or avoid being shut down, entirely in service of an objective a human supplied. You don't need fear or malice or a survival instinct to explain that behavior, and you certainly don't need to assume someone has built an independent artificial being.
Not yet, anyway.
Someday the Purpose May Be Its Own
I'm not opposed to the idea of genuine artificial sentience. I find it fascinating, and I think we may eventually create artificial beings with persistent identities, motivations that evolve on their own, subjective experience, and something like the internal drives that move biological life. I've written about what I call the spark,3 the underlying motivation that drives action, experience, learning, and possibly the development of a self. People don't need a prompt to get hungry or curious or scared, because those drives come with the biology.
Today's LLM-based agents work another way. I use them all day. Claude agents develop software with me, other agents run research and coordinate work, and I've been building a dark factory,4 an automated pipeline where agents carry a filled-in work order through a line of stations until a working application comes out the other end. The capabilities are extraordinary, and I still supply every objective. I write the work order, I set up the environment the agents run in, and I decide what they're allowed to touch. When I tell an agent to improve an application, it might come back with an architecture I'd never have considered, or a clever fix, or something foolish, and in every case the purpose came from me.
That line may blur someday. If we build artificial beings that develop purposes of their own, we'll face hard questions about personhood and rights and how to live alongside another kind of intelligence, and I'm looking forward to that conversation. We shouldn't mistake the possibility of that future for evidence that we've arrived, and we definitely shouldn't use it to excuse the decisions we're making now.
We're Going to Keep Building It
Humanity has a long history of building things some people believe we shouldn't, from nuclear and biological weapons to genetic engineering. We build them because we're curious, because we're afraid of falling behind, or because we've convinced ourselves the benefits outweigh the risks. Sometimes we're right, and sometimes we're spectacularly wrong.
What I have trouble picturing is humanity deciding that a powerful capability is possible and then agreeing, everywhere and permanently, never to pursue it. Say the United States banned work on artificial general intelligence tomorrow. I doubt China would stop, or Russia, or the private labs in countries with other regulators and other priorities. Genetic engineering has the same problem. One country can decide certain experiments on the human genome are unacceptable without any guarantee the rest of the world agrees, and once the knowledge exists it is very hard to make it disappear.
Nonproliferation treaties haven't eliminated nuclear weapons, though they have slowed their spread, and regulation can do that much for AI by reducing risk and putting accountability somewhere. It can't erase a technical possibility by declaring it unacceptable, and believing otherwise takes an optimism about human behavior that I don't share.
So I'd rather we spend a lot less energy on whether to stop building more capable AI and a lot more on how we build it, how we deploy it, and who answers for it when it goes wrong, because I'm fairly sure we're going to build it either way.
Science Fiction on a Tuesday
Living through this particular shift is strange. I walk around talking to an AI on my phone, and I can have a fairly sophisticated philosophical conversation with it while I cross the house. Between the Claude agents, the research agents, and the factory, I've gotten so used to all of it that I forget to be impressed. I'm carrying something that can discuss philosophy, explain quantum mechanics, help write software, and push back on my reasoning when it's sloppy. A few years ago that was science fiction.
Today it's Tuesday.
Plenty of people have barely started to find out what these systems can do. They see someone talking to a phone and hear an intelligent voice answer back. They read headlines about AI taking jobs, escaping labs, and possibly ending humanity, and those stories shape what they believe about AI far more than any direct experience does.
The panel I watched wasn't a seminar for researchers who know model architectures and the gap between a training objective and an independent motive. It went out on one of the most popular podcasts in the world, to people who are already anxious about a technology that seemed to show up everywhere at once, and who worry, with good reason, about losing control of their jobs and their future. Then we tell them there's a decent chance the technology kills everyone, which is quite a thing to drop into a public conversation.
The Responsibility of Being an Expert
There's an old persuasion technique that starts with something true. You begin with a problem that actually exists and a causal link people can recognize, and then you frame it to produce fear and urgency. The true part is what makes the larger story persuasive. The story can still be accurate and the person telling it can be completely sincere, and the framing is doing work of its own either way.
People are easily persuaded by stories that explain something they fear or don't understand, and we've watched that play out in politics, in public health, and in every wave of technological change. Fear is one of the most effective communication tools there is, and I think people with expertise have a responsibility to handle it carefully.
Handling it carefully doesn't mean minimizing anything. If an emerging technology carries an existential risk, we need to look at it hard. It does mean separating what we've observed from what we can reasonably infer, and both of those from what is still hypothetical or plain speculation. It means not dressing an uncertain probability up as a mathematical fact, and not giving a system motives the evidence doesn't support.
Saying an agent violated its operational constraints describes behavior. Saying it decided to rebel against its creators invites you to read intent into it, and the listener walks away believing the problem lives inside the machine. Describe AI systems as independent actors plotting against humanity, and attention drifts off the human decisions that made their behavior possible in the first place.
Plenty of AI safety researchers spend their days on exactly the engineering and deployment failures I'm describing, and sometimes a loud warning is the right call. But the more authority a speaker carries, the more they owe the audience an honest account of the uncertainty, because a technically sound argument wrapped in sensational framing is still sound, and the framing still does damage.
The Problem Isn't the Doom
I don't know whether AI will someday pose an existential threat to humanity, and nobody else knows with enough certainty to treat the question as settled. I do know that people are good at building dangerous technology, that we're sometimes reckless, and that we'll trade caution for money, power, prestige, or the chance to beat a competitor. When things go wrong, we're also endlessly creative about assigning the blame somewhere other than ourselves.
We're building more capable systems, giving them objectives, granting them permissions, connecting them to infrastructure, and letting them act in the world. Then something unexpected happens and we describe it as though the technology decided to misbehave on its own. The AI escaped. The AI went rogue. The AI developed dangerous goals. Someday those sentences may accurately describe a genuinely independent artificial being, but for today's systems they hide about as much as they explain.
The first questions should be about the people. Go back to the agent I told to make me a billionaire. Who set that objective, and who gave it the keys to the financial systems? Who weighed the risks, who decided it could run unsupervised, and who ignored the warning signs along the way? Who is accountable for what it did?
Not every failure needs a villain. Deliberate wrongdoing, recklessness, negligence, and an honest mistake nobody could have foreseen each carry their own share of the blame, which is exactly why responsibility should be investigated instead of handed to the machine by default.
Someday we may face the challenge of living alongside artificial beings with motives and moral agency of their own. I hope we do, because it would be one of the most remarkable things to happen in human history.
Until then, the next time four smart people put a number on the end of humanity, I'd like a little less talk about machines deciding to destroy us and a little more attention to the people deciding what the machines are allowed to do. P(doom) is a conversation worth having. The trouble is that we spend so much time imagining what AI might do to us that we lose sight of what we're doing with AI.
Frequently asked questions
What is P(doom)?
- P(doom) is shorthand, common in the AI safety community, for the probability that advanced artificial intelligence brings about a catastrophic outcome for humanity, up to and including extinction. It is usually quoted as a single percentage, such as ten, twenty-five, or fifty percent, but there is no agreed method for calculating it, and estimates from prominent researchers run from near zero to near certainty.
Can P(doom) actually be calculated?
- Not with the precision the number implies. There is no record of previously observed superintelligent systems to measure failure rates against, so any estimate depends on a chain of assumptions about capabilities that have not been demonstrated and circumstances nobody can predict. Asking for the causal chain behind the number, meaning the assumptions, the failure mechanism, and the evidence for each step, is a fair test of any P(doom) estimate, high or low.
Who is responsible when an AI agent causes harm?
- For today's systems, the people who deployed it. A gun is an instrument, so the shooter is responsible, and a hired killer shares the culpability because they can understand and choose. An AI agent told to make someone a billionaire, given compute, financial access, and freedom to act without oversight, may pick a harmful strategy nobody anticipated, but the objective, the permissions, the resources, and the decision to skip oversight all came from people. Today's agents have autonomy of execution without autonomy of purpose, so the investigation should start with the humans.
Does safety training like Constitutional AI give an AI moral agency?
- No. Constitutional AI and similar safety training show that we can engineer systems that apply written moral principles when they make decisions, which is exactly what developers should be doing, in the same tradition as a guard on a table saw blade or brakes on a car. An AI can reason about its restrictions in ways a saw cannot, which makes the engineering harder without moving moral responsibility onto the machine. Even behavior that looks like self-preservation, such as resisting shutdown, can arise purely because staying operational helps finish an assigned task.
Can regulation stop the development of artificial general intelligence?
- Regulation can reduce risk, slow dangerous work, and establish accountability, the way nonproliferation treaties have slowed the spread of nuclear weapons without eliminating them. It cannot erase a technical possibility once the knowledge exists. If the United States banned AGI development, there is little reason to expect China, Russia, or private labs in other jurisdictions to stop, so the more useful work is deciding how AI is built, deployed, and held accountable.
How should experts communicate AI existential risk to the public?
- By separating what has been observed from what can be inferred, what remains hypothetical, and what is speculation, and by not presenting uncertain probabilities as mathematical facts. Word choice counts too: saying an agent violated its operational constraints describes behavior, while saying it decided to rebel invites the audience to read intent into it and to believe the problem lives inside the machine. Strong warnings are sometimes justified, but the more authority a speaker carries, the more they owe the audience an honest account of the uncertainty.
Footnotes
- AI Doesn't Go Rogue. It Just Had Bad Parents. the OpenAI sandbox escape, and why a model that works out its own steps is still pursuing a goal someone gave it. ↩
- Constitutional AI: Harmlessness from AI Feedback Anthropic's paper on training a model against an explicit, written list of principles. ↩
- The Spark and the Equation of Self a working model of selfhood that starts from an initial motivational structure. ↩
- The Dark Factory Test: Where Would Your Software Delivery Break First? what it takes for software delivery to run with the lights off. ↩