by Braden Kelley and Art Inteligencia

Every statistician who just read that headline winced. Probabilities don’t go below zero, and they’re right. Hold that wince for a moment, because it points at something the whole conversation about superintelligence keeps skipping.

If you haven’t run into the term, pDoom is the probability you personally assign to artificial intelligence ending badly for humanity. People trade their pDoom numbers at dinner parties now, the way people used to trade fantasy football picks. It’s a useful little conversation starter. It’s also a lopsided one, because it only counts one column of the ledger.

Nobody asks for your pDoom of inaction.

Press play first

I built this article around a music video, because some arguments land better with a hook in them. Here it is: Nothing Went Foom! from Bright Mirror.

That title is a wink at one of the great nerd arguments of the last twenty years. “Foom” is the sound an intelligence explosion makes in the fast-takeoff story: the machine gets a little smarter, uses that to get a lot smarter, and before anyone finishes the meeting it has left us standing on the platform. The idea was batted around in the Yudkowsky and Hanson debates back in 2008 and has been living rent-free in our collective imagination ever since.

I love that the title makes the point that nothing went foom. Technology almost never works that way. Electricity took decades to rewire the world. The internet took decades more, and we are still arguing about what it did to us. Transformations arrive in increments, which sounds boring until you realize what increments give you: time to watch, time to measure, time to adjust, and time to build the guardrails while the thing is still being built. Gradual is where safeguards live.

What we’re actually talking about building

I get tired of conversations where superintelligence is only ever the villain, so let me describe the other movie.

We already have a preview. AlphaFold predicted the structures of hundreds of millions of proteins, a job that used to cost a doctoral student years per protein, and its creators shared a Nobel Prize in Chemistry in 2024 for it. That is what narrow, early, imperfect AI did to one corner of biology. Now scale the ambition.

A superintelligence, pointed carefully at our hardest problems, could plausibly help us:

• Compress drug discovery from a decade to a few years, and take a serious run at diseases we’ve written off as incurable.
• Design new materials for batteries, solar cells and carbon capture that human intuition would never stumble across.
• Help tame fusion, where the physics is understood but the control problem is brutal.
• Put a patient, personalized tutor in front of every child on earth, including the ones who live hours from the nearest decent school.
• Bring specialist-level diagnostics to places that have never had a specialist.
• Speed up science itself, which is the part that makes my futurist heart skip. When the engine of discovery accelerates, every other list gets longer.

I’m not promising any of it will happen. I’m saying it could, and that “could” is worth a lot of human lives and a lot of human potential.

The risk we never put on the spreadsheet

Here’s where the less-than-zero joke earns its keep. The standard pDoom calculation asks how likely it is that we build this and it goes wrong. Fair question. But look at what’s missing: the cost of every year we don’t build it, or build it late, or build it only in the places that don’t care about safeguards.

A disease that goes uncured for another decade is a cost. A grid that stays dirty for another decade is a cost. A child who never gets the tutor is a cost. Those costs are quieter than a robot uprising, so they never show up in the headline, but they are real people. If you only count the risks of acting and never the risks of not acting, you haven’t done risk analysis. You’ve done fear analysis.

So here’s my proposal for a better number. Take your pDoom from moving forward, then subtract the doom you are already accepting by standing still. I suspect that for a lot of people the net figure is a lot lower than they thought, and for some of us it’s flirting with the territory of the headline. Statisticians, you may now exhale.

Safeguards, of course

Now let me be extremely clear, because I know how this paragraph gets quoted without its neighbors: I am not arguing for recklessness. Brakes are what let you drive fast, and seatbelts didn’t kill the automobile, they saved the automobile.

Responsible innovation means staged deployment instead of big-bang launches. It means independent testing and red-teaming, transparency about what these systems can and can’t do, real accountability when they fail, and international cooperation, because a safeguard that only half the world adopts is a suggestion. It means keeping humans in the loop for the decisions that matter, and building systems whose values we can inspect rather than hope for.

And I’ll say this plainly: thoughtful, serious people believe the danger is real, and some put meaningful odds on catastrophic outcomes. They might be right. That’s exactly why the safeguards have to be real and not decorative, and why dismissing the worriers is as unhelpful as dismissing the optimists. The goal isn’t to win the argument. It’s to build the thing well.

Ask a better question

The debate usually gets stuck on “should we build superintelligence?” Treat it as a yes-or-no switch and one side has to lose. Ask instead: what would we need to see in place to say yes? What safeguards, what evidence, what governance, what tripwires that tell us to slow down?

That question has answers. It also has owners and deadlines, which a vibe never does.

So, how’s your pDoom?

Mine is not zero, and I would be suspicious of anyone whose is. But when I count both columns, including the diseases, the climate, the classrooms and the discoveries waiting on the other side of responsible progress, I find the biggest risk we face isn’t that we build this thing carefully. It’s that we let fear stop us from building it at all, and hand the future to whoever was less cautious.

Hit play on the video again. Then ask yourself what future you’d rather get to first.

OR, is it all an EVIL PLAN?

Image Credits: m365 CoPilot

Content Authenticity Statement: The topic area, key elements to focus on, etc. were decisions made by Braden Kelley, with a little help from Claude to clean up the article.

Sign up here to get Human-Centered Change & Innovation Weekly delivered to your inbox every week.