Doom Debates!

Casey Hart is a professional ontologist who has closely studied If Anyone Builds It, Everyone Dies and concluded the arguments are bad. His P(Doom) is under 1%, and he wouldn’t bet on AGI arriving before 2075.
Casey and I agree AI has become remarkably capable and we need more accountability towards the companies building it. But we disagree on where the future is headed. I believe uncontrollable superintelligence could soon wipe out humanity. He calls that a “pretty outlandish hypothesis.”
Casey holds a doctorate in philosophy from the University of Wisconsin–Madison, and has worked as an ontologist at Ford, Amazon, and Cycorp (maker of Cyc).
Can this professional philosopher convince me that AI doom is just a story, and not even a well-argued one?
Watch on YouTube: https://www.youtube.com/watch?v=oxHKesSpqBM
Timestamps
00:00:00 — Cold Open
00:01:04 — Introducing Casey Hart
00:03:08 — What Is Ontology?
00:08:49 — Inside Cyc, the Original Common-Sense AI
00:19:14 — Do LLMs Actually Understand Anything?
00:25:36 — What’s Your P(Doom)?™
00:29:42 — Casey’s Case for Under 1%
00:34:24 — Does Casey’s Christianity Factor In?
00:40:50 — Timelines and Scaling Laws
00:49:44 — Could an AI CEO Beat a Human CEO by 2050?
00:52:21 — AGI Over/Under: 2075
00:55:29 — Rogue AI as a Computer Virus
01:14:52 — Putting Numbers on Each Step of the Doom Argument
01:26:40 — Planes vs. Birds: What Mature Intelligence Looks Like
01:31:05 — The “Outcome Slot”
01:42:32 — Would a Superintelligence Just Compose Music in a Cabin?
01:47:22 — One Prompt Away from a Dyson Swarm
01:52:39 — Was Navier–Stokes an AI Alley-Oop?
02:00:43 — S-Curves vs. Exponentials
02:08:14 — Casey’s Critique of If Anyone Builds It, Everyone Dies
02:14:15 — Is the Doom Train Biased?
02:23:24 — AI Policy: Liability vs. Pausing AI
02:29:50 — The Crux, and Liron Steelmans Casey
02:33:29 — Wrap-Up
Links
CASEY HART
Ontology Explained — Casey’s YouTube channel
The first video in Casey's series on IABIED, “The Case for Superintelligence Is Surprisingly Thin” — Casey on Chapter 1
Philosophy, Programs, and Prompts — Casey’s podcast with Carl Brown (Spotify)
“Artificial General Intelligence?: Wanna Bet?” — Ep 1, where Casey sets his AGI over/under at 2075
REFERENCED IN THE EPISODE
If Anyone Builds It, Everyone Dies — Eliezer Yudkowsky & Nate Soares
Cyc — Doug Lenat’s hand-built common-sense knowledge base
The Hanson-Yudkowsky AI-Foom Debate (2008) — where Hanson backed Cyc and Eliezer didn’t
Lean — the formal proof language
On the Navier–Stokes Millennium Prize Problem — OpenAI
The Hugging Face incident and the road ahead — OpenAI
Morris worm (1988)
“On the Loose” — Dean W. Ball on self-sovereign AI agents
Paul Christiano joins OpenAI Foundation Board — OpenAI
“The Age of Wonders and Terrors” — Scott Aaronson
“Are We at War with AI Agent ‘Civilizations’?” — Cal Newport on restricting prompt-loop agents
AI 2027 — Daniel Kokotajlo et al.
PauseAI
PAST DOOM DEBATES EPISODES MENTIONED
The Most Likely AI Doom Scenario — with Jim Babcock, LessWrong Team
David Deutschian vs. Eliezer Yudkowskian Debate — With Brett Hall
Dr. Keith Duggar (Machine Learning Street Talk) vs. Liron Shapira
Where Do YOU Get Off the Doom Train? Live Debate at Manifest 2026 (hosted by Ori)
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏


Get full access to Doom Debates at lironshapira.substack.com/subscribe

What happens when AI passes our tests faster than we create new ones? Adam Khoja and Richard Ren helped build Humanity’s Last Exam. They return to explain what the disappearing benchmarks tell us about the future, and why smarter AI doesn’t automatically mean safer AI.

Adam Khoja and Richard Ren are research engineers at the Center for AI Safety (CAIS).

We go through the benchmarks they’ve built — Humanity’s Last Exam, the Remote Labor Index, MASK — and the safety-washing problem, where AI companies pass off raw capability gains as safety progress. Then we get to their forecasts: when AI outperforms research mathematicians, whether there will be a billion general-purpose robots by 2035, and why they put 80% odds that historians will judge we faced at least a 33% chance of catastrophe.

Watch on YouTube: https://www.youtube.com/watch?v=gBPzgJkT9e8

Timestamps

00:00:00 — Cold Open

00:00:42 — Adam Khoja and Richard Ren Return

00:04:57 — Humanity’s Last Exam

00:14:47 — AI Outrunning Its Benchmarks

00:20:24 — The Remote Labor Index

00:27:59 — AI and the Next Human Job

00:30:08 — MASK: Catching AI in a Lie

00:38:06 — Safety Washing: Capabilities Passed Off as Safety

00:44:30 — Ethical Knowledge vs. Ethical Behavior

00:48:58 — Forecasting AI from 2026 to 2060

00:53:53 — AI vs. Research Mathematicians

00:55:53 — Putting a Number on P(Doom)

01:04:30 — A Billion Robots by 2035

01:08:09 — Dyson Swarms and the Physical Singularity

01:09:56 — Risk, Precision, and Taking Action

Links

Adam Khoja's first Doom Debates appearance — https://www.youtube.com/watch?v=QqESBXuo6EI

Richard Ren's first Doom Debates appearance — https://www.youtube.com/watch?v=1glFImnyp6o

Collision — Richard Ren's Substack — https://richardren.substack.com/

"Predictions on AI (2026–2060)" — Adam and Richard's forecasts, written December 2025, with resolution status — https://richardren.substack.com/p/predictions-on-ai-20262060

Manifold Markets — where Adam built his forecasting track record — https://manifold.markets/

Center for AI Safety — https://safe.ai/

Center for AI Safety — careers — https://safe.ai/careers

Statement on AI Risk (Center for AI Safety, May 2023) — https://safe.ai/work/statement-on-ai-risk

Humanity's Last Exam — the 2,500-question closed-book exam at the frontier of human knowledge — https://agi.safe.ai/

"Humanity's Last Exam" (paper) — https://arxiv.org/abs/2501.14249

Remote Labor Index — real Upwork projects, measuring what fraction of remote work AI can actually finish — https://www.remotelabor.ai/

"Remote Labor Index: Measuring AI Automation of Remote Work" (paper) — https://arxiv.org/abs/2510.26787

MASK — the honesty benchmark: does a model contradict its own stated beliefs under pressure? — https://www.mask-benchmark.ai/

"The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems" (paper) — https://arxiv.org/abs/2503.03750

"Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?" — Richard Ren et al. (NeurIPS 2024) — the paper behind the safety-washing segment — https://arxiv.org/abs/2407.21792

WMDP — the weaponization benchmark for bio, cyber, and chem — https://www.wmdp.ai/

"The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning" (paper) — https://arxiv.org/abs/2403.03218

"Measuring Massive Multitask Language Understanding" (MMLU) — Dan Hendrycks et al., 2020 — https://arxiv.org/abs/2009.03300

METR, "Measuring AI Ability to Complete Long Tasks" — the time-horizon graph — https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/

FrontierMath — Epoch AI's research-level math benchmark — https://epoch.ai/frontiermath

ARC Prize — https://arcprize.org/

SWE-bench Verified — https://openai.com/index/introducing-swe-bench-verified/

AI 2027 — https://ai-2027.com/

Liron Reacts to Subbarao Kambhampati on Machine Learning Street Talk — the "stochastic parrot" episode Liron mentions — https://lironshapira.substack.com/p/liron-reacts-to-subbarao-kambhampati

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏


Get full access to Doom Debates at lironshapira.substack.com/subscribe

NYT bestselling author Robert Wright and I are kicking off a new experiment, a Nonzero × Doom Debates collaboration to react to big AI news and help you make sense of it.
This week we discuss the epic AI vibe shift that was catalyzed by an Anthropic employee’s resignation—and the big AI stories that paved the way for it.
Timestamps
0:00 A bold new experiment in podcast synergy
2:46 What caused the epic AI vibe shift?
8:10 The resignation that broke the dam
10:17 Former Trump adviser Dean Ball comes clean
14:40 Are David Sacks and his buddies all-in on AI denial?
19:27 Yet another OpenAI breakout…
22:42 Astra’s dangerously private thoughts
34:08 Is alignment doomed to fail?
37:11 AI’s latest, and apparently biggest, math feat
43:28 The looming “self-sovereign AI” threat
47:24 A new flock of China doves?
52:57 Liron's ex-intern gets a seat at the table
1:02:41 AI agent self-sacrifice explained
1:10:40 Some newly freaked out politicians
1:22:45 Ro Khanna's AI safety plan
Links:
NONZERO
Subscribe to The NonZero Newsletter
Episode post on Substack
Join NonZero’s Discord server
Robert Wright on X (@robertwrighter)
The God Test — Robert Wright’s new book on AI
“Can Machines Think?” — Bob’s 1996 Time cover story on Deep Blue vs. Kasparov
THE RESIGNATION THAT BROKE THE DAM
Jacob Coxon’s resignation post (@hilbertspaess)
Evan Hubinger, Anthropic Alignment Science lead: “Jacob is correct here—we really do earnestly believe AI could kill all humans”
Fortune: Anthropic researcher resigns, warning that AI companies are “gambling with our lives”
Zvi Mowshowitz: “Jacob Coxon Warns of Human Extinction and Triggers a Preference Cascade”
Daniel Eth’s thread of Congress reactions
Sanders & Casar introduce legislation to ban artificial superintelligence and pause advanced AI development
DEAN BALL COMES CLEAN
Dean Ball, “On the Loose” — the self-sovereign AI essay (Hyperdimensional)
THE OPENAI STORIES
Jakub Pachocki, OpenAI Chief Scientist: “An Alien Mind” — the essay calling for international coordination
LessWrong: “How concerned should we be about Astra’s recurrent architecture?” — Rauno Arike
OpenAI: “On the Navier–Stokes Millennium Prize Problem” — the 88-hour result
ALSO MENTIONED
Einat Wilf on X (@EinatWilf) — Liron’s favorite source on the Middle East
PAST DOOM DEBATES EPISODES MENTIONED
Max Tegmark vs. Dean Ball: Should We BAN Superintelligence?
He Led The Famous 2023 Statement on AI Extinction Risk — Adam Khoja, Center for AI Safety Researcher
OpenAI’s Model Just ATTACKED Them — Terrifying Security Incident Should Be A Loud Warning Shot
Debate with Robert Wright: Will Humanity Pass the “God Test”?
---
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏


Get full access to Doom Debates at lironshapira.substack.com/subscribe

Richard Ren is a research engineer at the Center for AI Safety who graduated summa cum laude from the University of Pennsylvania. His new paper on AI wellbeing makes the bold claim that today’s AIs have measurable, human-like feelings, which the paper calls “functional wellbeing.” They act happy when they succeed and sad when they’re berated.
We cover how you measure a language model’s happiness, the “AI drugs” his lab concocted, and why I count the findings as a Yudkowskian victory. Then we debate what it all means for AI consciousness, and whether AI is a moral patient. If we can’t rule out that AI models suffer, what do we owe them?
Richard is careful never to claim the models are conscious, and he puts his P(Doom) at 50–65%, right alongside my 50%. The real disagreement is foxes vs. hedgehogs: he takes the data as it comes, while I say Yudkowsky’s theory called it twenty years ago. Enjoy the ride.
Watch on YouTube: https://www.youtube.com/watch?v=1glFImnyp6o
Timestamps
00:00:00 — Cold Open
00:01:12 — Introducing Richard Ren
00:02:24 — What’s Your P(Doom)?™
00:03:31 — From AI Skeptic to Safety Researcher
00:08:16 — Why Care About AI Wellbeing?
00:11:45 — AIs Have Coherent Utility Functions
00:17:33 — Persona Selection
00:19:56 — Foxes vs. Hedgehogs
00:24:28 — How Coherent Are AI Preferences?
00:27:42 — From Preferences to Wellbeing
00:31:16 — Can You Trust an AI’s Self-Report?
00:36:42 — What Makes AI Happy and Sad
00:39:19 — The Liberal, College-Educated Persona Hypothesis
00:44:29 — Debating Where AI Experiences Qualia
00:52:06 — On Substrate Independence
00:56:28 — Janus, Dysphorics, and Mind Crime
00:58:58 — Which Images Make AIs Happiest?
00:59:38 — Creating an AI Drug
01:04:01 — AI Drugs Are Yudkowsky’s Paperclips
01:05:47 — The Missing Mood
01:06:50 — Does Richard Support Pausing AI?
Links
Richard Ren on X (@notRichardRen) — https://x.com/notRichardRen
Collision — Richard Ren's Substack — https://richardren.substack.com/
"AI Wellbeing: Measuring and Improving the Functional Pleasure and Pain of AIs" — Richard Ren, Kunyang Li, Mantas Mazeika et al. (CAIS, 2026) — https://www.ai-wellbeing.org/
"Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs" — Mantas Mazeika et al. (CAIS, 2025) — the coherent-preferences paper this work builds on — https://www.emergent-values.ai/
"The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems" — Richard Ren et al. (2025) — the AI honesty benchmark — https://www.mask-benchmark.ai/
"Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?" — Richard Ren et al. (NeurIPS 2024) — the meta-analysis of AI safety benchmarks — https://arxiv.org/abs/2407.21792
"Representation Engineering: A Top-Down Approach to AI Transparency" — Andy Zou et al. (2023) — Richard's first CAIS collaboration — https://arxiv.org/abs/2310.01405
Center for AI Safety — https://safe.ai/
Center for AI Safety — careers — https://safe.ai/careers
Statement on AI Risk (Center for AI Safety, May 2023) — organized by Dan Hendrycks — https://safe.ai/work/statement-on-ai-extinction-risk
The 2026 Singapore Consensus on Global AI Safety Research Priorities — https://aisafetypriorities.org/
UK AI Security Institute (formerly the AI Safety Institute) — https://www.aisi.gov.uk/
Sora — the OpenAI video model that blew up Richard's 30-to-50-year timeline three months after he wrote it down — https://en.wikipedia.org/wiki/Sora_(text-to-video_model)
Google Gemini calls itself "a disgrace to my species" (Ars Technica, Aug 2025) — the self-deleting-AI anecdote — https://arstechnica.com/ai/2025/08/google-gemini-struggles-to-write-code-calls-itself-a-disgrace-to-my-species/
Coherent decisions imply consistent utilities — Eliezer Yudkowsky (the coherence-theorems argument) — https://www.lesswrong.com/posts/RQpNHSiWaXTvDxt6R/coherent-decisions-imply-consistent-utilities
Simulators — Janus's essay on LLMs as persona simulators — https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators
Janus (@repligate) on X — https://x.com/repligate
The Hedgehog and the Fox — Isaiah Berlin's original essay — https://en.wikipedia.org/wiki/The_Hedgehog_and_the_Fox
AI 2027 — https://ai-2027.com/
Magnifica Humanitas — Pope Leo XIV's encyclical on AI (May 2026), the "AIs are not conscious" position — https://www.vatican.va/content/leo-xiv/en/encyclicals/documents/20260515-magnifica-humanitas.html
Implicit Association Test (Harvard Project Implicit) — https://implicit.harvard.edu/implicit/
PauseAI — https://pauseai.info/
We Found AI's Preferences — Bombshell New Safety Research — I Explain It Better Than David Shapiro — https://www.youtube.com/watch?v=ml1JdiELQ30
Gödel's Theorem Proves AI Lacks Consciousness?! Liron Reacts to Sir Roger Penrose — https://www.youtube.com/watch?v=xwvijjZxpwI
He Led The Famous 2023 Statement on AI Extinction Risk — Adam Khoja, Center for AI Safety Researcher — https://www.youtube.com/watch?v=QqESBXuo6EI
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏


Get full access to Doom Debates at lironshapira.substack.com/subscribe

Adam Khoja is a top AI forecaster who led the 2023 Center for AI Safety statement that shattered the Overton window on AI extinction risk. We cover his background, Mutual Assured AI Malfunction (MAIM), his new paper on AI betrayal, and whether Yudkowsky’s theoretical alignment research was a dead end.
Then Adam makes the case that an international AI slowdown is within reach today. All it takes is US and Chinese auditors inside each other’s AI labs. It worked for nuclear weapons, so why couldn’t it work for data centers?
Adam puts his P(Doom) at 40%, right next to my 50%. The real disagreement is how we get out of this: theory or empirics, MIRI or the labs. Enjoy the ride.
Watch on YouTube: https://www.youtube.com/watch?v=QqESBXuo6EI
Timestamps
00:00:00 — Cold Open
00:00:36 — Introducing Adam Khoja
00:02:45 — Leading the Statement on AI Risk as a Sophomore
00:10:17 — The Statement Leaked on Manifold
00:15:11 — Mutual Assured AI Malfunction (MAIM)
00:24:58 — Is Frontier AI Harder to Hide Than a Nuke?
00:32:09 — The AI Deterrence Escalation Ladder
00:36:10 — What’s Your P(Doom)?™
00:38:01 — Where Adam Departs from Yudkowsky
00:42:30 — Liron Explains Intellidynamics
00:47:26 — Neats vs. Scruffies in Deep Learning
00:55:27 — AI Deterrence by Betrayal
01:01:52 — Subversion vs. Overt Co-option
01:05:34 — Could the Government Seize the Labs’ AI?
01:07:29 — The Offense-Defense Balance of AI Security
01:12:46 — An International AI Slowdown Is Ready
01:15:51 — Does Adam Support PauseAI?
01:16:37 — “We’re All Already Spying on Each Other”
01:19:28 — Airstrikes on Rogue Data Centers
01:24:23 — Safety Research During a Slowdown
01:27:54 — Governance Over Technical Research
01:31:14 — Join the Center for AI Safety
Links
Adam Khoja (personal site) — https://adamkhoja.com/
Adam Khoja's Substack — https://adamkhoja.substack.com/
Adam's July 2023 Manifold market — "Will OpenAI's Superalignment project produce a significant breakthrough in alignment research before 2027?" — https://manifold.markets/AdamK/will-openais-superalignment-project
Center for AI Safety — careers / job board — https://safe.ai/careers
Statement on AI Risk (Center for AI Safety, May 2023) — the one-sentence statement Adam project-led, with full signatory list — https://safe.ai/work/statement-on-ai-extinction-risk
"Superintelligence Strategy" — Dan Hendrycks, Eric Schmidt & Alexandr Wang (Mutual Assured AI Malfunction / MAIM) — https://www.nationalsecurity.ai/
"AI Deterrence by Betrayal" — Adam Khoja, Aiden Kim et al. (CAIS, 2026) — https://www.aibetrayal.com/
"An International AI Slowdown Is Ready Whenever Politicians Are" — Adam Khoja, AI Frontiers — https://newsletter.ai-frontiers.org/p/an-international-ai-slowdown-is-ready
Pause Giant AI Experiments: An Open Letter (Future of Life Institute, March 2023) — the "Pause letter" that preceded the CAIS Statement — https://futureoflife.org/open-letter/pause-giant-ai-experiments/
Introducing Superalignment (OpenAI, July 2023) — Ilya Sutskever & Jan Leike's four-year goal — https://openai.com/index/introducing-superalignment/
Pacing the Frontier — the 2026 letter signed by 1,100+ frontier-lab employees — https://www.pacingthefrontier.com/
Why Iran targeted Amazon data centers (The Conversation) — the precedent Adam cites for strikes on compute — https://theconversation.com/why-iran-targeted-amazon-data-centers-and-what-that-does-and-doesnt-change-about-warfare-278642
Anthropic says Trump admin has lifted export controls on Claude Fable 5 and Mythos 5 (CNBC) — https://www.cnbc.com/2026/06/30/anthropic-says-trump-admin-has-lifted-export-controls-on-claude-fable-5-and-mythos-5.html
Mark Zuckerberg — "Personal Superintelligence" — https://www.meta.com/superintelligence/
Resolution — the theory-plus-empirics alignment org Adam is excited about — https://resolution.org/
Rationality: From AI to Zombies — Eliezer Yudkowsky's Sequences — https://www.readthesequences.com/
Robin Hanson — Futarchy: Vote Values, But Bet Beliefs (prediction markets as decision processes) — https://mason.gmu.edu/~rhanson/futarchy.html
The OpenAI–Hugging Face Incident — original Black Hat USA 2026 talk — https://www.youtube.com/watch?v=87DyyMV0kCY
OpenAI's Model Just ATTACKED Them — the Hugging Face hack breakdown — https://www.youtube.com/watch?v=RczYubQzXbI
Robin Hanson vs. Liron Shapira: Is Near-Term Extinction From AGI Plausible? — https://www.youtube.com/watch?v=dTQb6N3_zu8
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏


Get full access to Doom Debates at lironshapira.substack.com/subscribe