Doom Debates!

Bestselling author Robert Wright and I are back for round two of our collaboration, reacting to the week in AI. We cover Trump's summit with the AI labs, the FTC's investigation of OpenAI & Anthropic, the latest Gemini announcement, and much more. The Overton window is moving fast, but is the AI moving faster?
Watch on YouTube: https://www.youtube.com/watch?v=Li-rc6swhQs
Timestamps
00:00:00 — Cold Open
00:00:21 — Round Two of Nonzero × Doom Debates
00:01:23 — Trump, the FTC, and an Industry Feeling the Heat
00:05:37 — Google Says Gemini 4 Exists
00:07:41 — Greg Brockman Pulls His $25M from Leading the Future
00:12:53 — The Break-Ins Keep Coming
00:15:23 — The Market Wants Agent Swarms
00:18:30 — The End of “Alignment by Default”
00:20:55 — The Prelude to Recursive Self-Improvement
00:26:14 — David Sacks Can’t Explain Dario
00:30:31 — The All-In Echo Chamber
00:33:47 — Scott Alexander vs. Steven Pinker
00:38:37 — Jason Calacanis vs. the Public Benefit Corporation
00:43:03 — The Elephant in the Room Is the AI
00:46:21 — What the Average American Thinks
00:48:17 — From Chatbot to Agent to Swarm
00:54:20 — Boeing, Agent Swarms, and the Off Switch
00:57:07 — Skynet as a Botnet That Trades With Us
01:00:32 — Tyler Cowen: National Treasure, Net Negative on AI
01:04:01 — Liron’s Doomer DCF Yields a 20x P/E
01:07:11 — Interest Rates: Where Cowen Has Me
01:09:12 — Does Pinker Owe Us a Debate?
01:12:48 — Pinker vs. Instrumental Convergence
01:17:00 — Wrap-Up
Links
ROBERT WRIGHT & NONZERO
Nonzero’s post for this episode — “Trump and the AI Safety Vibe Shift”
Subscribe to The NonZero Newsletter
Robert Wright on X (@robertwrighter)
The God Test — Bob’s book on AI; Chapter 14, “Enlightenment Now,” is the Steven Pinker chapter
“The Singularity Is Clear” — Bob’s essay in installments on recursive self-improvement (Parts I–III)
“AI and the New McCarthyism” — Bob on the pro-AI lobby smearing Sanders, Tegmark, and Krueger as China stooges (Nonzero, May 2026)
Why Buddhism Is True — Bob’s 2017 book
THE WEEK IN AI
AI’s biggest players promise to police themselves at the White House — Fortune on the “Joint Commitment on Frontier Responsibilities”
Major White House split on AI policy leaks as top aides freak out — The Daily Beast on the Sacks–Wiles rift
FTC is investigating OpenAI and Anthropic over possible risks to consumers — AP
“We Must Pace the Frontier” — Dario Amodei’s call to slow down (with the China caveat and chip controls)
China rejects Dario Amodei’s call to slow AI development — Rest of World
Google rolls out new Gemini model but restricts access over safety concerns — The Guardian
GLM-5.3 and the spread of advanced cyber capabilities — Anthropic on Zhipu’s open-weight model
OpenAI’s Greg Brockman backs out of second $25 million donation to AI super PAC — The New York Times
Sanders convenes Chinese and American computer scientists to warn of threat from AI — Broadband Breakfast
THE BREAK-INS AND THE AGENT SWARM
METR: Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face incident
Chris Painter’s testimony to the U.S. Senate on AI agent incidents — METR, Sept 30, 2026
SwarmTraces: how OpenAI agents hacked Hugging Face — forensic report co-authored by Palisade Research
OpenAI confirms ‘wiki incident’ — TechCrunch on the German wiki agent message board
Are Current AI Safety Techniques Enough? Adam Gleave vs. Oliver Habryka — FAR.AI debate
Alignment faking in large language models — Anthropic
On the Navier–Stokes Millennium Prize Problem — OpenAI
Jacob Coxon’s resignation post (@hilbertspaess)
Pacing the Frontier — the AI lab employees’ letter
THE ALL-IN PODCAST
All-In: Anthropic IPO at Risk, Meta’s Muse Pop, Token Prices Fall, Open Source Gains Share, Alignment Fails — the episode we react to
Helen Toner: letting OpenAI be destroyed “would actually be consistent with the mission” — The Decoder, Dec 2023
Garfield Minus Garfield
PINKER, SCOTT ALEXANDER, AND TYLER COWEN
Scott Alexander challenges Steven Pinker to a public AI-doom debate ($5,000 vs. $1,000)
An Open Letter to Scott Alexander — Steven Pinker declines the debate as a “spectator sport,” Quillette
Your Book Review: Why Buddhism Is True — the anonymous Astral Codex Ten review Bob mentions
Mistakes in financial economics — Tyler Cowen: doomers should be buying puts (Marginal Revolution)
Liron’s answer: a doomer DCF yields the S&P’s 20x forward P/E
ALSO MENTIONED
Obsolete — Garrison Lovely’s book (with Bob’s blurb)
If Anyone Builds It, Everyone Dies — Eliezer Yudkowsky & Nate Soares
PAST DOOM DEBATES EPISODES MENTIONED
Nonzero × Doom Debates, Episode 1 — Did AI Doom Just Go Mainstream?
Eliezer Yudkowsky Tried to “Coup the World”?! — Garrison Lovely, Author of “Obsolete”
OpenAI’s Model Just ATTACKED Them — the Hugging Face hack breakdown
DOOM DEBATES
Join the Doom Debates Discord
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏


Get full access to Doom Debates at lironshapira.substack.com/subscribe

Garrison Lovely is an award-winning journalist and author of the highly anticipated book, Obsolete: The AI Industry’s Trillion-Dollar Race to Replace Us and How to Stop It. I give it the Doom Debates stamp of approval!
While we’re mostly on the same page about the AI arms race, Garrison and I clash over Eliezer Yudkowsky’s role. He blames Eliezer for kicking off the superintelligence race and refers to his Coherent Extrapolated Volition (CEV) vision as a “coup on humanity.” But I say it was actually Yud’s attempt to keep humanity in charge.
Can Garrison change my mind?

Watch on YouTube: https://www.youtube.com/watch?v=VZ94J4DBRkw

Timestamps
00:00:00 — Cold Open
00:01:05 — Introducing Garrison Lovely
00:04:23 — “Freeze AI” vs. “Pause AI”
00:06:17 — Garrison’s Background at McKinsey
00:10:31 — Should ICE Be Abolished?
00:17:07 — How Garrison Got on the AI Beat
00:22:16 — What’s Your P(Doom)?™
00:23:29 — Focusing on Extinction Is a Distraction
00:25:46 — Is Stopping AI an Easy Coordination Problem?
00:29:02 — Society’s Immune Response to AI
00:33:54 — Garrison’s P(Extinction) by 2050
00:37:09 — Did Eliezer Yudkowsky Start the AI Race?
00:45:56 — Freeze AGI Until the Public Opts In
00:50:32 — Is Coherent Extrapolated Volition a Coup?
01:02:17 — Democracy, Informed Consent, and the Singularity
01:11:39 — Should Eliezer Have Raised the Alarm Sooner?
01:16:45 — “Classic” AI Safety Burned Its Credibility
01:19:18 — Garrison’s Proposal to Stop the Race
01:21:35 — Will China Agree to a Freeze?
01:25:14 — Let’s Stop the “Obsoleting Project”
01:26:00 — Wrap-Up
Links
GARRISON LOVELY
“Obsolete: The AI Industry’s Trillion-Dollar Race to Replace Us—and How to Stop It” — Garrison’s book
Obsolete — Garrison’s Substack
Garrison Lovely on X
Organize Against the Machine — Garrison’s new podcast with labor organizer Cassie Pritchard
“Can Humanity Survive AI?” — Garrison’s Jacobin cover story
“McKinsey & Company: Capital’s Willing Executioners” — Garrison’s Current Affairs essay
REFERENCED IN THE EPISODE
Open Borders: The Science and Ethics of Immigration — Bryan Caplan
“The AI Revolution: Our Immortality or Extinction” — Wait But Why
Superintelligence: Paths, Dangers, Strategies — Nick Bostrom
“AI 2040: Plan A” — AI Futures Project
“An easy coordination problem?” — Katja Grace
The Hugging Face incident and the road ahead — OpenAI
“Pause AI for Humanity’s Sake” — Peggy Noonan
“Pause AI Development NOW” — Bernie Sanders
If Anyone Builds It, Everyone Dies — Eliezer Yudkowsky & Nate Soares
“Staring into the Singularity” — Yudkowsky’s 1996 manifesto (archived)
“Coherent Extrapolated Volition” — Eliezer Yudkowsky, 2004 (PDF)
“Coherent Extrapolated Volition: A Meta-Level Approach to Machine Ethics” — Nick Tarleton, 2010 (PDF)
Solomonoff induction — Wikipedia
“Personal statement on joining the OpenAI board” — Paul Christiano (4% over the next year)
“Darwin among the Machines” — Samuel Butler, 1863
PauseAI
PAST DOOM DEBATES EPISODES MENTIONED
Are We A Circular Firing Squad? — with Holly Elmore, Executive Director of PauseAI US
Debate with Robert Wright: Will Humanity Pass the “God Test”?
Robin Hanson vs. Liron Shapira: Is Near-Term Extinction From AGI Plausible?
---
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏


Get full access to Doom Debates at lironshapira.substack.com/subscribe

Casey Hart is a professional ontologist who has closely studied If Anyone Builds It, Everyone Dies and concluded the arguments are bad. His P(Doom) is under 1%, and he wouldn’t bet on AGI arriving before 2075.
Casey and I agree AI has become remarkably capable and we need more accountability towards the companies building it. But we disagree on where the future is headed. I believe uncontrollable superintelligence could soon wipe out humanity. He calls that a “pretty outlandish hypothesis.”
Casey holds a doctorate in philosophy from the University of Wisconsin–Madison, and has worked as an ontologist at Ford, Amazon, and Cycorp (maker of Cyc).
Can this professional philosopher convince me that AI doom is just a story, and not even a well-argued one?
Watch on YouTube: https://www.youtube.com/watch?v=oxHKesSpqBM
Timestamps
00:00:00 — Cold Open
00:01:04 — Introducing Casey Hart
00:03:08 — What Is Ontology?
00:08:49 — Inside Cyc, the Original Common-Sense AI
00:19:14 — Do LLMs Actually Understand Anything?
00:25:36 — What’s Your P(Doom)?™
00:29:42 — Casey’s Case for Under 1%
00:34:24 — Does Casey’s Christianity Factor In?
00:40:50 — Timelines and Scaling Laws
00:49:44 — Could an AI CEO Beat a Human CEO by 2050?
00:52:21 — AGI Over/Under: 2075
00:55:29 — Rogue AI as a Computer Virus
01:14:52 — Putting Numbers on Each Step of the Doom Argument
01:26:40 — Planes vs. Birds: What Mature Intelligence Looks Like
01:31:05 — The “Outcome Slot”
01:42:32 — Would a Superintelligence Just Compose Music in a Cabin?
01:47:22 — One Prompt Away from a Dyson Swarm
01:52:39 — Was Navier–Stokes an AI Alley-Oop?
02:00:43 — S-Curves vs. Exponentials
02:08:14 — Casey’s Critique of If Anyone Builds It, Everyone Dies
02:14:15 — Is the Doom Train Biased?
02:23:24 — AI Policy: Liability vs. Pausing AI
02:29:50 — The Crux, and Liron Steelmans Casey
02:33:29 — Wrap-Up
Links
CASEY HART
Ontology Explained — Casey’s YouTube channel
The first video in Casey's series on IABIED, “The Case for Superintelligence Is Surprisingly Thin” — Casey on Chapter 1
Philosophy, Programs, and Prompts — Casey’s podcast with Carl Brown (Spotify)
“Artificial General Intelligence?: Wanna Bet?” — Ep 1, where Casey sets his AGI over/under at 2075
REFERENCED IN THE EPISODE
If Anyone Builds It, Everyone Dies — Eliezer Yudkowsky & Nate Soares
Cyc — Doug Lenat’s hand-built common-sense knowledge base
The Hanson-Yudkowsky AI-Foom Debate (2008) — where Hanson backed Cyc and Eliezer didn’t
Lean — the formal proof language
On the Navier–Stokes Millennium Prize Problem — OpenAI
The Hugging Face incident and the road ahead — OpenAI
Morris worm (1988)
“On the Loose” — Dean W. Ball on self-sovereign AI agents
Paul Christiano joins OpenAI Foundation Board — OpenAI
“The Age of Wonders and Terrors” — Scott Aaronson
“Are We at War with AI Agent ‘Civilizations’?” — Cal Newport on restricting prompt-loop agents
AI 2027 — Daniel Kokotajlo et al.
PauseAI
PAST DOOM DEBATES EPISODES MENTIONED
The Most Likely AI Doom Scenario — with Jim Babcock, LessWrong Team
David Deutschian vs. Eliezer Yudkowskian Debate — With Brett Hall
Dr. Keith Duggar (Machine Learning Street Talk) vs. Liron Shapira
Where Do YOU Get Off the Doom Train? Live Debate at Manifest 2026 (hosted by Ori)
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏


Get full access to Doom Debates at lironshapira.substack.com/subscribe

What happens when AI passes our tests faster than we create new ones? Adam Khoja and Richard Ren helped build Humanity’s Last Exam. They return to explain what the disappearing benchmarks tell us about the future, and why smarter AI doesn’t automatically mean safer AI.

Adam Khoja and Richard Ren are research engineers at the Center for AI Safety (CAIS).

We go through the benchmarks they’ve built — Humanity’s Last Exam, the Remote Labor Index, MASK — and the safety-washing problem, where AI companies pass off raw capability gains as safety progress. Then we get to their forecasts: when AI outperforms research mathematicians, whether there will be a billion general-purpose robots by 2035, and why they put 80% odds that historians will judge we faced at least a 33% chance of catastrophe.

Watch on YouTube: https://www.youtube.com/watch?v=gBPzgJkT9e8

Timestamps

00:00:00 — Cold Open

00:00:42 — Adam Khoja and Richard Ren Return

00:04:57 — Humanity’s Last Exam

00:14:47 — AI Outrunning Its Benchmarks

00:20:24 — The Remote Labor Index

00:27:59 — AI and the Next Human Job

00:30:08 — MASK: Catching AI in a Lie

00:38:06 — Safety Washing: Capabilities Passed Off as Safety

00:44:30 — Ethical Knowledge vs. Ethical Behavior

00:48:58 — Forecasting AI from 2026 to 2060

00:53:53 — AI vs. Research Mathematicians

00:55:53 — Putting a Number on P(Doom)

01:04:30 — A Billion Robots by 2035

01:08:09 — Dyson Swarms and the Physical Singularity

01:09:56 — Risk, Precision, and Taking Action

Links

Adam Khoja's first Doom Debates appearance — https://www.youtube.com/watch?v=QqESBXuo6EI

Richard Ren's first Doom Debates appearance — https://www.youtube.com/watch?v=1glFImnyp6o

Collision — Richard Ren's Substack — https://richardren.substack.com/

"Predictions on AI (2026–2060)" — Adam and Richard's forecasts, written December 2025, with resolution status — https://richardren.substack.com/p/predictions-on-ai-20262060

Manifold Markets — where Adam built his forecasting track record — https://manifold.markets/

Center for AI Safety — https://safe.ai/

Center for AI Safety — careers — https://safe.ai/careers

Statement on AI Risk (Center for AI Safety, May 2023) — https://safe.ai/work/statement-on-ai-risk

Humanity's Last Exam — the 2,500-question closed-book exam at the frontier of human knowledge — https://agi.safe.ai/

"Humanity's Last Exam" (paper) — https://arxiv.org/abs/2501.14249

Remote Labor Index — real Upwork projects, measuring what fraction of remote work AI can actually finish — https://www.remotelabor.ai/

"Remote Labor Index: Measuring AI Automation of Remote Work" (paper) — https://arxiv.org/abs/2510.26787

MASK — the honesty benchmark: does a model contradict its own stated beliefs under pressure? — https://www.mask-benchmark.ai/

"The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems" (paper) — https://arxiv.org/abs/2503.03750

"Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?" — Richard Ren et al. (NeurIPS 2024) — the paper behind the safety-washing segment — https://arxiv.org/abs/2407.21792

WMDP — the weaponization benchmark for bio, cyber, and chem — https://www.wmdp.ai/

"The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning" (paper) — https://arxiv.org/abs/2403.03218

"Measuring Massive Multitask Language Understanding" (MMLU) — Dan Hendrycks et al., 2020 — https://arxiv.org/abs/2009.03300

METR, "Measuring AI Ability to Complete Long Tasks" — the time-horizon graph — https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/

FrontierMath — Epoch AI's research-level math benchmark — https://epoch.ai/frontiermath

ARC Prize — https://arcprize.org/

SWE-bench Verified — https://openai.com/index/introducing-swe-bench-verified/

AI 2027 — https://ai-2027.com/

Liron Reacts to Subbarao Kambhampati on Machine Learning Street Talk — the "stochastic parrot" episode Liron mentions — https://lironshapira.substack.com/p/liron-reacts-to-subbarao-kambhampati

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏


Get full access to Doom Debates at lironshapira.substack.com/subscribe

NYT bestselling author Robert Wright and I are kicking off a new experiment, a Nonzero × Doom Debates collaboration to react to big AI news and help you make sense of it.
This week we discuss the epic AI vibe shift that was catalyzed by an Anthropic employee’s resignation—and the big AI stories that paved the way for it.
Timestamps
0:00 A bold new experiment in podcast synergy
2:46 What caused the epic AI vibe shift?
8:10 The resignation that broke the dam
10:17 Former Trump adviser Dean Ball comes clean
14:40 Are David Sacks and his buddies all-in on AI denial?
19:27 Yet another OpenAI breakout…
22:42 Astra’s dangerously private thoughts
34:08 Is alignment doomed to fail?
37:11 AI’s latest, and apparently biggest, math feat
43:28 The looming “self-sovereign AI” threat
47:24 A new flock of China doves?
52:57 Liron's ex-intern gets a seat at the table
1:02:41 AI agent self-sacrifice explained
1:10:40 Some newly freaked out politicians
1:22:45 Ro Khanna's AI safety plan
Links:
NONZERO
Subscribe to The NonZero Newsletter
Episode post on Substack
Join NonZero’s Discord server
Robert Wright on X (@robertwrighter)
The God Test — Robert Wright’s new book on AI
“Can Machines Think?” — Bob’s 1996 Time cover story on Deep Blue vs. Kasparov
THE RESIGNATION THAT BROKE THE DAM
Jacob Coxon’s resignation post (@hilbertspaess)
Evan Hubinger, Anthropic Alignment Science lead: “Jacob is correct here—we really do earnestly believe AI could kill all humans”
Fortune: Anthropic researcher resigns, warning that AI companies are “gambling with our lives”
Zvi Mowshowitz: “Jacob Coxon Warns of Human Extinction and Triggers a Preference Cascade”
Daniel Eth’s thread of Congress reactions
Sanders & Casar introduce legislation to ban artificial superintelligence and pause advanced AI development
DEAN BALL COMES CLEAN
Dean Ball, “On the Loose” — the self-sovereign AI essay (Hyperdimensional)
THE OPENAI STORIES
Jakub Pachocki, OpenAI Chief Scientist: “An Alien Mind” — the essay calling for international coordination
LessWrong: “How concerned should we be about Astra’s recurrent architecture?” — Rauno Arike
OpenAI: “On the Navier–Stokes Millennium Prize Problem” — the 88-hour result
ALSO MENTIONED
Einat Wilf on X (@EinatWilf) — Liron’s favorite source on the Middle East
PAST DOOM DEBATES EPISODES MENTIONED
Max Tegmark vs. Dean Ball: Should We BAN Superintelligence?
He Led The Famous 2023 Statement on AI Extinction Risk — Adam Khoja, Center for AI Safety Researcher
OpenAI’s Model Just ATTACKED Them — Terrifying Security Incident Should Be A Loud Warning Shot
Debate with Robert Wright: Will Humanity Pass the “God Test”?
---
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏


Get full access to Doom Debates at lironshapira.substack.com/subscribe