Doom Debates!

Stephen Casper, PhD, is an incoming tenure-track professor of public policy at the Harvard Kennedy School.
Prof. Casper is relatively unworried about the threat of human extinction from rogue superintelligence, but he is concerned about a handful of companies wielding extremely powerful AI and enfeebling the public. That’s why he’s an AI governance hawk who advocates for taxes and regulation to keep frontier labs in check.
We agree that slowing down AI development would make the world safer. But you know his position is unique when he says he’d prefer to have less research on AI alignment!
Timestamps
00:00:00 — Cold Open
00:00:50 — Introducing Stephen “Cas” Casper
00:04:21 — Cas’s MIT Research: Model Tampering Attacks
00:07:14 — What’s Your P(Doom)?™
00:08:34 — Why a 5–10% P(Doom) Feels Patronizing
00:13:07 — Crux: “The Intelligence Ceiling Might Not Be That High”
00:17:18 — Power Structures vs. the Shortcut-Finding AI
00:26:03 — Thought Experiment: A Data Center From the Year 2126
00:28:25 — Is a Data Center as Dangerous as a Nuke?
00:33:41 — Cas’s Mainline Scenario: Slow-Burn Gradual Disempowerment
00:40:49 — A Multipolar Powder Keg: 10 Companies, Thousands of Open Models
00:43:48 — Boarding the Doom Train at a Later Station
00:45:49 — Why Cas Is an “Anti-Timelines” Person
00:52:26 — Sycophancy, MechaHitler & Nudification: Foreseeable Failures
00:57:18 — Crux: Does Humanity Get Retries?
01:02:39 — The Scariest Superintelligent Optimizers Are Companies
01:06:08 — Lessons From the Deepfake Ecosystem: DALL-E 2 vs. Stable Diffusion
01:12:42 — The Case for a Pause Treaty
01:16:35 — Whac-A-Mole Forever: What Winning Looks Like
01:20:51 — Enshittification and the 84% Who Don’t Use AI
01:30:30 — Alignment Research as Safety Washing
01:34:25 — The Jevons Paradox of AI Safety
01:36:48 — Cas Would Press the Button to Halt Superalignment
01:39:21 — Wrap-Up: The Unsexy Path to Lowering P(Doom)
Links
Cas’s links
* Cas’s website
* Cas on X
* Cas’s Google Scholar
* Cas’s MATS stream
Things referenced
* Superintelligence by Nick Bostrom
* Gradual Disempowerment (Duvenaud et al.)
* If Anyone Builds It, Everyone Dies
* AI 2027
* UK AI Security Institute
* Center for Human-Compatible AI (Berkeley)
* Internet Watch Foundation
Doom Debates episodes mentioned
* Mike Israetel Returns — AI’s Gonna Kill Everyone vs. AI Will Make Everything Awesome
* Top AI Professor Has 85% P(Doom) — David Duvenaud
* He Leads a Top AI Research Program, But He’d Hit the PAUSE Button — Kevin Zhu
* Alignment is EASY and Roko’s Basilisk is GOOD?! — Roko Mijic
* Andrew Critch vs. Liron Shapira: Will AI Extinction Be Fast Or Slow?
* AI Could Give Humans MORE Control — Ozzie Gooen
* Steven Byrnes Part 1
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏


Get full access to Doom Debates at lironshapira.substack.com/subscribe

Kevin Zhu walked away from a lucrative quant career to build Algoverse, one of the most productive mentorship programs for aspiring AI researchers. Can we agree on the best research path to a safe future? Let’s take a ride on the Doom Train!
00:00:00 — Cold Open
00:00:58 — Introducing Kevin Zhu
00:02:10 — From Citadel Quant to AI Researcher
00:09:14 — The Story of Founding AlgoVerse
00:12:53 — Discovering AI Safety: LessWrong & ARENA
00:17:22 — Emergent Misalignment Research
00:22:37 — Yudkowsky, MIRI & “Intellidynamics”
00:26:50 — What’s Your P(Doom)?™
00:29:37 — Kevin’s Timeline to AGI + AI 2027
00:30:42 — Would You Slow Down AI?
00:37:44 — Coming Out of the P(Doom) Closet
00:45:49 — Should We Shame AI Company Workers?
00:52:27 — OpenAI’s Superalignment Team Collapse
00:55:01 — Riding the Doom Train™
00:55:46 — First Stop: Instrumental Convergence
01:04:58 — Does Kevin Agree with the Orthogonality Thesis?
01:07:08 — “It’s Just Math.” Just Unplug It.
01:08:28 — “We Have a Safe Development Process”
01:11:49 — Group Dynamics & Laws Will Save Us
01:13:44 — Superintelligence Will Spare Us
01:14:25 — Is P(Doom) Just Bad Epistemology?
01:15:53 — China Will Race No Matter What
01:17:45 — Maybe Human Extinction Is Good?
01:20:36 — Wrap-Up
Links
Algoverse AI Research — https://algoverseairesearch.org/
Kevin Zhu on Instagram — https://www.instagram.com/kevinzhu.ai/reels/
Emergent Misalignment (Betley, Owain Evans et al.) — https://arxiv.org/abs/2502.17424
Emergent Misalignment via In-Context Learning (Algoverse follow-up, ACL 2025) — https://arxiv.org/abs/2510.11288
Agentic Misalignment (the blackmail result) — https://www.anthropic.com/research/agentic-misalignment
LessWrong — https://www.lesswrong.com/
PauseAI (global) — https://pauseai.info/
OpenAI Superalignment — disbanded (CNBC, May 2024) — https://www.cnbc.com/2024/05/17/openai-superalignment-sutskever-leike.html
Evan Hubinger — “Why I’m joining Anthropic” — https://www.alignmentforum.org/posts/7jn5aDadcMH6sFeJe/why-i-m-joining-anthropic
AI 2027 — https://ai-2027.com/
Bentham’s Bulldog / Matthew Adelstein on Doom Debates — https://doomdebates.com/p/benthams-bulldog-ai-doom-debate
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏


Get full access to Doom Debates at lironshapira.substack.com/subscribe

Liron explains the 80-year-old Erdős conjecture that a GPT model breakthrough, Scott Aaronson ponders "the last days of human relevance”, multiple live callers debate me, plus my 7-year-old son drops in to share his P(Doom)!
Timestamps
00:00:00 — Cold Open
00:00:42 — First Donation & “Why Is a Duck?”
00:02:54 — Can AI Draw “Colorless Green Ideas Sleep Furiously”?
00:08:26 — Doom Debates Sponsors Less Online — Be Our Intern
00:10:43 — Who We Still Need to Get on the Show
00:26:44 — Claude Opus 4.8 & the Rising Waterline of Intelligence
00:28:48 — The 80-Year-Old Geometry Conjecture a GPT Model Cracked
00:44:50 — My 7-Year-Old Ezra Joins: ChatGPT, Minecraft & His P(Doom)
00:58:46 — How the AI Actually Beat Erdős’s Grid
01:03:50 — Scott Aaronson: “The Last Days of Human Relevance”
01:07:53 — Why the Foom Is Taking Years, Not Hours
01:10:47 — Is Ori Just a Yes Man?
01:12:41 — Does P = NP + AI?
01:23:43 — METR’s Beth Barnes: “We Are Not On Top Of It”
01:28:31 — Liron’s Vibe Coding Confession
01:31:21 — Alverin Joins: Can We Wait, Then Hit the Off Switch?
01:48:01 — AI in a Box & the Super-Persuasion Threshold
02:06:24 — Brian Joins: The Ways It Could Go Right
02:11:30 — Jack Joins: How Fast Will the Foom Be?
02:18:48 — 80,000 Hours, Inventing Erdős Problems & Holly Elmore’s Warning
02:29:42 — Wrap-Up
Links
LessOnline 2026 — June 5–7, Berkeley — https://less.online/
Manifest 2026 — June 12–14 — https://manifest.is/
Shtetl-Optimized — “Dispatches from the possibly last days of human relevance” — https://scottaaronson.blog/?p=9782
Global Call for AI Red Lines (Bengio, Hinton, Harari) — red-lines.ai
Order the 80,000 Hours book — Benjamin Todd (Penguin) — https://80000hours.org/book/
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏


Get full access to Doom Debates at lironshapira.substack.com/subscribe

***ATTENTION: We’re looking for paid help at LessOnline from June 5-7. Apply now!***
OpenAI pushes the frontier of mathematics, Anthropic hires Andrej Karpathy to work on recursive self-improvement, live callers challenge the doom argument, and we've got a soundboard!
Doom Debates Paid Gig in Berkeley!
Doom Debates is proudly sponsoring The LessOnline conference this year and we got a merch table in the courtyard! We need 2 people to help represent the show — hand out T-shirts, talk to attendees, and spread the good word about Doom Debates. If you’re a regular viewer of the show, you probably know enough. Just be knowledgeable, friendly and helpful!
* 💰We pay you $500 to help out during the entire Fri-Sun event
* 🎟️ Plus you get FREE admission to the conference ($675 value)
This is best suited for someone local to the SF Bay Area or who can arrange accommodations. If you’re interested, send an email to internship@doomdebates.com or DM Producer Ori on Discord. Tell us a little about yourself and your availability for all 3 days. Thanks!
Links
Less Online conference (June 5–7, Berkeley) — https://less.online
Doom Debates Discord — https://discord.gg/g2X8h5UCrU
OpenAI — “An OpenAI model has disproved a central conjecture in discrete geometry” — https://openai.com/index/model-disproves-discrete-geometry-conjecture/
Timestamps
00:00:00 — Welcome + New Soundboard!
00:09:27 — Doom Debates Is Sponsoring Less Online
00:17:30 — Ben Goertzel Episode Recap
00:27:30 — Goertzel's Alignment Blind Spot
00:34:21 — OpenAI Disproves an 80-Year-Old Erdős Conjecture
00:41:00 — David Deutsch Says "Amazing"
00:50:31 — Eliezer Yudkowsky's Job Replacement Progression
00:54:07 — Anthropic's $44 Billion Run Rate
01:07:45 — Schadenfreude Week: Liron Loses $10K
01:13:04 — Teaching Ezra Prediction Markets
01:18:11 — Caller Williawa: Is the Goal Engine Argument Effective?
01:37:00 — Caller Will: Brain Rot and Idiocracy
01:55:04 — Anthropic Hires Karpathy for Recursive Self-Improvement
01:57:16 — Rob Wiblin Says Recursive Self-Improvement Should be "Illegal"
02:01:37 — Caller Henry: The Chicken Riddle
02:08:41 — David Sacks Kills the AI Executive Order
02:11:20 — The Philanthropy Tidal Wave
02:15:56 — Donate to Doom Debates
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏


Get full access to Doom Debates at lironshapira.substack.com/subscribe

Now that Doom Debates is almost 2 years old, enough time has passed that we can revisit my claims from 2024. I claim they’ve been aging well!
In this early episode, I react to Martin Casado, a General Partner at Andreessen Horowitz (a16z) who claims that AI is basically just a buzzword for statistical models and simulations. As a result of this worldview, he only predicts incremental AI progress that doesn’t pose an existential threat to humanity, and he sees AI regulation as a net negative.
From my perspective, Martin’s problem is that he needs to go beyond analyzing AI as just statistical models and simulations, and analyze it using the more predictive concept of “intelligence” in the sense of hitting tiny high-value targets in exponentially-large search spaces.If Martin appreciated that intelligence is a quantifiable property that algorithms have, and that our existing AIs are getting close to surpassing human-level general intelligence, then hopefully he’d come around to raising his P(doom) and appreciating the urgent extinction risk we face.

Links
Watch the original episode of the Cognitive Revolution podcast with Martin and host Nathan Labenz — https://www.youtube.com/watch?v=oZ7788oKoss
Follow Martin — https://x.com/martin_casado
Follow Nate — https://x.com/labenz
Follow Liron — https://x.com/liron

Timestamps
00:00 Introducing Martin Casado
01:42 Martin’s AGI Timeline
05:39 Martin’s Analysis of Self-Driving Cars
15:30 Heavy-Tail Distributions
38:03 Understanding General Intelligence
38:29 AI's Progress in Specific Domains
43:20 AI’s Understanding of Meaning
47:16 Compression and Intelligence
48:09 Symbol Grounding
53:24 Human Abstractions and AI
01:18:18 The Frontier of AI Applications
01:23:04 Human vs. AI: Concept Creation and Reasoning
01:25:51 The Complexity of the Universe and AI's Limitations
01:28:16 AI's Potential in Biology and Simulation
01:32:40 The Essence of Intelligence and Creativity in AI
01:41:13 AI's Future Capabilities
02:00:29 Intelligence vs. Simulation
02:14:59 AI Regulation
02:23:05 Concluding Thoughts
---
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏


Get full access to Doom Debates at lironshapira.substack.com/subscribe