The Information Bottleneck

Ion Stoica has done what almost no academic ever does — repeatedly turned university research into billion-dollar companies. He co-founded Databricks (now valued at over $100 billion), Anyscale, Arena AI and Conviva, while his Berkeley lab produced the open source projects the entire AI industry runs on: Ray, vLLM, and SGLang.
In this episode, we ask him how it's actually done. His answer is surprisingly unromantic: solve a problem people already care about, build an artifact good enough that they adopt it, and pay attention to the moment users start asking "who maintains this after the students graduate?" - that's when a project becomes a company. He's also insistent that the credit belongs to his students.
From there, the conversation goes deep into what he's watching now: why the AI stack has become an order of magnitude more complex than the Hadoop/Spark era, why maximizing GPU utilization is "the name of the game" for any enterprise, and why coding agents will struggle with distributed systems long after they've mastered web apps. He shares a memorable reward-hacking story — a load balancer that maximized throughput by dropping requests — explains why the gap between open and closed models sits at about six months, and closes with his case for regulating AI by outcomes, not capabilities.
Timeline
00:00 — Introduction: welcoming Ion Stoica
01:21 — The playbook: how research projects become companies
05:22 — Will vLLM and SGLang stay open source?
07:47 — The real bottleneck in the AI stack: complexity, not just hardware
14:31 — Should algorithms follow infrastructure, or the other way around?
16:13 — Can AI coding tools write distributed systems and GPU kernels?
21:09 — Verifiers, harnesses, and the limits of outsourcing understanding
25:41 — Reward hacking: the load balancer that dropped requests
25:58 — How should enterprises consume GPUs? Utilization as the name of the game
30:23 — GPU scarcity: will the compute crunch ever end?
35:27 — Hyper-optimization and the risk of locking in today's architectures
37:17 — Open vs. closed models: why every company wants to own the stack
40:35 — The six-month gap, and the rising cost of training frontier models
43:58 — Kimi, Qwen, and who's incentivized to keep open models alive
45:39 — Regulation: outcomes, not capabilities
47:41 — Self-regulation, concentration of power, and auditing open models
48:32 — Wrap-up
Music
"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.
About
The Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.

Jean-Francois Puget is a Director and Distinguished Engineer at NVIDIA, where he leads the Kaggle Grandmasters team, and he's ranked third on Kaggle's all-time list. We caught him on the day NVIDIA announced Nemotron Ultra and its new agent skills repo. We talk about what skills actually are, why they beat MCP tools on context cost, and how NVIDIA built an evaluation pipeline to separate skills that help from skills that don't.
From there we talk about the thing JFP cares about most: evaluation. He explains why most LLM benchmarks reward overfitting, how his team discovered O3 could pick the right files to fix SWE-bench issues without reading them, and why the only benchmarks he trusts are the ones where you commit before you see the score, which is exactly how Kaggle works. He predicts a "bloodbath" for the wave of competitors letting coding agents chase leaderboard scores with no notion of validation.
We also get into what coding agents are actually good for ("a mix of a genius and a dumb person"), the multi-agent system at NVIDIA that built a working PyTorch clone that runs 10x slower than the real thing, his unfiltered take on frontier lab PR and the Mythos release, whether AI is a bubble, and the story of how his team won ARC-AGI with a 4-billion-parameter model at 20 cents a task, including jumping from third to first in the final hours of a seven-month competition.

Timeline
00:00 — Intro
01:05 — NVIDIA's announcements: Nemotron Ultra and the agent skills repo
07:21 — Skills vs MCP tools, and progressive disclosure
10:24 — Agents that write their own skills: a new form of learning
13:33 — When overfitting is fine (and when it isn't)
15:47 — Why most LLM benchmarks reward overfitting
17:06 — The SWE-bench contamination story: O3 picks files without reading them
19:45 — How LLMs changed Kaggle, and the coming "bloodbath"
25:40 — What makes a good data scientist: evaluation and one-bit experiments
28:56 — Running Codex at scale: the top token consumers at NVIDIA
29:37 — Did coding agents kill AutoML?
30:16 — Genius and dumb at once: the limits of coding agents
35:21 — Humans in the loop, sandboxing, and the teenage hacker who never wrote code
37:42 — Mythos, frontier lab PR, and open source
40:08 — Why NVIDIA builds open models, and where it's already frontier
43:48 — World models, robots, and the coffee test
49:20 — Why agents still can't play Dota
50:24 — Is AI a bubble?
53:14 — Winning ARC-AGI with a 4B model at 20 cents a task
57:39 — Kaggle is a legal drugMusic:
"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.
About: The Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.

In this episode, we talked with Dimitris Papailiopoulos, researcher at Microsoft Research's AI Frontiers lab and professor at the University of Wisconsin, about doing research in the age of agents. Dimitris told us about the Sunday morning that changed how he works: he handed Claude Code and Codex a question he'd been sitting on for years, went about his day, and came back to an answer. After a few days of dread about what's left for humans, he landed somewhere more optimistic, calling this the golden age of asking questions.
We talked about his "smallest transformer that can add" leaderboard, a symbolic GSM8K solver built from if-else statements, and what happened when he put two Claude Code instances in the same file system and told them to do something cool (one pair invented a communication protocol, the other played Battleship). We also got into diversity and slop in agent-generated ideas, why agents get stubborn after a million tokens, harness overfitting on Terminal-Bench, continual learning and world models, whether agents need vision, and where information theory actually helps in AI and where it's a katana used to make coffee.

Timeline
00:00 Intro
01:45 How agents changed the way Dimitris does research
04:30 A Sunday morning with Claude Code, Codex, and GSM8K
07:15 The dread, then the golden age of asking questions
08:20 Taste and verification, and how we train students now
09:53 Will models make human verification obsolete?
11:30 The smallest transformer that can add 10-digit numbers
13:40 Humans as initializers for gradient descent in idea space
15:32 Allen on diversity, slop profiles, and high temperature research
21:44 When Claudes meet: Battleship, invented protocols, and a grokking paper
25:53 Single agent vs multi-agent under fixed compute
30:28 Auto-research benchmarks and what agents actually accelerate
35:14 Inside the symbolic GSM8K solver (with a live progress check)
40:04 Idea overfitting and why agents refuse to change course
44:00 Learning from failure traces and harness overfitting
48:04 Continual learning, memory files, and world models
51:30 Why don't labs personalize models on your own history?
57:52 Agent-to-agent communication: is Jira the right tool?
1:01:25 Multimodality: vision as a tool vs one unified model
1:05:40 Information theory and AI, or making coffee with a katana
1:11:23 Closing thoughts: ask bigger questions
Music:
"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.
About: The Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.

Phillip Isola, professor at MIT, joins us to talk about representation learning: what makes a representation good, why different models seem to converge on similar representations, and whether pre-training is really over.
We discuss the platonic representation hypothesis and its limits, why clustering structure matters more than global geometry, and Phillip's new neural thickets paper arguing that post-training is easier than people think because pre-trained weights already sit near solutions to downstream tasks. Phillip also explains why he thinks LLMs are already world models, why he's betting on RNNs making a comeback, and why his most exciting current direction is artificial life: putting LLM agents in open environments with no fixed task and studying them like new organisms.

Timeline:
00:00 Intro song
00:13 Intro
01:05 What is representation learning and why it matters
04:09 What makes a representation good: minimality and sufficiency
10:03 How cross entropy and contrastive learning shape representations
14:35 Dimensionality reduction and why dimension isn't the right complexity measure
16:35 Compression and geometric clustering during training
19:27 The platonic representation hypothesis and what actually converges
22:53 Local neighborhoods vs global structure: the Aristotelian follow-up
24:33 When convergence is strong: truth vs the space of possibility
28:09 Is there true similarity in the world? The Bouba-Kiki effect
30:56 World models vs autoregressive LLMs
32:14 Diffusion LLMs as a special case of autoregressive models
33:42 What architectures win in five years: the case for RNNs
36:11 Grad student descent, or do we actually have principles?
40:51 Feathers and wings: what to take from biology
43:17 How close are we to brain-like models? Marr's three levels
47:01 Are better models becoming less human-like?
49:38 Is pre-training all you need? The neural thickets paper
54:18 LoRA, low rank fine-tuning, and why post-training is easier than we thought
56:01 RL environments and what our benchmarks actually test
1:01:11 Artificial life: LLM agents as new organisms
1:07:20 What's overlooked in AI research right now
1:08:36 Why stay in academia, and doing science in the age of Opus

Music:
"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

About: The Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.

Most AI-for-science companies are selling shovels. Qichao Hu wants the gold.
In this episode, we talk with Qichao, the founder and CEO of Molecular Universe, the AI-for-science platform that grew out of SES AI, a high-energy-density battery developer he's run for fourteen years. His core distinction is that companies from the AI world build tools, such as foundation models that predict properties, while companies from the science world care about the final product, such as the new battery or material that actually ships. Molecular Universe sits firmly on the science side, and the difference shows up everywhere from what they publish to what they refuse to.
We get into the actual workflow of materials discovery and where AI compresses it. A single trial in a traditional lab can take a year with maybe a 40% success rate; the goal is to run a thousand candidates in parallel and turn that year into a week. Qichao walks through improving low-temperature fast-charging for EV batteries:  from hypothesis generation through molecule-, material-, and device-level property prediction, down to autonomous labs that synthesize and test the top candidates without a human touching a pipette.
The hardest problem, it turns out, isn't predicting molecular properties or measuring device performance, but it's the black box connecting the two. In batteries, that's the solid-electrolyte interface, which the field has been hand-waving about since the seventies. And the thing standing in the way of cracking it isn't a clever training trick but data: companies sitting on twenty years of records are finding it too messy, incomplete, and poorly labeled to train on, and are having to start collecting from scratch with new protocols and robots.
Timeline
00:13 — Intro and welcome;
01:19 — Shovel vs. gold
05:18 — Why the world's smartest scientist doesn't automatically give you a better battery
07:25 — The discovery workflow
09:37 — Exploration vs. exploitation
11:54 — Safety and filtering: screening novel molecules against banned and toxic-substance lists
17:55 — How hypotheses get generated, and where frontier LLMs help
20:29 — From hypothesis to ~400 formulations: property prediction, ranking, and handing off to autonomous labs
26:37 — "A foundation model for everything" — and the black box between molecular properties and device performance
30:01 — World models and physics
33:09 — The great unknown in batteries
37:08 — Simulation vs. reality: calibrating massive simulated datasets with a sliver of experimental data
41:47 — Lab robotics: how fast the hardware has caught up, and what a floor of autonomous labs looks like
43:50 — The real bottlenecks
50:21 — Pre-training from scratch vs. post-training LLMs, and why training tricks haven't reduced the need for good data
52:42 — Evaluation
55:42 — Publish the B+ model, keep the A model
58:05 — Five years out
1:00:37 — Closing thoughts and wrap
Music:
"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.
About: The Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.