Beyond Coding

How does Amazon build its agentic AI? Michael Giannangeli, Head of Product for Amazon Nova and Agentic AI, breaks down evals, RL gyms, and model routing.
He also explains why the bottleneck in software has shifted away from engineering hours and what takes its place.
In this video, we cover:
The eval lifecycle: building from real failure modes, saturation, and why 100% means delete
RL gyms: training models on real environments like migrations, DevOps, and pen testing
Model routing, cost-per-token trade-offs, and why routing isn't solved
The agent stack of an Amazon product lead: Claude Code, Codex, and Kiro
Autonomous migrations, trust, and how much human-in-the-loop survives
For engineers and product people building with AI agents who want to see how a frontier lab actually closes its feedback loops.
Recorded at the AI4 conference 2026.

Timestamps:
00:00:00 - Intro
00:00:36 - The Agents an Amazon Product Lead Uses Daily
00:03:36 - Why Nobody's Heard of Amazon Nova
00:04:55 - Model Costs and the Routing Problem
00:08:10 - Why Building Good Evals Is So Hard
00:10:05 - When Evals Saturate and Get Deleted
00:12:17 - Turning Real Failure Modes Into Hundreds of Evals
00:15:26 - Improving Models Without Training on Customer Data
00:18:26 - If Everyone Uses Agents, You Need Agents
00:20:22 - The Bottleneck Is No Longer Engineering Hours
00:23:20 - Ship Fast to Validate the Right Thing
00:26:44 - Staying at the Frontier Amid Constant Noise
00:29:37 - Spend 10-20% of Your Time Experimenting
00:32:54 - RL Gyms: How Models Learn From Failure
00:37:09 - Will Migrations Become Fully Autonomous?

Guest: Michael Giannangeli - Head of Product, Agentic AI & Amazon Nova at Amazon
#AmazonNova #AgenticAI #AIEngineering

AI is changing what developers build, but code alone is no longer enough to prove what you can do. Wes Bos explains why engineers need to solve problems beyond syntax, how agent workflows are reshaping software development, and what still requires human thinking.
In this conversation:
The limits of generative UI and AI-generated design
Agent loops, harnesses, and cheaper AI models
The rising cost of AI coding and the case for local hardware
Why developer education is shifting from syntax to problem-solving
Personal branding, conferences, newsletters, and AI-generated content
For developers navigating AI-assisted coding, this episode explores the skills and signals that still help you stand out.
This podcast was recorded at JSNation, the key web dev conference.

OUTLINE
00:00:00 - Code Is Not Enough for Developers
00:00:32 - Why Generative UI Still Feels Unfinished
00:04:35 - How Agent Loops Improve AI Coding
00:07:06 - When Agent Workflows Become Standard Tools
00:08:19 - Are Cheaper AI Models Good Enough?
00:10:44 - Can AI Coding Costs Stay Sustainable?
00:12:24 - What Engineers Need To Learn Now
00:14:23 - Why Fundamentals Matter Beyond Syntax
00:15:34 - How Non-Coders Are Building Production Tools
00:16:21 - Why In-Person Conferences Still Matter
00:18:11 - Personal Branding When Code Isn't Enough
00:20:37 - Can Newsletters Beat The Attention Crisis?
00:22:02 - Why AI-Generated Content Feels Insulting
00:24:12 - Use AI To Scaffold, Not Think

Answering engineer questions on AI pressure, career growth, product thinking and impact. Including the production incident I'm glad happened, and the mindset I refuse to accept when things break.

In this video, we cover:
- Whether managers are really demanding more output because of AI
- Balancing fundamentals with AI coding tools and agents early in your career
- Specialist vs generalist and when to lean into each
- Visibility, personal branding and who gets credit for your work
- Product thinking, evaluating impact and what I got wrong about content being king

For software engineers at any level who want honest answers on career strategy in the agent era, from someone doing both engineering and product.

Timestamps:
00:00:00 - How to Spot the Next Big Thing
00:03:15 - The Saying I Hate Most
00:04:27 - The Production Mistake I'm Glad I Made
00:07:32 - Are Managers Demanding More Because of AI?
00:13:39 - Learning Fundamentals vs AI Coding Tools
00:19:00 - Will AI Ever Get Good at Distributed Systems?
00:20:51 - Specialist vs Generalist: When to Lean In
00:26:35 - How to Become More Visible in Your Org
00:31:49 - I Was Wrong: Content Isn't King
00:35:03 - Workflows, Priorities and Hiring an Editor
00:37:08 - What Being a Force Multiplier Really Means
00:41:26 - How to Evaluate What's Worth Building
00:45:01 - Product Thinking Without Years of Experience
00:48:13 - Energy Management, Curiosity and Defining Success
00:54:21 - Hair Talk

How do you prove AI is shipping more features? Amos Haviv leads the Developer Workflow teams at Booking.com, supporting 4000 engineers operating 8000 repos.
Everybody is burning through their AI budget right now and almost nobody can answer what it bought them. Amos can, because his team spent four years building an event system to debug their own SDLC before AI upped the urgency.
In this video, we cover:
Why verification is the bottleneck right now, and where it moves next
Building an event store that separates KTLO from real feature delivery
Why static dashboards create the metric they measure, and the cobra story behind it
Agent cost, model routing, and why Booking ignores token maxing entirely
Running a developer survey with a 92% response rate across 3k+ engineers
Who should own skills and MCPs: a central platform team or the domain experts?
For platform engineers, engineering leaders, and anyone being asked to prove ROI on AI tooling this quarter.

Timestamps:
00:00:00 - Everyone is burning through their budget
00:00:32 - Verification Is the Bottleneck Every Team Hit
00:03:35 - 4,000 Engineers and 8,000 Repos at Booking.com
00:06:48 - Why Copying Google and OpenAI Will Break You
00:09:21 - Verification Is a Stack of Agents, Not One Review
00:13:27 - Cost Is Becoming a Bottleneck of Its Own
00:17:14 - Was the Internet a Bubble? What That Teaches Us
00:25:32 - What Working With the Frontier Labs Looks Like
00:28:26 - Debugging the SDLC With Four Years of Event Data
00:30:24 - Do Engineers Using AI Actually Ship More Features?
00:37:13 - Where to Start If You Measure Nothing Today
00:45:01 - The Cobra Effect: When a Metric Becomes a Target
00:52:23 - Everyone Is a Builder Now, and Everything Needs Support
01:01:21 - Is AI Turning Every Engineer Into a Manager?
01:03:46 - The Developer Survey With a 92% Response Rate
01:10:09 - Who Owns Skills, MCPs, and the Enterprise Harness
01:17:46 - Great Developer Experience Is High Velocity

Mentioned in the episode:
High Output Management by Andy Grove
The Sovereign Individual (1997)
The story of General Magic

Views expressed are Amos's own and do not represent Booking.com.
#AI #SoftwareEngineering #DeveloperExperience

"I need to stop using Opus. This doesn't work." That was Heitor Lessa's conclusion after a refactor cost him 200 million tokens, and it forced him to rebuild the entire agent workflow now available for 1400 engineers. Heitor spent 11 years at AWS, built Lambda Powertools to 230 billion API calls a week, and in this episode he walks through the full SDLC workflow on screen, from discovery to merge check.
In this episode, we cover:
The product loop: discovery, whiteboarding, and the /roadmap command
Spec-driven development with Open Spec and why vanilla setups fail
Three model tiers: SOTA for planning, mid-tier for implementation, cheap models for reviews
Merge checks with adversarial reviewers and attestations that catch agents fabricating test results
The /retro command: using the Socratic method to make your workflow more deterministic
If you're an engineer figuring out how to work with agents at team scale without losing trust in your codebase, this is the workflow to steal. This is also the first Beyond Coding episode with visuals on screen, so let me know what you think of the format.
Timestamps:
00:00:00 - The Math Doesn't Add Up
00:00:43 - Amazon Hypergrowth: 11 Years, 8 Different Roles
00:03:29 - Learning From the Trenches as a Technical Account Manager
00:08:38 - Developer Identity and the Birth of Lambda Powertools
00:10:20 - The Hard Parts of Working in Public
00:13:12 - How Powertools Hit 230 Billion API Calls a Week
00:16:42 - Career Advice: Learn Adjacent Roles, Not More Tech
00:19:37 - When Leadership Decisions Don't Make Sense to You
00:23:21 - The Product Loop Starts With Discovery
00:25:22 - From Whiteboard to /roadmap
00:27:37 - Why Humans Plan First and Agents Come Second
00:30:33 - Commands vs Skills Across 32 Different Models
00:33:38 - Adversarial Reviewers on Every Plan
00:36:07 - The Socratic Method, Explained
00:40:29 - Why He Only Takes Paper Notes
00:44:43 - The Five-Line Paper Trick for High-Stakes Meetings
00:48:18 - /new-work: Capturing Scope Creep Without Derailing
00:54:03 - The Dev Loop Begins: Open Spec Explore
00:56:34 - Three Model Tiers: SOTA, Mid, Cheap
00:57:43 - The $5,000/Month Per Engineer Question
00:58:57 - Guardrails vs Autonomy for 1,400 Engineers
01:04:22 - Auto-Sizer: Does This Task Even Need a Spec?
01:07:26 - Decision Fatigue and Why Frameworks Win
01:09:10 - The Plan Phase: Specs, Design, Formal Verification
01:13:07 - The Refactor That Cost 200 Million Tokens
01:15:11 - When Agents Forge Evidence They Ran Your Tests
01:17:27 - Local-First Architecture Explained
01:23:04 - The Apply Phase: Fully Autonomous Loops
01:24:30 - Coding Was Never the Bottleneck
01:26:39 - Why This Workflow Is an Investment
01:27:39 - Decision Logs and the /onboarding Command
01:29:06 - Running Agents Locally With Enterprise Governance
01:32:42 - Hooks: Making Quality Gates Deterministic
01:36:02 - Merge Checks: 15 Adversarial Reviewers Per Change
01:38:30 - /retro: Interviewing Yourself to Improve the Loop
01:43:12 - Trust, Loss of Trust, and Recovery With Agents
01:48:02 - Experience, Scars, and Critical Thinking
01:49:32 - Why Right Now Is the Time to Experiment
01:52:04 - Conviction Comes From Being in the Loop
#softwareengineering #aiagents #aws