TestGuild Automation Podcast

In this episode 600 of the TestGuild Automation Podcast, Joe Colantonio talks with Jason Arbon, founder of Testers.ai, Jank.AI and IcebergQA and author of the new book Testing AI: Engineering Confidence in Non-Deterministic Systems.
Take Our 2027 Survey Now: https://testgld.link/27data
Jason makes a case most testers have not heard yet. Coding is being absorbed by AI. Specification work is thinning out. Product, development, and test roles are converging into one. And when the music stops, the only seat left belongs to the person who can look at what the machine produced and make an evidence backed call on whether it ships. He calls that confidence engineering, and he argues it is not a rebrand of QA. It is what QA was always supposed to be.
Along the way, Joe and Jason get into the containment problem and why alignment, not lockdown, is now the real safety goal. They dig into why testing cost scales quadratically, meaning ten times more generated code creates roughly a hundred times more testing demand. Jason also pushes back hard on skeptics of agentic testing, pointing out that almost nobody has run the obvious experiment of testing a site themselves for a week and comparing their results against what AI finds.
You will also hear Jason's most practical piece of advice in the whole conversation. If you are not running the same suite five times against the same build and looking at the actual results, not just flake, you are not testing seriously in an AI world.
Plus a detour into grokking, the Chinese Room, and Geoffrey Hinton, because it would not be a Jason Arbon episode without one.
Listen up!

Your developers just got supercharged by AI coding agents. Your test coverage did not. So how do you keep quality high when product code is shipping faster than any test team can follow?
In this episode Andrew Knight, the Automation Panda and Senior Director of Product and Engineering at Cycle Labs, shares how his six person team is running the biggest release quarter in company history using AI coding agents, spec driven development, and Playwright.
You will discover:
How Playwright turned itself into an AI automation platform with the MCP server, the planner, generator, and healer agents, and the new CLI and skills approach that cuts your token usage way down.
Why Andy uses Spec Kit to codify his testing strategy once, in markdown, so quality standards get baked into every single pull request instead of being caught in review.
How to decide which AI generated tests are actually worth running when compute time and budget are finite.
What AI slop looks like from a manager's seat, and how to build a team culture that catches it before it ships.
Why Andy believes AI coding tools are the new compiler and markdown is the new programming language, plus my pushback on what that means for everything testers were trained to care about.
What Andy really thinks about token costs, subscription tiers, and what happens when the AI subsidies run out.
Plus the one piece of advice he gives to any tester still sitting on the sidelines of the AI shift.
Whether you are an automation engineer, a QA lead, or an engineering manager trying to figure out where testing fits in an AI first workflow, this episode gives you a practical playbook you can start using this week.

Keith Klain has spent 25+ years leading enterprise quality programs in financial services and is one of the testing industry's most respected voices. In this episode, he joins Joe to discuss his BrowserStack talk, "The Death of Test Engineering," why testing isn't dying—but outdated testing roles are—and how AI is reshaping the future of software quality.
In this episode:
Why "The Death of Test Engineering" isn't what you think
The difference between checking and testing
Will AI replace software testers?
Why confidence engineering matters
How to communicate testing in business terms
Where automation adds real value
Why agentic AI testing is overhyped
Notable Quotes
"If you look at AI as a competitor, you're screwed. If you look at it as just another tool... it's not a threat."
"Uncertainty in testing is a feature, not a bug."
"If an organization doesn't value testing, you're doing it wrong."
"You can be incredibly confident about something and incredibly wrong."
One Thing to Try This Week
Ask yourself after every critical defect:
"What would the business impact have been if this had gone live?"
It's a simple way to shift conversations from test cases to business risk.
Resources Mentioned
Rethinking Expertise — Harry Collins
How Complex Systems Fail — Richard Cook
Measuring and Managing Performance in Organizations — Robert Austin
Emily Bender
Alex Hanna
Timnit Gebru
Dagmar Monett
Olivia Gambelin
Ed Zitron
Joe's Recommendation: Prophecy

Nandini Srinivasan has spent 25 years in the quality industry and leads a global QA organization of over 150 engineers across the US, Canada, India, and Pakistan.
In this episode, she breaks down exactly how she built a QA AI acceleration charter, ran a train-the-trainer model, and used a phased proof-of-concept approach to separate real AI from what she calls "powerful automation dressed up as AI. 
We get into:
How she frames quality metrics for executives
Using language around revenue protection
Risk mitigation
Feature velocity instead of test coverage percentages.
She talks about the four pillars she uses to present her team's value: quality, scalability, performance, and availability.
Nandini also shares her take on the future of QA hiring, why the judgment layer will always require a human, and what skills testers need to stay relevant as AI agents take over the more mechanical parts of automation.
She is also writing a five-part LinkedIn series called "The Voice of QA in the AI Era" if you want to follow along.

Matt Wynne, co-creator of Cucumber and BDD practitioner, joins Joe for the first time in over a decade to talk about what two years inside a Silicon Valley AI startup taught him about the future of software testing.
Matt spent time at Mechanical Orchard working alongside experienced XP practitioners to modernize legacy COBOL mainframes using LLMs, and then spent a week with the team that coined the term "software factory," where the rule was simple: humans never write the code, never read the code.
In this episode, Matt breaks down what harness engineering actually means, why shared understanding is still the real bottleneck even in an agentic world, and how testers can use multiple LLMs to review AI-generated pull requests without reading every line.
He also gets honest about the grief that comes with realizing you can encode years of hard-won expertise into a Markdown file, and why that does not mean your skills are worthless.
If you are working in a brownfield codebase, wondering how to handle the flood of agentic PRs, or trying to figure out where testers fit in a world where agents write the code, this conversation is worth your time.
Find Matt at:
mattwynne.net
leansoftware.ai
Also check out his course:
Build a Software Factory: Hands-off agentic coding for experienced engineers
https://testgld.link/mattcourse