Data Engineering Podcast

Summary
In this episode, I spoke with Steven Renwick, CEO and co-founder of Tilores, about the realities of entity-oriented data processing and the practical role of LLMs in entity resolution. We covered the confusing terminology around entity resolution, identity resolution, record linkage, named entity recognition, and disambiguation, and why these problems show up almost everywhere once organizations reach a certain scale. Steven explained why entity resolution is especially critical in domains like financial services, public sector systems, logistics, and any environment where messy, duplicated, or inconsistent records can create compliance, operational, or analytical risk.
We also dug into the question of whether LLMs can replace traditional entity resolution systems. Steven shared why the answer is only partly yes: LLMs can be very useful for discovery, entity extraction, weak-match review, onboarding, and configuration, but they struggle as the core production matching engine because of cost, scale, consistency, explainability, and state management requirements. We explored why graph structures are so useful for representing entities, how TeloRes approaches scale without relying on graph databases directly, and why teams should think carefully about build-versus-buy when tackling entity resolution. The conversation closed with lessons from real-world data quality, the growing importance of trustworthy identity layers for AI and analytics, and how clean entity data can strengthen everything from knowledge graphs to AI applications.

Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
Today’s episode is sponsored by Parallel - where agents find answers. Most engineers today closely follow new model releases, but don’t pay attention to their agent’s most important tool: web search. Parallel develops enterprise-grade infrastructure for agents to retrieve high-quality context from the web. Their core products are a suite of APIs for retrieving high quality information from the web with Pareto-optimal quality, cost, and speed. Whether you work on voice agents that need 200 millisecond latency, chat bots that balance speed, depth, and quality, or long-horizon agents to do thorough, overnight research for you, Parallel is a single platform for all your agentic research. Get started for free at dataengineeringpodcast.com/parallel
Your host is Tobias Macey and today I'm interviewing Steven Renwick about whether and where to use LLMs in entity-oriented data processing
Interview
Introduction
How did you get involved in the area of data management?
Can you start by giving your definition of what constitutes an "entity"?What are some examples of the types of data work that rely heavily on entity resolution/disambiguation?

How do LLMs alter or augment the functional capabilities of entity-oriented data processing?
As with many areas of knowledge work, LLMs can easily create a compelling demo, but quickly degrade when trying to go from demo to production. What are the traps and pitfalls that become evident when trying to do entity resolution in production?What are your heuristics for what constitutes the threshold beyond which naive approaches become untenable?

Can you describe what a production grade pipeline for entity resolution looks like?What role can/should LLMs play in that process?

As LLMs become more powerful and faster, how/does that change the equation of their role in entity-oriented data processing? (particularly with diffusion-based models)
What are the most interesting, innovative, or unexpected ways that you have seen LLMs used in entity extraction/resolution/disambiguation?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on entity resolution at scale?
When is entity resolution overkill or unwarranted?
What are some predictions or wishes that you have for the future directions/applications of entity resolution?
Extra - have we looked at the new "System One" AI decision models,like Jev?
Contact Info
LinkedIn
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.
Links
Tilores
Named Entity Recognition (NER)
Entity Resolution
Neo4J
AWS Neptune
Apache Spark
Graph Traversal Algorithms
Entity Resolution Blocking
GraphQL
Clique-Based Graph Compression
Splink
Levenshtein Distance
Cosine Similarity
GraphRAG
Entity-Resolved Knowledge Graph
JEV Decision Model
Laya
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Summary
In this episode, I spoke with Keerti Kulkarni about the data and AI systems behind physical logistics automation at Amazon. She explained how her team supports dock-door operations across more than 100 North American facilities by turning telemetry from physical workflows into actionable operational intelligence. We discussed the end-to-end data path from CloudWatch, Firehose, and S3 into Redshift and AI-powered analysis, along with the realities of handling petabyte-scale event streams, late and duplicate data, reconciliation, cost controls, and the need to treat physical telemetry as a first-class signal rather than noisy exhaust. Kirti also shared how these systems are designed to protect worker privacy while still surfacing the right information for safety and operational decisions.
We also dug into the role of LLMs and agentic workflows in helping operations teams diagnose failures quickly through conversational interfaces. Kirti described how AI-generated summaries help identify root causes such as authorization problems, hardware issues, or connectivity failures, while human feedback loops, structured outputs, and strict vocabulary controls help manage hallucination risk. Beyond troubleshooting, we explored how the platform is evolving toward proactive maintenance, training insights, per-site cost attribution, and broader modular agent architectures that could be shared across logistics teams. It’s a detailed look at what it takes to combine data engineering, real-time operations, and AI in a large-scale physical environment.

Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
Today’s episode is sponsored by Parallel - where agents find answers. Most engineers today closely follow new model releases, but don’t pay attention to their agent’s most important tool: web search. Parallel develops enterprise-grade infrastructure for agents to retrieve high-quality context from the web. Their core products are a suite of APIs for retrieving high quality information from the web with Pareto-optimal quality, cost, and speed. Whether you work on voice agents that need 200 millisecond latency, chat bots that balance speed, depth, and quality, or long-horizon agents to do thorough, overnight research for you, Parallel is a single platform for all your agentic research. Get started for free at dataengineeringpodcast.com/parallel
Your host is Tobias Macey and today I'm interviewing Keerti Kulkarni about scaling insights for physical logistics automation with AI-enabled intelligence

Interview

Introduction
How did you get involved in the area of data management?
Can you start by unpacking what "physical logistics automation is"?What are the technical and organizational systems involved in the production and consumption of that data?

What are some of the ways that inaccuracies or increased latency manifest in real-world impact?
You recently worked on a substantial system for managing this data. Can you describe the challenges and limitations of the system that you started from?
What was your process for identifying the weak points of the system and how to evolve it?
What is the design of the system that you have running today?What are some of the design choices that had the biggest impact on the final result?

What are the most complicated aspects of managing cost attribution to the different physical locations, data flows, and model usage?
What is the role of AI in providing insights to the organizational users?How do you measure and manage accuracy of the models as the system evolves?

What are the most interesting, innovative, or unexpected ways that you have seen your system used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on physical logistics automation?
What do you have planned for the future of your platform?

Contact Info
LinkedIn
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links
Amazon Key
AWS Cloudwatch Analytics
AWS Firehose
AWS Kinesis
AWS Quick
OSHA == Occupational Safety and Health Administration
OmniGraphPodcast Episode

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Summary

In this episode, I spoke with Christian Bremeau and Simon Dynin about their decades-long work in metadata management and how that experience shaped the evolution of Meta Integration Technology and its direct-to-customer MetaKarta platform. We explored the recurring challenges of fragmented data ecosystems, why every tool inevitably creates its own metadata silo, and why organizations still need a vendor-neutral system to stitch lineage, semantics, and change history across databases, ETL/ELT pipelines, BI tools, and cloud platforms. Christian and Simon explained why deep lineage, versioning, configuration management, and deterministic understanding of transformations matter far more than a polished catalog UI when teams need to trust numbers, assess impact, or satisfy audits.

We also dug into the growing importance of metadata in the age of AI. Christian and Simon shared their view that metadata management is becoming foundational for both BI and agentic systems, especially as organizations struggle with semantic drift, tool sprawl, and non-deterministic AI behavior. They introduced Metacarta’s new Semantic Hub as a way to reconcile business logic across legacy and modern tools, compile shared semantics into downstream systems, and provide consistent answers for both human analysts and AI agents. Along the way, we discussed metadata ops, the organizational processes needed to make metadata programs succeed, and the practical situations where a cross-platform metadata system is—or isn’t—the right fit.

Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
Today’s episode is sponsored by Parallel - where agents find answers. Most engineers today closely follow new model releases, but don’t pay attention to their agent’s most important tool: web search. Parallel develops enterprise-grade infrastructure for agents to retrieve high-quality context from the web. Their core products are a suite of APIs for retrieving high quality information from the web with Pareto-optimal quality, cost, and speed. Whether you work on voice agents that need 200 millisecond latency, chat bots that balance speed, depth, and quality, or long-horizon agents to do thorough, overnight research for you, Parallel is a single platform for all your agentic research. Get started for free at dataengineeringpodcast.com/parallel
Your host is Tobias Macey and today I'm interviewing Christian Bremeau and Simon Dynin about his long history of dealing with metadata at Meta Integration Technology, and how they are adapting to the age of AI

Interview

Introduction
How did you get involved in the area of data management?
Can you start by sharing some of the highlights of your career in metadata and how the industry's focus on that subject has changed over the years?
What are the major sources of friction and fragmentation in metadata management in your experience?
There are numerous metadata management, metadata integration, data catalog, and other governance/discovery/semantic platforms in the ecosystem. How do you explain the differentiating factors of your system to people who are evaluating their options?
One of the constant challenges in data systems is the tension between data locality and data consistency. What is your approach to managing a canonical, reliable definition of a data asset, while making it available and accessible to all of the consuming systems?
There was a surge of interest in metadata systems in the 2018 - 2022 time-frame as the industry invested in cloud data warehouses, ELT, and all of the other "modern data stack" patterns. Now AI is driving a new surge of interest. What are the categorical differences between these periods in your experience?What are the new stressors that you anticipate from the continued growth in LLM-mediated and agentic data interactions?

Can you describe the overall architecture of the MetaKarta platform and how it has evolved in the latest version?
For a team who is trying to tame their data estate, what does the adoption of MetaKarta look like?How does it change their day-to-day work?

You're promoting the idea of "MetaDataOps" as a dedicated concern for an organization. What are the skills and activities of someone who is focused on that capability?
What are the most interesting, innovative, or unexpected ways that you have seen MetaKarta used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Metadata and MetaKarta?
When is MetaKarta the wrong choice?
What do you have planned for the future of MetaKarta?

Contact Info
ChristianLinkedIn

SimonLinkedIn

Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links
Meta Integration Technology
MetaKarta)
CognosCognos Framework Manager

Informatica
ER Studio
Talend
Data Catalog
ETL == Extract, Transform, Load
ELT == Extract, Load, Transform
ThoughtSpot

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Summary
In this episode Christopher Doidge talks about his Agile Ledger Architecture (ALA) approach to data warehousing and how it aims to reduce data debt while shortening the path from raw data to trustworthy business insight. Christopher explained that ALA is not a replacement for existing warehouse patterns like medallion architecture, star schemas, or other modeling approaches, but a complementary discipline focused on pushing business definitions upstream, enforcing cleaner ledger-style transformations, and producing gold-layer tables that stakeholders can actually use without relying on analysts to repeatedly rebuild the same logic. He also discussed his “15-minute litmus test” for time-to-insight, the importance of durable documentation through a data dictionary, and why undocumented business logic living in analyst scripts is often one of the biggest hidden forms of data debt. Overall, this was a thoughtful conversation about designing warehouse systems that serve not just analysts, but the broader business as well.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
Today’s episode is sponsored by Parallel - where agents find answers. Most engineers today closely follow new model releases, but don’t pay attention to their agent’s most important tool: web search. Parallel develops enterprise-grade infrastructure for agents to retrieve high-quality context from the web. Their core products are a suite of APIs for retrieving high quality information from the web with Pareto-optimal quality, cost, and speed. Whether you work on voice agents that need 200 millisecond latency, chat bots that balance speed, depth, and quality, or long-horizon agents to do thorough, overnight research for you, Parallel is a single platform for all your agentic research. Get started for free at dataengineeringpodcast.com/parallel
Your host is Tobias Macey and today I'm interviewing Christopher Doidge about a data warehousing approach called Agile Ledger Architecture that is designed to drive down data debt
Interview
Introduction
How did you get involved in the area of data management?
Can you describe what the agile ledger architecture is and the story behind it?
There are numerous patterns and practices that have been developed for data warehousing over the past 40 years. How does the agile ledger architecture fit in that ecosystem? (e.g. is it compatible with Kimball, Inmon, Data Vault, Anchor Modeling, etc.?)
What are some examples of the types of debt that accumulate in current approaches to warehouse implementation, and the impact that it has on the utility of that asset?
Digging into the architecture itself, what are the core principles that it is built on?
What are the technologies or practices that it is best suited to? (e.g. event streams, lakehouse, ELT workflows, etc.)
For someone who has already invested a substantial amount of effort into building a warehouse, what does adoption of the agile ledger architecture look like?
Once you have started that adoption, what are the technical controls that you can put in place to ensure that this architecture is maintained and doesn't regress or get sidestepped in another portion of the warehouse?
As LLMs and agents grow to become the predominant consumers (and often producers) of a warehouse, what are the benefits that the agile ledger architecture provides to ensure appropriate context and grounding to produce useful insights and accurate answers?
What are the most interesting, innovative, or unexpected ways that you have seen the agile ledger architecture used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on data warehouse design and implementation?
When is the agile ledger architecture the wrong choice?
What do you have planned for the future of this architecture?
Contact Info
LinkedIn
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links
Agile Ledger Architecture Book (affiliate link)
Ralph Kimball
Bill Inmon
Data Vault
Star Schema
Anchor Modeling
Surrogate Key
Data Lakehouse
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Summary
In this episode Soham Mazumdar, co-founder and CEO of Wisdom.ai, talks about what “context” really means in data engineering and AI systems. He explores why context has become such an overloaded term, spanning everything from semantic layers and data catalogs to tribal knowledge, query logs, dashboards, and even agent memory. Soham explained that the big shift is that context is no longer being prepared primarily for human analysts, but for LLMs and agents that can’t reliably fill in missing gaps on their own. That change raises the bar for how context is represented, validated, benchmarked, and maintained so that AI systems can produce trustworthy outcomes.

Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
Your host is Tobias Macey and today I'm interviewing Soham Mazumdar about what "context" actually means in data engineering
Interview
Introduction
How did you get involved in the area of data management?
One of the perennial challenges of engineering in all forms is building a shared understanding of what a given word means. "Context" is one that is being used for an increasing number of purposes with the introduction of AI agents. Can you start by sharing some of the ways that this terminology overload has caused problems in your own experience?
Data engineering has arguably always been about context engineering, but at the scale of human consumers. What are the substantive changes that AI/agentic consumers bring to the discipline?
While we all understand the notion of "context", turning it into a useful and re-usable component is a different matter entirely. What are some of the ways that "business context" or "technical context" manifests as a tangible artifact?
This also brings up the question of data modeling. What are some of the key attributes that are necessary when storing, enriching, evolving, and joining into that context?
One could argue that the entire history of data warehousing is about building organizational context. What are the real differences in approach for today's work of capturing and activating that context?
How does your work at Wisdom AI address the technical and operational burdens of capturing, modeling, and exposing context at the speed necessary to keep up with organizational demands?
What are the most interesting, innovative, or unexpected ways that you have seen Wisdom AI used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Wisdom AI/context engineering?
When is Wisdom AI the wrong choice?
What do you have planned for the future of Wisdom AI?
Contact Info
LinkedIn
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.
Links
Wisdom AI
Context Engineering
Knowledge Graph
Ontology
Snowflake Open Semantic Interchange (OSI)
Semantic Layer
Palantir Foundry
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA