The Production AI Playbook: Deploying Agents at Enterprise Scale — Sandipan Bhaumik, Databricks

At-a-Glance

• Verdict: Watch fully
• Core thesis: Enterprises fail to productionize AI agents because they focus on model selection and demos rather than building the five foundational pillars: evaluation systems, observability/tracing, data infrastructure, multi-agent orchestration, and governance—and the speaker provides a tested framework to fix this, drawn from AWS/Databricks production deployments.
• Why it matters: This talk addresses the gap between AI demos and production at enterprise scale with concrete patterns, real incident playbooks, and cost/risk trade-offs gleaned from regulated industries (financial services, retail banking) where AI accountability is mandatory.
• Best use: Use as the operational playbook for production AI systems: adopt the five-pillar framework, integrate the evaluation pipeline and tracing architecture, reference the production incident playbook, and apply the three advanced lessons (test case governance, prompt versioning rigor, behavioral eval cost control) before scaling agents.

Executive Summary

Sandipan Bhaumik (Sandy), a technical lead at Databricks with prior principal architect experience at AWS, presents a production AI deployment framework distilled from years of working with enterprise customers in B2B software and regulated industries. He opens by diagnosing a common failure pattern: organizations rush to choose models, build demos with controlled data, get leadership approval, then face mysterious failures in production because they lack observability (can't see what the AI does), evaluation (can't measure success), and governance (no accountability when things break). This cycle wastes money and effort without ROI.

To solve this, Sandy proposes five pillars that must be designed before writing code, ideally in sequence but often implemented in parallel: (1) Evaluation—define success numerically and build automated test pipelines with golden datasets from domain experts; (2) Observability—trace every agent decision for regulators, debugging, and performance; (3) Data Foundation—architect both question data (for answering queries) and tracking data (tracing logs) with proper governance; (4) Orchestration—manage complexity when scaling from one agent to many, choosing patterns like orchestrator-worker, choreography, or human-in-the-loop; (5) Governance—establish audit trails, PII validation, prompt versioning as code, and model change management. He illustrates these pillars with a retail banking chatbot case study where the team selected the model in week 7 of an 8-week POC, after building evaluation and data layers first.

The case study customer had 20,000 monthly calls, wanted to deflect 60% of simple queries to an AI agent, and had already spent $85K on a failed 6-month POC. By implementing Sandy's framework—starting with 200 human-agent answer examples as the eval dataset, defining 85% accuracy and 60% deflection targets, building an automated evaluation pipeline, setting up tracing, and only then testing models—they launched successfully. Post-launch, they caught a production issue (outdated policy document in the vector database causing wrong answers) within weeks because tracing and CSAT monitoring revealed the problem, which would have been invisible without the system. Sandy shares practical lessons: treat the test case library as a living, governed system with categorized rows; version prompts with detailed commit messages explaining the failure being fixed; and control costs of expensive behavioral evals (layer 3) by running subsets during CI and full tests only on main branch merges.

Sandy emphasizes that enterprises won't run AI in one framework or cloud—they'll use LangChain, CrewAI, AWS, Azure, etc.—so a centralized tracing data strategy (e.g., Databricks Unity Catalog, MLflow, Delta Lake) is essential to serve dashboards, first-line support, auditors, and online monitoring. He provides a production incident playbook (detect via dashboards, diagnose via tracing, contain via prompt rollback or circuit breakers, fix via test case library, then add new test cases) and offers downloadable artifacts including evaluation checklists, tracing setup guides, and templates. The talk concludes with three easily missed lessons: govern the growing test case library with owners and categories, treat prompt commits like code changes with rigorous documentation, and architect CI pipelines to avoid runaway costs from re-running behavioral evals on large datasets.

Key Takeaways

• Claim: Enterprises fail at production AI because they start with model selection and demos instead of building evaluation, observability, and governance systems first. | Evidence: Sandy observed a repeating pattern across customers: teams debate GPT vs Claude, build features in controlled environments with predictable data, get leadership sign-off on demos, then face mysterious production failures with no ROI and wasted money. A retail banking client spent $85K over 6 months on a failed POC before engaging Sandy's team. | Caveat: The speaker does not quantify how many organizations exhibit this pattern or provide comparative success rates, though he states it appeared 'in every customer conversation' over two years at AWS and Databricks. | Implication: Operators must design the five-pillar framework (evaluation, observability, data, orchestration, governance) before touching code or choosing models, ideally in sequence, to avoid costly demo-to-production failures and ensure measurable business outcomes. | Timestamp: 01:30
• Claim: The three critical gaps blocking production AI are observability (can't see what AI does), evaluation (can't measure success), and governance (no accountability when AI fails). | Evidence: In customer engagements, Sandy found teams couldn't trace agent decisions for regulators, lacked systems to continuously measure business-relevant metrics (accuracy, deflection, latency), and had no playbook for 3am incidents or data ownership when agents misbehaved. | Caveat: The speaker focuses on regulated industries (financial services) and B2B software; less-regulated contexts may face different governance pressures, though observability and evaluation gaps apply universally. | Implication: Production AI systems require mandatory tracing infrastructure (especially in Europe and regulated sectors), automated evaluation pipelines tied to business KPIs, and incident playbooks that define who owns what when failures occur. | Timestamp: 03:45
• Claim: Evaluation is the specification for your AI system: define success numerically (e.g., 85% accuracy, 60% deflection) and build a golden dataset from domain experts before any coding. | Evidence: In the banking chatbot project, the team collected 200 real human-agent answer examples, defined 85% accuracy and 60% query deflection as success metrics, then built an automated pipeline comparing agent responses to the golden dataset. This living dataset grew post-launch as new edge cases emerged. | Caveat: The speaker does not specify how to determine the 'right' number of initial test cases (he says 'there's no correct number, but we started with 200') or how to balance coverage versus cost when scaling the dataset. | Implication: Operators should treat evaluation datasets as living systems that grow with production use, categorize test cases by failure type (security, logic, PII) for governance, and use domain experts—not just engineers—to capture real-world edge cases and gray areas. | Timestamp: 06:20
• Claim: Evaluation must be layered across three tiers: deterministic checks (formats, regex), semantic checks (LLM-as-judge for groundedness/relevance), and behavioral checks (tool call efficiency, duplicate API calls, loops). | Evidence: Sandy describes layer 1 as 'easy, cheap stuff' (email format, PII detection with classic ML), layer 2 as LLM-as-judge prompts checking safety and groundedness, and layer 3 as behavioral analysis catching duplicate database calls that are fine in demos but expensive at scale (thousands of daily queries). The banking chatbot made 3 API calls to answer one balance query—acceptable in testing, costly in production. | Caveat: Layer 3 behavioral evals can become very expensive as the test dataset grows (300-500 rows); Sandy recommends running subsets during CI and full tests only on main branch merges, but does not provide cost benchmarks or thresholds. | Implication: Operators should implement all three layers but architect CI pipelines to control layer 3 costs, prioritize deterministic checks to reduce LLM judge load, and monitor behavioral patterns (API call duplication, retries) to prevent runaway cloud spend at scale. | Timestamp: 08:50
• Claim: Observability via tracing is mandatory for regulators and debugging: enterprises must capture every agent decision (intent classification, API calls, retrieval, reasoning, guardrails) with timestamps and confidence scores. | Evidence: Sandy shows a simplified trace of a banking chatbot handling an overdraft dispute: intent classification (with confidence score and latency), customer DB API call, policy document retrieval from vector DB, reasoning step, final guardrail checks. Without this, customer disputes have no audit trail, and regulators (especially in Europe) block production deployment. | Caveat: The speaker notes real tracing data 'is not as beautiful as this slide'—he simplified it for presentation. He does not discuss schema design, storage costs, or query performance for high-volume tracing data at scale. | Implication: Operators must implement structured tracing (e.g., MLflow, OpenTelemetry) from day one, design tracing schemas to support auditors/regulators and online monitoring, and use traces to detect issues like duplicate calls or stale data retrieval before customer satisfaction drops. | Timestamp: 11:40
• Claim: Data foundation must cover both 'question data' (for answering queries) and 'tracking data' (tracing logs for monitoring and audits), with a centralized strategy because enterprises use multiple frameworks and clouds. | Evidence: Sandy estimates 60% of project time goes to data foundation because 'data was built for humans, agents don't forgive.' Agents confidently return wrong answers from stale data. In the banking case, an outdated policy document in the vector DB caused wrong answers; tracing revealed embeddings weren't updated. Databricks Unity Catalog centralizes metadata tagging, permissions, and discovery so agents query tables with context (PII tags, column descriptions). | Caveat: The speaker is a Databricks employee and heavily promotes Databricks tools (Unity Catalog, Delta Lake, MLflow); he does not compare alternatives or discuss vendor lock-in, though he mentions open-source foundations (Spark, Delta, MLflow) and multi-cloud support. | Implication: Operators should architect a centralized data catalog with metadata tagging for agent context, design tracking data pipelines that unify tracing from multiple frameworks (LangChain, CrewAI, etc.) and clouds, and implement incremental data loading with versioning to catch stale embeddings before production failures. | Timestamp: 14:30
• Claim: Multi-agent orchestration requires choosing patterns based on dependencies: orchestrator-worker for centralized control, choreography for parallel independence, human-in-the-loop for low-confidence thresholds. | Evidence: Sandy contrasts orchestrator-worker (one orchestrator distributes work to specialized agents, all requests flow through central logs) with choreography (autonomous agents listen to a message bus, work in parallel—e.g., mortgage app agents for customer details and approval running independently to reduce latency). He states one agent is simple, but five agents increase complexity exponentially. | Caveat: The speaker does not provide decision criteria for choosing patterns or discuss failure modes (e.g., orchestrator as single point of failure, message bus ordering/idempotency in choreography). He refers to a separate deep-dive video on fault tolerance patterns (saga, compensation, circuit breaker) but doesn't detail them here. | Implication: Operators should select orchestration patterns based on task dependencies and latency needs, implement fault-tolerance patterns (circuit breaker to limit retries, compensation for rollbacks) as agent count grows, and use human-in-the-loop fallback when confidence drops below thresholds to maintain CSAT. | Timestamp: 18:15
• Claim: Governance for production AI means audit trails for every action, pre-validation of PII, prompt versioning as code with detailed commit messages, and model change management independent of vendor benchmarks. | Evidence: In the banking project, pre-validation with NER and regex caught 47 PII breaches during testing. Sandy emphasizes prompt changes must go through enterprise change management (not just git commit), with commit messages explaining what failure the change fixes. Model provider benchmark boards don't reflect enterprise-specific performance, so teams must test new model versions against their own eval datasets before switching. | Caveat: The speaker does not specify how to enforce detailed commit message standards or integrate prompt versioning with existing ITSM systems, though he mentions integration is necessary for alerting the right person during incidents. | Implication: Operators must treat prompts as code with rigorous versioning and change justification, validate PII at ingestion to avoid compliance breaches, test model upgrades against internal eval datasets (not public benchmarks), and maintain flexibility to switch models to reduce vendor risk and regulatory exposure. | Timestamp: 20:45
• Claim: In the retail banking case study, the team deferred model selection to week 7 of an 8-week POC—after building eval, data, and tracing systems—and successfully launched with measurable results. | Evidence: The customer had 20K monthly calls, wanted 60% deflection of simple queries, spent $85K on a failed 6-month POC. Sandy's team spent weeks 1-2 collecting 200 human-agent answer examples and defining success (85% accuracy, 60% deflection, latency targets). Weeks 3-6 built data foundation (API connections, distributed storage, tracing) and tested for duplicate calls. Week 7 they ran models against the eval dataset to pick the best. Post-launch, they caught a policy document staleness issue via tracing and CSAT monitoring. | Caveat: The speaker does not disclose the final cost, timeline, or quantitative post-launch CSAT/deflection numbers ('six weeks post-launch, we calculated operational metrics'), so ROI and improvement magnitude are unclear. | Implication: Operators should resist pressure to choose models early, invest first in eval pipelines and data infrastructure so model selection is data-driven, and plan for living eval datasets that catch issues like stale retrieval sources after launch. | Timestamp: 23:10
• Claim: Three commonly missed lessons are: govern the growing test case library with owners and categories, version prompts with detailed failure-context commit messages, and control layer 3 behavioral eval costs via CI subset testing. | Evidence: Sandy notes test case libraries grow over time (from 200 cases to hundreds more) and need categorization (security, login, logic) so teams can trace changes. Prompt commit messages are often generic ('simple commit messages'), but must document what failure caused the change. Layer 3 evals (tool call checks) cost money at scale; he recommends running subsets during CI, full tests only on main branch merges. | Caveat: The speaker does not provide cost benchmarks, subset sizing heuristics, or tooling for test case categorization/governance, leaving operators to design their own systems. | Implication: Operators should assign test case library owners, tag cases by failure type for efficient root cause analysis, enforce prompt commit standards (document failure and expected fix), and architect CI eval budgets to run cheap deterministic checks always, expensive behavioral checks selectively, to avoid runaway cloud costs as datasets scale. | Timestamp: 30:20

Detailed Brief

The Failure Pattern: Why AI Demos Don't Reach Production

• Claims: Organizations start with model selection debates (GPT vs Claude), build features in controlled environments, get leadership buy-in on demos, then face mysterious production failures without ROI.; This pattern appeared in every customer conversation over two years across B2B software and regulated industries.; A retail banking client spent $85K over 6 months on a failed POC before Sandy's team intervened.
• Evidence: Teams debate models, build demos with predictable data, leadership signs off, then after a few weeks people ask why AI isn't answering correctly—leading to wasted money and effort.; No one knew why the banking POC failed: couldn't see what AI was doing, couldn't measure success, couldn't assign accountability when things broke.
• Caveats: The speaker does not quantify failure rates, provide comparative success metrics, or name specific customer organizations (likely under NDA).; Focus is on regulated industries and B2B; less-regulated consumer contexts may have different risk/governance profiles.
• Implications: Operators must resist the market's model-first narrative and design systems for observability, evaluation, and governance before writing code.; Leadership pressure to 'do something with AI' drives demo culture; aligning exec expectations with production requirements (tracing, eval pipelines) is critical to avoid wasted investment.

Pillar 1: Evaluation as Specification

• Claims: Evaluation is the AI system's specification: define success numerically (accuracy %, deflection rate, latency), not vaguely ('good accuracy').; Build a golden dataset from domain experts showing how humans answer real queries, including gray areas and edge cases.; Automate evaluation pipelines to compare agent responses to the golden dataset continuously in production.
• Evidence: Banking chatbot example: defined 85% accuracy, 60% deflection of simple queries, collected 200 human-agent answer examples as ground truth.; Three evaluation layers: (1) deterministic (formats, regex, PII detection), (2) semantic (LLM-as-judge for groundedness/relevance), (3) behavioral (tool call efficiency, duplicate API calls, loops).; Layer 3 caught the banking agent making 3 database calls for one balance query—fine in demos, expensive at scale with thousands of daily queries.
• Caveats: No 'correct number' of initial test cases specified (team started with 200); coverage vs cost trade-off is left to operator judgment.; Layer 3 behavioral evals are expensive; running full evals on large datasets (300-500 rows) during every CI run can cause runaway costs.; LLM-as-judge accuracy depends on prompt quality and judge model selection; speaker does not discuss judge model failures or adversarial cases.
• Implications: Operators should co-design eval datasets with domain experts (support agents, compliance teams) to capture real-world nuance, not just engineer intuition.; Treat eval datasets as living systems: add new test cases after every production incident, categorize by failure type (security, logic, PII) for governance.; Architect CI pipelines to run cheap deterministic checks always, LLM judges selectively, and expensive behavioral checks only on main branch merges to control cloud spend.; Define business-relevant KPIs (deflection rate, CSAT) as eval targets, not just technical metrics, so success aligns with ROI.

Pillar 2: Observability and Tracing

• Claims: Tracing every agent decision (intent classification, API calls, retrieval, reasoning, guardrails) is mandatory for regulators, especially in Europe and regulated industries.; Without tracing, customer disputes have no audit trail, and teams can't debug production failures (e.g., why AI gave wrong answers).; Tracing enables detection of issues like duplicate API calls, stale data retrieval, and policy document mismatches before customer satisfaction drops.
• Evidence: Banking chatbot trace example: user asks to waive overdraft fee, agent does intent classification (confidence score, latency), calls customer DB API, retrieves policy docs from vector DB, reasons, applies guardrails, responds.; Without tracing, when customer raised dispute, team had 'nowhere to go' and would have to give discounts to placate customer.; Post-launch, tracing caught an issue where agent queried outdated policy doc (embeddings not updated) causing wrong answers and CSAT drop.
• Caveats: Real tracing data 'is not as beautiful as this slide'—speaker simplified for presentation, does not discuss schema design, storage costs, or query performance at scale.; No mention of tracing data retention policies, GDPR/privacy implications of storing detailed user interaction logs, or anonymization strategies.
• Implications: Operators must implement structured tracing (MLflow, OpenTelemetry, vendor SDKs) from day one, not as an afterthought.; Design tracing schemas to support multiple consumers: auditors/regulators, first-line support dashboards, online monitoring/alerting, LLM judges running over traces.; Use tracing to detect behavioral patterns (retries, duplicate calls, tool call loops) that are invisible in metrics dashboards but expensive at scale.; In regulated industries, tracing is a blocker for production approval—build it before pilots, not after.

Pillar 3: Data Foundation Strategy

• Claims: Data foundation has two parts: 'question data' (for answering queries) and 'tracking data' (tracing logs for monitoring/audits).; 60% of project time goes to data foundation because 'data was built for humans, agents don't forgive'—agents return wrong answers confidently from stale/bad data.; Enterprises use multiple frameworks (LangChain, CrewAI, etc.) and clouds (AWS, Azure, GCP), requiring a centralized data strategy to unify tracing and serve multiple teams.
• Evidence: Banking case: outdated policy document in vector DB caused wrong answers; tracing revealed embeddings weren't updated, so agent retrieved stale data.; Databricks Unity Catalog example: centralizes metadata (table/column descriptions, PII tags), permissions, and delta sharing; agents query tables with context, improving groundedness.; Architecture: raw data on cloud storage → Delta Lake (database-like properties, incremental loading) → Unity Catalog (permissions, discovery) → apps (Mosaic AI, Genie text-to-SQL, BI).; Centralized tracking data strategy: collect traces from multiple frameworks/clouds, serve to operational dashboards, first-line support, auditors, online monitoring, LLM judges.
• Caveats: Speaker is Databricks employee; heavy promotion of Unity Catalog, Delta Lake, MLflow without comparing alternatives (Snowflake, BigQuery, AWS Glue, etc.).; No discussion of vendor lock-in, migration complexity, or cost at scale; mentions open-source foundations (Spark, Delta, MLflow) and multi-cloud support but doesn't detail portability.; Schema design, storage costs, and query performance for high-volume tracing data not addressed.
• Implications: Operators should invest in metadata cataloging (table/column descriptions, PII tags) so agents have semantic context when querying, reducing hallucinations.; Architect tracking data pipelines early: unify tracing from multiple agent frameworks and clouds into one queryable store (Delta Lake, data warehouse) to serve auditors, dashboards, and eval pipelines.; Plan for incremental data loading and versioning to catch stale embeddings or outdated retrieval sources before they cause production failures.; Consider data governance (who owns customer embeddings, policy docs, tracing logs) and privacy (PII in traces) as part of the foundation, not bolt-on later.

Pillar 4: Multi-Agent Orchestration Patterns

• Claims: One agent is simple; five agents increase complexity exponentially due to coordination, dependencies, and failure modes.; Three orchestration patterns: (1) orchestrator-worker (centralized control, all requests through orchestrator), (2) choreography (autonomous agents, message bus, parallel execution), (3) human-in-the-loop (fallback when confidence drops below threshold).; Pattern choice depends on task dependencies and latency needs; choreography reduces latency for independent parallel tasks, orchestrator provides central logging for debugging.
• Evidence: Orchestrator-worker: one orchestrator distributes work to specialized agents, all logs go to orchestrator, good for debugging but potential bottleneck.; Choreography: agents listen to message bus for events (e.g., mortgage app triggers), work in parallel (one agent gets customer details, another approval details), reduces latency but harder to trace.; Human-in-the-loop: when agent confidence falls below threshold, route to human to review agent's work, take action, avoid low CSAT.
• Caveats: Speaker does not provide decision criteria or failure mode analysis (orchestrator single point of failure, message bus ordering/idempotency).; Refers to separate deep-dive video on fault tolerance patterns (saga, compensation, circuit breaker) but doesn't detail them here.; No discussion of state management complexity, distributed transaction consistency, or observability challenges in choreography pattern.
• Implications: Operators should choose orchestrator-worker for early-stage multi-agent systems where centralized debugging is critical, then migrate to choreography for latency-sensitive parallel tasks.; Implement fault-tolerance patterns (circuit breaker to limit retries, compensation for rollback, saga for long transactions) as agent count scales to prevent cascading failures.; Use human-in-the-loop as safety valve for low-confidence responses, not as excuse to skip evals—monitor handoff rate to detect agent degradation.; Plan for state management (where is conversation state stored, how long is it retained, how do agents share context) and tracing across distributed agents before deploying multiple agents.

Pillar 5: Governance (Regulatory, Security, Change Management)

• Claims: Governance is not just data governance; it's audit trails for every action, PII validation, prompt versioning as code, model change management.; Pre-validation (NER, regex for PII) caught 47 breaches during testing in the banking project.; Prompt changes must go through enterprise change management like code, not casual git commits, with detailed messages explaining what failure is being fixed.; Model providers' benchmark boards don't reflect enterprise-specific performance; teams must test new model versions on their own eval datasets before switching.
• Evidence: Banking case: named entity recognition and regex for PII detected 47 breaches before production.; Prompt versioning: commit messages are often generic, but must document failure cause and expected fix so teams can trace why changes were made.; Model change management: can't rely on one model due to risk; must have flexibility to switch models and test them on internal eval datasets, not public benchmarks.
• Caveats: No specifics on enforcing detailed commit message standards or integrating prompt versioning with ITSM tools.; Doesn't discuss model fallback strategies, A/B testing for model changes, or rollback procedures if new model degrades performance.; Regulatory focus (audit trails, PII) may be less critical in non-regulated industries, though observability and accountability still apply.
• Implications: Operators must treat prompts as first-class code artifacts with version control, change approval workflows, and documentation linking changes to incidents.; Implement PII validation at data ingestion (not just at agent output) to catch compliance breaches early and avoid costly remediations.; Test model upgrades (e.g., GPT-4 to GPT-4.5) against internal eval datasets before production cutover; vendor benchmarks (MMLU, HumanEval) are not proxies for business-specific accuracy.; Design incident playbooks (who to alert, ITSM integration, fallback strategies) before production, not during 3am outages, to reduce mean time to recovery and reputation damage.

Case Study: Retail Banking Chatbot (Model Selection in Week 7)

• Claims: Customer had 20K monthly calls, wanted to deflect 60% of simple queries to AI agent, spent $85K on failed 6-month POC before Sandy's team.; New approach: weeks 1-2 collect 200 human-agent answer examples, define 85% accuracy and 60% deflection targets; weeks 3-6 build data foundation and tracing; week 7 test models on eval dataset.; Post-launch success: caught policy document staleness issue via tracing and CSAT drop within weeks, fixed by updating vector DB embeddings.
• Evidence: Failed POC symptoms: no visibility into AI decisions, no measurement system, no accountability.; New POC structure: eval system built first, automated pipeline comparing agent responses to golden dataset, rating system with human review for low scores, test cases added to dataset for next iteration.; Foundation work: API connections traced, distributed storage set up, duplicate API calls detected during testing (agent made 3 calls for one balance query).; Post-launch issue: interest rate policy changed, notifications sent to customers, but agent retrieved outdated policy doc from vector DB, causing wrong answers and CSAT drop—detected via tracing, fixed by updating embeddings.
• Caveats: No final cost, timeline, or quantitative post-launch metrics disclosed (CSAT improvement, deflection %, cost savings).; Speaker does not explain how they chose 200 as initial test case count or how they prioritized which 60% of queries to target.; Customer org and POC vendor not named (likely NDAs), limiting ability to verify claims or compare approaches.
• Implications: Operators should defer model selection until eval and data systems are in place, so the decision is data-driven, not driven by vendor hype or executive preference.; Living eval datasets are critical: the policy doc issue would not have been caught without tracing + CSAT monitoring + eval pipeline that could quickly add new test cases.; Detecting issues post-launch requires the full system (tracing, CSAT tracking, automated eval) to be operational, not bolted on after complaints escalate.; Spending weeks on eval and data before coding is counterintuitive but de-risks production: the 'slower' start enables faster, safer scaling.

Three Commonly Missed Lessons

• Claims: Test case library is a living, growing system requiring governance: owners, categorization by failure type (security, logic, PII), documentation.; Prompt versioning commit messages must be detailed, explaining what failure caused the change and what it will fix, not generic 'updated prompt' messages.; Layer 3 behavioral evals (tool call checks) are expensive at scale; run subsets during CI, full tests only on main branch merges to control costs.
• Evidence: Test case library grows from 200 to hundreds as production uncovers edge cases; categorizing (e.g., security issues, login failures) helps trace what changed.; Prompt changes in git often have simple commit messages, but teams need to understand why the change was made—what failure it addresses—for root cause analysis.; Running layer 3 evals on 300-500 test cases during every CI run costs money; selecting subsets (e.g., 50 cases) for quick checks, full suite for merges, controls cloud spend.
• Caveats: No cost benchmarks, subset sizing heuristics, or tooling recommendations for test case governance.; Doesn't discuss how to enforce commit message standards (code review, linting, templates) or integrate with existing change management systems.; Layer 3 eval cost control strategy (subsets in CI) may miss edge cases; no guidance on which subset to run or how to rotate coverage.
• Implications: Operators should assign test case library owners, use tagging or folder structure to categorize by failure type, and review growth patterns to identify systemic issues.; Enforce prompt commit standards via templates, code review, or linting; link commits to incident tickets (Jira, ITSM) so changes are traceable to business impact.; Design CI eval budgets: run deterministic checks (layer 1) always, LLM judges (layer 2) selectively, behavioral checks (layer 3) on main merges or scheduled nightly runs, to avoid cloud bill surprises.; Living datasets + version control + cost controls together enable continuous improvement without runaway costs or lost context.

Production Incident Playbook

• Claims: When AI fails in production, follow: detect (via dashboards), diagnose (via tracing), contain (rollback prompts, fallback to human, apply circuit breakers), fix (using test case library and LLM judge reports), then add test cases to eval dataset.; Playbook must integrate with ITSM systems (PagerDuty, ServiceNow) to alert the right person at the right time and prevent downstream system impact.; Without this playbook, teams lack accountability and waste time during 3am incidents.
• Evidence: Detect: use dashboards monitoring CSAT, latency, deflection rate.; Diagnose: trace logs show which step failed (intent classification, API call, retrieval, reasoning, guardrails).; Contain: version prompts so you can roll back; use fault-tolerance patterns (saga, compensation, circuit breaker) from multi-agent orchestration video; deflect to human if confidence drops.; Fix: review LLM judge reports, eval dataset results, identify root cause, apply fix (prompt change, data refresh, tool call logic update).; Test: add incident as new test case in eval dataset so it's caught in future CI runs.
• Caveats: Speaker does not detail ITSM integration mechanics, alerting thresholds, or escalation paths.; Fault-tolerance patterns (saga, compensation, circuit breaker) mentioned but not explained; referred to separate video.; No discussion of incident retrospectives, blameless postmortems, or knowledge sharing across teams.
• Implications: Operators must define the playbook before production, not during incidents: who is on-call, what dashboards to check, what fallback strategies to apply.; Integrate AI monitoring with existing ITSM/alerting systems so AI incidents trigger the same escalation paths as infrastructure/app failures.; Treat every production incident as a test case: document root cause, apply fix, add to eval dataset, verify fix with automated tests, close the loop.; Without the playbook, teams will panic during incidents, apply ad-hoc fixes, and repeat the same failures—defeating the purpose of observability and eval systems.

Notable Concepts & Terms

• LLM-as-Judge: Using a secondary LLM to evaluate the primary LLM's output for safety, groundedness, relevance, by providing a judge prompt specifying criteria; automates semantic evaluation at scale (layer 2 evals).
• Layer 3 Evals (Behavioral Evaluation): Beyond deterministic (formats) and semantic (groundedness) checks, behavioral evals analyze agent tool call efficiency, duplicate API calls, loops—critical for cost control at production scale but expensive to run.
• Golden Dataset / Eval Dataset: Collection of real-world questions with expected answers from domain experts (e.g., 200 human-agent responses), used to benchmark agent performance; treated as a living system that grows with production incidents.
• Orchestrator-Worker Pattern: Multi-agent orchestration where one orchestrator distributes work to specialized agents, provides centralized control and logging, but can be a bottleneck; contrasts with choreography (message bus, parallel agents).
• Choreography Pattern: Multi-agent orchestration where autonomous agents listen to a message bus for events, work in parallel without central orchestrator, reduces latency but harder to trace and debug.
• Unity Catalog (Databricks): Centralized data catalog providing permissions, metadata tagging, discovery, delta sharing; enables agents to query tables with semantic context (PII tags, column descriptions) to improve groundedness.
• Delta Lake: Open-source storage layer that brings database-like ACID properties (incremental loading, versioning) on top of raw data files (Parquet, JSON) in cloud storage; foundation for Databricks' data management.
• Circuit Breaker Pattern: Fault-tolerance pattern that limits retries (e.g., max 3 attempts) to prevent cascading failures or runaway API calls; mentioned as part of multi-agent orchestration but detailed in separate video.
• Question Data vs Tracking Data: Sandy's distinction: question data = data needed for agents to answer user queries (pre-training, retrieval, APIs); tracking data = tracing logs for observability, audits, monitoring—both need data strategy.
• Deflection Rate: Percentage of user queries handled by AI agent without human escalation; key business KPI for chatbots (e.g., 60% deflection target = 60% of simple queries answered by agent, reducing support costs).
• Agent Bricks: Databricks product for building production AI agent applications, providing out-of-the-box LLM judges, tracing, evaluation, monitoring; Sandy positions it as operationalizing the five-pillar framework.
• Production Incident Playbook: Framework for handling AI failures: detect (dashboards), diagnose (tracing), contain (rollback/fallback), fix (test case library), close loop (add test case to eval dataset)—must integrate with ITSM.

Operator Notes / Why Ken Should Care

• This is a practitioner's playbook, not vendor fluff: Sandy spent 60% of project time on data foundation, deferred model selection to week 7 of 8, and caught 47 PII breaches pre-launch—signals real trench experience.
• For agent system operators: adopt the five-pillar sequence (eval → observability → data → orchestration → governance) as your design checklist before writing code; resist exec pressure to ship demos without tracing and eval pipelines.
• For AI ops teams: the three-layer eval framework (deterministic → semantic → behavioral) is actionable; implement layer 1 (regex, PII) and layer 2 (LLM-as-judge) first, then add layer 3 (tool call checks) with cost controls (CI subsets, main branch full tests).
• For content/workflow ops: the living eval dataset concept applies beyond AI—treat every production incident as a test case, categorize by failure type, version control with detailed commit messages linking to incidents.
• For investing/due diligence: ask SaaS/AI companies if they version prompts as code, have incident playbooks, separate question data from tracking data strategies, and test model upgrades on internal evals (not public benchmarks)—gaps indicate production risk.
• For GTM/product teams: deflection rate and CSAT as KPIs (not just accuracy) align AI metrics with business outcomes; tracing and eval systems are table stakes for enterprise buyers, especially in regulated industries (Europe, finserv).
• Critical gotcha: layer 3 behavioral evals (tool call checks) can cause runaway cloud costs if run on full datasets during every CI run—architect eval budgets and subset strategies before scaling.
• Multi-agent orchestration complexity is exponential: one agent is simple, five agents require state management, fault tolerance (circuit breakers, saga, compensation), and centralized tracing—don't skip pillar 4.
• Data quality is the bottleneck: 'agents don't forgive' stale/bad data; invest in metadata catalogs (table/column descriptions, PII tags) and versioning (embeddings, policy docs) so retrieval is traceable and agents have semantic context.
• Governance is not optional: regulators (especially Europe) block production without tracing/audit trails; prompt versioning and model change management reduce vendor lock-in and enable rapid response to model drift or failures.

Watch Map

• 00:00: Intro: speaker background (Databricks, AWS principal architect, 5 years data/AI at scale), session goal (production AI playbook from trenches)
• 01:30: The failure pattern: orgs start with model selection, build demos, get leadership buy-in, then face mysterious production failures (no ROI, wasted $85K example)
• 03:45: Three critical gaps: observability (can't see AI decisions), evaluation (can't measure success), governance (no accountability at 3am)
• 05:00: Five-pillar framework intro: evaluation → observability → data → orchestration → governance (design before code, ideally in sequence)
• 06:20: Pillar 1: Evaluation as specification—define success numerically (85% accuracy, 60% deflection), build golden dataset from domain experts, automate eval pipeline
• 08:50: Three eval layers: deterministic (formats, regex), semantic (LLM-as-judge), behavioral (tool calls, duplicate APIs)—layer 3 catches expensive patterns at scale
• 11:40: Pillar 2: Observability/tracing—capture every agent decision (intent, API calls, retrieval, reasoning, guardrails) for regulators and debugging; banking chatbot trace example
• 14:30: Pillar 3: Data foundation—60% of project time; question data (answering queries) vs tracking data (tracing logs); agents don't forgive stale data; Unity Catalog example
• 18:15: Pillar 4: Multi-agent orchestration—one agent simple, five agents exponential complexity; orchestrator-worker, choreography, human-in-the-loop patterns; fault tolerance (circuit breaker, saga, compensation) in separate video
• 20:45: Pillar 5: Governance—audit trails, PII validation (47 breaches caught in banking case), prompt versioning as code, model change management independent of vendor benchmarks
• 23:10: Case study: retail banking chatbot—20K calls/month, 60% deflection goal, $85K failed POC; new approach: eval first (weeks 1-2), data foundation (3-6), model selection week 7
• 26:30: Case study results: post-launch, tracing caught policy doc staleness (embeddings not updated), causing wrong answers and CSAT drop—fixed via eval pipeline and data refresh
• 30:20: Three missed lessons: (1) govern growing test case library with owners/categories, (2) prompt commit messages must explain failure context, (3) control layer 3 eval costs via CI subsets
• 33:00: Production incident playbook: detect → diagnose → contain → fix → add test case; integrate with ITSM; treat every incident as eval dataset entry
• 35:00: What to do tomorrow: define success, collect example answers, build automated eval pipeline; QR codes for downloadable artifacts (eval checklists, tracing setup, templates) and LinkedIn newsletter

Source/Metadata

• Title: The Production AI Playbook: Deploying Agents at Enterprise Scale — Sandipan Bhaumik, Databricks
• Transcript words: 7212
• Duration seconds: 2225
• Timestamp note: Timestamps manually reconstructed from transcript structure; actual video timestamps may differ slightly. QR codes mentioned for artifacts (eval templates, tracing guides, LinkedIn newsletter).