Your Agent Failed in Prod. Good Luck Reproducing It. - Tisha Chawla & Susheem Koul, Microsoft

At-a-Glance

• Verdict: Watch fully
• Core thesis: Chasing bitwise determinism in production AI agents is futile; instead, instrument boundary-level recording (input/output pairs at each node) to capture execution traces, then replay and stub those traces offline for deterministic debugging and regression testing without model calls.
• Why it matters: Ken's systems will fail non-deterministically in production—this shows a concrete engineering pattern (Chronicle) to debug and test agent failures you can't reproduce via prompt re-runs.
• Best use: Study the Chronicle boundary annotation pattern, adopt trace-based regression tests for agent ops, and incorporate the 'replayability not determinism' mindset into your agent CI/CD workflows.

Executive Summary

Tisha Chawla and Susheem Koul from Microsoft present a hard-won lesson: when an agent misbehaves in production—selling 1,000 shares instead of $1,000 worth—engineers reflexively lower temperature to zero and try to reproduce the exact output. They show this fails because temperature=0 doesn't guarantee determinism at the system level due to GPU floating-point variance, batch invariance in inference servers, and mixture-of-experts routing dynamics. Running the same prompt 1,000 times can still yield dozens of different outputs even at temperature zero, making traditional 'pull logs, rerun locally' debugging impossible.

The speakers reframe the problem: the question isn't 'how do I make the model deterministic?' but 'how do I debug a run I can't reproduce?' They introduce Chronicle, a proof-of-concept framework that annotates code boundaries (LLM calls, tool calls, retrieval steps) to record input/output pairs, model versions, and metadata as a trace. When a production failure occurs, Chronicle freezes the entire agent run—including all intermediate node states—so teams can replay the exact execution offline, stub the non-deterministic LLM nodes with their recorded outputs, run the modified code (e.g., new guardrails) live against the frozen context, and assert correctness without calling the model again. This turns every production trace into a free, rerunnable regression test.

The talk distinguishes deterministic testing (guardrails, tool logic) from behavioral testing (tone, trajectory), arguing Chronicle solves the former by kicking probability out of the test loop. Key takeaways: stop chasing bitwise determinism through APIs; log session variables (model version, build ID, RAG chunks); capture the full envelope not just prompts; use replays to debug and verify fixes; and keep temperature variation alive in production because that's what enables agency. The speakers provide a QR code for Chronicle code and articles.

Key Takeaways

• Claim: Setting temperature to zero does not make agent outputs deterministic at the system level. | Evidence: Hard data from Reddit/Hacker News threads shows running the same prompt 1,000 times at temperature=0 still returns dozens of different responses due to GPU non-determinism, floating-point math non-associativity, batch invariance (requests grouped with random traffic alter matrix ops), and mixture-of-experts routing (tokens rerouted if batches overflow expert capacity). | Caveat: Temperature zero guarantees greedy argmax sampling, but underlying scores can shift run-to-run; the speakers do not quantify how many different outputs occur per 1,000 runs, only that variance exists. | Implication: Ken cannot rely on temperature=0 + prompt replay for debugging; he needs trace-based observability to capture what the agent actually did, not what it should deterministically output. | Timestamp: timestamp unavailable
• Claim: The wrong question is 'how do I make the model deterministic?'; the right question is 'how do I debug a run I can't reproduce?' | Evidence: Teams waste weeks chasing bitwise determinism and conclude 'the system is unknowable.' The speakers argue determinism was never the north star—debugging was—and replayability (rebuilding a run well enough to debug it) is achievable via recording, not via freezing the model. | Caveat: No specific team or timeline cited for the 'weeks burned' claim; this is anecdotal positioning. | Implication: Ken should prioritize instrumentation and trace capture over temperature tuning or prompt engineering for reliability; the engineering investment belongs in observability, not in futile determinism chasing. | Timestamp: timestamp unavailable
• Claim: Chronicle annotates code boundaries (methods) to record input/output pairs and metadata, freezing the entire agent run as a trace that can be replayed offline with zero model calls. | Evidence: Demo shows a stock-selling agent with three boundaries annotated: initial planning (LLM), place_order tool, and finalize_agent (LLM). When the agent mistakenly sold 1,000 shares (quantity) instead of $1,000 worth, Chronicle captured the bad tool call JSON and all node I/O. In the test replay, Chronicle stubbed the LLM's bad output but ran the updated guardrail code live, asserting the order was blocked this time. | Caveat: Chronicle is a proof-of-concept; no mention of production maturity, performance overhead, or integration with existing observability stacks (e.g., OpenTelemetry). The speakers do not discuss how Chronicle handles streaming outputs, async workflows, or distributed traces across services. | Implication: Ken can adopt the boundary annotation pattern in his agent code to auto-generate regression tests from production failures; every production trace becomes a free, deterministic test case once the model calls are stubbed. | Timestamp: timestamp unavailable
• Claim: Deterministic testing (guardrails, tool logic) should use Chronicle-style trace replay; behavioral testing (tone, trajectory) should use LLM-as-a-judge techniques. | Evidence: Speakers distinguish two test categories: deterministic nodes where Chronicle 'kicks probability out of the window' by stubbing LLM outputs, and subjective behavioral checks where replay is insufficient and evaluator models are better suited. | Caveat: No examples or tooling provided for the behavioral side; the talk focuses exclusively on deterministic replay and does not integrate the two testing modes into a unified workflow. | Implication: Ken should use Chronicle for regression/guardrail tests and separate LLM-as-a-judge pipelines for qualitative checks; the two are complementary, not substitutes. | Timestamp: timestamp unavailable
• Claim: Record at the boundary (what enters/leaves each node), not at the network layer, because half your agent never touches the network (local retrieval, in-process tools, memory). | Evidence: The speakers state that network-layer logging misses local steps and breaks under streaming/async; boundary recording captures the meaning of each step, not the packets. | Caveat: No quantification of 'half your agent' or discussion of how to handle async boundaries, partial failures, or non-serializable state (e.g., open file handles, DB cursors). | Implication: Ken's instrumentation strategy should annotate agent code boundaries—not just API calls—to capture the full execution context; relying on HTTP middleware or LLM provider logs will leave blind spots. | Timestamp: timestamp unavailable

Detailed Brief

Why temperature=0 fails to deliver determinism

• Claims: Temperature zero means always pick argmax, but underlying scores shift run-to-run.; Floating-point math is non-associative; timing shifts in matrix ops flip the winning token.; Batch invariance: requests grouped with random traffic alter computation paths.; Mixture-of-experts routing: tokens rerouted if batches overflow expert capacity.
• Evidence: Reddit/Hacker News threads show 1,000 runs at temperature=0 yield dozens of different outputs.; Running the same matrix multiply alone on GPU 1,000 times yields identical bits, proving it's not a concurrency bug.; The real culprit is batch invariance—your request's computation depends on what else hit the server that millisecond.
• Caveats: No hard numbers on variance magnitude (e.g., 5% different? 30%?).; No distinction between open-source local inference (more controllable) vs. hosted API (less controllable).; The speakers do not discuss whether batch size=1 or single-request inference eliminates variance.
• Implications: Ken cannot debug agent failures by re-running prompts locally; he must capture the original execution state.; Hosted APIs (OpenAI, Anthropic) will never offer bitwise determinism; self-hosted inference might reduce but not eliminate variance.; Teams should abandon temperature tuning as a debugging strategy and invest in trace-based observability instead.

Chronicle: boundary annotation and trace replay

• Claims: Chronicle annotates methods with @boundary to record input/output pairs and metadata (model version, build ID).; Each production run is frozen as a trace; nodes can be stubbed during replay to isolate code changes.; Replay mode enables deterministic CI: rerun the exact failure offline with zero model calls.
• Evidence: Demo: stock-selling agent with three boundaries (planning LLM, place_order tool, finalize LLM).; Production failure: agent sold 1,000 shares instead of $1,000 worth; Chronicle captured the bad tool call JSON.; Test replay: Chronicle stubbed the LLM's bad output, ran the updated guardrail live, and asserted the order was blocked.; Speakers show JSON hyper-detail for each node: input schema, output, metadata (model version, sampling params).
• Caveats: Chronicle is a proof-of-concept; no discussion of production readiness, performance overhead, or storage costs.; No mention of how Chronicle handles streaming responses, async/concurrent nodes, or distributed traces across microservices.; No integration examples with existing observability stacks (OpenTelemetry, Honeycomb, Datadog).; The speakers do not address how to version or manage large trace datasets or how to decide which traces to keep.
• Implications: Ken can auto-generate regression tests from production failures by annotating agent code boundaries.; Every production trace becomes a free, rerunnable test case once the model calls are stubbed.; Chronicle's boundary pattern is orthogonal to framework (LangGraph, Autogen, custom); it wraps any Python method.; Teams can iterate on guardrails/tools without burning API credits or waiting for model inference during test runs.

Deterministic vs. behavioral testing

• Claims: Deterministic testing applies to guardrails and tool logic; Chronicle freezes LLM outputs to kick probability out of the test loop.; Behavioral testing measures subjective qualities (tone, trajectory) and requires LLM-as-a-judge techniques.
• Evidence: Chronicle demo shows deterministic assertion: tool output == 'blocked'.; Speakers state 'tone of the agent or whether the trajectory it took was right' is more subjective and better suited to evaluator models.
• Caveats: No examples or tooling for behavioral tests; the talk does not show how to integrate LLM-as-a-judge into the Chronicle workflow.; No discussion of when to use which test type or how to balance coverage.
• Implications: Ken should use Chronicle for regression/guardrail tests and separate LLM-as-a-judge pipelines for qualitative checks.; Deterministic tests are fast, free (no model calls), and repeatable; behavioral tests are slow, expensive, and probabilistic.; A complete agent test suite requires both modes; Chronicle solves only the deterministic half.

Five key takeaways from the speakers

• Claims: Stop chasing bitwise determinism through the API—fundamental API design makes it impossible.; Log session variables: LLM version, build ID, RAG chunks.; Capture the full envelope, not just the prompt—there are many ingredients in the final response.; Use replays to debug, fix failures, then use the same trace as a test case.; Keep generation-time variation alive; don't pin temperature to zero—that's what brings agency to the agent.
• Evidence: Speakers provide a QR code for Chronicle code and articles.; Final slide: 'We hope the traces you put in today make your on-call cycle tomorrow much better.'
• Caveats: No discussion of how to balance temperature variation (for creativity) with reliability/safety requirements.; No guidance on which session variables matter most or how to surface them in dashboards.
• Implications: Ken should treat temperature/top_p as inference-time knobs, not debugging levers.; Session variable logging (model version, code version, RAG chunks) should be instrumented from day one; retrofitting is painful.; Trace-based debugging is the path to SLAs and on-call sanity for agent systems.

Notable Concepts & Terms

• Bitwise determinism vs. replayability: Bitwise determinism = same input, same output (controllability); replayability = rebuild a run that already happened well enough to debug it (observability). Speakers argue you don't need the former, you need the latter.
• Boundary annotation: Chronicle's core abstraction: annotate any method (LLM call, tool call, retrieval) to record input/output pairs and metadata; boundaries become the unit of replay and stubbing.
• Batch invariance: The real culprit for non-determinism: your request's computation depends on what else hit the inference server that millisecond, altering matrix operation order and final token scores.
• Mixture-of-experts (MoE) routing: Experts have strict capacity limits; if a batch overflows a subnetwork, tokens get rerouted. Whether a token makes the cut depends on the traffic you got batched with, introducing non-determinism.
• Deterministic CI: Chronicle's replay mode: rerun the exact production failure offline with zero model calls by stubbing LLM nodes; enables fast, free regression tests.
• Deterministic vs. behavioral testing: Deterministic tests (guardrails, tool logic) use Chronicle trace replay; behavioral tests (tone, trajectory) use LLM-as-a-judge. Both are necessary; Chronicle solves only the former.

Operator Notes / Why Ken Should Care

• Ken's agent systems will fail non-deterministically in production; Chronicle's boundary annotation pattern is the most concrete mitigation strategy in this talk—adopt it for trace-based debugging.
• The 'temperature=0 still varies' claim is critical for agent ops: if Ken's team is burning cycles tuning temperature for reproducibility, redirect that effort to instrumentation and observability.
• Chronicle is proof-of-concept, not production-ready; Ken should evaluate whether to fork/extend Chronicle or build similar boundary recording into his own agent framework (LangGraph, Autogen, custom).
• Trace-based regression tests (replaying production failures with stubbed LLM outputs) are a major unlock for agent CI/CD; every production failure becomes a free test case.
• The speakers do not address cost/storage of traces, which matters at scale; Ken should plan retention policies and sampling strategies (e.g., keep all failures, sample 1% of successes).
• Chronicle's pattern is orthogonal to LLM provider and agent framework; it wraps any Python method, so it can be retrofitted into existing codebases.
• The talk implies Chronicle is open-source (QR code for code access); Ken should check if it's on GitHub and whether Microsoft is actively maintaining it or if it's a research artifact.
• For Ken's investing/GTM use cases: if agents interact with financial APIs (brokerage, trading), the $1,000 vs. 1,000 shares bug is a real risk; Chronicle-style guardrails + replay tests are essential for compliance and customer trust.

Watch Map

• timestamp unavailable: Intro: production agent failure scenario (sell $1,000 becomes sell 1,000 shares at $190/share = $190k disaster)
• timestamp unavailable: Why temperature=0 fails: GPU non-determinism, batch invariance, MoE routing, floating-point math
• timestamp unavailable: Reframe: wrong question (determinism) vs. right question (debugging/replayability)
• timestamp unavailable: Chronicle overview: boundary annotation, trace recording, metadata capture
• timestamp unavailable: Live demo: stock agent failure trace, JSON hyper-detail, replay with stubbed LLM + live guardrail, assertion passed
• timestamp unavailable: Deterministic vs. behavioral testing distinction
• timestamp unavailable: Five key takeaways, QR code for Chronicle code/articles

Source/Metadata

• Title: Your Agent Failed in Prod. Good Luck Reproducing It. - Tisha Chawla & Susheem Koul, Microsoft
• Transcript words: 2747
• Duration seconds: 849
• Timestamp note: Timestamps not present in transcript; watch_map is conceptual