AI Engineer

Shipping AI to a Million Patients Without an A/B Test — Jared Joselowitz, Ufonia

2032 summary words 9 min summary Watch video

Start with the signal

9 min read

Summary

At-a-Glance

Verdict: Watch fully
Core thesis: For patient-facing clinical agents, safe deployment cannot rely on A/B tests, rollback, or vendor benchmarks; it requires a traceable evidence system that simulates defined hazards, evaluates behavior at scale, optimizes against clinical cost functions, and expands autonomy only as evidence accumulates.
Why it matters: This is a concrete blueprint for replacing reactive production experimentation with a pre-deployment evaluation and governance loop in domains where an erroneous agent response is irreversible and potentially harmful.
Best use: Use it as a reference architecture for high-stakes agent evaluation: hazard taxonomy, synthetic edge-case generation, LLM-as-judge validation, cost-sensitive optimization, version traceability, and staged deployment gates.

Executive Summary

Jared Joselowitz describes how Ufonia evaluates DORA, a regulated voice agent that performs clinical conversations such as post-operative follow-ups and pre-operative checks. Ufonia has completed roughly 200,000 UK clinical calls across 20 hospitals and plans to reach one million patients within two years. The central operating constraint is that conventional software-release safety nets do not apply: patients cannot ethically be randomized into an inferior variant, a spoken clinical call cannot be rolled back, and a model vendor's benchmark claims provide no defense after an incident.

The company therefore begins with harm rather than model capability. It identifies clinical hazards with clinicians—such as missing sudden vision loss or severe pain, hallucinating medical advice, or failing to acknowledge distress—and builds tests specifically intended to trigger those failure modes. Its Matrix framework uses a simulated patient agent, PatBot, to generate large volumes of clinically grounded conversations with DORA. Ufonia tested whether this simulator was plausible through a patient-and-public-involvement study, where participants frequently judged simulated-patient conversations as more realistic than real-patient examples.

At scale, the system relies on an LLM judge, Bev Judge, rather than engineers manually reviewing transcripts. The judge receives the dialogue, expected behavior, and clinician-defined hazards, then returns a pass/fail result and explanation. Against a 240-example corpus labeled by 10 clinicians from 10 specialties, the cited top model, Gemini 2.5 Pro at the time of the paper, achieved 0.96 F1 and near-perfect sensitivity—an intentionally prioritized metric because missing a true red flag is much worse than over-escalating one.

Simulation is not presented as proof of real-world benefit. It is the fast inner loop that earns the right to progress into supervised real-patient evaluation and monitored deployment. Ufonia combines real-call data with manufactured rare events and transcription failures, automatically optimizes prompts with a cost-sensitive metric, reruns the simulation gate, and deploys in stages. Its governing principle is that the regulated artifact is not merely the model or prompt: it is the evidence chain linking every call, data set, prompt version, judge verdict, and deployment decision back to a specific hazard.

Key Takeaways

Claim: Reactive release practices are unsuitable for patient-facing agents because their safety model assumes an organization can afford a limited number of failures. | Evidence: Joselowitz identifies three broken safety nets: A/B testing can be unethical or illegal when it exposes patients to an inferior change; a spoken call cannot be undone after delivery; and model-card benchmark scores are not adequate evidence in a post-incident review. Even a 5% rollout can expose hundreds or thousands of patients to unproven care. | Implication: Ken should treat high-consequence agent releases as an evidence-gated assurance problem, not as an observe-and-roll-back product experiment. | Caveat: The argument is specifically strongest where outputs can cause irreversible or material harm; lower-stakes products may still appropriately use controlled production experimentation.
Claim: The evaluation program should be organized around a clinician-defined hazard model, not generic model accuracy or broad benchmark performance. | Evidence: The talk names concrete hazards: failure to detect red-flag symptoms such as sudden vision loss or severe pain, fabricated medical answers, and failure to respond appropriately to patient distress. Ufonia reports documenting dozens of such hazards and grounding test scenarios in specific clinical workflows rather than abstract prompts. | Implication: Build a failure taxonomy before selecting models or metrics; each system capability, test case, and deployment control should map to a distinct operational harm.
Claim: Synthetic patient simulation enables ethically safe, scalable testing of rare and dangerous conversational cases before real patients are exposed. | Evidence: Matrix conditions its PatBot simulated patient on a target scenario—for example, asking whether the agent is human or AI—and runs conversations within a defined clinical context. Ufonia chose simulation over hired actors because actors would not scale with rapid system iteration. In a four-pair patient/public realism study, the majority judged the simulated patient more realistic in three of four comparisons. | Implication: Use synthetic personas and adversarial scenario generation as the pre-production inner loop, especially to manufacture infrequent harms that production data will not surface quickly enough. | Caveat: Realistic simulation is not equivalent to clinical validity: synthetic patients can miss real-world behaviors, so passing simulations only earns the right to conduct carefully supervised testing with real people.
Claim: LLM-as-judge evaluation can scale safety review only if it is validated against domain experts and tuned toward the relevant error asymmetry. | Evidence: Bev Judge evaluates simulated dialogues against expected behaviors and clinician-defined hazardous scenarios, producing pass/fail outcomes and failure reasons. Ufonia evaluated it on 240 examples with labels from 10 clinicians across 10 specialties; the cited Gemini 2.5 Pro judge achieved 0.96 F1 and almost perfect sensitivity. | Implication: Before relying on automated evaluation, create an expert-labeled adjudication set and measure the judge on the error type that matters most—often recall/sensitivity for severe hazards rather than average agreement. | Caveat: The reported score is tied to Ufonia's 240-example validation corpus and the then-current model, not a general proof that any LLM judge is clinically reliable.
Claim: Prompt optimization should be automated and driven by an explicit cost function, because manual prompt tuning is brittle, subjective, and difficult to audit. | Evidence: Joselowitz cites results where formatting changes shifted benchmarks by 76 percentage points and few-shot example ordering moved performance from near-random to near state-of-the-art in some cases. Ufonia uses JEPA (Genetic Pareto), associated with the DSPy creators: it identifies failed examples, has a strong model reflect on them and update prompts, and retains a Pareto frontier of candidate prompts. The reported iteration time falls from hours or days of manual work to roughly 30–60 minutes. | Implication: Make the objective function—not hand-authored prompt intuition—the core control surface; preserve reproducible prompt versions, optimizer inputs, and evaluation outcomes. | Caveat: Automation does not remove clinical judgment: clinicians must still define the data, hazards, and success metric that the optimizer is allowed to pursue.
Claim: Safety optimization must be cost-sensitive because different mistakes have radically different clinical consequences. | Evidence: For red flags, correctly detecting a true red flag is rewarded highly, while missing one is treated as potentially catastrophic. A false positive is framed as a relatively mild burden—such as asking a patient additional questions—so the system can deliberately optimize for sensitivity rather than a flat aggregate accuracy score. | Implication: Define asymmetric loss matrices for agent decisions; do not accept a single aggregate quality number where severe false negatives can be hidden by high performance on routine cases. | Caveat: Higher sensitivity can increase false positives and workflow burden, so the selected trade-off must be clinically and operationally acceptable for the specific use case.
Claim: The deployable product is an evidence-backed operating system, with autonomy expanded only through staged proof rather than a one-time model launch. | Evidence: Ufonia's loop combines real-call data, synthetic rare symptoms and mistranscriptions, prompt optimization, and Matrix simulation gating before deployment. It then moves through user testing, supervised clinical evaluation with clinicians in the loop, and monitored deployments. Every call, data set, pinned prompt, and judge verdict is intended to trace to the hazard it addresses. | Implication: Implement release gates that tie allowed agent autonomy to accumulated, auditable evidence, and treat post-deployment data as input to the next safety-evaluation cycle rather than as permission to rely on passive monitoring. | Caveat: The work is never complete: new modalities and languages introduce new hazards, requiring ongoing data collection and reassessment.

Detailed Brief

DORA's operational scope and regulatory framing

Claims: DORA performs clinical conversations that clinicians would otherwise conduct, rather than being positioned as a replacement for doctors.; Providing symptom guidance and answering medical questions makes the system a medical device and brings it within a regulated safety obligation.; The speaker reduces the regulatory problem to three questions: what the software does, what can go wrong, and how the organization ensures those failures do not occur.
Evidence: The demonstration concerns a cataract-surgery follow-up in which DORA asks clarifying questions about blurry near vision, explains that it should improve in the first days after surgery, and advises avoiding swimming for a month.; Ufonia reports deployment across 20 UK hospitals, approximately 200,000 completed clinical calls, an intended one-million-patient scale-up over two years, and US launch in two live clinics with six further clinics signed across four states.
Caveats: The transcript gives operational scale claims but does not provide comparative clinical outcomes, incident rates, or independent validation of DORA's patient benefit.
Implications: A conversational interface becomes a substantially different governance category when it can answer domain questions or advise users, even if its intended role is administrative or supportive.

Evidence lineage as the regulatory deliverable

Claims: The speaker's central governance standard is full traceability from observed or simulated behavior back to the hazard being controlled.; New deployment data is not just telemetry; it feeds a recurring improvement flywheel alongside deliberately constructed edge cases.
Evidence: The stated lineage includes every call, data set, pinned prompt, judge verdict, and the exact hazard addressed.; The testing corpus incorporates real-call data plus synthetic cases such as rare symptoms and mistranscriptions that may not naturally appear at sufficient frequency.
Caveats: The presentation does not detail the specific documentation format, approval authority, retention policy, or thresholds used to approve a deployment gate.
Implications: For auditability, agent teams need provenance across data, prompts, evaluators, model versions, and releases—not only a final quality score or a static model card.

Notable Concepts & Terms

DORA: Ufonia's voice-based clinical conversational agent for workflows such as post-operative follow-up and pre-operative checks.
Matrix: Ufonia's simulation framework for generating and evaluating clinically grounded agent-patient dialogues before real-patient exposure.
PatBot: The LLM-based simulated patient in Matrix, conditioned on specific test scenarios and personas.
Bev Judge: The LLM-based evaluator that assesses conversations against expected behaviors and clinician-defined hazards, returning structured failure judgments.
Sensitivity: The priority safety metric for detecting true hazards; the speaker favors overcalling potential red flags over missing dangerous ones.
JEPA (Genetic Pareto): A prompt-optimization approach that iteratively reflects on failures, generates revised prompts, and preserves a Pareto frontier of high-performing candidates.
Cost matrix: An explicit representation of asymmetric error costs used to optimize agent behavior for the consequences that matter clinically rather than for average accuracy.
Simulation inner loop / real-patient outer loop: A staged assurance model: simulation supports rapid pre-deployment iteration, while supervised real-world evaluation supplies the only meaningful evidence of actual patient benefit.

Operator Notes / Why Ken Should Care

Create a hazard register for each consequential agent workflow, with severity, expected response, test scenario, responsible domain reviewer, and release-blocking status.
Build an expert-labeled evaluation set before adopting an LLM judge; measure false-negative behavior separately from overall agreement or F1.
Require prompt, model, evaluator, test-data, and policy versions to be pinned and linked to each release decision.
Use an asymmetric loss function for safety-critical workflows, then explicitly approve the added false-positive and human-escalation load that higher recall will create.
Establish a staged autonomy ladder: synthetic evaluation, supervised user testing, monitored limited deployment, then broader operation only after predefined evidence thresholds are met.
Add manufactured rare cases, ambiguity, distress signals, and transcription faults to regression suites rather than waiting for production incidents to generate coverage.

Source/Metadata

Title: Shipping AI to a Million Patients Without an A/B Test — Jared Joselowitz, Ufonia
Transcript words: 3415
Duration seconds: 1154
Timestamp note: No timestamps or chapters were provided in the transcript.

Search this video Resume transcript at 0%

Full transcript 3615 words · 15 min read