AI Engineer

Realtime Voice Agents with Frontier Intelligence — Bohan Li, EliseAI

1804 summary words 8 min summary Watch video

Start with the signal

8 min read

Summary

At-a-Glance

  • Verdict: Watch fully
  • Core thesis: EliseAI achieves real-time voice agent performance with frontier-model intelligence by cascading a fast streaming transcriber with corrective batch transcription, using background tool-calling agents to eliminate round-trip latency, and caching TTS audio prefixes to hide generation lag—demonstrating production-grade latency optimization for voice AI in healthcare/housing domains.
  • Why it matters: These are concrete architectural patterns Ken can reuse for any latency-sensitive agent orchestration system: speculative execution layers, background tool pre-fetching, and output caching to hide LLM inference time without sacrificing model quality.
  • Best use: Watch for the architecture diagrams and trace-through examples; implement the streaming speculative transcriber, background tool-calling, and prefix-cache TTS patterns in OpenClaw or other real-time agent harnesses.

Executive Summary

Bohan Li from EliseAI presents a cascaded voice agent architecture inspired by self-driving car design: perception (transcription), planning (LLM), and control (TTS). The core challenge is hiding latency from slow frontier models while maintaining conversational naturalness. EliseAI solves this with three interlocking techniques: a streaming speculative transcriber that layers fast streaming ASR under slower batch-corrective ASR; background tool-calling agents that pre-fetch tool results and inject them into the main agent's context to eliminate round-trip tool-calling latency; and a prefix cache for TTS that plays previously generated audio segments immediately while the full sentence generates in parallel, suppressing duplicate audio and hiding generation time.

The speculative transcriber works by emitting fast detections from a streaming layer (e.g., Flux) and correcting them with a slower, context-aware batch layer (Scribe V2) when inaccuracies appear—for example, distinguishing "Elise's trial" from "Elise trial" and extracting structured name/DOB fields. Background tool-calling triggers eager agent generations on every transcription update, but only emits once user speech ends; meanwhile, background agents parse incoming text and populate tool results so the main agent believes it called tools itself, eliminating inference round-trips. The TTS prefix cache detects common phrase prefixes across generations, plays cached audio instantly after three tokens, pipes unique tokens to Cartesia's WebSocket TTS, and suppresses the cached portion of Cartesia's output to avoid duplication—resulting in seamless sub-second audio playback that hides full-sentence generation latency.

The live demo call (OBGYN appointment scheduling with insurance upload, calendar checks, and slot booking) shows all three layers working together: rapid turn-taking, natural interruptions handled gracefully, and no perceptible lag. EliseAI focuses on healthcare and housing verticals with production deployments, contrasting with typical SF AI startups by targeting critical life domains. The company is hiring and expanding in the Bay Area.

Key Takeaways

  • Claim: EliseAI uses a cascaded voice architecture modeled on self-driving cars: perception (transcription), planning (LLM), and control (TTS). | Evidence: Bo explicitly compares voice agent layers to autonomous vehicle stacks—bounding boxes/LIDAR become transcription, trajectory planning becomes LLM generation, vehicle controls become text-to-speech. | Implication: This framing suggests Ken can borrow robotics/AV design patterns (speculative execution, sensor fusion, control smoothing) for agent orchestration.
  • Claim: The streaming speculative transcriber layers a fast streaming ASR (Flux) under a slower, context-aware batch ASR (Scribe V2) to balance speed and accuracy. | Evidence: User says name and DOB; streaming layer emits "Elise trial 2303" first, then Scribe V2 corrects to "Elise's trial, date of birth 2303" because it understands the question context. Corrections cancel older detections when new audio arrives. | Implication: Ken can implement a similar two-tier transcription pipeline in OpenClaw: emit fast preliminary transcripts for responsiveness, then overlay corrective updates when context-aware models finish.
  • Claim: Background tool-calling agents eliminate LLM round-trip latency by pre-fetching tool results and injecting them into the main agent's context. | Evidence: On each transcription update, EliseAI triggers an eager agent generation without tool calls; a background agent parses the text in parallel, extracts name/DOB, performs phonetic matching for misspellings, and pushes results into context so the main agent believes it already made the tool call. | Implication: Ken should design agent control planes to run background workers that speculatively execute tools or fetch data, then merge results into the main generation context to avoid blocking the critical path.
  • Claim: The TTS prefix cache plays previously generated audio for common phrase prefixes while the full sentence generates in parallel, hiding LLM and TTS latency. | Evidence: Agent generates "you said your name is [name]..." streaming. After three tokens, cache hits on "you said your name is," plays that audio immediately, then pipes unique tokens ([name]) to Cartesia's WebSocket. Cartesia generates the full sentence with natural prosody, but EliseAI suppresses the cached portion and plays only the unique tail, creating seamless concatenation. | Implication: Ken can build a phrase-level output cache for agent responses, especially in domains with repeated phrases (greetings, confirmations, error messages), to achieve sub-second audio playback and hide generation lag. | Caveat: The speaker notes there may be a "tiny bit of a hiccup" but claims users won't notice; production audio quality and cache hit rate are not quantified.
  • Claim: EliseAI targets healthcare and housing verticals with production voice agents, contrasting with typical SF AI startups by focusing on "life's most critical areas." | Evidence: Demo call is an OBGYN appointment scheduler handling insurance upload, calendar checks, and slot booking. Bo emphasizes the company is "very focused on helping people where they need it" in housing and healthcare. | Implication: Ken should evaluate EliseAI as a case study for vertical-specific agent deployment and GTM strategy: high-stakes domains with clear ROI, repeatable workflows, and regulatory/compliance requirements that reward robust engineering over rapid iteration.

Detailed Brief

Streaming Speculative Transcriber Implementation

  • Claims: The streaming layer emits partial transcripts as audio arrives; the corrective layer cancels and replaces detections when new audio context invalidates earlier guesses.; Scribe V2's contextual understanding allows it to parse structured fields (name vs. DOB) from conversational input, not just transcribe words.
  • Evidence: First detection: "Elise trial." Corrective layer doesn't fire because text matches. Next detection: streaming text updated. Corrective layer cancelled because new audio arrived. Final detection: Scribe V2 emits "Elise's trial, date of birth 2303" with structured field separation.; Bo notes punctuation corrections are ignored ("we don't care") because they don't change semantic content.
  • Caveats: No performance metrics given for streaming vs. corrective accuracy, latency distributions, or cancellation rates.
  • Implications: Ken can build a similar cancellation queue in OpenClaw: streaming detections flow through, but slower high-fidelity models can cancel/replace them before emission to downstream agents.; Structured field extraction at the transcription layer reduces LLM prompt complexity and tool-calling overhead.

Background Tool-Calling Pattern Detail

  • Claims: Eager agent generations are triggered on every transcription update but held until end-of-utterance confirmation.; Background agents use phonetic matching to correct misspelled names from transcription errors.; Main agent receives tool results in context, so from its perspective it successfully called tools without waiting.
  • Evidence: User says "sure"—agent knows more is coming, doesn't emit. Next update: "still not really a name," agent plays along. Final corrected detection triggers background agent to find name/DOB, cancels earlier eager generation, re-triggers with tool context injected.; Bo mentions "correcting mistranscriptions of name, doing some phonetic matching."
  • Implications: Ken should design agent harnesses to treat tool execution as a speculative background task, not a blocking call, and merge results optimistically into the main generation's context window.; Phonetic/fuzzy matching in tools reduces brittleness when ASR makes predictable errors on proper nouns.

TTS Prefix Cache Mechanics

  • Claims: Cache waits for three tokens before checking for a hit to avoid single-word false matches.; Cartesia receives the full transcript and generates with natural prosody, unaware of the cache.; EliseAI suppresses the cached portion of Cartesia's output and plays only the unique tail to avoid duplication.
  • Evidence: "you said your name is" is cached. Once "[name]" token arrives (cache miss), cached audio plays immediately. Cartesia generates full sentence, but only the name portion is played after cache completes.; Bo states: "The user, there might be a tiny bit of a hiccup... you probably won't be able to notice."
  • Caveats: No quantification of cache hit rate, latency savings, or prosody quality at the cache/TTS boundary.; Potential for prosodic mismatch if cached audio and live TTS differ in intonation, though speaker claims it's seamless.
  • Implications: Ken can implement a phrase-level output cache keyed on token sequences, with a minimum sequence length threshold to avoid false hits.; This pattern is most effective in domains with high phrase repetition (customer service, scheduling, onboarding scripts).; Suppressing duplicate audio requires precise alignment between cache playback and live TTS generation; WebSocket streaming TTS (like Cartesia) enables this.

Notable Concepts & Terms

  • Streaming Speculative Transcriber: EliseAI's term for a two-tier ASR pipeline: fast streaming layer (Flux) for immediate detections, slower batch layer (Scribe V2) for context-aware corrections.
  • Background Tool-Calling Agents: Worker agents that run tool execution in parallel with LLM generation and inject results into the main agent's context to eliminate round-trip latency.
  • Prefix Cache (TTS): Audio cache keyed on token sequences; plays previously generated audio for common phrase prefixes while unique tokens generate, hiding TTS and LLM latency.
  • Eager Agent Generation: Triggering LLM generation on every transcription update (before end-of-utterance) to start inference early, but holding emission until speech is confirmed complete.
  • Cascaded Voice Agent: Architecture pattern splitting voice agent into perception (ASR), planning (LLM), and control (TTS) layers, analogous to self-driving car design.
  • Scribe V2: Context-aware batch ASR model used by EliseAI for corrective transcription with structured field extraction (e.g., distinguishing name from DOB).
  • Cartesia: WebSocket-based text-to-speech engine used by EliseAI for streaming audio generation with natural prosody.

Operator Notes / Why Ken Should Care

  • Implement streaming speculative transcription in OpenClaw: fast preliminary ASR layer for responsiveness, slower corrective layer that can cancel/replace earlier outputs before downstream agents consume them.
  • Design agent control plane to run background workers for tool execution and data fetching, then merge results into main generation context to avoid blocking inference on I/O.
  • Build phrase-level output cache for agent responses, keyed on token sequences with a minimum length threshold (3+ tokens), to hide LLM and TTS latency in repetitive conversational flows.
  • Evaluate EliseAI as a vertical deployment case study: high-stakes domains (healthcare/housing), production voice agents with regulatory compliance, and GTM strategy focused on critical workflows vs. general-purpose tooling.
  • Investigate Cartesia or similar WebSocket TTS providers to enable streaming audio generation and precise cache/TTS alignment for seamless playback.
  • For any real-time agent system, adopt the perception/planning/control framing from robotics/AV to structure latency optimization: speculative execution at each layer, cancellation queues, and output caching.
  • Monitor prosodic quality and user perception at cache boundaries if implementing TTS prefix caching; Bo claims hiccups are imperceptible but provides no quantitative validation.

Source/Metadata

  • Title: Realtime Voice Agents with Frontier Intelligence — Bohan Li, EliseAI
  • Transcript words: 3057
  • Duration seconds: 793
  • Timestamp note: No timestamps or chapters provided in transcript.
Search this video Resume transcript at 0%
Full transcript 1814 words · 13 min read
0:00

My name is Bo.

0:12

I'm going to be here presenting real-time voice agents with Frontier Intelligence. I'm going to be talking a little bit about how we at Elise AI architected our voice agent harness to get real-time voice with the Frontier level of intelligence that we need. So before I start, I think I wanted to draw some parallels about why we decided to go with Cascaded Voice Agents, and especially comparing that to self-driving cars, which I was working in before.

0:24

So, to me, Cascaded Voice Agents makes sense when you view it through the lens of breaking it down into perception, which for self-driving cars is the bounding boxes, the camera, the LIDAR. For voice, it's going to be the transcription, effectively turning these signals from the real world into elements of data that the language model or whatever brain you're working on can process. Second is the planning stack, which is straightforward. This is where the language model will take in the outputs from the perception stage and produce the outputs that you want to produce back out into the real world.

0:32

And finally, there's the controls layer where in self-driving, you would be taking the trajectory that the planner would output and turn it into the real controls to drive the car. Here, we're turning the text into audio that we use to express our voice agent's thoughts. So here I'll be diving into each one of these elements. We've made a few interesting tricks on each of these areas to improve the speed of our voice agents without sacrificing the intelligence.

0:47

So the first one is going to be the transcriber layer. We came up with this concept called the streaming speculative transcriber, where effectively we are layering a fast streaming transcriber, like Flux, on top of or below a Scribe V2 or an accurate batch transcription, which takes in more context. It's a little bit slower, but it will give you more accurate detections. So we're going to walk through a setting now. In this case, the agent just asked, "Can you write your name and date of birth?" and the user is going to say this, and we'll see how that plays out timing-wise.

0:54

First we're going to get the short detection from the streaming layer. The accurate layer, the corrective layer is not going to fire because it's the same text. We're going to get some more streaming text detections. And in this case, the corrective layer is actually cancelled because we got new text. So more context, more audio is going to beat the old accurate one. And here's where the first correction comes in. Because the Scribe V2 layer understands the context of the question, it's able to understand that this is talking about a name and this is a date of birth. And then a couple more detections. These are just punctuation. We don't care.

1:03

And so in the end, we release this text over to the agent. Moving on to the language model. So here, since we're using these slow but intelligent LLMs, we really want to reduce the number of round trips. And the thing that causes us to do a lot of inferences is tool calling. So one way to get rid of that is by having background agents do the tool calling for you and push the tools back into the context of the main agent so that it thinks it made the tool call, but it really didn't.

1:15

So we remember from detections from before. What will happen is each one of these detections is going to trigger an early generation of the agent. But we won't actually emit this out until we're confirming that the user has finished speaking. So in this case, the user says "sure." The agent knows that the user is about to say something else. Our background tool calling here, which is going to be helping us figure out the name and the date of birth from the user detection, is not firing. So nothing much there. The next instant detection comes in. It says that, you know, still not really a name. Our agent plays along and continues there.

1:28

Now more context comes back. The agent feels like there should be a name. It's going to ask to spell it out because it's probably thinking there's some transcription error here. Still no name or date of birth. And then finally this—remember this is our corrected final instant detection from the transcriber from the Scribe V2. Here, our eager agent generation that was made without any tool calls is going to get cancelled because the background agent finally is able to find the name and date of birth it's looking for. So it's going to re-trigger and now the agent actually has the context it needs.

1:39

And you see here we're doing it. The tool call here has some intelligence there. We're correcting mistranscriptions of name, doing some phonetic matching here. And then, once we've understood that this is the end of the user utterance, we'll emit it out. So pretty standard.

1:47

Okay, and then the next layer here is going to be text-to-speech. So with text-to-speech, the goal is to take what the agent said and the agent is going to be emitting this in a streaming fashion. So we're going to need to produce audio as quickly as possible. And ideally what you can do is before the agent has even finished generating the full text, you can have the audio play. So it's hiding the latency of finishing the generation. So I'm going to play the streaming agent output now. It starts with "you." And actually before I dive further, there's this new concept that we're introducing here called the prefix cache.

1:57

So the prefix cache is going to be looking at the agent stream and seeing if we already have generated audio for that sequence of words from a prior generation or maybe the same generation in this call as well. So it sees the word "you." For this prefix cache, we don't want to immediately hit on every single word. We're going to be waiting for a little bit more words. So after three words, the prefix cache gets our first hit.

2:09

And over here on the right, this is our text-to-speech standard provider. Cartesia is a text-to-speech engine with WebSocket support. So we're piping the agent through the cache and also piping it through WebSocket. More tokens come in, more cache, more sending through WebSocket. Not much to say here. And okay, so now we get our first unique thing, which is we found a token that actually causes a cache miss. It makes sense. If we're caching previous generations, "you said your name is" is a pretty common thing. But once we add in the name, suddenly that's going to result in a cache miss.

2:17

At this point, we're actually going to yield out our cached audio. So "you said your name is" is going to be emitted as the rest of the streaming text is coming back. So at this point, the user hears the agent. The user doesn't really know what's going on. It just looks like really fast response times to them. And now the remaining text flows through. And at this point, we've already emitted from the cache. The cache has done its job. The rest we can throw into Cartesia.

2:35

And here's the trick where Cartesia has seen the entire transcript up to this point. To Cartesia, it doesn't know about the existence of this prefix cache. It's just going to generate this full sentence with standard natural prosody. But what we do is when the generation comes back, since we've already played the audio here, we can actually suppress the audio from Cartesia here and just play out the remaining stuff.

2:38

So the user, there might be a tiny bit of a hiccup. I'll play some audio later. And you'll know that you probably won't be able to notice. And effectively, we're taking this part and just emitting it directly after this audio has finished playing. So it looks seamless to the user. Yeah, so at this point, this is a lot of technical details. I think the best way is just to play a call and see this in action. So here I recorded a call and let's just hear what happens. [CALL RECORDING] This is Bo, OBGYN, Elise speaking. Hi, I think I might be pregnant and I want to schedule an ultrasound to confirm. That's exciting news. Can you provide your name and date of birth?

3:05

Sure, it's Elise's trial, date of birth 2303. Thank you. Are you a new patient with us? Yes. Welcome to Bo, OBGYN. Can I text you a link to upload your insurance information? Sure. Okay, sent. Let me know when you've received it. I've received your insurance information. The earliest appointment I have is Thursday, July 2nd at 10 a.m. Actually, give me a second. Let me check my calendar. Sure. Take your time. Do you have something for next week? For next week, I can offer an ultrasound with Dr. Avery Stone at North Clinic on Tuesday, July 7th at 2 p.m. or 3 p.m. Do either of those work for you? Yeah, 2 p.m. works.

3:43

Great. Your appointment has been booked. We look forward to seeing you then. Thanks. Bye-bye. [END CALL RECORDING] All right. Yeah, that's pretty much it. Yeah, you can see all this streaming and a lot of things happening in the background. And this is what really makes voice agents interesting. And there's a lot of effort that can be done in the harness to really get a natural conversation, which is what we're after.

3:59

Okay, so briefly, in the last part, I want to talk a little bit about Elise. So Elise, our headquarters are in New York and we're trying to expand our presence here in the Bay Area. We're a different style of company than what people think of when they think about AI startups in San Francisco, where we're actually very focused on helping people where they need it, in life's most critical areas. We work on housing, healthcare, and we're doing really well. And here's a link to join our team, and we're going to be posting a lot on Twitter, so you can follow us at Elise.ai as well. Yeah, that's it. Thank you.

4:22

So what will happen is each one of these detections is going to trigger an early kind of generation of the agent. And we, but we won't actually admit this out until we're confirming that the user has finished speaking. So in this case, the user says sure. The agent kind of knows that the user is about to say something else. Our background tool calling here, which is going to be helping us figure out the name and the date of birth from the user detection, is not firing. So nothing much there. The next instant detection comes in. It says that, you know, still not really a name. Our agent kind of plays along and continues there. Now kind of more context comes back.

5:09

The agent kind of feels like there should be a name. It's going to ask to spell it out because it's probably thinking there's some transcription error here. Still no name or date of birth. And then finally this, you remember this is kind of our corrected final instant detection from the transcriber from the Scribev2. Here, our eager kind of agent generation that was made without any tool calls is going to get cancelled because the background agent finally is able to find the name and date of birth it's looking for. So it's going to re-trigger and now the agent actually has the context it needs. And you see here it's kind of we're doing it.

5:46

The tool call here is a little bit some intelligence there. We're going to be, you know, correcting mistranscriptions of name, kind of doing some like phonetic matching here. And yeah, and then we'll kind of, once we've understood that this is the end of the user utterance, we'll kind of emit it out. So pretty standard. Okay, and then the next layer here is going to be text-to-speech. So with text-to-speech, the goal is to kind of take what the agent said and the agent's going to be admitting this in a streaming fashion. So we're going to need to produce audio as quickly as possible.

6:22

And ideally what you can do is before the agent has even finished generating the full text, you can have the audio play. So it's kind of hiding the latency of finishing the generation. So I'm going to kind of play the streaming agent output now. So it starts with U. And actually before I dive further, there's this new concept that we're introducing here called the prefix cache. So the prefix cache is going to be looking at the agent stream and seeing if we already have generated audio for that sequence of words. From like a prior generation or maybe like the same generation in this call as well. So it sees the word you.

7:12

For this prefix cache, we're going to be, you know, we don't want to like immediately hit on every single word. We're going to be waiting for a little bit more words. So after three words, the prefix cache gets our first hit. And over here on the right, this is kind of our text-to-speech standard provider. You know, Cartesia is a text-to-speech engine with WebSocket support. So we're piping the agent through the cache and also piping it through WebSocket.

7:45

More tokens come in, more cache, more sending through WebSocket. Not much to say here. And okay, so now we get our first unique thing, which is we found a token that actually causes a cache miss. It makes sense. If we're kind of caching previous generations, you said your name is is a pretty common thing. But once we add in the name, suddenly that's going to result in a cache miss. At this point, we're actually going to yield out our cached audio. So you said your name is is going to be admitted as the rest of the streaming text is coming back. So at this point, the user hears the agent. The user doesn't really know what's going on.

8:27

It just looks like really fast response times to them. And now the kind of remaining text flows through. And at this point, we've already admitted from the cache. The cache has done its job. The rest we can kind of throw into Cartesia. And here's kind of the trick where Cartesia has seen the entire transcript up to this point. To Cartesia, it doesn't know about the existence of this prefix cache. It's just going to generate this full sentence with standard natural prosody. But what we do is when the generation comes back, since we've already played the audio here, we can actually suppress the audio from Cartesia here and just play out the remaining stuff.

9:14

So the user, there might be a tiny bit of a hiccup. You know, I'll play some audio later. And you'll know that you probably won't be able to notice. And effectively, we're kind of taking this part and just emitting it directly after this audio has finished playing. So it looks seamless to the user. And yeah, so I guess at this point, kind of this is a lot of technical details. I think the best way is just kind of play a call and see this in action. So here I recorded a call and let's just hear what happens. This is Bo, OBGYN, Elise speaking. Hi, I think I might be pregnant and I want to schedule an ultrasound to confirm. That's exciting news.

10:07

Can you provide your name and date of birth? Sure, it's Elise's trial, date of birth 2303.

10:21

Thank you. Are you a new patient with us? Yes. Welcome to Bo, OBGYN. Can I text you a link to upload your insurance information? Sure. Okay, sent. Let me know when you've received it.

10:48

I've received your insurance information. The earliest appointment I have is Thursday, July 2nd at 10 a.m. Actually, give me a second. Let me check my calendar.

11:01

Sure. Take your time. Do you have something for next week? For next week, I can offer an ultrasound with Dr. Avery Stone at North Clinic on Tuesday, July 7th at 2 p.m. or 3 p.m. Do either of those work for you? Yeah, 2 p.m. works. Great. Your appointment has been booked. We look forward to seeing you then. Thanks. Bye-bye. All right. Yeah, that's pretty much it. Yeah, you can kind of see all this streaming and a lot of things happening in the background. And yeah, this is what really makes voice agents interesting. And there's a lot of effort that can be done in the harness to really kind of get a natural conversation, which is what we're after.

11:59

Okay, yeah, so I guess briefly, you know, in the last part, I want to just talk a little bit about Elise. So I think Elise, you know, our headquarters are in New York and kind of we're trying to expand our presence here in the Bay Area. We've, I think it's maybe like a different style of company that I think people are, like, think of when I think about AI startups in San Francisco, where we're actually very focused on just, like, helping people and helping people where they need it, like kind of the life's most critical areas. We work on housing, healthcare, and we're doing really well.

12:38

And, you know, here's a link here to kind of join our team, and there's going to, we're going to be posting a lot on Twitter, so you can follow us at Elise.ai as well. Yeah, that's it. Thank you.

Reading tools

Type to find a passage

Appearance
Ask this transcript

Notebook

Export Markdown

Add a note