Voice In, Visuals Out: The Agony and the Ecstasy - Allen Pike, Forestwalk Labs

At-a-Glance

• Verdict: Watch fully
• Core thesis: Voice-in + visual-out AI experiences are now feasible and superior to text-based chat if you solve the latency challenge with fast models, eager inference intervals, and aggressive caching.
• Why it matters: Ken's operator stack needs low-latency voice agents for real-time workflow—this is a practical engineering brief on how to build them using existing models (Haiku, GPT-4o Mini) and prefix caching to stay under 1s response times.
• Best use: Reference when designing real-time voice agent architecture; note the specific latency thresholds, model choices, and caching strategy for production voice workflows.

Executive Summary

Allen Pike from Forestwalk Labs argues that Andre Karpathy's thesis—voice is the preferred human input for AI, visuals the preferred output—is now practically achievable. The traditional text-in/text-out paradigm has given way to rich visual outputs (HTML, tool calls, illustrations) and voice input that works in real time. Pike's team built a voice agent that lives in their calls and can file Linear issues within a second of spoken intent, feeling 'perfectly natural' without interrupting conversation.

The central engineering challenge is latency. For seamless voice-to-visual experiences, responses must appear within ~1 second (the human attention threshold), not the 200ms needed for voice-to-voice conversation. Pike rejects slow models like GPT-4o Mini (P95 latencies of 5–10 seconds on OpenAI) in favor of Claude Haiku-class models, which reliably hit sub-second response times. He proposes a two-tier architecture: a fast 'real-time' model (Haiku or small open-source) handles continuous inference every 1–2 seconds as users speak, then hands off heavier tasks asynchronously to larger models.

Pike shares three critical techniques: (1) Use Haiku-class models on latency-optimized inference platforms, not just small models on slow platforms. (2) Send inference requests eagerly every 1–2 seconds during speech, not after waiting for silence, so the system reacts to partial sentences. (3) Implement stable prefix caching—keep the first 90% of the context identical across requests to get 90% cost and latency savings, then append only the new 10% and minimize output tokens. These techniques let Forestwalk deliver real-time voice-driven workflow automation that feels 'instant' rather than 'awkward and confused' like current Siri or ChatGPT voice experiences.

Key Takeaways

• Claim: Voice-in + visual-out AI experiences are now feasible and preferable to text-in/text-out, per Karpathy's thesis. | Evidence: Forestwalk built a voice agent that lives in calls and files Linear issues within a second of spoken intent ('I said let's file that as a linear issue, and the voice agent within a second responded that it had done so'). Rich visual outputs (HTML, tool calling, illustrations) are now possible with recent model advances. | Caveat: Most current voice interfaces (Siri, ChatGPT voice) are 'slow and dumb'—Pike acknowledges users are down on voice because existing experiences are awkward and confused. Success depends on solving latency. | Implication: Ken should prioritize voice-in + visual-out for agent workflows, not voice-in + voice-out or traditional chat UX. The architecture is proven at Forestwalk and delivers higher bandwidth than typing (more words/min, more meaning per word). | Timestamp: 00:00–03:30
• Claim: Seamless voice-to-visual experiences require <1s response latency, not the 200ms needed for voice-to-voice conversation. | Evidence: Pike cites human attention thresholds: 100ms feels 'instant,' 1000ms is the limit before people lose train of thought. For voice-to-voice, the limit is 200ms or conversation feels broken. For voice-to-visual, the more forgiving 1s envelope is sufficient ('if something appears on screen within a second of what you've said, it feels seamless'). | Caveat: Achieving even 1s is 'a ridiculous amount of work' when you chain network requests, speech-to-text, model inference, and network return. Most teams will miss this budget without careful architecture. | Implication: Ken's agent stack must target sub-1s end-to-end latency for voice-triggered visual responses. Voice-to-voice is harder (200ms) but voice-to-visual is achievable now with the right model and caching choices. | Timestamp: 03:30–05:30
• Claim: GPT-4o Mini is too slow for real-time voice; Claude Haiku-class models are required. | Evidence: Pike reports P95 latencies of 5–10 seconds for GPT-4o Mini on OpenAI ('5,000ms, 7,000ms, P95 sometimes 10,000ms latency for the small model'). Haiku is 'much better in terms of that P95 latency.' The fast model handles real-time turns; heavy work is handed off asynchronously to larger models. | Caveat: No specific Haiku latency numbers are given, and Pike does not name the inference platform (likely Anthropic API or a low-latency provider). Open-source small models are mentioned as alternatives but not specified. | Implication: Ken should default to Haiku (or small open-source models on low-latency inference) for the real-time agent layer, not GPT-4o Mini. Use a two-tier architecture: fast model for continuous inference, larger model for heavy async tasks. | Timestamp: 06:00–07:30
• Claim: Send inference requests every 1–2 seconds as the user speaks, not after waiting for silence. | Evidence: Traditional voice apps wait for seconds of speech + 1s of silence before inference, blowing the latency budget. Pike's system 'eagerly responds as the person is talking' every 1–2 seconds ('you have to have a model and infrastructure where you're able to get these fast turns going'). | Caveat: This requires the model and prompt to handle partial/incomplete sentences gracefully. Sentences like 'oh hey we're going to want to change this and actually also let's do this other thing' must be parsed incrementally. | Implication: Ken's voice agent should use streaming STT (e.g. Deepgram, AssemblyAI) with inference triggers every 1–2s, not end-of-utterance detection. The system must tolerate mid-sentence inference and update visuals as speech continues. | Timestamp: 07:30–09:00
• Claim: Stable prefix caching is essential: keep the first 90% of context identical across requests to get 90% cost and latency savings. | Evidence: Pike describes 'prefix caching' (available on multiple platforms) where identical context prefixes are cached. 'You can get up to 90% cheaper, faster inference… the first 90% of the context window should be the same from request to request, then just use that final 10%, and minimize output tokens.' | Caveat: This assumes long-running or frequently running agents where context is stable. No guidance on prompt design to ensure prefix stability or how to handle context drift over time. | Implication: Ken's agent prompts must be structured so system instructions and static context (e.g. user profile, tool schemas) are identical across turns. Only append new user speech/events in the final 10%. Minimize output tokens to keep inference fast. | Timestamp: 09:00–10:30

Detailed Brief

Voice-in + visual-out as the new AI UX paradigm

• Claims: Voice is the human-preferred input (per Karpathy), visuals the preferred output; Humans communicate faster and with more bandwidth via speech than typing (more words/min, more meaning per word—'okay' vs 'okay'); Rich visual outputs (HTML, tool calling, illustrations) are now feasible with recent model advances; Forestwalk built a voice agent that files Linear issues within a second of spoken intent during calls
• Evidence: Karpathy's recent argument (last month) on voice-in, visuals-out; One-third of the human brain is dedicated to visual processing; Models can now generate 'rich HTML, tool calling, visualizations, interactive controls, beautiful illustrations'; Pike's team mentioned a Slack bug on a call, spoke 'let's file that as a linear issue,' and the agent responded within a second that it had done so; Voice is 'the ultimate way that as humans we communicate'; people 'jump on a call' for high-bandwidth communication
• Caveats: Most current voice interfaces (Siri, ChatGPT voice) are 'slow and dumb'—users are down on voice because experiences are awkward and confused; Traditional text-in/text-out is still the dominant AI UX; voice-in, visuals-out is emerging but not yet mainstream
• Implications: Ken should design agent workflows for voice-in + visual-out, not voice-in + voice-out or text chat; Visual outputs allow richer feedback (charts, controls, images) than voice responses; Voice input enables hands-free, high-bandwidth agent interaction during calls, meetings, or while multitasking

The latency challenge: hitting the 1s envelope for voice-to-visual

• Claims: Voice-to-visual experiences require <1s response latency to feel seamless; Voice-to-voice requires 200ms or less; voice-to-visual is more forgiving (1s); 100ms is the threshold for computer reactions to feel 'instant'; 1000ms is the limit before people lose train of thought; Achieving sub-1s latency across speech-to-text, model inference, and network round-trip is 'a ridiculous amount of work'
• Evidence: Pike cites research from the 1960s on the 100ms instant-reaction threshold; 1000ms (1 second) is the limit before 'people start to lose their train of thought'; For voice-to-voice conversation, 200ms latency is required for full back-and-forth without interruption; Thinking Machines/Neolab recently demoed a time-sliced architecture that does continuous inference in 200ms chunks for voice-to-voice; Voice-to-visual benefits from the 1s envelope because visuals appearing on screen within a second 'feel seamless'
• Caveats: Novel architectures (like time-sliced continuous inference) exist for voice-to-voice but are not yet widely adopted; No specific latency numbers are given for Forestwalk's production system—only qualitative descriptions ('within a second'); Achieving 1s end-to-end requires optimization across STT, model, network, and rendering layers
• Implications: Ken's agent stack must measure and optimize end-to-end latency from speech start to visual update; Voice-to-visual is the pragmatic choice today; voice-to-voice requires bleeding-edge architecture; Latency budget must account for STT, inference, network round-trip, and rendering—each layer must be sub-300ms to stay under 1s total

Three critical techniques for sub-1s latency

• Claims: 1. Use a Haiku-class model on a latency-optimized inference platform, not just a small model on a slow platform; 2. Send inference requests eagerly every 1–2 seconds as the user speaks, not after waiting for silence; 3. Implement stable prefix caching: keep 90% of context identical, append only new 10%, minimize output tokens
• Evidence: GPT-4o Mini on OpenAI had P95 latencies of 5–10 seconds despite being small and cheap; Claude Haiku is 'much better in terms of P95 latency' and can respond in hundreds of milliseconds; Fast model handles real-time turns; heavy work is handed off asynchronously to larger models; Traditional voice apps wait for speech + 1s silence before inference, blowing the budget; Forestwalk infers every 1–2 seconds as people talk, handling sentences like 'oh hey we're going to want to change this and actually also let's do this other thing'; Prefix caching on multiple platforms offers 'up to 90% cheaper, faster inference' if the first 90% of context is stable; Pike recommends structuring context so the first 90% is static (system instructions, tools) and only the final 10% changes per turn
• Caveats: Haiku latency numbers are not specified; no mention of inference provider (Anthropic API vs. self-hosted vs. low-latency vendor); Open-source small models are mentioned as alternatives but not named or benchmarked; No guidance on prompt engineering to ensure prefix stability or how to handle context drift over long conversations; Eager inference every 1–2s requires handling partial/incomplete sentences; no detail on how this is implemented
• Implications: Ken should benchmark Haiku vs. GPT-4o Mini vs. small open-source models on target inference platforms (Anthropic, Together, Fireworks, etc.) for P95 latency; Use a two-tier architecture: fast model (Haiku) for continuous inference, larger model (Opus, GPT-4) for async heavy tasks; Integrate streaming STT (Deepgram, AssemblyAI) with inference triggers every 1–2s, not end-of-utterance detection; Structure agent prompts with static prefix (system, tools, user profile) and append only new speech/events in the final 10%; Minimize output tokens (short responses, JSON tool calls) to keep inference fast

Notable Concepts & Terms

• Voice-in, visuals-out (Karpathy thesis): Preferred AI UX paradigm: users speak (high-bandwidth input), AI responds with visual outputs (charts, HTML, tool calls) rather than voice or text
• Tyranny of latency: The central engineering challenge for real-time voice AI: achieving sub-1s (voice-to-visual) or sub-200ms (voice-to-voice) response times across the full stack
• 100ms instant-reaction threshold: Human perception threshold (since the 1960s): computer reactions under 100ms feel 'instant'; over 1000ms causes loss of train of thought
• Prefix caching: LLM inference optimization where identical context prefixes are cached across requests, offering up to 90% cost and latency savings (available on Anthropic, OpenAI, others)
• Two-tier agent architecture: Fast 'real-time' model (Haiku) handles continuous inference every 1–2s; larger model (Opus, GPT-4) handles heavy async tasks handed off by the fast model
• Eager inference intervals: Sending inference requests every 1–2 seconds as the user speaks, rather than waiting for silence, to keep the experience feeling real-time
• Haiku-class model: Pike's term for Claude Haiku or equivalent small models with sub-second P95 latency on low-latency inference platforms (contrasted with GPT-4o Mini on OpenAI)

Operator Notes / Why Ken Should Care

• Pike's talk is a practical engineering brief, not theory—Forestwalk ships a production voice agent that files Linear issues in <1s. The latency thresholds (100ms instant, 1s attention, 200ms voice-to-voice) are critical for Ken's agent UX design.
• The GPT-4o Mini latency data (5–10s P95 on OpenAI) is a red flag: even 'small' models can be slow if the inference platform prioritizes throughput over latency. Ken must benchmark on low-latency providers (Anthropic, Together, Fireworks) before choosing a model.
• The two-tier architecture (fast real-time model + async heavy model) is a repeatable pattern for Ken's operator stack: Haiku for continuous monitoring/response, Opus/GPT-4 for deep reasoning when triggered.
• Prefix caching (90% static context, 10% dynamic) is a must-have for real-time agents. Ken should audit current prompts to ensure system instructions, tool schemas, and user profiles are identical across turns, with only new events appended.
• Eager inference every 1–2s requires streaming STT and prompts that tolerate partial sentences. This is a departure from traditional voice apps (wait for silence) and likely requires custom integration work.
• Pike's call for collaboration ('I'd love to hear from anybody who has been exploring and experimenting') suggests Forestwalk is ahead of the curve but not alone—Ken should monitor the real-time AI agent space (Neolab/Thinking Machines, Fixie, others) for emerging architectures.

Watch Map

• timestamp unavailable: Timestamps unavailable in transcript
• 00:00–03:30: Intro: Karpathy's voice-in/visuals-out thesis, why visuals are preferred output, why voice is controversial input (Siri/ChatGPT pain), Forestwalk demo (file Linear issue in 1s)
• 03:30–05:30: The latency challenge: 100ms instant, 1s attention, 200ms voice-to-voice, why voice-to-visual is more forgiving (1s envelope), time-sliced architectures for voice-to-voice
• 06:00–07:30: Technique 1: Fast model on latency-optimized platform (GPT-4o Mini too slow at 5–10s P95, Haiku is better, two-tier architecture)
• 07:30–09:00: Technique 2: Eager inference intervals (send requests every 1–2s as user speaks, not after silence, handle partial sentences)
• 09:00–10:30: Technique 3: Stable prefix caching (90% static context, 10% dynamic, minimize output tokens, 90% cost/latency savings)
• 10:30–end: Closing remarks: invitation to share learnings, call to build delightful experiences

Source/Metadata

• Title: Voice In, Visuals Out: The Agony and the Ecstasy - Allen Pike, Forestwalk Labs
• Transcript words: 1943
• Duration seconds: 785
• Timestamp note: Timestamps unavailable—transcript does not include chapter markers or MM:SS annotations. Watch_map estimated from flow of argument.