AI Engineer

I Monitored Crime Audio. Voice Agents Scare Me More. — Sumanyu Sharma, Hamming AI

2210 summary words 10 min summary Watch video

Start with the signal

10 min read

Summary

At-a-Glance

  • Verdict: Watch fully
  • Core thesis: Voice agents are scaling faster than their reliability/safety infrastructure, creating a larger blast radius than decentralized crime because centralized architecture changes can instantly impact millions of users—and adversarial actors will exploit verification, PHI/PII leaks, and trust assumptions at scale.
  • Why it matters: Ken is building orchestration and agent control planes; this speaker ran production-scale monitoring (Citizen crime alerts) and now monitors 10,000 voice agents, reporting 10% real-world error rates and 1-in-5 adversarial break rates. The video provides a tested debugging loop, failure taxonomy, and red-teaming threat model directly applicable to Ken's agent systems and security posture.
  • Best use: Extract the six-step debugging loop (identify, prioritize impact, understand, execute, check for regressions, monitor), the crime-vs-voice comparison framework for blast radius thinking, and the red-teaming findings (1-in-5 agent break rate, bypass verification, PHI leaks) to inform OpenClaw's pre-deployment testing, monitoring architecture, and adversarial scenario planning.

Executive Summary

Sumanyu Sharma, CEO of Hamming AI, previously worked at Citizen monitoring thousands of hours of police radio for real-time crime alerts. He argues that voice agents now scare him more than crime because agents are graduating from POCs to production at massive scale—he estimates over a trillion calls per year will be handled by conversational voice agents within five years. With a 1% error rate, that would mean 10 billion incidents annually; in practice, Hamming monitors 10,000 agents and observes a 10% real-world error rate across financial services, healthcare, and consumer domains. These failures range from annoyances (repetition, mishearing) to safety risks (mishandling peanut allergies at drive-throughs, failing to book medical appointments, bypassing verification steps).

The key difference between crime and voice agents is centralization and blast radius. Crime is hyperlocal and decentralized—a robbery affects a finite set of people. Voice agents are centralized: a single prompt or architecture change instantly propagates to millions of users. Sharma recounts his own experience showing up to a doctor's appointment that a voice agent failed to schedule, wasting two hours. He asks the audience to imagine the same failure happening to a grandparent or for a procedure instead of a checkup. The confident, natural-sounding voices make errors harder to detect, and users assume correctness.

Sharma presents a six-step debugging loop borrowed from Facebook growth teams: (1) identify all challenges in the conversation experience, (2) prioritize by impact size using frequency and severity analysis, (3) understand the root cause, (4) execute the fix, (5) validate that the fix works and does not introduce regressions, and (6) continuously monitor in production. He emphasizes cross-conversation pattern analysis—not just checking known rubrics on individual calls, but discovering emergent behavior across thousands of interactions. Teams should start by manually listening to calls for texture, then scale with evals and LLM-as-judge scoring, but must invest in cross-call analysis to catch systematic issues.

On adversarial threats, Sharma warns that as voice agents gain access to more data and tools, their attack surface expands. Hamming's red-teaming product, shipped in April, can break one in five agents—bypassing verification, extracting PHI/PII, and tricking agents into unauthorized actions. He predicts that when adversarial callers learn to exploit voice systems at scale, incidents will spike and directly impact individuals, unlike crime which feels distant. His recommendation is threefold: deep pre-deployment testing (text-to-text or voice-to-voice adversarial scenarios), robust monitoring (per-call scoring, manual evals, cross-call analysis tracking both agent and user behavior for adversarial intent), and 24/7 red-teaming for high-stakes deployments.

Key Takeaways

  • Claim: Voice agents are scaling to handle a trillion calls per year within five years, but real-world error rates are 10% (not the theoretical 1%), meaning 100 billion potential incidents annually. | Evidence: Hamming monitors 10,000 agents in production across financial services, healthcare, and consumer verticals and observes a 10% error rate. Failures include agents claiming they booked appointments when they did not, skipping eligibility verification, applying unauthorized discounts, mishearing users, and providing incorrect information. | Implication: Ken's orchestration and control planes must assume high baseline error rates and prioritize systematic pattern detection and regression testing over optimistic 1% assumptions.
  • Claim: Voice agents have a centralized blast radius: a single prompt or architecture change can instantly impact millions of users, unlike decentralized crime which affects finite, hyperlocal groups. | Evidence: Sharma contrasts crime (robbery, theft affecting local individuals) with voice agents where one change propagates globally. He cites his own failed appointment booking—a waste of time for him, but potentially life-threatening if it had been a grandparent's procedure. | Implication: Ken should design OpenClaw's control plane with rollback, canary deployment, and blast-radius containment in mind. Pre-deployment validation and A/B testing are not optional for centralized agent changes.
  • Claim: Most teams start with manual call listening and spreadsheet rubrics (greetings, closing, validation, core logic), then scale with evals and LLM-as-judge, but miss emergent cross-conversation patterns that are only visible across thousands of calls. | Evidence: Sharma describes the typical progression: manual listening for texture, then deterministic/stochastic scoring for known problems, but teams remain stuck checking consistency on individual calls rather than discovering novel failure modes that appear systematically across conversations. | Implication: Ken should invest in cross-conversation pattern analysis infrastructure early—not just per-call scoring—especially for OpenClaw's orchestration layer where systemic workflow bugs may only surface across many executions.
  • Claim: To validate a fix, replay the exact failing conversation 5-50 times, then test the same intent with varied wordings, accents, styles, and added intents to ensure the fix is net positive without regressions. | Evidence: Sharma gives the example of his failed appointment booking: replay that exact conversation multiple times, then vary the phrasing, add another intent, and change accents to get coverage. He also notes that some hypotheses (e.g., outbound agent vocal quality in the first five seconds) can only be validated via real-world A/B testing, not simulation. | Implication: Ken should build replay and variation testing into OpenClaw's CI/CD pipeline, and plan for A/B testing infrastructure for high-stakes changes that cannot be fully validated offline. | Caveat: Pre-deployment synthetic testing cannot catch all issues; A/B testing in production is required for certain vocal and timing-sensitive changes.
  • Claim: Hamming's adversarial red-teaming product can break one in five voice agents in production, bypassing verification, extracting PHI/PII, and tricking agents into unauthorized actions. | Evidence: Shipped in April, tested across financial services, healthcare, and consumer domains. Sharma states: 'We can probably break one in five agents at this point. We've bypassed verification. We've gotten data we should not have. This is not theoretical. This is actually a real concern.' | Implication: Ken must assume that adversarial callers will exploit OpenClaw-orchestrated agents at scale. He should implement continuous adversarial testing, monitor for user intent (earnest vs. adversarial), and design verification steps that resist social engineering and prompt injection over voice.
  • Claim: Voice agents sound confident and natural, which makes users assume correctness even when the information provided is wrong, creating a trust gap that is hard to detect and expensive to repair. | Evidence: Sharma cites a Twitter example where a customer was confused by incorrect trade-in information from a voice agent, and the human (Alex) had to call back to rebuild trust: 'Voices sound very confident. They sound very natural, but the information provided is often not correct. That's the biggest problem here.' | Implication: Ken should design OpenClaw's agent interfaces to surface confidence scores, flag uncertain responses, and log all agent commitments (bookings, verifications, data lookups) for human auditing, rather than relying on vocal naturalness to imply correctness.
  • Claim: The only real defense against adversarial voice exploits is (1) deep pre-deployment testing (text-to-text or voice-to-voice), (2) comprehensive monitoring (per-call scoring, manual evals, cross-call analysis for both agent and user behavior), and (3) 24/7 red-teaming for high-stakes deployments. | Evidence: Sharma outlines three defenses after demonstrating the 1-in-5 break rate. He emphasizes that monitoring must track not just agent behavior but also user intent—whether users are being adversarial, annoying, or trying to trick the agent. | Implication: Ken should build or integrate red-teaming and adversarial intent detection into OpenClaw's monitoring and evaluation stack, especially for financial, healthcare, or high-PII workflows.

Detailed Brief

Six-step debugging loop for voice agent reliability

  • Claims: Step 1: Identify all challenges and problems in the conversation experience.; Step 2: Prioritize by impact size using frequency (one-off vs. systematic) and severity (low impact vs. high impact/safety risk).; Step 3: Understand the root cause of the prioritized failure.; Step 4: Execute the fix.; Step 5: Validate that the fix works and does not introduce regressions by replaying the exact conversation with variations.; Step 6: Continuously monitor in production for emergent patterns and regression.
  • Evidence: Sharma borrowed this framework from early Facebook growth team members.; He uses a 2x2 matrix: y-axis is known problems vs. emerging behavior, x-axis is few conversations vs. many conversations (coverage).; He categorizes issues as one-off low impact (ignore), systematic low impact (annoying but worth solving in bake-offs), one-off high impact (hope it is a fluke), and systematic high impact (P0 target).; Example of P0: a financial services agent that fails to freeze a credit card when requested.
  • Caveats: Some hypotheses cannot be tested synthetically and require real-world A/B testing (e.g., outbound agent vocal quality in the first five seconds).
  • Implications: Ken should map this loop to OpenClaw's incident response and regression testing pipeline, ensuring cross-call pattern detection and A/B testing infrastructure are in place before scaling deployments.

Adversarial threat model and red-teaming findings

  • Claims: As voice agents gain more capabilities, data access, and tool integrations, their attack surface expands proportionally.; The more natural and human-sounding agents become, the easier it is to trick humans into revealing PII/PHI or trusting fraudulent agents.; Hamming's red-teaming product has successfully bypassed verification steps, extracted unauthorized data, and rejected agents across financial services, healthcare, and consumer domains.
  • Evidence: Sharma references the Mythos NSA trade secret extraction case as a proof point that voice agents can be exploited to reveal sensitive data.; 1-in-5 agent break rate observed in real testing.; Red-teaming product shipped in April 2024.
  • Implications: Ken must design OpenClaw to resist adversarial voice inputs, including social engineering, prompt injection over voice, and multi-turn exploits.; He should implement monitoring that flags adversarial user behavior (attempts to bypass verification, repeated probing, unusual intent sequences) and triggers human review or stricter gating.

Notable Concepts & Terms

  • Blast radius: The scope of impact from a single change. Voice agents have centralized blast radius (one prompt change affects millions of users) vs. decentralized crime (localized impact).
  • Cross-conversation analysis: Analyzing patterns across thousands of calls, not just scoring individual calls, to discover emergent failure modes that are systematic but not obvious in single-call rubrics.
  • LLM-as-judge: Using language models to score call quality and adherence to rubrics at scale, replacing manual review for known problems but insufficient for discovering novel patterns.
  • Red-teaming (24/7 adversarial testing): Continuous automated adversarial probing of voice agents to identify security, verification, and data leakage vulnerabilities before attackers exploit them in production.
  • Replay testing with intent variation: Taking a real failing conversation, replaying it 5-50 times, then testing the same intent with varied wordings, accents, styles, and added intents to validate fixes without regressions.
  • Frequency-severity matrix: 2x2 prioritization framework: one-off vs. systematic (frequency) crossed with low impact vs. high impact/safety risk (severity). Systematic + high impact = P0.
  • Earnest users vs. adversarial users: Distinction between users who just want their problem solved (earnest) and those trying to trick, bypass, or exploit the agent (adversarial). Monitoring must track both.
  • PHI/PII leakage via voice: Risk that adversarial callers or compromised agents reveal protected health information or personally identifiable information through conversational exploits.

Operator Notes / Why Ken Should Care

  • Map Hamming's six-step debugging loop to OpenClaw's incident response workflow, especially step 5 (regression checking) and step 6 (continuous cross-call monitoring).
  • Build or integrate cross-conversation pattern analysis into OpenClaw's monitoring stack; do not rely solely on per-call scoring or known rubrics.
  • Design canary deployment and rollback for OpenClaw's orchestration layer to contain blast radius from prompt or workflow changes.
  • Implement replay testing with intent variation in CI/CD: take failing interactions, replay 5-50 times, then vary wording/accents/intents to validate fixes.
  • Plan for A/B testing infrastructure to validate changes that cannot be fully tested synthetically (e.g., vocal quality, timing-sensitive flows).
  • Assume 10% baseline error rate for production agents and budget monitoring/red-teaming accordingly; do not optimize for theoretical 1% assumptions.
  • Build or integrate 24/7 adversarial red-teaming for high-stakes OpenClaw workflows (financial, healthcare, high-PII domains).
  • Monitor for adversarial user behavior: repeated verification bypass attempts, unusual intent sequences, probing for data leakage.
  • Surface agent confidence scores and flag uncertain responses in OpenClaw's user interfaces; do not rely on vocal naturalness to imply correctness.
  • Log all agent commitments (bookings, verifications, data lookups) for human audit trails, especially in regulated domains.
  • Reach out to Hamming if OpenClaw needs architecture or evals validation for voice-adjacent orchestration (speaker provided WhatsApp and direct contact).

Source/Metadata

  • Title: I Monitored Crime Audio. Voice Agents Scare Me More. — Sumanyu Sharma, Hamming AI
  • Transcript words: 2453
  • Duration seconds: 964
  • Timestamp note: Timestamps were not present in the transcript; all timestamp fields are null.
Search this video Resume transcript at 0%
Full transcript 2449 words · 11 min read
0:00

Suman Yu Reviewer Reviewer My name is Suman Yu, and I'm the founder and CEO of Hamming. Before working on voice agent reliability and safety, I worked at a company called Citizen out of New York. Anybody here use Citizen app? Awesome. Thank you. At Citizen, we listened to crime, thousands of hours of police radio station data, and sent millions of alerts to users in San Francisco, New York, LA, Chicago, Baltimore, and so on. Some obviously gory and pretty sad, but others more funny, like a person stealing bags of ice cream from Safeway. A report of a man hanging off the side of the house after a woman stole his ladder.

1:07

If I actually take a look at the Citizen app right now, for those who are customers or users, I can see that there is a man yelling at person. There is indecent exposure. This is real. This is real time. This is a couple hours ago. These are real time alerts that we're sending.

1:27

Now, voice agents scare me more because they're finally graduating from demos and POCs to production. We should be super excited, but I'm nervous. I'm personally nervous. They're talking to users at a scale that would make Gary Tan and Paul Graham proud.

1:43

When I got started in voice agent reliability in early 2024, voice was just starting to work. It was not quite good yet, but it was just starting to work. You would have to pay me a lot of money for me to stop using Aqua Voice, Super Whisper, Whisper Flow, and so on. These products are just getting super good. And a big reason is because the underlying infrastructure is getting better and the orchestration layer is getting meaningfully better. It's getting much faster to build products and voice experiences that maybe are 60% good in a pretty short period of time, but the long tail is still wide away. I think speech-to-speech models are getting better.

2:24

Teams are experimenting with hybrid architectures of combining voice-to-voice modalities and cascading stacks to make the experience reliable but still pretty low latency. Things are obviously getting better. Agents are being connected to calendars, CRMs, EHRs, reservation systems, and so on. Voice agents can now take actions. However, reliability is still the number one problem holding back most voice agent deployments at scale. This is still the number one problem. This is an example I found on Twitter two weeks ago. A person is trying to get information for a trade-in and gets absolutely confused with the information that they're receiving.

3:09

Alex now has to correct for this loss of trust by trying to call the person and see what happened and fix the situation. Let me see if audio works here. I'm screwed up with another customer. We're getting it fixed, but I got to call them and see if I can work it out. I'm like, dude, half the time I'm like, I don't know if I'm talking to AI. I don't know if I'm talking to a person. It was just confusing, but we got there. It probably is AI and human. So I think voices sound very confident.

3:36

They sound very natural, but the information provided is often not correct. That's the biggest problem here. This example is more personal. I had booked an appointment with a physician a couple weeks ago, or I thought I did. I showed up to the appointment and turns out I was not actually on the schedule. So the front desk made it turn me away. I wasted two hours. For me, this was a waste of time. But what if this was actually your parent? What if this was your grandparent? What if this appointment was for a procedure instead of a regular checkup? The costs for these different permutations of the same failure mode can actually be super high.

4:12

Now let's compare crime to voice agents. I think observation number one is crime is actually decreasing over time. This is a good thing. And I hope it crosses the x-axis at some point in the future. Voice, on the other hand, is generally taking off, right? We're seeing a pretty fast takeoff of voice agents being deployed in production. There's at least a trillion calls that are done every single year. And majority of these will be done by conversational voice agents over the next five years. If you assume a one-person error rate, that is still 10 billion incidents per year. That's a lot.

4:54

In practice, we currently monitor 10,000 agents, and the error rate is closer to 10% in practice. These range from agents saying they found the right policy when they actually skipped the eligibility or verification steps, or applying discounts when they were not really supposed to, mishearing what the person said, providing incorrect information, or claiming they booked an appointment when they actually did not, just like it happened for me. Now, not every single call has an equally bad cost. Some range in the crime land, some range from trash fires, which are funny, annoying, not really hurting somebody. For a voice equivalent, that would be annoyances like repetition,

5:36

or not quite understanding what the user is saying. All the way to safety risks like mass shootings, or in the voice agent equivalent, it would be a drive-through that's deploying voice agents at scale, like a Taco Bell or McDonald's, and a person orders a vegan burger with peanut allergies. If one of those two situations are not handled correctly, that is definitely a safety concern at scale. The other big difference between crime and voice agent deployments is crime generally tends to be pretty hyperlocal. It tends to be very decentralized, right? Things like robbery, or motor vehicle theft, or larceny.

6:19

They're impacting a finite set of individuals that are involved in that situation. On the other hand, voice agents are much more centralized.

6:32

A single prompt change, or an architecture change, can have pretty massive implications downstream for all of the millions of users that are in the crossfire. So the blast radius is quite massive. So the natural question is, how do you make these incidents much more visible and obvious? That's the obvious question here. I'll borrow a framework from a couple of my friends who were OG growth folks at Facebook. Step one is to identify, okay, what are all the challenges and problems that exist in your conversation experience? Step two is to prioritize impact size. There's a frequency and severity analysis that's pretty important.

7:11

Step three is to understand, okay, how do we actually fix this? Step four, execute. Step five, okay, did my change actually work? And did it cause any regressions somewhere else? And lastly, we continue to monitor in production. On the y-axis, I think it's important to highlight there are known problems that already exist. Things like turnover latency, interruptions, maybe some ASR problems you're aware of. And these are known problems that exist that the team should track over time. On the other axis is actually emerging behavior or patterns that are only obvious across lots of conversations. On the x-axis, you have coverage, just like insurance.

7:55

Are you analyzing few conversations? Are you analyzing many conversations? Most teams will typically start by listening to calls manually. And I think that's the best place to start. I don't think you should skip that step. There's a lot of depth and insights you get by actually listening to specific conversations and building that texture that comes from that intuition. However, it's obviously not scalable. So most teams end up having a spreadsheet of five or ten different rubrics around greetings, closing, validation, core logic, and so on. To scale that up even further, you then end up investing in some evals product, right?

8:34

You might run some LMs as a judge and compute classic metrics and also more deterministic and stochastic scoring logic. But there, you're still stuck with checking for consistency of known problems, but you're not really discovering novel insights that are actually happening across conversations. We're spending a ton of time on performing cross-conversation analysis, not a pattern on a single call, but across conversations. And some of the best teams that we work with are doing the same. Now, to prioritize impact size, I think there's problems that are one-off that are low impact. I mean, who cares?

9:15

Even low impact and systematic problems in the crime world, that would be a trash fire. In a voice agent world, it could be some repetitions the team is experiencing. They're still annoying at scale, and if you are doing a bake-off, it's still worth solving for them. I would not ignore these kinds of problems. One-off and high impact? Well, hope it is a big chronic. And I think systematic and high impact are obviously the P0 target areas for the team to solve. An example of that would be in a FinServe capacity, there's a voice agent that helps users freeze their credit cards. And if it doesn't do that, well, that's a massive fail. All right, so understand and execute.

9:56

I'm pretty sure everyone's doing this. Please fix my agent. I think fixing, or rather attempting to make a fix, is the simplest and the lowest effort component of this debugging pipeline and loop. The next step is, all right, I made a change to my system. How do I actually know this thing works for real? A great way that's naive is to take a real call. For example, in my case, I booked an appointment and it didn't get scheduled, and replay that exact conversation and run that maybe 5, 10, 20, 50 times and see, okay, what is my probability of passing this type of issue?

10:24

A better way is to keep the same intent, but change the wordings, change the patterns, change the accents, change the style, add one more intent to the mix. And that gives teams much more coverage to feel confident that yes, I actually made a change, and my changes are net positive instead of net negative. There are certain fixes and hypothesis that are very difficult to test in a pre-deployment synthetic setting. And so A-B testing ends up being pretty critical for those circumstances. For example, if you have an outbound agent, the first five seconds of a conversation tends to be the most important.

11:14

And so the vocal quality and the specific words you end up using, they matter the most. And so A-B testing that is the only way in real life setting to get results. You can't really do it through simulations alone. And so there we have the loop. Identify, prioritize impact size, understand the fix, execute, check, make sure it didn't break anything, and then continue monitoring. So I think making voice agents useful is already hard as it is, even when dealing with earnest users on the other line, right?

11:51

These are people who just want their problem solved. They're not trying to mess with you. These are normal people. Now what happens when Mythos learns how to dial? So if we can extract trade secrets from the NSA, it can certainly seduce you into revealing PHI and PII data as well.

12:13

And I think both voice agents and humans will be targeted here.

12:26

Voice agents because there's a pressure to make these more capable, give them access to more data, give them access to more tools, deploy them quickly. The more the capability, the bigger the surface area.

12:44

This is common sense. And the more the voice agents become natural and human sounding, the more humans will be tricked along the way as well for those who are weaponizing. We shipped our red teaming product back in April just to test out this hypothesis for how many agents can we actually break from an adversarial capacity.

13:01

And we can probably break one in five agents at this point. We've tested this across financial services, healthcare, consumer, and so on. We've bypassed verification. We've definitely had agents, and we've been able to reject several agents and gotten data we should not have. So this is not theoretical. This is actually a real concern. I think the only real defense against the dark arts is step one to invest deeply in pre-deployment testing.

13:37

This could be text to text. This could be voice to voice. There's pros and cons to both. Have to chat offline if folks are interested. And this is just making sure you're not self-owning when you're talking to real people who just want to get their problem solved. Step two is to have a great monitoring system of all kinds. And I've highlighted different flavors of monitoring. Per-call scoring, manual evals, listening to conversations and cross-call analysis. And this is helpful both for monitoring what the agent is saying and behaving and how it's actually doing, but also the users. Are the users being adversarial? Are they being annoying?

14:20

Are they trying to trick the agent into doing things it's not supposed to be doing? And I think our new recommendation now is to run 24-7 red teaming for your agents, especially if you believe the cost of bad interactions can be rather large. So I think voice agents have this awesome potential of making the world feel much more human compared to interacting with clunky IVR trees or chatbots or worse, being stuck on a hold. And when we think about crime, we often think of crime happening to somebody else. Crime does not happen to you, typically.

15:00

With voice agents, especially bad actors, as these agents are deployed and as bad actors start to exploit a lot of the vulnerabilities, the number of incidents is about to go way up. And so the reason I fear voice agents more than crime is that one of these incidents is going to impact you. It already did for me. Awesome. So it's time for me to shill. We brought a lot of tokens. If you are interested in working in this space, please come and talk to us. And if you are deploying voice agents and want to validate whether your architecture or your evals are set up correctly, please come and talk to us. We'll be outside. And here's my number. Here's my WhatsApp.

15:43

Thanks, everyone. Thanks, everyone. Thank you. Thank you. Thank you.

16:02

Thank you. Thank you. Thank you. Thank you. Thank you. Thank you.

Reading tools

Type to find a passage

Appearance
Ask this transcript

Notebook

Export Markdown

Add a note