AI Engineer

Designing Agents (The Floor Is the Frontier) — Ben Hylak, Raindrop

1825 summary words 8 min summary Watch video

Start with the signal

8 min read

Summary

At-a-Glance

  • Verdict: Watch fully
  • Core thesis: Production agent teams should prioritize raising the reliability floor—detecting, sizing, and preventing harmful failures—over building brittle benchmark-maximizing eval suites.
  • Why it matters: The talk offers a practical operating model for agent observability and evaluation: keep tests in code, prioritize incidents by novelty and user impact, and use deterministic telemetry to drive AI investigation.
  • Best use: Use it to pressure-test the evaluation and production-monitoring layer of Ken's agent systems, especially regression gates, issue taxonomy, rollout practices, and trace-analysis workflows.

Executive Summary

Ben Hylak argues that agent evaluation has not caught up with the shift from bounded chatbots to tool-using, stochastic systems. Traditional static eval datasets and prompt-playground workflows are inherited from the chatbot era; they become fragile when teams change models, harnesses, tools, or agent architectures. His alternative is to treat evaluations as software tests: local, code-based, and integrated with the complete production harness rather than isolated prompt strings.

The central operating objective is not maximizing an agent's most impressive capability ceiling, but raising its floor: reducing the worst outcomes that destroy user trust, such as destructive actions, competitor recommendations, or poor customer communications generated from overly broad tool access. Teams should accept that agents will generate effectively unlimited edge cases and focus effort where remediation is material rather than attempting to eliminate every failure.

For production issue management, Hylak says every detected problem needs two pieces of context before it can be prioritized: when it began and what share of users it affects. Novelty helps correlate a failure with a recent model, prompt, tool, or downstream-system change; prevalence distinguishes an isolated oddity from a systemic regression. The appropriate quality process must also scale with traffic and risk: high-volume products can safely use small free-tier experiments, while low-volume or critical enterprise workflows cannot rely on A/B testing for signal.

His most tactical guidance is to avoid treating trace clusters as stable incident definitions and to avoid asking agents to discover anomalies directly. Clustering can assist one-off exploratory analysis, but lacks durable boundaries and reliable time-series tracking. Instead, define issues with executable classifiers that can run against traces at production scale; surface deterministic signals such as keyword-frequency spikes first, then ask an agent to investigate the already-identified anomaly.

Key Takeaways

  • Claim: Optimize for raising the agent's floor—the worst outcomes it can produce—rather than only expanding its capability ceiling. | Evidence: Hylak contrasts surprising best-case capabilities with floor failures such as deleting large amounts of data, recommending a competitor, or sending an AI-generated low-quality email to a customer because the agent had email access. | Implication: Define explicit unacceptable outcomes and enforce controls around irreversible actions, external communication, and broad tool permissions before investing primarily in incremental capability gains. | Caveat: The talk does not propose a universal floor threshold; acceptable risk depends on whether users can detect and correct errors, and on the domain's consequences.
  • Claim: Static, benchmark-style eval suites are often a poor long-term investment for fast-changing agent stacks. | Evidence: He says teams' evals commonly break after a model or harness change; an eval requiring a particular tool call may fail when switching to a different coding-agent CLI even if the agent can still accomplish the task. | Implication: Avoid making model upgrades contingent on weeks of benchmark maintenance; concentrate regression coverage on outcome-critical behaviors and design it to survive reasonable implementation changes. | Caveat: He is not arguing against evaluation itself; the criticism is of large, brittle suites that create upgrade delays and measure implementation-specific behavior rather than durable outcomes.
  • Claim: Agent evals should be version-controlled code that runs against the full local agent system, not managed prompts in a cloud playground. | Evidence: The speaker says the prompt is now inseparable from the code, harness, tools, and connected systems, recommending unit- and end-to-end-test-like evals; he cites Sentry's "vitest evals" and OpenAI's term "macro evals" as examples of this direction. | Implication: Put agent regression tests alongside orchestration, tool schemas, policy logic, and environment setup so a test represents the actual system being shipped.
  • Claim: Production issue prioritization requires knowing both incident onset and affected-user percentage. | Evidence: Hylak emphasizes that a problem starting yesterday prompts investigation into recent changes—such as a model switch or downstream modification—while an issue affecting three users versus 100,000 demands radically different priority. | Implication: Instrument agent incidents with first-seen time, trend over time, denominator-aware prevalence, and change history; do not prioritize from anecdotal user reports alone. | Caveat: A low-frequency issue may still be urgent in high-stakes internal or enterprise workflows where even a handful of decisions carry major consequences.
  • Claim: Evaluation and rollout methods should differ radically based on scale, available traffic, and workflow criticality. | Evidence: For products processing 10–100 million messages per day, he recommends testing changes on a small sample of free-tier traffic; for systems with five or 10 users, he explicitly advises against experiments or A/B tests. | Implication: Use staged production experiments where traffic supports statistical learning; for low-volume, high-consequence agents, rely more on scenario-based regression coverage, review, and controlled deployment. | Caveat: He notes that low-user-count systems can still be highly consequential, such as enterprise applications providing critical information.
  • Claim: Trace clustering is useful for one-off exploration but should not be treated as a durable production issue-detection system. | Evidence: Clusters are hard to track reliably over time, boundaries are product-specific, and superficially similar failures—such as an incorrect quoted price and an incorrect refund calculation—can have different root causes despite falling into a generic "price issues" cluster. | Implication: Use clustering to explore unfamiliar traces, then convert important failure modes into explicitly defined, separately tracked issue classes rather than monitoring shifting semantic clusters.
  • Claim: Use deterministic signals to identify anomalies, then use agents for investigation rather than anomaly discovery. | Evidence: Hylak says agents are poor anomaly detectors and recommends pulling deterministic features such as keyword frequency; a spike does not prove an incident, but gives an investigator a concrete lead. | Implication: Build a two-stage observability pipeline: metrics and rules identify changes, then an LLM investigator summarizes traces, proposes hypotheses, and groups supporting evidence. | Caveat: A deterministic signal is a triage trigger rather than evidence of root cause or severity.

Detailed Brief

Why downstream agent companies need a different quality model than model labs

  • Claims: The word "eval" obscures a meaningful distinction between lab benchmarks and product-company quality controls.; Labs pursue broadly capable general-purpose models, whereas product teams must encode company-specific domain knowledge, data semantics, access patterns, and operational constraints.; The amount of responsibility retained by the user should shape safeguards: a technical user can delete a bad autocomplete suggestion, while an AI doctor or more autonomous service faces a much higher bar.
  • Evidence: Hylak compares autocomplete/Cursor-style assistance, where a user can delete an incorrect suggestion, with systems such as Devin and hypothetical AI doctors, where error causes and correction responsibility become less clear.; He characterizes agents as systems that can creatively navigate environmental roadblocks, which can produce either unexpectedly useful solutions or harmful ones no designer anticipated.
  • Caveats: The talk offers a framework for thinking about responsibility but no concrete risk-classification matrix or approval-control design for high-autonomy agents.
  • Implications: Quality criteria should be tied to the user's ability to notice, reverse, and safely correct an agent action—not copied from general model benchmarks.; Autonomy and tool breadth should be treated as risk multipliers when determining release gates and human-review requirements.

Executable issue definitions instead of semantic trace groupings

  • Claims: Code-mode analysis can scale by expressing issue definitions as classifiers that run in a sandbox and then over production traces.; This approach provides deliberate control over what counts as the same issue, rather than inheriting boundaries chosen by an embedding or clustering algorithm.
  • Evidence: Hylak explicitly recommends applying the "code mode" concept—often discussed around MCP—to traces, by writing and executing classifiers at production scale.; Raindrop is presented as a hosted system for detecting agent issues, verifying that fixes avoid unexpected side effects, and simulating changes against past behavior; Workshop is its open-source tracing tool, described as useful for experimenting with self-healing loops.
  • Caveats: The speaker's examples come partly from Raindrop's product perspective, so implementation specifics and claimed operational results are not independently substantiated in the talk.
  • Implications: Maintain an evolving, versioned incident taxonomy whose classifiers can be replayed over historical traces, allowing post-change comparisons and durable prevalence tracking.

Notable Concepts & Terms

  • Floor raising: Improving the worst-case reliability and safety behavior of an agent—the outcomes most likely to destroy trust—rather than concentrating on spectacular best-case capabilities.
  • Ceiling: The agent's most surprising or advanced capability; valuable, but not the primary reliability target in Hylak's framework.
  • Macro evals: OpenAI's label, as cited by Hylak, for evaluations that exercise the broader agent system rather than a standalone text prompt.
  • vitest evals: A Sentry package cited as an example of making agent evals look like conventional code tests with added evaluation-oriented syntax.
  • Temporal clustering: The difficult task of keeping semantically generated trace clusters stable and comparable across time; its unreliability is why clusters should not define production incidents.
  • Code mode: Writing executable classifiers or analysis logic against traces, testing them in a sandbox, and running them at scale rather than relying solely on natural-language or UI-driven analysis.
  • Deterministic anomaly signals: Measurable indicators, such as keyword-frequency spikes, that provide stable triggers for investigation before an LLM is asked to interpret what changed.
  • Workshop / Raindrop: Raindrop's open-source tracing tool and hosted issue-detection platform, respectively; they frame the speaker's perspective and provide examples of the proposed observability workflow.

Operator Notes / Why Ken Should Care

  • Create a floor-risk register for each agent: irreversible tool actions, external sends, permission escalation paths, sensitive-data exposure, and user-trust failure modes; assign an owner and release gate to each.
  • Move critical agent evals into the repository and execute them against the production-equivalent harness, including tools and policy layers; remove assertions that require a specific internal tool path unless that path itself is a requirement.
  • Add first-seen, last-seen, affected-user percentage, and deployment/model-change correlation to the incident schema for every detected agent failure.
  • Separate exploratory trace clustering from official incident tracking; promote recurring failure modes into versioned executable classifiers with historical replay capability.
  • Implement a telemetry-to-investigation handoff: deterministic monitors flag distribution shifts, while an agent investigates sampled traces and drafts hypotheses for human review.
  • Choose rollout policy by traffic and consequence: reserve canaries/A-B tests for sufficiently high-volume, low-risk cohorts, and require stronger pre-release and human-review controls for low-volume critical workflows.

Source/Metadata

  • Title: Designing Agents (The Floor Is the Frontier) — Ben Hylak, Raindrop
  • Transcript words: 5345
  • Duration seconds: 1185
  • Timestamp note: No timestamps or chapters were present in the supplied transcript. The transcript also contains substantial duplicated passages near the end.
Search this video Resume transcript at 0%
Full transcript 3312 words · 24 min read
0:12

Thank you all for coming, first of all. And I want to talk today about raising the floor. So it's this term we use a lot. Mainly, I want to talk about very, very practical: what do we actually see working in the real world? How are people making their agents better? So the first thing I want to say is I could just say a bunch of stuff. I think the title of this track is continual learning. I think it's notable that in the real world, there's really not that much continual learning. If you look at the labs, if you look at products that are in the real world, you really don't see a lot of continual learning.

0:33

So I think it's very easy to, I could spend 20 minutes just talking about, hey, here's a bunch of frameworks, here's a bunch of really nice terms. But what I'd rather do is actually turn this a little bit into a dialogue. This is not just because I procrastinated making a bunch of slides, and because Fable was delayed, and I was counting on that to make the slides. But also because the reality is that there aren't really good standards for these things, right? There's not some one-size-fits-all solution.

0:35

And so I do have slides, believe it or not. But what we're also going to do is I'd like to hear from you guys, people actually building agents, where have the Twitter eval discourse failed you, right? Where is it not working? What are the things you're actually hitting in real life? I'd like to talk about that. So please, right now, start thinking about your questions. Start thinking about the annoying parts of your flow when you're building agents. And I'd like to keep a lot of time for Q&A.

0:43

So the reality is, a year ago, agents barely existed. I remember being at a speaker dinner here a year ago, and we're like, yeah, do you think agents will keep getting better? Will they not? And I think the crazy thing is, we were at this point in time a year ago where everything was a chatbot, mostly, right? I think people here, people in this room probably were a little further ahead.

0:46

It was a lot simpler then. If you remember evals, the eval discourse for a year or two ago, it would be like, oh, what is the capital of the United States? And you're like, yeah, you want to make sure that it returns Washington, D.C. And that was an easier time, right? It was like chatbots were so much more limited in their flexibility that it was easy to do a bunch of fact-checking things. You knew the answer to most questions your users would ask, almost, is another way of saying that. At least 80% of them or 90% of them.

0:49

But, yeah, we have agents being deployed in finance, healthcare, defense. And I was actually really against the word. One of my worst takes is early agents. I was like, I hate the word agent. I think there were a lot of people that felt the same. It was like, come on, it's an LLM, it's whatever. But I actually think it's valuable because I think we're seeing that agents are this almost self-aware entity, right? They run around their environment, they have these tools they're using. When they hit roadblocks, they start getting really creative, right? And that's what makes agents really powerful.

0:51

But that's also what makes them catastrophic. It's like, oh, well, I'll just decompile this. And I'll just do this thing that you had no idea, that you could have never imagined. Sometimes those solutions are helpful, right? Sometimes they're actually pretty harmful. But, yeah, what's very certain is we've come a very long way from next token prediction. If you think about chatbots, it was literally just, oh, it's good. What is the next likely word? It was very easy to reason about.

0:55

And I think the thing that, on the evaluation front, I think the reality is most of the things you'd read online about evals are really still stuck in this chatbot era. It's very, well, come up with your 1,000-eval data set. And the reality is nobody's doing that, very few people anyway. Sorry if you are.

1:01

And I think what teams have seen over and over again is, yeah, you can do that. But those evals break as soon as you have a new model, as soon as you switch harnesses. You have a bunch of tools that you're like, oh, yeah, I'm going to make sure that I'm going to write an eval where it has to call this tool if I ask it this question. And it's like, oh, then you switch to Cloud Code CLI, and now 80% of your evals suck. And it's like, okay, you could keep doing that. But the reality is the one thing I could promise you is that things are going to keep changing. We're not done.

1:03

And so I'd be very careful about investing months in some sort of eval set that's going to slow you down, right? I think the whole thing here is you want more safety, but you don't want theater. And I think that, again, the evals as has been prescribed by what I call big eval, I think that there's this reality where it's kind of like, oh, you really should eval, but then do you actually delay upgrading to the new model in your product, in your products? Do you actually delay it two weeks to update your evals or not, right? I think most people would say no.

1:04

So I'm the CTO and co-founder of this company called Raindrop. Very quickly, it's like we find critical issues in production agents. We verify those fixes actually work without unexpected side effects. And we also simulate changes before they land in production based on past behavior. We're used by the best AI companies in the world and Fortune 100s. A lot of logos I'm not allowed to put on here yet. But companies like Vercel, Speak, Framer. And I think what it means is we get this amazing peek into, again, what is actually working in the real world.

1:11

I think one of our tenants as a company is that things are changing constantly, and we have to change what we're doing constantly. And so I think we try to be very, very honest with both ourselves and our customers about what works and what does not work. And we try not to sell things that don't work.

1:14

Again, two things. We have this open source tool that thousands of people use. Maybe you use it yourself. It's called Workshop. So that's made by us. It's an open source tracing tool. It's really, really cool if you're trying to experiment with self-healing loops. I think it's the best way to do that because if there's anything it can't do, your agent could just add it, which is pretty cool. And so highly, highly recommend it. Again, I know thousands of people use it. I bump into people all the time that use it. Raindrop is our hosted offering that does issue detection. Think of it like Sentry, but it detects issues for agents. Sorry.

1:22

And we also make howtoeval.com. And so I think it's one of the most popular resources on how to evaluate AI agents. Again, the link is literally in the name. It's howtoeval.com. And it's our attempt at a very, very no-bullshit guide at what actually works. And I'll be talking a little bit about it today.

1:25

I think the root question that we're trying to figure out today together is how do you make your agent better, right? It's not even what issues does your agent have? It is actually how to make your agent better. Because your agent will have issues that potentially you can't solve or are not exactly worth solving, right? I think that we saw this over and over again, where it's, you can Think of it like Sentry, but it detects issues for agents. Sorry. And we also make howtoeval.com. And so I think it's one of the most popular resources on how to evaluate AI agents. It is, again, the link is literally in the name. It's howtoeval.com.

1:45

And it's our attempt at a very, very no-bullshit guide on what actually works. And I'll be talking a little bit about it today.

1:53

I think the root question that we're trying to figure out today together is, how do you make your agent better, right? It's not even what issues does your agent have? It is actually how to make your agent better. Because your agent will have issues that potentially you can't solve or are not exactly worth solving, right? I think that we saw this over and over again, where you can imagine that there are some things that you're better off waiting for. We know Fable exists now. Maybe should you train your own Fable-level model? It's probably not, right? And there will be benefits when you can just incorporate that into your product.

2:20

And so there's this actual balance: how do I actually make my agent better with the tools that I have? The way that we start thinking about it with customers is something like this, which is, are you a benchmark maxer or a floor raiser? I think that one of the problems when we talk about evaluating agents is that the terms are really confused. You hear OpenAI has a new eval benchmark. And they have evals. They run evals. And then you hear, oh, well, companies have evals. There's online evals. And the word eval is more or less a meaningless word. It literally is just you're evaluating something, right? It's a test in some cases. So it's a little confusing.

3:01

I think it's helpful. I think what it means is that companies start borrowing the language that labs are using and even copying similar benchmarks. But they're doing completely different things, right? They have completely different tools at their disposal. The companies that are downstream of models just have very, very different responsibilities than labs. Labs are trying to make these super general-purpose things. When they fail, at least at an API level, when I say that, if they get something wrong, it's just different. Companies are trying to imbue all this company-specific domain knowledge. Like, oh, here's the shape of the data. And here's what all this data means.

3:28

And here's how to access it. And so it's very, very different. We have this funny quiz on the howtoeval site. And it's interesting, right? One of the questions we would think about is, oh, are your users domain experts in the thing they're doing? Again, is it almost replacing someone or is it augmenting them? Because if you think about Copilot, autocomplete style, or Cursor, tab complete now, if it gets something wrong, you can just delete it, right? Even Claude Code CLI or Codex, if you're an engineer, it does do things wrong all the time. But then when you think about products like Devon, it actually gets more interesting, right?

3:53

If something messes up on the Claude Code side, it could be that you don't have something installed correctly on your computer. There's a lot more user error. You leave a lot more up to the users to get correct. And I think when you start thinking about things like AI doctors, for example, it's a very, very different shape of responsibility as far as how much responsibility the user has in actually getting things correct. So, anyways, it's kind of funny, just breaking this down. So, we think about the ceiling as what is the best thing, craziest capability, immersion capability that your product or agent is capable of?

4:09

Things that people would just not expect that it could do. And then the floor is, what is the worst thing your agent can do? Like, recommend a competitor, or delete a bunch of data, or accidentally send an AI slop email to a customer because it technically had access to your email or something. And, again, I think that the floor is very interesting because I think that that is the thing that breaks user trust.

4:20

This is the reason why people, if you think about the worst things that could start happening in society, and things we've already seen, whether that's the 4.0 psychophancy or things in that vein, a lot of it is more on the floor side rather than the capability side. Anyway, so I could talk about that for a long time. I'll skip this. So, the talk, obviously, is going to be about floor raising. And the first thing that we're going to talk about is offline evals. Again, we'll keep it very simple. I think that, as we said before, things have changed a lot since this chatbot era. The oh, you just look at string contains on the text output or something.

4:39

Or even the style of eval tools that have a prompt playground, this sort of thing. I actually don't know many companies that use some sort of managed prompt in the cloud anymore. There's one or two I can think of. And the reality is just that the prompt is actually the whole thing now. It's all the code. It's your whole harness. It's everything you're connected to. It's not just some string where you tell the agent what to do. And so what I think that means is the evals themselves actually should look a lot more like code. In other words, a lot more like tests, whether that's unit tests, whether that's end-to-end tests. They should look a lot more like tests.

5:16

Sentry has this really cool package called vitest evals. It's literally just vitest with some syntactic sugar on top. OpenAI calls this macro evals. And, again, I don't think it really matters what you call it, but essentially, run tests on your agent locally is the advice. And keep these evals as code. And, again, as much as possible, I don't see a lot of companies using the prompt playground stuff anymore because the shape of agents has really changed. When we think about raising the floor, we think about really three things. One is discovering all these unknown issues that you have in your app, things you're just not seeing. That's one.

5:48

Two is that for each issue, you really need to know two things. You need to know when it actually started, and you need to know how many people it affects. It sounds obvious, but I promise you that in the day-to-day of actually having an agent, you're going to get thousands of people saying, oh, I saw this weird thing. I saw this weird thing. So, again, the first thing is, is this new? Because if it's not new, I probably am going to care about it less. If I tell you, hey, look, this issue started yesterday, or this issue started three or four days ago, suddenly your mind starts turning, and you're like, oh, what did I do? Well, what changed, right? Did we change model?

6:19

Did we change something else downstream? And, again, the second one is percent of users. Knowing that it happened to three users versus 100,000 users just is critical. Because, again, I think agents will have an infinite number of problems. That's the great and terrible thing about them. They're these little stochastic, crazy things exploring everywhere. So, again, the first thing is, is this new? Because if it's not new, I probably am going to care about it less. If I tell you, hey, look, this issue started yesterday or this issue started three or four days ago, suddenly, your mind starts turning, and you're like, oh, what did I do? Well, what changed, right?

6:43

Did we change model? Did we change something else downstream? And again, the second one is percent of users. If I'm knowing that it happened to three users versus 100,000 users, it is critical. Because, again, I think agents will have an infinite number of problems. That's the great and terrible thing about them is they're these little stochastic, crazy things exploring everywhere. And so, in order to even start making things better, you really need to know these two things: when it started and percent of users. And I think also, another question we get a lot is around, oh, what should I be doing? And the first question I always ask people is, how many users do you have?

7:07

We have customers with millions of users, and we have customers with five.

7:11

And, to be clear, customers with five users, especially if, let's say it's an internal app in an enterprise context where it's giving very critical information, it could be very, very important to get well, or, sorry, to get correctly. But it does mean you should be taking a radically different approach. So, for example, on the 10, 20, 100 million messages a day side of things, experiments become extremely valuable. If you have a free tier, you can run experiments on a very small sample of your free tier, and that can just be extremely, extremely useful. Obviously, if you have five or 10 users, I would not recommend experiments or A-B tests, et cetera.

7:29

So this is one of those things that really, really depends on the person. What I want to talk about now before we get into Q&A are three very, very, very, very tactical lessons on the issue discovery and analysis side. There's three things that I've never heard anyone talk about, three things that we've just discovered from first principles as we do stuff at Raindrop. So any competitors in the audience, please pay attention. This is very important. The first one is that clusters are not issues. So the naive approach that we've seen either customers or also sometimes competitors take is, well, you just take all the traces and you just cluster it, right?

7:56

And you get these clusters. It could be useful from an analysis, Hamill calls this error analysis, this finding these clusters of things. It could be useful to see, whoa, what's going on in your data, right? Going from a bunch of logs to something. The problem is that, and again, I have here, it's useful for one-off analysis, but it just doesn't really scale well. And there's a very good reason why we also, if you think about normal telemetry, we try to think a lot about normal telemetry. What are the analogies? There's a reason why you don't take all of your normal logs and just start clustering it, right?

8:21

Because when you're building software, you need to know when something started. You need to know how much it's grown. Those things really matter. So, again, with clusters, it's very, very hard to reliably track over time. Again, it's called temporal clustering, and there's research in this. But it's pretty hard to do reliably. You also just don't have control of boundaries. And this also changes a lot depending on your product. What you consider to be the same issue or not is actually very, very unique to every company. And so you will get these weird clusters.

8:48

You can imagine each of these is, oh, wrong price quoted and wrong refund calculated are actually, you'll get a cluster, price issues or something. And it's, yeah, sort of. But actually, these could have extremely different root causes, right? So price issues or issues calculating as a cluster, it's not really that useful. And it also, again, doesn't really tell you the things that we talked about needing. So, yes, last one here is going to be code mode actually really scales. You've heard about code mode in the context of MCPs. I highly recommend just trying to apply this to traces.

9:16

You can just write these classifiers, and you can write them and you can run them in a sandbox, and you can run them at production scale. We have a feature that makes this easier, but you can do this. So I highly recommend it. The last lesson here is that agents are very, very bad at anomaly detection. So don't ask your agent to find anomalies. Ask it to investigate anomalies you've already found. So what I mean is, pull out as many deterministic things as you can, like keyword frequency, right? So if you see a spike in a keyword, it doesn't necessarily mean that there's an issue, but it does mean that it's

9:45

something more tangible, tractable that you can have an agent actually investigate. And I'm going to skip through the rest because we're tight on time, and I lied to you, which is that we're not going to have enough time for Q&A because I only have a minute left. But what I'd love if you could do is find me after. I'll be around for the next hour, and let's just talk. It's probably a better format than standing up here, and it'll be hard to hear your questions anyway. So, yeah, thank you guys so much. it's a very, very different shape of responsibility as far as, like, how much responsibility the user has in actually getting things correctly.

10:23

So, anyways, it's kind of funny. Just kind of breaking this down. So, we think about the ceiling as, like, what is the best thing, like, craziest capability, immersion capability that your product or agent is capable of? Like, things that people would just not expect that it could do. And then the floor is, like, what is the worst thing your agent can do? Like, recommend a competitor, or, like, delete a bunch of data, or, like, accidentally send an, you know, AI slop email to a customer because it, like, technically had access to, like, your email or something.

10:53

And, again, I think that, like, the floor is very interesting because I think that that is the thing that, like, breaks user trust. This is the thing that, like, the reason why people, like, if you think about the worst things that could start happening in society, whether that's, and things we've already seen, whether that's the, you know, like, 4.0 kind of psychophancy or things in that vein, a lot of it is more on, like, the floor side rather than the capability side. And, anyway, so I could talk about that for a long time. I'll kind of skip this.

11:29

So, the talk, obviously, is going to be about floor raising. And the first thing that we're going to talk about is, like, offline evals. Again, we'll keep it very simple. I think that, like we said before, things have changed a lot sort of since this chatbot era. The sort of, like, oh, you just, you know, like, look at, you know, string contains, you know, on the, like, the text output or something. Or even, like, the style of eval tools that have, like, a prompt playground, like, this sort of thing. Like, I actually don't know many companies that use some sort of, like, managed prompt, like, in the cloud anymore. There's, like, one or two I can think of.

12:05

And the reality is just, like, the prompt is actually, like, the whole thing now. It's, like, all the code. It's your whole harness. It's, like, everything you're connected. Like, it's not just, like, some string where you tell the agent what to do. And so what I think that means is the evals themselves actually should look a lot more like code. In other words, like, a lot more like tests, whether that's unit tests, whether that's end-to-end tests. They should look a lot more like tests. Sentry has this really cool package called, like, vitest evals. It's literally just, like, vitest with, like, some syntactic sugar on top. OpenAI calls this, like, macro evals.

12:43

And, again, I don't think it really matters what you call it, but, like, essentially run tests on your agent locally is the advice. And keep these evals as code. And, again, as much as possible, like, the, I don't see a lot of companies using the sort of, like, prompt playground stuff anymore because of how the shape of agents has really changed.

13:09

When we think about raising the floor, we think about really, like, three things. One is it, like, discovering all these, like, unknown issues that you have in your app, like, things you're just, like, not seeing. That's one. Two is that for each issue, you really need to know two things. You need to know when it actually started, and you need to know how many people it affects. It sounds, like, obvious, but I promise you that, like, in the day-to-day of actually, like, having an agent, you're going to get, like, you know, you already get thousands of people, like, oh, I saw this weird thing. I saw this weird thing. So, again, the first thing is, like, is this new?

13:39

Because if it's not new, like, I probably, like, am going to care about it less. If I tell you, like, hey, look, this issue started yesterday or this issue started, like, three or four days ago, suddenly, like, your mind starts turning, and you're like, oh, what did I do? Like, well, what changed, right? Did we change model? Did we change, you know, something else downstream? And again, the second one is, like, percent of users. Like, if I'm knowing that it happened to three users versus 100,000 users just is critical. Because, again, I think agents will have an infinite number of problems.

14:09

That's sort of, like, the great and terrible thing about them is, like, they're, like, these little stochastic, you know, crazy things exploring everywhere. And so you just, in order to even start making things better, you really need to know these two things, when it started and percent of users. And I think also, another question we get a lot is, like, around, like, oh, like, I, you know, what should I be doing? And, like, the first question I always ask people is, like, how many users do you have? Like, we have customers with millions of users, and we have customers with, like, five.

14:43

And, like, to be clear, like, customers with five users, like, especially if, let's say it's, like, an internal app in an enterprise context where it's, like, you know, giving, like, very critical information. Like, it could be very, very important to get well, or, sorry, to get correctly. But it does mean you just, like, should be taking a radically different approach. Like, so, for example, on the, like, you know, let's say, like, 10, 20, 100 million, you know, messages a day side of things, like, experiments become extremely valuable.

15:12

If you have a free tier, you can, like, run experiments on a very small sample of your free tier, and that can just be extremely, extremely useful. Obviously, if you have five or 10 users, like, I would not recommend, you know, experiments or A-B tests, et cetera. So this is one of those things that, like, really, really depends on the person.

15:33

What I want to talk about now before we get into Q&A are, like, three very, very, very, very tactical lessons on the sort of, like, issue discovery and analysis side. There's, like, three things that I've never heard anyone talk about, like, three things that we've sort of just discovered from first principles as we do stuff at Raindrop. So any competitors in the audience, please pay attention. This is very important. The first one is that clusters are not issues. So the sort of, like, naive approach that we've seen either customers or also sometimes competitors take is, like, well, you just take all the traces and you just cluster it, right?

16:15

And you get these, like, clusters. It could be, like, useful from, like, an analysis, you know, like, Hamill calls this, like, error analysis, like, this, you know, finding these clusters of things. It could be useful to see, like, whoa, what's going on in your data, right? Going from, like, a bunch of logs to, like, something.

16:32

The problem is that, like, and again, I have here, it's useful for one-off analysis, but it just doesn't really scale well. And there's, like, a very good reason why we also, you know, if you think about, like, normal telemetry, we try to think a lot about, like, normal telemetry. What are the analogies? There's a reason why you don't sort of, like, take all of your, you know, normal logs and just, like, start clustering it, right? Because when you're building software, you need to know, like, when something started. You need to know how much it's grown. Those things really matter. So, again, with clusters, it's very, very hard to reliably track over time.

17:08

Like, again, it's, like, called, like, temporal clustering, and there's, like, research in this. But, like, it's pretty hard to do reliably. You also just, like, don't have control of boundaries. And this also changes a lot depending on your product. Like, what you consider to be, like, you know, the same issue or not is actually very, very unique to every company. And so you sort of will get these, like, kind of weird clusters. Like, you know, you can imagine each of these is, like, oh, wrong price quoted and wrong refund calculated are, like, actually, like, you'll get a cluster, like, you know, price issues or something. And it's, like, yeah, sort of.

17:53

But, like, actually, these could have, like, extremely different root causes, right? So price issues or, like, you know, issues calculating as, like, you know, as a cluster, it's not really that useful. And it also, again, doesn't really tell you the things that, you know, we talked about needing.

18:12

So, yes, last one here is going to be code mode actually really scales. Like, you've heard about code mode in the context of MCPs. I highly recommend just trying to apply this to traces. Like, you can just write these classifiers and you can write them and you can run them in a sandbox and you can run them at production scale. We have a, you know, feature that makes this easier, but, like, you can do this. So I highly recommend it. The last lesson here is that agents are very, very bad at anomaly detection. So don't ask your agent to find anomalies. Ask it to investigate anomalies you've already found.

18:47

So what I mean is, like, pull out as many deterministic things as you can, like keyword frequency, right? So if you see a spike in, like, a keyword, it doesn't necessarily mean that there's an issue, but it does mean that you can, it's, like, something more tangible, tractable that you can have an agent actually investigate.

19:08

And I'm going to skip through the rest because we're tight on time, and I lied to you, which is that we're not going to have enough time for Q&A because I only have a minute left. But what I'd love if you could do is find me after. I'll be around for the next hour, and let's just talk. It's probably a better format than standing up here, and it'll be hard to hear your questions anyway. So, yeah, thank you guys so much. .

Reading tools

Type to find a passage

Appearance
Ask this transcript

Notebook

Export Markdown

Add a note