Hi everyone. I don't know if you're familiar with what I'm about to show, but remember when someone from the product is saying that some page is too slow and maybe we can optimize it. And then someone from engineering would say, yeah, probably, but we'd have to dig in to find out. So we just leave it as is. And then a few weeks later, Jenny would say, no, no, no, but it's actually way too slow now. We have to prioritize it. How long is it going to take?
And we would say something like, well, somewhere between an hour and a week. We'd have to look at it to find out. And then when she asks who can dig into it, the answer is only Dave. He's the only one who knows that code, and everyone else who wrote it left a decade ago. And amazingly, this still happens in teams all the time because we would know how much time it takes to do a specific optimization, but we never know how long it's going to take to investigate it and to find it. And miraculously, every time we look, we actually find things that can be done around performance and stability, but we never stop to proactively look for them because it doesn't make sense.
So hi, I'm Mai, co-founder and CTO at HUD. We're building a runtime intelligence layer for coding agents that captures function-level context and deep forensic context on things that matter so that coding agents can help you fix what's going on with production. And as a part of that, we built an agent workflow that helps with continuously optimizing performance of production applications. And I'm going to share a bit more about what we built, what were the challenges along the way, and hopefully you can take something out of it and apply it in your day-to-day. So we're surrounded by a lot of agency. So we're surrounded by a lot of agent PRs that all look good to us.
And one agent wrote them, and the other one says, yeah, I went over it and it looks fine. And we still feel that urge to verify before we just merge it to production. And then when it doesn't work, we end up asking ourselves, how the hell did we get to this place? And if you feel like that, I want you to know that you're not alone. So Google just published their Dora metrics for 2026. And we can see that the biggest impact of AI adoption on engineering is individual effectiveness, or that feeling of, oh my God, I'm so fast. I can do everything in the world. And for me personally, lack of sleep is another symptom of that.
But the second one is software delivery instability. And the throughput is actually not impacted as much as we expected. So we feel more effective. We're more effective individually. But as a team, our throughput is the same, and our software breaks more often, which is not exactly what we were hoping for with this revolution just yet. But maybe there is an agent for that. And if we can build faster with AI and we can fix faster with AI, then we can get those gains that we were talking about.
So I'm going to start with why we even wanted to do this. Then we're going to go over the tech and the process, which I think is also really important, and then share some gotchas and takeaways along the way. So debt leaks faster than we can bail. We have these issues that we ignore because they're not important enough, then they degrade to the point where they are important enough. And then we reach this crisis mode where we are all hands on deck. We fix it in emergency mode, and then we go straight back to ignoring, which is a leaky bucket by definition.
And that mostly happens because the research phase is a black box. It could take an hour or weeks, and we have to pay that debt and make sure that we spend time and engineering time on it in order to even know what can be done about it. And that's hard. It's legitimately hard to prioritize something where you're not sure exactly what you're going to get out of it. But what if we can automate that investigation? So we can run on a weekly basis with real production context, analyze the sweet spot, and flag the high-ROI opportunities in a way that's scored.
So it runs automatically without us having to stop and do something about it. It has the production context in mind and can give high-ROI scored performance opportunities of the things that are easy and impactful. Kind of like that performance sprint that you run every few months, just automated. Now let's talk about how we can actually do it because the dream is very nice, but the devil's in the details. So first of all, we wanted an infrastructure for the agentic workflows that would be vendor-neutral in terms of compute and where it runs in terms of the harness and also in terms of the model.
Things are changing all the time. We wouldn't want to constrain ourselves to a specific vendor, specific model, or anything like that. And especially with HUD, we want our customers to be able to use whatever agents they wish. We also want it to be secure in terms of the tool calls, permissions, and authentication. We want some trigger system, whether those are scheduled runs like the weekly run or a set of webhooks. For example, if we see some SLO breach, we would want to investigate it. And we really wanted it to be easy to maintain and update over time.
I think one of the biggest learnings we've had with agentic workflows is that even if they work out of the box or we get to a point where we're happy about them, in time, we evolve and our expectations go up. So just being able to maintain and update those logics and build that feedback loop was very important for us so that it's reliable and that people actually trust the outcome. Specifically, we chose to work with GitHub agentic workflows for that. And we can choose whatever agent we want to work with and build the workflows on top of that. But there are many other great tools that could be used for that. So this is how it looks.
You can see that there's a description of the job and the goal and the analysis. And then we go over the GitHub repository and generate that weekly deep insight report analyzing production data and finding those low-hanging, high-ROI opportunities. So for this specific setup, GitHub Actions runs weekly, and it uses cloud code. That was our specific choice. And then captures the runtime intelligence overhead via MCP so that we can look at the different endpoints, connect to the function level of what happens there, and send a report to Slack. Of course, that could have been Teams or an email or anything like that. It could have been cursor or co-pilot.
It's just the setup that we started with to make sure that we have something that runs without us in the loop and sends that report to somewhere we actually live in, which is Slack. So we want to take the production context, the traces, the queries, the latencies, then analyze them with the agent, score and flag which opportunities matter, because if you can optimize something, but it runs every three weeks, or you can reduce 20 milliseconds, then it doesn't matter. And then the most important part here is the difference.
So the agent actually fixes it, reruns the test, and sees the impact that it had on that specific flow that was optimized so that the human gets something after we already detected it. We understood the root cause. We understood why it matters to the business. And we verified that the fix actually impacted that time. And then a human reviews that, and the loop is actually closed. So it's not, hey, I have this idea of something you can do. It's, here's something that works and we believe would make an impact on these specific business flows in production that are running 7000 times a week. And then the human gets in the loop as a review gate.
And of course, it didn't work out of the box, if you were wondering. So the first hurdle we had along the way is what we call plausible unverified. So the agent would suggest a fix. It sounds right. It would look fairly real. And then after we verified it, it just didn't work. And it's true that the agent suggested something that could theoretically cause that slowdown. But what we wanted is to ground it on what's actually happening in production. Second part was complex queries. We specifically use ClickHouse. It's just an amazing columnar database, but it's also slightly different than the classic SQL patterns. And I'll talk about that in a bit.
And third one is the lazy fix. I'm sure you guys also experienced that when something throws an exception and the agent says, well, maybe we can just catch that exception and then everything will be fine. But what we really want is to understand why that exception was even thrown in the first place or why the results are lagging. So those were things that we found that methodology could be very, very impactful with. So that's not just the understanding of the data and how to connect it. It's also being quite thorough on what we want to do in that process and building the playbook of how a senior engineer would do that. And of course, we need the context to be right.
So the problem with context, there are only two problems with context. You either have too much of it or you have too little of it. And I'll talk about that in a bit. And the third one is the lazy fix. I'm sure you guys also experienced that when something throws an exception and the agent says, well, maybe we can just catch that exception and then everything will be fine. But what we really want is to understand why that exception was even thrown in the first place or why the results are lagging. So those were things that we found that methodology could be very, very impactful with. So that's not just the understanding of the data and how to connect it.
It's also being quite thorough on what we want to do in that process and building the playbook of how a senior engineer would do that. And of course, we need the context to be right. So the problem with context, there are only two problems with context. You either have too much of it or you have too little of it. And what we found around runtime context, especially from production, is that more often than not, you actually have these two problems together because, on one hand, you have a lot of low-signal data that is hard for the agent to reason over.
And on the other, you might not have all the logs and traces and metrics that you need in order to investigate that issue, which would still leave some room for assumptions and theories that are not necessarily what our users are experiencing in production. And also, when we talk about metrics like service level, CPU and memory, or endpoints in the P90s, they are not connected to the function level. So our coding agents reason over code and they look at these metrics, and there are some relations between them, but they don't exactly speak the same language.
So when we ask questions about what's taking time and what can I do about it, we're often finding that there are some gaps there. And again, accuracy could drop from that. So what we did is what we call prod to code, which is to be able to explain what's going on in production at the same level that agents reason over because the agent context lies on a function and file level, not on a service and endpoint level. So our context is running on a function level, and it is also connected to the endpoint or event consumer or cron job that ended up starting this task.
So you can ask a question like, hey, this endpoint that sometimes takes seven seconds, where is the time spent and what can be done about it, whether that's a function or an outbound call to a database, an LLM, or another microservice? And with that, you basically have the complete function-level context for every single function, like this service map but on a function level, the different invocations, where they come from, how often they run, and the deep forensic context only when it's needed.
Only when we see requests that are taking longer than the P99 or some threshold that we can define, then we will capture that forensic evidence so that we can say, hey, here is an example of a request that took longer. Let's find out why. Let's find out why. And we also have the ability to see that on top of the code, which is the HUD, the heads-up display, but I guess it's a way to explain how that data set is actually structured in a way that is much more comprehensive for a coding agent to read and reason over. And then we talked about the complex queries and what to do with them.
So the basic layer gave us the HUD query language, which is basically ClickHouse queries over that structure of functions and endpoints and forensics. On top of that, we also added a set of skills. We found that sometimes just querying the data is enough, but being able to get to the right query and to ask it again and again really created a lot of variance in our evals. And the skills actually help work with that data so that agents can use it. So, for example, if we're talking about a 500, an HTTP 500, we would want to understand where that error came from.
If we're talking about a memory spike, we want to understand what was running on those specific pods at that specific time where memory was higher and compare it to a baseline so that we see the diff. And all of these things were extremely helpful to be able to be a bit more methodological around how that works and to create more consistent results. And on top of that, there are a set of automations. So if we have the data, the query language, and a set of skills, we can build automations on top of them, like auto-fixing issues as they arise or detecting code that isn't even running for the last 60 days and eliminating it.
So, for this automated performance improvement automation that we are talking about right now, for performance, one example of that is looking for artificial delays like timeouts and sleeps, n plus one queries, missing indexes, sequential asyncs, and so on and so forth. These are specific things that are much easier to find when you actually look for them. And in a code base that's 20 years old and has hundreds of contributors, it makes sense that you'll find quite a lot of those. And removing them is fairly easy and impactful.
And then when you're asking something like, why are my endpoints taking long, you can actually find the specific reasons and not just guess a bunch of static code analysis, which could get you some result. I'm not saying it's never going to work. But when you're talking about an automation, we have to think about how to build something that is robust enough for us to trust over time. And then we said, okay, now that our evals are looking good, we run weekly, we find real slow endpoints from production that are invoked, we score them, we test them, we verify them. Maybe we can just open tool requests and everyone will fix everything and the world will be amazing.
Well, that's not exactly how that worked. Because people are still people, and no one wants to wake up for a rain of 80 pull requests, as small as they can be. That's just not how people operate. And no one has time for that. We're too busy building other things, and it's fine. So what we actually do is we use that priority to make sure that we only flag the ones that matter. And we actually started with one at a time to create that appetite and that habit. So we look at whether this is a hot path in terms of how often it runs and how critical it is for the business. The business impact, as in if this is something that has to do with payments or sign-up flows.
Obviously, we are more sensitive to that. Like asking ourselves, would we be able to convince the product manager to prioritize it? And we also looked at the risk. If it's a risky change that requires a migration or anything like that, then obviously it would require more time from the human who's reviewing it. And in that case, we're not necessarily looking for the highest-impact ones, but for the highest-ROI ones. And because we automate the investigation, we can look at the impact and the risk together and only surface and require attention on the ones that we believe are the right ones that are worth the engineering time.
So instead of opening 80 PRs, we built this human-friendly report that basically tries to convince you that it's worth your while. Something like, hey, this endpoint, it's usually taking around 200 milliseconds, but every once in a while, it takes 45 seconds. And it happens because you're using distinct and not the search index of Mongo. There's a very short explanation of what's happening and what's the fix. And then you can either create a ticket and fix it on your own and create a PR or just look at it.
And we found that building these small gists that are humanly readable and easy to understand made a huge, huge difference because we're still living in this hybrid world where humans are in the loop. And we want to respect our place in people's lives and to make sure that we flag the things that really matter and that we have some confirmation, not only that the fix is good, but also that this issue is worth fixing. And then when you look at that endpoint and you deploy that change and all of a sudden it's flat again, it is pretty satisfying. And then next time you'll get that report, maybe you'll have a look, and it will be easier to convince you.
So four things that I learned that could be relevant for you. One is we need to define what matters, and the scoring and the guardrails are what makes this reliable. We can automate a bunch of things, and it becomes easier to just create some sloth. But when we start with what are the things that are worth it, even though it's much cheaper to fix these things than it was a few years ago, it's still not free. And therefore, we need to understand that it's worth the impact, the human reviewer, and the risk that it entails if it does. The second part is that a lot of what we're talking about today is accelerating developers and what they do.
And I think what's interesting about this experience is that we automated something that is not just doing it faster. We're automating a phase that just did not happen in the day-to-day life of engineers. No one actually stopped every week and had a look at whether there are low-hanging fruits that could be relevant. One is we need to define what matters, and the scoring and the guardrails are what makes this reliable. We can automate a bunch of things, and it becomes easier to just create some sloth. But when we start with what are the things that are worth it, even though it's much cheaper to fix these things than it was a few years ago, it's still not free.
And therefore, we need to understand that it's worth the impact, the human reviewer, and the risk that it entails if it does. Second part is that a lot of what we're talking about today is accelerating developers and what they do. And I think what's interesting about this experience is that we automated something that it's not just doing it faster. We're automating a phase that just did not happen in the day-to-day life of engineers. No one actually stopped every week and had a look on whether there are low-hanging fruits that could be relevant.
But now we can use the agents to not only do the things that we're doing faster, but also to help us surface opportunities that we would probably never do without them. And this sounds simple, but it's hard to actually apply it in the day-to-day. But context over cleverness works almost every time. If they have the right context, if they have the right skills, if they know exactly what they need, these agents get much, much more useful. And every time a new model comes out, things get better. And yet I do believe that, at least for most of the cases, the models are good enough already to be able to automate that.
But it is up to us to help steer and guide to the right directions to make sure that we get the results that we want. And they're not necessarily the absolute right thing because it doesn't necessarily exist. And when we know our domain and our business, we understand which issues matter more, what's worth fixing, what latencies actually impact our customers' experience in the most significant way. And that helps us make sure we focus on the right things. And in that aspect, agentic engineering is not like coding with an agent.
If something works 80% of the time and you're using it with your cursor and your IDE, that's fine because you're there, you're in context, and you can help fix and steer. If we're talking about an automation that runs autonomously, we have to have very high confidence that we're doing the right thing. And that it's not going to just stray off and hand us a bunch of things that we would either waste time on reviewing or just not be able to trust over time. And I think the hardest part of this automation was to get to a point where we feel confident enough that the issue is worth fixing and that the fix is verified in runtime.
So that it is handed to a person when we have fairly good confidence that it actually works. And yet they still review it, but we know it's worth their time. And getting to that agentic engineering automation level requires crossing towards the 80, 90% trust. And it's something that is dramatically different than using an agent directly as an engineer. So I hope that was helpful. And if there are any questions or anything like that, I'm available and always love to geek out on the AI in the SDLC. And looking forward to hear about cool automations that you build on your own. Thank you. Thank you.