Fighting AI with AI — Lawrence Jones, Incident

Transcript

[SPEAKER_00] Hi, everyone. I'm here to talk today about how we use AI to manage the complexity of the AI products that we build at Incident.io and to share with you some of the tips and tricks and the internal tools that we use when we're building our AI SRE product. But first, who am I? So I'm Lawrence. I'm a founding engineer at a company called Incident.io. So we build, if you haven't heard of us, an incident response management platform. So we're used by companies like Netflix, Etsy, Skyscanner, and actually probably a few of you in the room. We page you when things go wrong, and we help you run your incident. And as you're running your incident, we help you communicate with your customers. And you might be thinking, where does AI actually come into this? And actually, we don't just want to help people respond to these incidents. Our goal is actually to fully automate production investigations, so whether it's a big incident or if you just have some ticket and you wanted to look into production, we want to be the place that you turn to ask us questions about what's going on. Now, it turns out we've been building this for about a year and a half, two years now, and that's actually a really big ask. And the systems that we've had to build to try and support this have been really quite complicated and have been on the edge of what you can do with all of the AI technology that's out there. And they often pose a challenge for humans to debug them. They are now complicated enough that you can't, as a human, really tractably dig into how these things are performing. You need assistance to help you. So, for example, this is one of the investigations that we would actually produce for you when you have an incident. We will end up running right at the start of the incident this investigation which will go through hundreds of telemetry queries. It's going to look at your logs, your metrics, your traces, any historical incident data that we have. And it's going to try and cross-reference this with your code base and go, hey, I'm pretty sure that the problem is this, and you should probably do this to fix it. But I want to pause here and go, how would you, if you were building this system, how would you actually figure out how to tell me if this was a good or a bad report? How do you know if it's right? How do you know if it's wrong? And there are a lot of things that you might do. You might jump into the incident, you look at everything that happened, you might look at the post-mortem if there was one that was written. But all of this might actually take you a really long time to do. In fact, it normally takes you an hour or so to get a real full understanding of this incident. And it's only at that point that you could then look at this investigation and go, I think it's right, or it gave me the information that was really useful. And as I said, behind this investigation are hundreds if not thousands of prompts. So how on earth do we scalably understand how this system is performing, especially across all of our customer accounts when they all have very different things going on? You end up with a lot of stuff, a lot of AI, and you've got to use AI to try and actually tractably get a handle on this. So I actually did a talk a year ago at LDX about becoming AI engineers, where I went through some of the core constructs that hopefully a lot of you in the room, given that we're an AI engineering conference, are familiar with. So things like prompts, evals, scorecards, traces, data sets, back tests. This talk is going to be about if you assume that you have these constructs put together, and you're building these complicated AI systems, how can you use AI with the internal tools that you use to understand them to get a better handle of how your system is performing? So yeah, in this talk, I'm going to talk about how you can use AI to help you manage and curate your eval data sets and making it easier for you to work with them, making it easier for coding agents to actually work with your eval tool. I'm going to talk about probably what was the biggest unlock for us when we were building these systems, which was starting to translate the UIs that we built to try and debug them into downloadable file systems, which has actually helped us massively using tools like Code Code and Codex to dig into how the system is performing. And then I'm going to talk about how you can build repeatable analysis pipelines using AI agents to run through them. But first, evals. So evals for me are AI unit tests. So each eval takes a prompt and it goes here is some input data. It runs the prompt, it gets the output, and then it has some grading criteria that says does this eval pass, does it fail? And for us, evals live in YAML files right next door to our Go prompts. So we do everything in Go at Incident.io, including all of the AI work that we do. And this is how we prove when we make a change to a prompt before we ever go and merge it that the prompt is actually going to do the thing that we want it to do. So for us, this is what a prompt looks like. This is a contrived prompt. I would hope no one actually has this in production anywhere. It takes a message and tries translating it into pirate speak. So really simple, bit silly. But what we do for evals is if this is the prompt, we would then define on the left some grading criteria for this prompt where we will go there are two things that we care about. We care that the result actually looks like pirate speak. And we are going to care that the meaning is preserved between the input and what we actually produce as an output. So this is actually what we are going to use to tell us if the eval passed or failed. And then we have the evals up on the top right, which is just in a YAML file. Where we go here are three different test cases. And we will run through them and you can see the results of us actually running this on the bottom right. So this works and it works really quite well. But it does have some problems. And I am assuming that several people in the room have come across these themselves. So first, evals are really, really fiddly. Setting up realistic test data for your evals if you want to actually understand how this stuff is running is quite difficult to do. And especially in our environment, our production evals are including almost an entire incident. So you can imagine a full incident report and that is the only thing that can trigger the bad behaviour. Where we go here are three different test cases. And we will run through them and you can see the results of us actually running this on the bottom right. So this works and it works really quite well. But it does have some problems. And I am assuming that several people in the room have come across these themselves. So first, evals are really, really fiddly. Setting up realistic test data for your evals if you want to actually understand how this stuff is running is quite difficult to do. And especially in our environment, our production evals are including almost an entire incident. So you can imagine a full incident report and that is the only thing that can trigger the bad behaviour. This is hard for you to end up pulling them down and putting them in your eval test suite. They just become extremely unmaintainable very quickly. Now quite early on we created this little button that we have that allows you to steal an eval from production. So if anything was going wrong inside of our AI interactions, you could go in and you could just pull that down, put it in the code base, and you could run the eval against it. But the thing with this is production evals aren't great. So if you think about evals as a unit test for your prompts, you want a unit test suite to be reasonably understandable. An ideal unit test is very focused and just says I expect it to do this thing. You don't want to have two megabytes of all the YAML associated with it. It's just really, really hard to work with. And what we found was as these YAML files for the evals were growing really, really large, our coding agents weren't able to work with them. So if you want to do a quick read and modify the eval suite, you'd be boosting that into the context and you'd quickly hit your context limit, which is obviously a problem because then you can't work with it effectively. So what we ended up doing was we ended up creating a small CLI tool that we call eval tool that was designed to allow agents to leverage our eval suite files. So it's just a small CLI that can go what test cases do you have in here? I want to edit one. I want to replace one. I want to add one. And it was by doing this that we allowed agents to work effectively with our eval tooling. And that's why we were able to create this runbook to the right, which is actually a runbook that's designed for a coding agent to use. So either a runbook or a skill. It depends on how you want to package it. But the cool thing about this is that now that agents can work with the evals, you can end up in a situation where you just ask your coding agent to go, hey, I've got a problem here. Can you look at this prompt? I want it to do these things. And the coding agent is going to turn up and it will create an eval case where it proves that the thing has failed and then it will go modify the prompt so that the eval now passes. And then it will go through this runbook and one of the most important stages for us is checking at the end that the change that you've made to the prompt hasn't ended up breaking any of the other evals that you had in your test suite. And we also have a final pass that tries consolidating the prompt as well because if you end up doing this repeatedly, you end up with a prompt that is massive and really, really difficult to maintain. So you want every time you make an adjustment to try and simplify as well. So this has actually worked really well for us. And you can see here, this is me using it in core code where you can just point it at the eval and say, hey, have a look at the prompt. This is a real prompt for us which turns human queries into log queries for a log key system. And it ends up racing through and goes and adds a new eval. It checks out the passes with a certain number of repeats. And then it gets to the end and it's like, yep, I think I've added it. The pass rate is acceptable. You can go ahead and get going. But the problem with this is this solves one problem. And the problem that it solves is that if you know what the prompt is that you want to change, you can now change it very reliably. And that's very useful if you're working on these tools. But one of the biggest problems that you have now is that if you're building these systems, you'll know that they're not just one prompt anymore. In fact, most of the production AI systems you will use on a daily basis are many, many, many prompts. And to illustrate this problem, I've taken our chat bot. So this is the chat bot that you interact with inside of an incident. And what I've done is I've created a graph of all of the different prompts, tools, agents, and everything in the hierarchy that powers an interaction with our system. So you can actually see there's ten different agents there. There's 50. I don't even know. It's actually bigger than this. I couldn't put it on the screen. It's a lot of stuff. So even if you think that you know, even if you've got a bad interaction that came in from a customer, you don't necessarily know which part of your system is actually the problem and which part to go change. So even if you have this eval red-green cycle, you're going to struggle to know where to go to fix it. And this gets even worse for a system like our investigations. So if you think about trying to run through this process to debug what's going on in an incident, we have a ton of stuff that goes on inside that system. And you can see all the steps on the left. And each one of those steps unpacks into the trace that you have on the right. And really it's not about the details here. It's more about each one of these green blocks actually expands into possibly hundreds of different prompts and hundreds of different tool calls. And at any point if you make a slight subtle error, you can't then easily trace through the system where the error originated, even if it ends up resulting in you having totally the wrong picture on what you think that the incident was. And your RCA is totally wrong. So we built these UIs so that we could help humans look at them. And they've been really good for humans to look at them. But I guess going back to what I was saying before, we just feasibly don't have enough time to go through this stuff. So the problems that we had was we have all these UI tools, but agents can't properly use them. So how do we get to a place where the agents can use the tools properly? And I think Anthropic stumbled on this with Claude Code where they found that when they released Claude Code, these agents are fantastic at using file systems and just going through this data using standard tools. And so we thought, can we just download all of the UI that we have as a file system? And that's what we've done. And they've been really good for humans to look at them. But going back to what I was saying before, we just feasibly don't have enough time to go through this stuff. So the problems that we had was we have all these UI tools, but agents can't properly use them. So how do we get to a place where agents can use the tools properly? And I think Anthropic stumbled on this with Claude Code where they found that when they released Claude Code, these agents are fantastic at using file systems and just going through this data using standard tools. And so we thought, can we just download all of the UI that we have as a file system? And that's what we've done. So now for each of our different AI systems, you're able to download all of the content as a file system. And we drop that into a sandbox Claude Code, at which point you can just point Claude Code at it and go, hey, I've got a problem here. It's behaved in the wrong way. It can see everything that went into all the prompts. It understands the structure because it's self-documenting. And then it can tell you, because you have access to the code base as well, exactly where you should actually be making the modification to try and change it. And then you can lean on that red-green cycle from before to try and modify a prompt if you need to. Also, there's more stuff that you can put in this than you might think. There is really not much of a limit as to what you can put into ASCII. So traces like this can get translated exactly from how you would present them in the UI to a text file which then the LLM can consume in a really nice way. So this is something that has turned the way that we debug our application into we hear that there's a bad experience. We end up downloading that interaction into a sandbox Claude Code. You sit there in the session and you go, hey, have a look at this. Tell me what you think has gone wrong. What is your interpretation of the problem? And then you go, I really wanted to do this instead. What part of the system would you change? And then it will work its way through the hierarchy of all those tools and prompts that you just saw. And it will be able to tell you where you should be making the modification. And then all from that session, because you have access to the code base, you can just go, hey, can I make that change? And then you can prove it using the eval runbook that I mentioned before. So we've implemented these file system packages for a load of different AI interactions for us now. So it's really easy for us to just drop this in Claude Code and just get going. But we now have another problem, right? Because whilst you can do this on an individual basis, we are running thousands of investigations across hundreds of our customer accounts. And we're doing that daily because we need to know if this system is getting better or worse. So what you can see here is we have what we call a back test, which is essentially a batch of investigations that we run on a daily basis against our account and against a load of our customer accounts as well. And eventually you just get this rolled up number, which is 86% accurate RCA on our account, which is great. But this doesn't really tell you why the number went up and it doesn't tell you why it went down. And if you want to improve the system for someone, you're going to struggle. So what we've actually done is we've allowed ourselves to download all of these investigations into a file system that we can then provide into an analysis pipeline that, again, is run using Claude Code that can end up running a structured analysis like with markdown playbooks that help you run it repeatedly and reliably each time. So what that actually looks like is we created this repo called Scrapbook. And inside of Scrapbook, we have this very structured flow that explains exactly how a coding agent should go through all of the information that we've downloaded, how it should understand these investigations, and the process that it should go through to actually run them. Now, the key things that I think are very important to these flows are you start and you actually parallelise out all of your agents. So you start maybe 25 agents in parallel and they can all individually build their analysis of an investigation. And then you go into the next stage of the pipeline where you do some cohort clustering and you look at meta points around what are the same types of failure? How do we go wrong in different ways? And by clustering it together, you end up with a really useful report that doesn't just tell you how this has gone wrong, but it tells you why is your AI system performing well or badly on this customer account? And actually, what should you do to try and fix it or improve the system? And this is something that we've done several times over for several of our systems now. And I think it generalises really well for anyone who's building this type of thing. So the points that make a really good pipeline for this: you should leverage sub-agents to do that parallel per-entity analysis. You should store all of your analysis in files inside these downloads so that you have incremental analysis built as you run through it so that you can start and resume the analysis if you ever need to. And then you want to combine this analysis with the code base that is actually powering the system so that if it finds a problem, it can look in the code base and go, hey, I think that this is the problem and this is the place. And it can actually do some analysis to go, I think that you should change it like this. And then at the end, because you have this all loaded in your code session, you can just go fix it and ask the coding agent, Claude Code, to actually go make the change. And then you can use that eval red-green process to actually confirm it works. And yeah, this is a PR that is created after you do something like that where, because the backtest showed a couple of investigations that were going wrong and I knew exactly what the problem was, I could have a chat about a feature that we might change in the system. And then we can deploy that and we can test that out in production and see how the thing goes. So the key thing for me is these patterns do generalise. So for any of you in the room who are building complicated AI systems and you're finding it really hard to understand them or debug them or evolve them, you really need to be using AI just as effectively in your internal tools to try and understand these systems and grow them just as you are in the products that you're building yourself. So make sure that you prioritise any of the debugging tools that you have so that they work really well with the coding agents that you're leveraging on a day-to-day basis. File systems are exceptionally good agent context. We could have put an MCP on top of this or use human use agents. It wouldn't have been half as effective as this ability to just download in bulk all of the information that you need so that the coding agent can go through it and find the details. So, yeah, make sure that you prioritise any of the debugging tools that you have so that they work really, really well with the coding agents that you're leveraging on a day-to-day basis. File systems are exceptionally good agent context. We could have put an MCP on top of this or use human use agents. It wouldn't have been half as effective as this ability to just download in bulk all of the information that you need so that the coding agent can grab through it and find the details. And then, yeah, any time you are performing complex analysis, look at creating an AI runbook for it instead. It will save you literally days or maybe weeks of your life. And then one final point from me. We are hiring. We're in London. We have just done a fairly big raise last year and we're looking to expand the team so that we can build some of these systems. So, if any of this work looks interesting to you and you're interested in being on the edge of building some of this AI, AI SRE product, then just get in contact and let me know. I'd love to chat. All right. Thank you. It's behaved in the wrong way. It can see everything that went into all the prompts. It understands the structure because it's self-documenting. And then it can tell you, because you have access to the code base as well, exactly where you should actually be making the modification to try and change it. And then you can lean on that red-green cycle from before to try and modify a prompt if you need to. Also, there's more stuff that you can put in this than you might think. There is really not much of a limit as to what you can put into ASCII. So, like, traces like this can get translated exactly from how you would present them in the UI to a text file which then the LLM can consume in a really nice way. So, yeah, this is something that has turned the way that we debug our application into we hear that there's a bad experience. We end up downloading that interaction into a sandbox called code. You sit there in the session and you go, hey, like, have a look at this. Tell me what you think has gone wrong. Like, what is your interpretation of the problem? And then you go, I really wanted to do this instead. Like, what part of the system would you change? And then it will work its way through the hierarchy of all those tools and prompts that you just saw. And it will be able to tell you where you should be making the modification. And then all from that session, because you have access to the code base, you can just go, hey, can I make that change? And then you can prove it using the eval runbook that I mentioned before. So, yeah, like, we've implemented these file system packages for a load of different AI interactions for us now. So, it's really easy for us to just drop this in Claude Cove and just get going. But we now have another problem, right? Because whilst you can do this on an individual basis, we are running thousands of investigations across hundreds of our customer accounts. And we're doing that daily because we need to know if this system is getting better or worse. So, what you can see here is we have what we call a back test, which is essentially a batch of investigations that we run on a daily basis against our account and against a load of our customer accounts as well. And eventually you just get this rolled up number, which is like, oh, cool, 86% accurate RCA on our account, which is great. But, like, this doesn't really tell you why the number went up and it doesn't tell you why it went down. And if you want to improve the system for someone, you're going to struggle. So, what we've actually done is we've allowed ourselves to download kind of all of these investigations into a file system that we can then provide into an analysis pipeline that, again, is run using Claude Code that can end up running a structured analysis like with markdown playbooks that help you run it repeatedly and reliably each time. So, what that actually looks like is we created this repo called Scrapbook. And inside of Scrapbook, we have this, like, very structured flow that explains exactly how a coding agent should go through all of the information that we've gone and downloaded, how it should understand these investigations, and the process that it should go through to actually run them. Now, the key things that, like, I think are very important to these flows are you start and you actually parallelise out all of your agents. So, you start maybe 25 agents in parallel and they can all individually build their analysis of an investigation. And then you go into the next stage of the pipeline where you do some cohort clustering and you look at, like, meta points around, like, what are the same types of failure? How do we go wrong in different ways? And by clustering it together, you end up with actually, like, a really, really useful report that doesn't just tell you how this has gone wrong, but it tells you why is your AI system performing well or badly on this customer account? And actually, like, what should you do to try and fix it or improve the system? And, like, this is, this is, like, something that we've done several times over for several of our systems now. And I think it generalises really well for anyone who's building this type of thing. So, the points that make, like, a really good pipeline for this, you should leverage sub-agents to do that parallel per-entity analysis. You should store all of your analysis in files inside these downloads so that you have, like, incremental analysis built as you run through it so that you can start and resume the analysis if you ever need to. And then you want to combine this analysis with the code base that is actually powering the system so that if it finds a problem, it can look in the code base and go, hey, I think that this is the problem and this is the place. And it can actually do some analysis to go, I think that you should change it like this. And then at the end, because you have this all loaded in your code session, you can just go fix it and ask the coding agent, Claude code, code, codex, whatever you use, to actually go make the change. And then you can use that eval red-green process to actually confirm it works. And then, like, yeah, this is a PR that is created after you do something like that where, because the backtest showed a couple of investigations that were going wrong and I knew exactly what the problem was, I could have a chat to her about a feature that we might change in the system. And then we can deploy that and we can test that out in production and see how the thing goes. So, yeah, that's it. So, the key thing for me is, like, these patterns do generalise. So, for any of you in the room who are building kind of complicated AI systems and you're finding it really hard to understand them or debug them or evolve them, you really need to be using AI just as effectively in your internal tools to try and understand these systems and grow them just as you are in the products that you're building yourself. So, yeah, make sure that you prioritise any of the debugging tools that you have so that they work really, really well with the coding agents that you're leveraging on a day-to-day basis. File systems are exceptionally good agent context. We could have put an MCP on top of this or use, like, human use agents. It wouldn't have been half as effective as this ability to just download in bulk all of the information that you need so that the coding agent can grab through it and find the details. And then, yeah, any time you are performing complex analysis, look at creating an AI runbook for it instead. It will save you literally days or maybe weeks of your life. And then one final point from me. Like, we are hiring. We're in London. We have just done a fairly big raise last year and we're looking to expand the team so that we can build some of these systems. So, if any of this work looks interesting to you and you're interested in being on, like, the edge of building some of this AI, like, AI SRE product, then just get in contact and let me know. I'd love to chat. All right. Thank you. . . . . . . .