How I deleted 95% of my agent skills and got better results — Nick Nisi, WorkOS

Transcript

Nick Nisi Good morning everyone. Welcome to my talk, Building AI Systems That Ship. I'm Nick Nisi and I work at WorkOS. We've got a booth downstairs. Come check us out and talk to us. I would be happy to chat. But let me start that over. Hi, I'm the bottleneck. I'm a DX engineer at WorkOS and I work on 20 plus repos across eight different languages. It's all of our SDKs and open source things that we have. And it's AuthKit Next.js, AuthKit React, WorkOS Node, WorkOS Kotlin, WorkOS Ruby, PHP, everywhere. So there's a lot to do across a lot of different things. And I'm really good at working on those. And I've gotten really good over the last eight months working with those via agents. So I haven't written a line of code myself in probably eight months. I've gotten really good at just scaling that with agents and then reviewing what they do and instructing them and getting the work done faster and better while still maintaining good quality. But there was a big problem. Doing that with one agent at a time across all of these repos, I'm just constantly context switching over and over. And it just gets harder and harder. And that's okay. But the problem is that for every one of those, there's this little bit of setup time that I'm doing each time, which is giving it ten minutes of my time to set up and establish the problem. Let's look at this GitHub issue. Let's look at this linear ticket. Let's take a look at this Slack thread and figure out what's going on and see if we can reproduce the issue and then go. So that was a lot of my time just spent dealing with the agent, getting it basically the context that I already have and then getting it to work on it from there. Now on the other side, I'm also working on products that we want to build for agents because while I said I'm a developer experience engineer, the developer is still the most important in my job. But increasingly, the pipeline to get to that developer is through agents. And so I see the agentic experience as being equally as important because that's how we're going to get in front of the developers. So there's two different ways I needed to go AI native and two different directions for that. So on the internal side, building that, I started building this project called Case. This is a harness. If you've read Ryan LaPapolo's Harness Engineering, it's that. I just took those ideas and started building them. Basically, I give it a GitHub issue, a PR, a Slack thread, a linear ticket, anything, and I could just point it at it and it could figure out the context that it needs and go. And then it wouldn't stop until it has a PR with evidence that it actually did what I asked it to or what the problem was or fixed what the issue was. But most importantly, it had to provide that evidence. And this originally started as a Claude skill because why not? I thought Claude could do anything. And it was working really well. But as it got more complex, the context drop became very real. It would just start forgetting things or skipping over tasks. And I would ask Claude, why did you do that? I was, oh, yeah, you told me to do that. I decided not to. Not great. So I rebuilt it on top of Pi and using a TypeScript state machine to facilitate going through and stepping through these agents. So it has five different agents in it. An implementer, a verifier, a reviewer, a closer, and a retro agent. And those are important, but they're not the most important thing. The most important piece of case is the gates in between that. And that's what the state machine really enforces is the checks in between everything. So when we implement something, we can't move on to the reviewer until the verifier verifies it. And once the reviewer reviews it, if there's any issues, it has to send it back to the implementer to do those. And once all of that's done, the closer can work. But the closer can't work until it thinks that it's done, and the closer is there to provide evidence. And then the retrospective is there to analyze the entire performance. It looks at the logs of everything that Case did and says, what could I have done better? And then it updates its own memory system to ensure that the next time it can skip some steps if it went in circles for a little bit, and it can give itself some hints on where to go so that the next time it works in that project, it doesn't hit the same roadblocks. So the next agent doesn't really matter. Proving that the work matters. Proving that what happened in each of these states is what matters. And that word there, proving, is the most important piece of that. Because the agents, they would just lie to me all the time. I would ask it, hey, you need to run the test. And this was more when it was a skill, and I would be, hey, you need to run these tests and make sure that the tests actually pass. And one way to do that was I just had it check for a .case tested file. And if that file existed, great. It ran the tests. Perfect. Well, it figured it out pretty fast. Claude would just touch that file and be, yep, I ran the tests. Such a junior engineer, I swear. So I had to figure out a way to prove that. So one way to do that was just to actually take the test output and SHA-256 that and save that into the case tested file and then verify cryptographically, yes, you actually ran the tests. And really, the main piece there is that I just made it easier to just do the work that I wanted it to do rather than lie about it. And that's really the main thing. It stopped lying not because I asked it very nicely. I made it prove that it was going to actually do the work each time. Now, that was on the inward side. On the outward side with the WorkOS CLI, this is a tool that our customers use. And it can do lots of things, but its headlining feature is that it can install AuthKit for you. One of the biggest pain points when we're trying to ask someone to look at our product or they're interested in it is, oh, I'd have to go spend some time and get it set up and read the docs and all of that. Not anymore. With WorkOS install, it just goes and figures out what project you're in. Oh, you're in an XJS project, you're in a TanStack project, you're in a Ruby project. I'll figure that out. Oh, you've already got Auth0 set up? I can easily remove that and put it in AuthKit and we'll be good. And it does it in less than five minutes. If you don't have a WorkOS account, it will provision one for you that you can go claim later. So there is zero friction to getting it set up. And that's a really important piece of being agentic forward in our public facing persona and how our customers use us and how they perceive us. But there's problems with that, too. As I was building it, it would be overly confident, just these models always are, and say, yep, I did that. One of the cases of that was I was trying to install into a TanStack start project. TanStack start is relatively new, still in RC, and it's changing constantly. Well, case, sorry, the CLI made some changes, it installed it, and it made some changes to a file called start.ts. That file is implicit, it has an implicit contract with TanStack. It has to export certain things. And how our customers use us and how they perceive us. But there's problems with that, too. As I was building it, it would be overly confident, just like these models always are, and say, yep, I did that. One of the cases of that was I was trying to install into a TanStack start project. TanStack start is relatively new, still in RC, and it's changing constantly. Well, case, sorry, the CLI made some changes, it installed it, and it made some changes to a file called start.ts. That file is kind of implicit, it has an implicit contract with TanStack. It has to export certain things. And we messed that up. The code looked right to me, it looked right to Claude, but it did not look right to TanStack start. So, boom, it failed. And so we had to figure out a way to tell it when it failed or make it understand that. And I thought, oh, well, we just need some skills, right? Skills are the way to do that. So I started teaching it, making these skills. And, of course, I thought, you know what, we have these great docs. I can just take our docs and generate some skills. So I generated over 10,000 lines of skills that were all based on our docs. And I did it in this really elaborate way where it would take sections of our docs and make skills about them. And then it would put a little comment in the skill with the cryptographic cache of the current state of that section of the docs. And it basically, if I ran it again and that SHA didn't change, don't update the skill. So it wasn't just constantly updating all the time. I thought I was being really clever and awesome. And I generated this huge thing. And I even made some evals for it. I started making those. And it would take me 68 minutes to run those scenarios. It was just crazy. And it would fail over and over. And it would have these retries and get there eventually. But it was a lot of work, a lot of tokens. So I had more tokens. I thought, more tokens? Great. That's way better. But it ended up producing worse results. And it was really the measurement there, the evals that were telling me, hey, this isn't right. So I rewrote it by hand. And instead of focusing on covering comprehensively everything that we have in our docs, I was like, oh, I just have to cover some common gotchas for everything. So for our entire docs, instead of having 10,000 lines of that, I have 553 lines of gotchas. And these are just the most common things that came up as I was running these evals over and over and over. They ran faster, way smaller in terms of token count, only took six minutes per run. And I wasn't sending the models on these long goose chases by having it go check a whole bunch of different things. It would stay focused on things. And so by deleting 95% of that, the performance of it actually went up. And I really only knew that because I measured it. So looking at that, I had one skill in particular that I could see. And when I ran it with that skill, and I gave it a task and said, hey, load this skill and then do this task, it got it correct 77% of the time. But if I asked it to do the same task without loading the skill, it was correct 97% of the time. So I was actively making it worse. And I only knew about that because I was measuring it. And so evals are super important when you're working with this non-deterministic code. Claude makes it really easy now. They have evals, a Claude skill skill that will do evals for you. And it'll even set up, it'll create an HTML output of that and show you side by side. I ran a bunch like this and a bunch without the skill. And here's the results. Use that, measure, and see where you're actually falling apart. Because I thought I was making things a lot better by having a whole bunch of code. I just needed to trust that the model already knew how to code. And I just had to gently nudge it in the right direction in some cases. So what did I actually learn from both of these systems? Basically, you want to enforce things. Don't instruct. The model can lie about it. It can decide not to pull things, not to do certain things because either it forgot about it, it got distracted with other things. But if you actually set up a pipeline where it has to enforce itself and prove to you that it did what you asked it to do, then you're going to have a better time, for sure. And oftentimes with a lot less tokens. You want to guide the model. Don't prescribe it. So don't just give it, hey, here's a summary of all of my docs with a whole bunch of information. You want to prescribe it. Hey, when you're working in Next.js and you're in the proxy, you want to do this. If you're not in the proxy, you can't call redirects. That's a really big one that constantly comes up over and over and over. It would just put those everywhere. And so guide it, but don't prescribe to it. And then, of course, measure. Don't assume that it works. Just trust that it has a... Trust is a pass rate, a hash, a delta score, anything like that, so that you can prove to it. One of the things that case does at the end, as part of its reviewer script, I still read all of the code that it generates to make sure that it's actually code that I would be proud of shipping. But I'm not even going to waste my time looking at that code until it's proved to me that it did whatever I asked in a non-code way. And so the main way for that is, if it's working on a UI bug, I want it to use the Playwright CLI and record a video of itself doing something before and then doing it after the fix and showing me, hey, now it's fixed. It's working. And if it can prove that to me in those videos that it attaches to the PR, I'm way more inclined to look at that PR and say, yeah, okay, we can just fix some of the weird things that it did, but it did do the work correctly. And I'm way more incentivized to waste my time and become that bottleneck again for that. If not, I just ask it to do it again. So every failure became data for the next run. This is another important thing, is when things failed, and this goes back to that harness engineering thing, if you are working on a harness and it is making mistakes, don't go fix the mistakes that it made. Fix the harness so that it can fix the mistakes. In Ryan LaPapolo, I didn't see his talk here, but I saw a talk on Zoom and he talked about how their team would never work on the code itself. They would only work on the harness to fix the code itself. And I really took that to heart with Case, so I only work on Case itself to make sure that it's doing what I want. And if it fails, then we do it again, and that becomes part of its memory. And that's the other big piece of it, is that as Case is running, the final piece of it is this retrospective agent. Fix the harness so that it can fix the mistakes. In Ryan LaPapolo, I didn't see his talk here, but I saw a talk on Zoom and he talked about how their team would never work on the code itself. They would only work on the harness to fix the code itself. And I really took that to heart with Case, so I only work on Case itself to make sure that it's doing what I want. And if it fails, then we do it again, and that becomes part of its memory. And that's the other big piece of it, is that as Case is running, the final piece of it is this retrospective agent. And all it does is it looks at what it did, and it goes in and looks at the Claude and Codex transcripts, like the JSONL files, and it pulls out information. Hey, was I running a lot of tools at the same time? Did I run the same tool request three times in a row without any changes to anything? Was I getting in a doom loop there, trying to identify those things? And see what it can do better. And then internally, Case keeps a whole bunch of memory files as markdown files, and it understands, okay, I have a general memory file. If I'm working in Next.js, I have a Next.js memory file, a tan stack start memory file, et cetera. And it figures out where to put information about that, so that it won't make a mistake and break the start.ts in tan stack start again. It knows about that because it put it into its memory. And one thing that I want to add is that auto dream thing that Claude is now doing where it can prune its memory over time. That'll be the next piece that I add to it. But making sure that it can learn from its mistakes, and it can do it automatically, and then you can also provide feedback. Have a way for you to provide the feedback to it as well. And then the next time you give it a task, it's just going to be that much better. And eventually, you're just going to start trusting it more and more. And if you're making your product work for agents, there's a couple of important things as well. Figure out what the agents get reliably wrong about your product and focus on that. Don't focus on the product as a whole because it probably knows a lot about it, a lot more than you think about it. Write down those gotchas. Create skills around those. You can create tutorials too, but don't rely on that. The models can read the tutorials and learn from that. But remember that the models know how to code. They just need to know the intricacies of your product and where the landmines are in that. And of course, measure what you're shipping. You want to understand where the model is failing for your particular product and make sure that you focus on that. And the only way that you can do that is through things like evals. Otherwise, you just might be adding noise and sending the model on wild goose chases. And think about the consumers in the way that you think about developers. Think about those agents in the same way that you think about developers. What do they want to know? How can I make things better for them? Do I have a lot of JavaScript loading on my page after the fact that's adding a whole bunch of context that maybe is not getting added when whatever process they use to go pull and summarize the information on your page? Is that getting lost to them? Make sure that it's not. And if you're making agents work for you, like in the case of Case, you replace your trust with evidence. Never trust it. Always make it prove to you that it did something. If it ran the test, make it prove it. If it fixed a UI bug, it has to show it to you. Otherwise, don't waste your time on it. And enforce that with code, not prompts. So this is why I switched it to Python and used a state machine to force it, because I have full control over that state machine. And it's outside of the Python or Claude deciding, should I do this or not? No, you have to do it. I enforce that through that loop. And then every failure becomes a system bug. Each time it messes up on something, that's a bug in the harness. Go fix the harness. So the agent, you want to build the environment that you can work with the agent in and focus on that. The practices that we have haven't really changed. Our job hasn't really changed. We've just abstracted it a little bit. Your job was never really about writing code. It was always about building these systems. And now we just have a better abstraction to understand that. So take that into account and go forward from there. So that's the talk. Thank you. And I'd be happy to answer any questions with the time I have left. Thank you. Thank you. Thank you. Thank you. Thank you. We'll see you next time. And the only way that you can do that is through things like evals. Otherwise, you just might be adding noise and sending the model on wild goose chases. And think about the consumers in the way that you think about developers. Like think about those agents in the same way that you think about developers. What do they want to know? How can I make things better for them? Do I have a lot of JavaScript loading on my page after the fact that's adding a whole bunch of context that maybe is not getting added when whatever process they use to go pull and summarize the information on your page? Is that getting lost to them? Make sure that it's not. And if you're making agents work for you, like in the case of case, you replace your trust with evidence. Never trust it. Always make it prove to you that it did something. If it ran the test, make it prove it. If it fixed a UI bug, it has to show it to you. Otherwise, don't waste your time on it. And enforce that with code, not prompts. So this is why I switched it to pi and used a state machine to force it, because I have full control over that state machine. And it's outside of the pi or Claude deciding, should I do this or not? No, you have to do it. I enforce that through that loop. And then every failure becomes a system bug. Each time it messes up on something, that's a bug in the harness. Go fix the harness. So really, the agent just, you want to build the environment that you can work with the agent in and focus on that. The practices that we have haven't really changed. Our job hasn't really changed. We've just kind of abstracted it a little bit. Your job was never really about writing code. It was always about building these systems. And now we just have a better abstraction to understand that. So take that into account and go forward from there. So that's the talk. Thank you. And I'd be happy to answer any questions with the time I have left. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. We'll see you next time.