Building Great Agent Skills: The Missing Manual

Transcript

Hello friends, I was dearly hoping to be able to come to the AI Engineer World's Fair, but family matters have intruded and I'm not able to make it. However, I will not be leaving you empty-handed. I'm going to give you the talk that I would have given in San Francisco. This talk is called The Missing Manual, How to Write Great Skills, and I think that the ability to distinguish good skills from bad skills is only getting more important. As developers, we seem to be pretty talented at finding different forms of hell for us to go to. A few years ago, we had tutorial hell, which is where you would go into a bunch of tutorials trying to learn something, not be able to piece it together and just get into this cycle you couldn't get out of. We had framework hell, where every other 10 minutes there was a JavaScript framework being announced, and you had to learn the hot new thing all the time. And now I think we have another version of hell, which is skill hell. Skill hell is where you have all of these skills available, freely available, that you can download, contribute to, you can figure out on your own, but you don't really know how the pieces all work together. You can't tell a good skill from a bad skill. And this means that people are trying to piece together these frameworks, trying to try everything that's out there all at once, and they can't, or rather they don't get the results that the skills themselves promise. This is true at an individual level, but it's also true at an organization level too. Organizations have no way or no understanding on how to build good skills, how to take their operating procedures and turn them into things that an agent can do. And if you don't do that, then it's hard to get the bounty that skills can offer. Just one more skill, bro. That seems like what we're saying. And I feel a bit of guilt here too, because we have Matt Pocock skills, which is my skills repo, which is one of the most popular engineering skill sets out there. And so I feel like I want to help the people who use my skills get out of skill hell. So how do we do it? How do we get out of this? Well, what is actually missing here? Well, in my opinion, the thing that we're missing is we don't know what makes a skill great. We can't yet look at a skill and go, okay, this skill is doing these good things and these bad things. There's no shared rubric, no framework for looking at a skill and making it better. And so that's what I'm going to give you in this talk. I'm going to give you a skill checklist, a checklist of things you can look at inside the skill to make sure that it's doing what it says it's doing and ways you can improve it, where you can write skills. This checklist looks like this. We start with the trigger of the skill, how the skill is invoked and the decisions that you need to design there. Then the internal structure of the skill, how the skill is actually composed and laid out internally. Then number three is how do you actually steer using the skill? How do you get the skill to tell the agent what to do? And then four, how do you make the skill as small as possible? Because once we've got a working skill, we then need to maximize it, prune out all of the irrelevant stuff, prune out all of the no ops. There's one handy advantage of me not being in the room with you, which is you can immediately go and try this out because I've encoded all of this into a new skill in my repo called writing great skills. So if you've got an immediate use case for this, just go to my skills repo, close this browser, get out of here and go and use this skill to either improve your skills or write great new ones. But let's start going through the checklist then. We have number one, the trigger, the way the skill is invoked. And in order to talk about this, I'm actually going to do a bit of comparison here, which is that my skills are often compared to another set of extremely popular engineering skills called superpowers. And I'm really often asked the question, how do your skills compare to superpowers? What's the difference between them? To understand that, we need to understand the difference between user invoked and model invoked skills. Anytime you have a skill, you can always invoke it manually. So the skill sits on your file system, the agent will just be able to pull up the skill and understand what's in there. And you can always do that by communicating that to the agent. It doesn't always look like this forward slash depending on the harness, but you can always user invoke your skills. Another way that skills can be invoked is by the agent itself. These are called model invokable skills or model invoked skills. You can take a description. So the description of the skill always ends up in the agent's context. And the agent can look in that and go, OK, based on that description, I'm going to invoke the skill. And I end up reading the skill.md file, which is where the meat of the skill is, into my context window. That's how you invoke a skill. That's what happens when a skill is invoked. So this description serves as a kind of context pointer. It sits in the agent's context, pointing to another file where the agent can go if it wants more context. But that context pointer, you don't need to put it into the agent's context. It can just be invisible from the agent. And that is what we call a user invokable skill. So some skills can only be invoked by the user because they don't have this context pointer. It's optional. For instance, we can see in my code base design here, this is a model invokable skill. It has a description that ends up in the agent's context window. But if we look at my grill me skill instead, we can see it has disable model invocation true. This means that this little description here will only show to the user. It won't be visible to the agent. So this then is tip number one. Decide if your skill is user invoked or model invoked. Now, you might think that model invoked skills are better, right? Because either the model can invoke it itself or the user can invoke it. It's more flexible. But every time you add a model invoked skill into your agent's environment, it increases what I'm going to call the context load on that agent. It adds a new description, which is costing you tokens on every request, but also adding a different thing for the agent to think about. So if you have 100 model invoked skills, that's going to be 100 descriptions inside the context for your agent. So it seems to make sense then to either tamp down the number of model invoked skills or to just use all user invoked skills. But user invoked skills have a different load, which is the more user invoked skills you have, the higher cognitive load on the user. In other words, the more things the user needs to keep in their head, the more skill you require from the pilot. And so if we compare Matt Pocock skills to superpowers, superpowers is primarily model invoked skills. It gives the agent superpowers. Whereas my adding a different thing for the agent to think about. So if you have 100 model invoked skills, that's going to be 100 descriptions inside the context for your agent. So it seems to make sense then to either tamp down the number of model invoked skills or to just use all user invoked skills. But user invoked skills have a different load, which is the more user invoked skills you have, the higher cognitive load on the user. In other words, the more things the user needs to keep in their head, the more skill you require from the pilot. And so if we compare Matt Pocock skills to superpowers, superpowers is primarily model invoked skills. It gives the agent superpowers. Whereas my skills, I much prefer to be in full control. That means I get to keep the context load on the agent as small as possible, but it does impose more of a cognitive load on me. So I need to understand the skills really deeply in order to get the most use out. So why have I done this? Why did I prefer user invoked skills? Well, every time you have a model invoked skill, it basically you get a cost in unpredictability. Because every time you have a context pointer pointing from one resource to another, the model may just choose not to follow it. You know, even if it's absolutely perfect for the task, it may just choose not to invoke the skill. I much prefer removing that level of unpredictability, imposing a bit more cognitive load on the user. And what you get is you're removing a class of problem from even being a problem. Because this unpredictability leaves people to need to eval their skills to make sure they're being called at the right time, which is really nasty. And it's a problem that I prefer to avoid. But what I'm hoping to show you here is that model invoked skills and user invoked skills both have their same costs. So it's not an easy decision which one you choose. So that then is the trigger, how the skill gets invoked. Now let's talk about the structure, the internal layout of the skill. I think of there as being two main units that you need to put into most skills. These two units are the steps and the reference. The steps are the step-by-step procedure that the skill is going to walk through. And the reference is any supporting information that helps it walk through those steps. You can have skills that have no steps and are only reference. And you can have skills that are no reference and only a set of simple steps to walk through. But if you start thinking of skills as composed of these two units, it really helps just break them down a lot more. If we look at an example, one of my skills called 2PRD creates a product requirements document out of the current context window. It's got three steps in it. So it finds the relevant context. It confirms the test scenes with the user. So there's a little human in the loop checkpoint there to make sure we're not doing anything weird with the testing, which I find really important. And then we write the product requirements document. To handle those three steps, we've got two bits of reference material. We've got a little bit of reference on what is a test scene. And then we've got a product requirements document template. So a literal markdown template, which is used to write the PRD. So this is a great way to write a skill from scratch. You work out if you need some steps, then you write those steps and you work out what reference material those steps need. And you put it in a separate little spot in the skill, which is for reference material. However, there's a really important constraint that we need to think about, which is tip number three, we want to make the main skill.md file as small as possible. Every skill is composed of its description and then a skill.md file and then any reference material that branches off that. And this skill.md file, if we make it small, then we're saving in a bunch of different ways. Smaller skills are just easier to maintain, easier to audit, fewer words to think about. And every time you shave off a word that is a token shaved, that multiple tokens shaved from your skill's cost. So I do believe that small skills are really important, both for maintainers and for users. One really useful way you can make your skill smaller is by thinking about the different branches of the skill, the different ways the skill can be used. Because if you have reference material that's only used in one branch, then that's a candidate for being removed from the main skill.md. For instance, if we look at my 2PRD here, we have two pieces of reference material. What is a test seam and the PRD template? Well, we need the PRD template every single time because we are always creating a PRD. And we probably also need the what is a test seam information every time because we're always asking about the test seams. So 2PRD, there's only one branch and all the reference material belongs on that branch. So it probably also belongs in the skill.md file. However, if we look at a different skill of mine, which is domain modeling, domain modeling does two things. It updates a local glossary called context.md, and then it also creates architectural decision records. In other words, it's doing two different things. Or it might actually choose to do neither of these, in which case it doesn't need the template and it doesn't need the ADR template either. So in other words, domain modeling has two or maybe three branches. And this means that we don't need to include the ADR template or the context.md template into the main skill. They can be moved into separate zones. The way you do that is you have the skill.md file. Then you put it behind a context pointer and you point the context template to a separate markdown file inside the skills folder. That context pointer literally just says if you need the template or if you need to update the context.md file, go to this file. And I call that an external reference. It's a reference that's external to the skill.md that you can just easily reference. The agent can pull in very easily because it's bundled along with the skill. So this is a technique you can use for making the skill.md as small as possible, which has so many benefits. Hide branching reference material behind context pointers. In other words, if you feel like your skill is going to be used in lots of different ways, then take the reference material that's relevant for those branches and hide them behind context pointers. So that is structure. We need to think about making the skill.md super duper small. We need to think about the branches in our skill moving material out behind context pointers. And we need to think about steps and reference, which are the two main units inside a skill. Let's go next to steering, the actual ways we get the agent to do what we want it to do. And for me, steering comes down to one really cool technique, which is the kind of main thing I want you to get from this talk. This technique fixes this issue, which is the agent doesn't do what I want. In other words, I specify something in the skill. I think that I've been clear and then it just doesn't do the thing. Now, I think the main reason this happens is because you're not using a technique called leading words. The idea of leading words or Moving material out behind context pointers. And we need to think about steps and reference, which are the two main units inside a skill. Let's go next to steering, the actual ways we get the agent to do what we want it to do. And for me, steering comes down to one really cool technique, which is the main thing I want you to get from this talk. This technique fixes this issue, which is the agent doesn't do what I want. In other words, I specify something in the skill. I think that I've been clear and then it just doesn't do the thing. Now, I think the main reason this happens is because you're not using a technique called leading words. The idea of leading words or light vert, if you like literary theory, is that there are certain words that pack in a bunch of meaning into a very small space. These leading words are really powerful with agents because you put the leading word in the skill itself, in the text, and then the agent will repeat the leading word back to itself as part of its operations, as part of its thinking tokens, and as part of its output to you. And then because it's re-emphasizing that word and that word hopefully describes what you want from the agent, that then goes and changes its behavior. Let's make this more concrete with an example. So let's imagine that we have a problem which is a classic problem with agents, which is that they code layer by layer. In other words, if you give them a big tranche of work to do, they will generally code up all of the database layer, then all of the schemas, then all of the API endpoints, then all of the front end. They don't do the typical human thing, which is to seek feedback early on, get something small working, and then expand out from there. Now, we can try to encourage the agent to do that by just saying, you know, don't code layer by layer, make sure that you create a small slice first and then go from there. But what if instead we used a leading word? We said vertical slice is our leading word. We want to slice up the work instead of horizontal slices into vertical slices. A vertical slice is a pretty well-known terminology in development, and so this will hopefully trigger the agent's priors and it will understand what we mean. We don't just have to have a two-word skill where it just says vertical slice. What we're doing is we're packing lots of meaning into a relatively short phrase that we then repeat throughout the skill. The cool thing about this technique is you can know if it's worked because you say vertical slice in your skill, and then you'll notice in the reasoning traces that it's saying, okay, we're going to do this as a thin vertical slice, then you should get better implementation plans. Everyone I've explained this technique to feels they've been doing that for a while. They've been using these little phrases to try to encourage the agent to do what they want. All I'm asking you now is to use those consistently within your skills and watch in the thinking traces as the agent adopts your way of doing it. So often if the agent isn't doing what you want, you need to make your leading words more consistent, more powerful, and look for others because English is a pretty wide API in terms of different functions you can call, different things you can experiment with. And there are many leading word candidates out there and agents are actually pretty good at helping you think of them. Another little lever you can use with agents is sometimes the agent just doesn't do enough legwork. What I mean by this is that we're on a step, let's say, and maybe the step is to ask clarifying questions or to explore the code base. And the agent just doesn't do enough of it. It doesn't put enough effort into that particular step. A real classic case of this and something that I have found almost everywhere it exists is plan mode. Because in plan mode, we have two steps. We have ask clarifying questions and then create a plan. And what I have found in every single implementation of plan mode I've tried is that ask clarifying questions just doesn't ever do enough legwork. It sees that its ultimate goal is to create a plan. And so it just does a small amount of legwork with ask clarifying questions, ask you a couple of things, and then eagerly creates the plan. So what was my solution here? Instead of doing plan mode, I instead have a skill called grill with docs, which is my ask clarifying questions phase. And then I split that up into a separate skill. So I split the planning into its own skill. So grill with docs now is its own skill where the agent only sees that part of the process. And then after grill with docs completes, we then go and do PRD. In other words, we have step one and step two, but the agent only sees one step at a time. So this is a really cool technique for increasing legwork on the step that you're on by hiding the future goal, hiding the future steps. It's not always necessary to split skills into individual steps, but in particular cases where you really want an extra chunk of legwork, it really works very well. So that is steering, using leading words to capture what you want in small reusable tokens, and then making sure that it's doing the right amount of legwork per step. So let's head now into pruning. Now pruning really is just a quick fire set of failure modes, different things that you can get wrong. And the first is fairly obvious, is we do not want massive skills. Massive skills are usually a symptom of something else going wrong. So a symptom of one of these other failure modes. And the first one is pretty simple. Don't repeat yourself. You need to make sure you're watching out for duplication. And in general, I like to have every part of the skill to have a single source of truth. In other words, if you have a piece of reference material like the PRD template, let's say, or something even smaller, like what is a test seam, you make sure that you don't repeat that in several places or cover multiple steps in multiple places. Just make sure each part has a single source of truth and you're not repeating yourself, even across reference material. The next way that skills get big is via sediment. And sediment is a classic thing when people are working on the same set of docs, which is that everyone starts contributing to a shared markdown file. People add their own stuff. They don't feel brave enough to delete and modify anyone else's. And so you just end up with this huge amount of sediment with often irrelevant material for the skill, especially stuff that hasn't been laid out properly. With a skill with a lot of sediment, you really need to look at structure. That's the first thing you need to do. You need to make sure that the stuff that's been added is relevant for all branches. If it's not, then move it into the correct branches. Or if it's totally irrelevant, maybe just remove it. Or maybe there's stuff in there that's totally stale, in which case you just need to kill it. The next failure mode is really common when an agent Don't feel brave enough to delete and modify anyone else's. And so you just end up with this huge amount of sediment with often irrelevant material for the skill, especially stuff that hasn't been laid out properly. With a skill with a lot of sediment, you really need to look at structure. That's the first thing you need to do. You need to make sure that the stuff that's been added is relevant for all branches. If it's not, then move it into the correct branches. Or if it's just totally irrelevant, maybe just remove it or kill it. Or maybe there's stuff in there that's totally stale, in which case you just need to kill it dead. The next failure mode is really common when an agent writes your skills, which are no-ops. So things inside the skill that appear to do something actually influence the agent's behaviour inside the context of the skill. Let's imagine we have an implement skill and we have an entire paragraph of the skill that tells the agent to write a long detail commit message. What would happen if you just deleted that paragraph? Well, the agent would probably still write a decent long commit message. People ask me a lot how I get my skills so small. And it's just using these techniques, using deletion tests, making sure that I compact things into leading words. I don't have anything irrelevant in there, and I don't have any sediment. And that finally brings us to the full sweep of things. Number one, we check the trigger. We make sure that it's firing at the right times. We check whether we're imposing context load or cognitive load. With structure, we think about branches. We think about structuring things into steps and reference. And we make sure that material that's only relevant for one branch is outside of the main skill.md. For steering, we're thinking about condensing text down into leading words and watching those leading words appear in the reasoning traces. And we're also thinking about legwork. Should we break this skill down further to increase its focus on the current phase by hiding the future phase from it? And with pruning, we're doing a final pruning pass over the entire skill, watching out for sediments, watching out for crud, and watching out especially for no-ops. Now, all of this stuff, the best way to get started with this framework is inside this skill, inside the writing great skills skill. You can check it out from matpocot skills, download it, use it to improve your own skills, and maybe even use it to run over some community authored skills so that you can check that the skills that you're actually pulling in are any good. If you want to follow along with my stuff, then I have a newsletter up on AIHero.dev. And my plans for the next few months are to release an AI coding crash course, which is an intro to a lot of the stuff I've been talking about and how you get off the ground working with engineering and AI. I hope that what I've given you is enough to help you escape from skill hell, or at least try to make the bitter journey out of there. I'm so sorry not to be able to attend in person, but thanks for watching. I'll see you very soon. talented at finding different forms of hell for us to go to. In, like, a few years ago, we had tutorial hell, which is where you would go into a bunch of tutorials trying to learn something, not be able to piece it together and sort of just get into this cycle you couldn't get out of. We had framework hell, where every other 10 minutes there was a JavaScript framework being announced, and you had to learn the hot new thing all the time. And now I think we have another version of hell, which is skill hell. Skill hell is where you have all of these skills available, freely available, that you can download, contribute to, you can figure out on your own, but you don't really know how the pieces all work together. You can't tell a good skill from a bad skill. And this means that people are trying to piece together these frameworks, trying to try everything that's out there all at once, and they sort of can't, or rather they don't get the results that the skills themselves promise. This is true at an individual level, but it's also true at an organization level too. Organizations have no way or no understanding on how to build good skills, how to take their operating procedures and turn them into things that an agent can do. And if you don't do that, then it's hard to get the bounty that skills can offer. Just one more skill, bro. That's kind of seems like what we're saying. And I feel a bit of guilt here too, because we have Matt Pocock skills, which is my skills repo, which is one of the most popular engineering skill sets out there. And so I feel like I want to help the people who use my skills get out of skill hell. So how do we do it? How do we get out of this? Well, what is actually missing here? Well, in my opinion, the thing that we're missing is we don't know what makes a skill great. We can't yet look at a skill and go, okay, this skill is doing these good things and these bad things. There's no shared rubric, no framework for looking at a skill and making it better. And so that's what I'm going to give you in this talk. I'm going to give you a skill checklist, a checklist of things you can look at inside the skill to make sure that it's doing what it says it's doing and ways you can improve it, where you can write skills. This checklist looks like this. We start with the trigger of the skill, how the skill is invoked and the decisions that you need to design there. Then the internal structure of the skill, how the skill is actually composed and laid out internally. Then number three is how do you actually steer using the skill? How do you get the skill to tell the agent what to do? And then four, how do you make the skill as small as possible? Because once we've got a working skill, we then need to basically maximize it, prune out all of the irrelevant stuff, prune out all of the no ops. There's one handy advantage of me not being in the room with you, which is you can immediately go and try this out because I've encoded all of this into a new skill in my repo called writing great skills. So if you've got an immediate use case for this, just go to this my skills repo, you know, just close this browser, get out of here and go and use this skill to either improve your skills or write great new ones. But let's start going through the checklist then. We have number one, the trigger, the way the skill is invoked. And in order to talk about this, I'm actually going to do a bit of comparison here, which is that my skills are often compared to another set of extremely popular engineering skills called superpowers. And I'm really often asked the question, how do your skills compare to superpowers? What's the difference between them? To understand that, we need to understand the difference between user invoked and model invoked skills. Anytime you have a skill, you can always invoke it manually. So the skill sits on your file system, the agent will just be able to pull up the skill and understand what's in there. And you can always do that by communicating that to the agent. It doesn't always look like this forward slash depending on the harness, but you can always user invoke your skills. Another way that skills can be invoked is by the agent itself. These are called model invokable skills or model invoked skills. You can take a description. So the description of the skill always ends up in the agent's context. And the agent can look in that and go, OK, based on that description, I'm going to invoke the skill. And I end up reading the skill.md file, which is where the meat of the skill is, into my context window. That's how you invoke a skill. That's what happens when a skill is invoked. So this description serves as a kind of context pointer. It sits in the agent's context, pointing to another file where the agent can go if it wants more context. But that context pointer, you don't need to put it into the agent's context. It can just be invisible from the agent. And that is what we call a user invokable skill. So some skills can only be invoked by the user because they don't have this context pointer. It's optional. For instance, we can see in my code base design here, this is a model invokable skill. It has a description that ends up in the agent's context window. But if we look at my grill me skill instead, we can see it has disable model invocation true. This means that this little description here will only show to the user. It won't be visible to the agent. So this then is tip number one. Decide if your skill is user invoked or model invoked. Now, you might think that model invoked skills are better, right? Because either the model can invoke it itself or the user can invoke it. It's more flexible. But every time you add a model invoked skill into your agent's environment, it increases what I'm going to call the context load on that agent. It adds a new description, which is costing you tokens on every request, but also adding a different thing for the agent to think about. So if you have 100 model invoked skills, that's going to be 100 descriptions inside the context for your agent. So it seems to make sense then to either tamp down the number of model invoked skills or to just use all user invoked skills. But user invoked skills have a different load, which is the more user invoked skills you have, the higher cognitive load on the user. In other words, the more things the user needs to keep in their head, the more skill you require from the pilot. And so if we compare Matt Pocock skills to superpowers, superpowers is primarily model invoked skills. It gives the agent superpowers. Whereas my skills, I much prefer to be in full control. That means I get to keep the context load on the agent as small as possible, but it does impose more of a cognitive load on me. So I need to understand the skills really deeply in order to get the most use out. So why have I done this? Why did I prefer user invoked skills? Well, every time you have a model invoked skill, it basically you get a cost in unpredictability. Because every time you have a context pointer pointing from one resource to another, the model may just choose not to follow it. You know, even if it's absolutely perfect for the task, it may just choose not to invoke the skill. I much prefer removing that level of unpredictability, imposing a bit more cognitive load on the user. And what you get is just you're removing a class of problem from even being a problem. Because this unpredictability leaves people to need to eval their skills to make sure they're being called at the right time, which is really nasty. And it's a problem that I prefer to avoid. But what I'm hoping to show you here is that model invoked skills and user invoked skills both have their same costs. So it's not an easy decision which one you choose. So that then is the trigger, how the skill gets invoked. Now let's talk about the structure, the internal layout of the skill. I think of there as being two main units that you need to put into most skills. These two units are the steps and the reference. The steps are the step-by-step procedure that the skill is going to walk through. And the reference is any supporting information that helps it walk through those steps. You can have skills that have no steps and are only reference. And you can have skills that are no reference and only a set of simple steps to walk through. But if you start thinking of skills as composed of these two units, it really helps just break them down a lot more. If we look at an example, one of my skills called 2PRD creates a product requirements document out of the current context window. It's got three steps in it. So it finds the relevant context. It confirms the test scenes with the user. So there's like a little human in the loop checkpoint there just to make sure we're not doing anything weird with the testing, which I find really important. And then we write the product requirements document. To handle those three steps, we've got two bits of reference material. We've got a little bit of reference on what is a test scene. And then we've got a product requirements document template. So just a literal markdown template, which is used to write the PRD. So this is a great way to write a skill from scratch. You work out if you need some steps, then you write those steps and you work out what reference material those steps need. And you put it in a separate little spot in the skill, which is for reference material. However, there's a really important constraint that we need to think about, which is tip number three, we want to make the main skill.md file as small as possible. Every skill is composed of its description and then a skill.md file and then any reference material that branches off that. And this skill.md file, if we make it small, then we're saving in a bunch of different ways. Smaller skills are just easier to maintain, easier to audit, fewer words to think about. And every time you shave off a word that is a token shaved, that multiple tokens shaved from your skill's cost. So I do believe that small skills are really important, both for maintainers and for users. One really useful way you can make your skill smaller is by thinking about the different branches of the skill, the different ways the skill can be used. Because if you have reference material that's only used in one branch, then that's a candidate for being removed from the main skill.md. For instance, if we look at my 2PRD here, we have two pieces of reference material. What is a test seam and the PRD template? Well, we need the PRD template every single time because we are always creating a PRD. And we probably also need the what is a test seam information every time because we're always asking about the test seams. So 2PRD, there's only one branch and all the reference material belongs on that branch. So it probably also belongs in the skill.md file. However, if we look at a different skill of mine, which is domain modeling, domain modeling does two things. It updates a local glossary called context.md, and then it also creates architectural decision records. In other words, it's doing two different things. Or it might actually choose to do neither of these, in which case it doesn't need the template and it doesn't need the ADR template either. So in other words, domain modeling has two or maybe three branches. And this means that we don't need to include the ADR template or the context.md template into the main skill. They can be moved into separate zones. The way you do that is you have the skill.md file. Then you put it behind a context pointer and you point the context template to a separate markdown file inside the skills folder. That context pointer literally just says if you need the template or if you need to update the context.md file, go to this file. And I call that an external reference. It's a reference that's external to the skill.md that you can just easily reference. The agent can pull in very easily because it's bundled along with the skill. So this is a technique you can use for making the skill.md as small as possible, which has so many benefits. Hide branching reference material behind context pointers. In other words, if you feel like your skill is going to be used in lots of different ways, then take the reference material that's relevant for those branches and hide them behind context pointers. So that is structure. We need to think about making the skill.md super duper small. We need to think about the branches in our skill moving material out behind context pointers. And we need to think about steps and reference, which are the two main units inside a skill. Let's go next to steering, the actual ways we get the agent to do what we want it to do. And for me, steering comes down to one really cool technique, which is the kind of main thing I want you to get from this talk. This technique fixes this issue, which is the agent doesn't do what I want. In other words, I specify something in the skill. I think that I've been clear and then it just doesn't do the thing. Now, I think the main reason this happens is because you're not using a technique called leading words. The idea of leading words or light vert, if you like literary theory, I suppose, is that there are certain words that pack in a bunch of meaning into a very small space. These leading words are really powerful with agents because you put the leading word in the skill itself, in the text, and then the agent will repeat the leading word back to itself as part of its operations, as part of its thinking tokens, and as part of its output to you. And then because it's re-emphasizing that word and that word hopefully describes what you want from the agent, that then goes and changes its behavior. Let's make this more concrete with an example. So let's imagine that we have a problem which is a classic problem with agents, which is that they code layer by layer. In other words, if you give them a big tranche of work to do, they will generally code up all of the database layer, then all of the schemas, then all of the API endpoints, then all of the front end. They don't do the sort of typical human thing, which is to seek feedback early on, get something small working, and then expand out from there. Now, we can try to encourage the agent to do that by just saying, you know, don't code layer by layer, make sure that you create a small slice first and then go from there. But what if instead we used a leading word? We said vertical slice is our leading word. We want to slice up the work instead of horizontal slices into vertical slices. A vertical slice is a pretty well-known terminology in development, and so this will hopefully trigger the agent's priors and it will understand what we mean. We don't just have to like have a two-word skill where it just says vertical slice. What we're doing is we're packing lots of meaning into a relatively short phrase that we then repeat throughout the skill. The cool thing about this technique is you can know if it's worked because you say vertical slice in your skill, and then you'll notice in the reasoning traces that it's saying, okay, we're going to do this as a thin vertical slice, then you should get better implementation plans. Everyone I've explained this technique to sort of feels like, oh yeah, I've been doing that for a while. I've been using these little phrases to try to encourage the agent to do what I want. All I'm asking you now is to use those consistently within your skills and watch in the thinking traces as the agent adopts your way of doing it. So often if the agent isn't doing what you want, you need to make your leading words more consistent, more powerful, and look for others because, you know, English is a pretty wide API in terms of different functions you can call, different things you can experiment with. And there are many leading word candidates out there and agents are actually pretty good at helping you think of them. Another little lever you can use with agents is sometimes the agent just doesn't do enough legwork. What I mean by this is that, okay, we're on a step, let's say, and maybe the step is to ask clarifying questions or to explore the code base. And the agent just doesn't do enough of it. It doesn't put enough effort into that particular step. A real classic case of this and something that I have found almost everywhere it exists is plan mode. Because in plan mode, we have two steps. We have ask clarifying questions and then create a plan. And what I have found in every single implementation of plan mode I've tried is that ask clarifying questions just, you know, doesn't ever do enough legwork. It sees that its ultimate goal is to create a plan. And so it just does a small amount of legwork with ask clarifying questions, ask you a couple of things, and then eagerly creates the plan. So what was my solution here? Instead of doing plan mode, I instead have a skill called grill with docs, which is kind of my ask clarifying questions phase. And then I split that up into a separate skill. So I split the planning into its own skill. So grill with docs now is its own skill where the agent only sees that part of the process. And then after grill with docs completes, we then go and do two PRD. In other words, we have step one and step two, but the agent only sees one step at a time. So this is a really cool technique for increasing legwork on the step that you're on by hiding the future goal, hiding the future steps. It's not always necessary to split skills into individual steps, but in particular cases where you really want an extra chunk of legwork, it really, there's no technique like it. It works very, very well. So that is steering, using leading words to capture what you want in small reusable tokens, and then making sure that it's doing the right amount of legwork per step. So let's head now into pruning. Now pruning really is just a quick fire set of failure modes, different things that you can get wrong. And the first is fairly obvious, is we do not want massive skills. Massive skills are usually a kind of symptom of something else going wrong. So a symptom of one of these other failure modes. And the first one is pretty simple. Don't repeat yourself. You need to make sure you're watching out for duplication. And in general, I like to have every part of the skill to have a single source of truth. In other words, if you have a piece of reference material like the PRD template, let's say, or something even smaller, like what is a test seam, you make sure that you don't repeat that in several places or like cover multiple steps in multiple places. Just make sure each part has a single source of truth and you're not repeating yourself, even across reference material too. The next way that skills get big is via sediment. And sediment is just a classic thing when people are working on the same set of docs really, which is that everyone starts contributing to a shared markdown file. People add their own stuff. They don't feel brave enough to delete and modify anyone else's. And so you just end up with this huge amount of sediment with often irrelevant material for the skill, especially stuff that hasn't been laid out properly. With a skill with a lot of sediment, you really need to look at structure. That's the first thing you need to do. You need to make sure that the stuff that's been added is relevant for all branches. If it's not, then move it into the correct branches. Or if it's just totally irrelevant, maybe just remove it or kill it. Or maybe there's stuff in there that's totally stale, in which case you just need to kill it dead. The next failure mode is really common when an agent writes your skills, which are no-ops. So things inside the skill that appear to do something, actually influence the agent's behaviour inside the context of the skill. Let's imagine we have an implement skill and we have an entire paragraph of the skill that tells the agent to write a long detail commit message. What would happen if you just deleted that paragraph? Well, the agent would probably still write a decent, like, long commit message. People ask me a lot how I get my skills so small. And it's just using these techniques, using deletion tests, using, making sure that I compact things into leading words. I don't have anything irrelevant in there, and I don't have any sediment. And that finally brings us to the full sweep of things. Number one, we check the trigger. We make sure that it's firing at the right times. We check whether we're imposing context load or cognitive load. With structure, we think about branches. We think about structuring things into steps and reference. And we make sure that material that's only relevant for one branch is outside of the main skill.md. For steering, we're thinking about condensing text down into leading words and watching those leading words appear in the reasoning traces. And we're also thinking about legwork. Should we break this skill down further to increase its focus on the current phase by hiding the future phase from it? And with pruning, we're doing a final pruning pass over the entire skill, watching out for sediments, watching out for crud, and watching out especially for no-ops. Now, all of this stuff, the best way to get started with this framework is inside this skill, inside the writing great skills skill. You can check it out from matpocot skills, download it, use it to improve your own skills, and maybe even use it to run over some community authored skills so that you can check that the skills that you're actually pulling in are any good. If you want to follow along with my stuff, then I have a newsletter up on AIHero.dev. And my plans for the next few months are to release an AI coding crash course, which is an intro to a lot of the stuff I've been talking about and how you get off the ground working with engineering and AI. I hope that what I've given you is enough to help you escape from skill hell, or at least try to make the bitter journey out of there. I'm so sorry not to be able to attend in person, but thanks for watching. I'll see you very soon.