Harnesses in AI: A Deep Dive — Tejas Kumar, IBM

Transcript

[SPEAKER_00] Hello, everybody. Hello. Everybody's head turned up. Hello, hi. Hey. [SPEAKER_00] How was lunch? Was it good? Yep. Yes. You didn't like it, huh? It's British food. Anyway, hi. I'm Tejas. I'll be your first speaker this afternoon. Tejas, that's pronounced like a connoisseur. Tejas, that's pronounced like a connoisseur. [SPEAKER_00] It's really contagious. [SPEAKER_00] Don't worry, I'm not. [SPEAKER_00] Hopefully, my joy in AI is. [SPEAKER_00] And I've had the privilege of working at a number of different places over my career in one form or the other. [SPEAKER_00] It's just been an absolute joy to learn from the best. [SPEAKER_00] Today, I'm an AI developer advocate at IBM, where we do things with AI, believe it or not. We train frontier models. [SPEAKER_00] We build harnesses. [SPEAKER_00] It's really a fun lab to work in. [SPEAKER_00] But that's not what I'm here to talk to you about today. Today, I'm here to talk to you about AI harnesses. AI harnesses. Before I move forward, I would love to just have a show of hands. How many of you are confident in your understanding of AI harnesses? You're thinking, I could present this on stage today. Look around. Look around. No, seriously, look around. That's why we're doing this talk. This is my hope. I want you to, if I ask you this at the end of the talk, I want you to say, I get it now. That's the whole point. I have literally nothing to gain from this other than sharing knowledge. Because also, this term is everywhere. You may have heard it used 52,000 times today. And it means different things to different people. In the machine learning world, it means a glorified test suite for machine learning models. But in the AI world, it means something different. And so today, we're going to understand this in detail. It's a deep dive, but it's 18 minutes long. So let's move forward. I want to start by talking about why harness. Why do we use harnesses? And the reason for this is because we pay rent to companies that give us compute, give us inference, give us tokens in return. Some of you maybe work for companies that have frontier models like Anthropic or Google or whatever. And you maybe, what was the term? Token billionaires, yeah? I'm not that. I am maybe with Watson models. But the vast majority of us aren't token billionaires. We pay rent. We literally pay $20 a month for Cloud Pro. And then you get a context window that's limited and you don't get the full hog, so to speak. And the model you rent is a black box. They could at any time, I'm not saying they do, but they could if Opus is somehow not available, they could serve you Sonnet even though it says Opus. You would never know, right? And so it's a big issue. There are too many variables that we cannot control. So why harness? Because the name of the game with harness is reliability. I really hope I'm not supposed to stand in front of this white line and then I'm just nodding the camera. Anyway, whatever. It's reliability. It's making sure that the agents we build do what they do, period, irrespective of the black box model, irrespective of the thing we rent and so on, okay? Now that we understand why harness, let's talk about what a harness even is from first principles. Let's take it all the way back to harnesses that we know and understand. If you've ever climbed a mountain or you've seen someone, this is a harness. Mountain climbers will harness themselves to what? To a mountain because it's stable and they can't go off the rails, literally. They anchor themselves in something stable so that they can't drift too far. Okay? That's what a harness is by design. When you have any dog owners here, you have dogs, you walk your dog on a harness. Okay? Why? Because your dog doesn't go and bankrupt you with tokens. Okay? That's what a harness is. But the problem is, if we think about what harness is, there are really two types. There's one from the machine learning world, which as I mentioned is a test suite and a test runner. You give a model some inputs and you see the quality of the outputs. That's not this. This is not ML engineer Europe. We're going to talk today about the agent harness that is common in AI engineering. Okay? So what is an agent harness? An agent harness, and this is the key point here. I'm not making money off this. It's just an expression. The agent harness is everything around the model that gives it grounding in reality. It's literally the thing that ties it to a stable environment. An agent, so Claude code for example, can be considered an agent harness. And some of you will say, oh no, it's a coding agent. Absolutely, it's a coding agent. But it's a harnessed coding agent. An agent harness has more or less the same typical suspects, moving parts. Number one, it's got a tool registry. So Claude code, cursor, codex, they have tools to read from the file system, to write, to execute bash commands. They have a tool registry. They have a model. And some of them allow you to choose a model, some of them allow you to not. [SPEAKER_00] They have a model. They have primitives for managing context. Almost every harnessed agent runtime today will compact its own context. Right? That's the job of the harness. Guardrails are another part of a harness. For example, max steps. Anyone using max steps, do not do more than five tool calls. That's a guardrail. And so if you do that, you just kill the run. An agent loop is another part of an agent harness. Which is crucial. And some of them allow you to choose a model, some of them allow you to not. They have a model. They have primitives for managing context. Almost every harnessed agent runtime today will compact its own context. Right? That's the job of the harness. Guardrails are another part of a harness. For example, max steps. Anyone using max steps do not do more than five tool calls. That's a guardrail. And so if you do that, you just kill the run. An agent loop is another part of an agent harness. Which is crazy. This is where some people I've spoken to preparing this talk will say, wait, isn't a harness just the agent loop? No! It's the stuff around the agent loop. In fact, it could be a loop around your agent loop. It could be an NM loop. And we'll look at that a little bit in some code. And then finally, there's a verify step. This is, for example, in a coding agent. After the work is done, a verify step would be, hey, let's run lint. Let's run tests. Let's make sure nothing broke. Right? So almost every... I'll use coding agents as an example, but you could have a harness for anything. And it's amazing, because it really grounds black box models in a stable environment that you control. Okay? I'd like to show you a demo. And what we're going to do together is we're going to build a harness. A bare-bones baby's first harness. Let's call it a poor man's AI harness together. So we understand from first principles how this works. We're going to build a computer use agent that has a job. The job is go to Hacker News and upvote the first post. Okay? It's a computer use... It's a browser use agent. We're going to use a really bad model intentionally. We're using GPT 3.5 Turbo, which is from 2023. Right? But we're going to harness it so that it can actually do the job. And we're going to save money. So let's get into the demo. I've spoken too much. Welcome to my project. This is my project. Hello, everybody. This is the entry point. Can you see that? Is it too small? Yeah? You want it bigger? Let's do bigger. Okay. So this is not... Actually, this room is too bright. Let's do light mode. It's not my nature, but sometimes. That's better, yeah? Okay. So we have a model and we're trying an old model. I'll just... Sorry. We shouldn't have seen that. No, we'll try an old model and this is the prompt. This is the task. This is literally my prompt. Upvote a story. I just described it. For the purpose of this demo, we will not change the prompt at all. Because a lot of us think, hey, my agent is not doing what it's supposed to do, so I just need to prompt it harder. Right? That's not always true. I need to change the system prompt. We're not going to touch any prompts here. We're just going to build a harness and the outcome will change. We log some things to the console and then we start a browser session. Okay, what's a browser session? It's literally just Playwright. Not Playwright MCP. Like Playwright, Playwright. Where this is just a class I made with an open method that launches Chromium and gets a context and makes a page. And then navigate. We're just literally calling the Playwright functions. This is just traditional engineering. So we create a session. We open the session, meaning a browser window and a context. And then we create our tools and we give that browser session to the tools. And we create a context and we give the task, meaning the prompt here, to the context. Now, create tools is literally what it sounds. It's here. There's just some types. And create tools is a function that takes a browser session and gives you tools. And these tools are not, I didn't invent this. This is from OpenAI's SDK. Okay. So you have the name, the description, parameters and execute. The way you actually call the tool in your runtime. And there's just tools for, I made this. It's very easy. So that's my tools. And then create context. You may think, whoa, context engineering. Absolutely not. This is my context. There's nothing here. It's just a system prompt. Literally the most basic system prompt and the user's task. This is basic, basic. And then we have run loop, which is just running the agent in a loop. So what it's doing here, we can actually just look at this too. While true. So it is an agent loop. And we get a response from the agent. And we see if the response says stop, meaning if the LLM says I'm done, then we return the value. If we get any other response, we don't do anything except add these events into a trace. So we just push history into a big list of histories. Make sense? And so that's all we're doing here. This is just a loop where we just collect events until we're done. Okay? So this is super basic. Now let's see how it works. So I'm going to come over here. And I'm going to do, are you okay, sir? Do you need water? I'm going to npm run agent. And so it's going to open Chromium. If we get any other response, we don't do anything except add these events into a trace. So we just push history into a big list of histories. Make sense? And so that's all we're doing here. This is just a loop where we just collect events until we're done. Okay? So this is super basic. Now let's see how it works. So I'm going to come over here. And I'm going to do, are you okay, sir? Do you need water? I'm going to npm run agent. And so it's going to open Chromium. It's going to, okay, Hacker News, so far so good. Click upvote. Oh, no! So we hit a login screen and then it panicked and crashed. But look, it lies. You see this? This is a problem. And so what's the solution? Prompt it harder. No. Change the system prompt. Always log in with these credentials included in the system prompt. No. So how do we then solve this? And look, because of my logging, we can actually see it just clicks the upvote button and then considers it a success. It doesn't verify. This is the job of a harness. Okay? So now incrementally, we're going to slowly start building a harness. And so let's just move. I'm not going to write code here. I'm not going to live code because we don't write code anymore. We inspect diffs. Right? Anyone write code by hand? You don't? Maybe you actually do belong here. Anyway, so I'm kidding. So this is the first change we're going to make. This was our index file. And we have this run loop that I showed you, but now we're going to add one thing to it, which is default guardrails. We're going to create some guardrails. Okay? What do our guardrails look like? Well, let's go and look at it in the editor with guardrails over here. And so we have some types, but these are our guardrails. We have two. Max iterations, meaning if you do more than six steps, I'm going to kill you. And max messages, meaning if you have more than this many messages, I will compress the context. These are just guardrails. Okay? A little utility to combine them, and we can compose them here. We can do as many as we want. So now let's go back to our changes. That's the guardrails. If we go back to the agent loop, we actually use the guardrails here in this diff. And so we include the guardrail functions, and we can see that here, what we're doing is we're checking how many messages have we accumulated, and we just trim the context if it's too much. But what I did want to show you is here at the end, we push context size. We just have some more metadata about what we've done with our guardrails. Okay? Our context compressor is extremely basic and extremely naive. This is what it does. Let me actually open this with syntax highlighting to spare you. This is what it does. So what we're doing is if we always keep the system prompt and the user prompt, and the most recent two messages. So if the guardrail is triggered, we always remove everything after the system prompt and the user prompt in the middle, and we keep the last two messages. This is super naive. Don't do that. There's better ways, but this is where we start. We're getting there. So we're starting to have a harness, but it's not called a harness. But this is really a pregnant harness. Like, it's almost born. Okay? And so what we're going to do is let's just call it a harness now. So I'm going to show you another diff where we, here, check this out. Index, we've deleted almost everything. And we've moved it into this file called harness. Let's go look at our entry point now. Index, it's all gone. So the prompt is there, but this is 19 lines of code, and we just have run harness. We've taken all the logic from here and hidden it in a function called run harness. And as you would expect, run harness does exactly the same thing as we did in the index. Okay? Nothing new is here except maybe a print function, which is just console log. Is this clear so far? Yeah, we just moved stuff. Now that we have something called a harness, we can actually use it. And let's solve the problem of lying first before we solve the problem of logging in as me. Yeah? Because it says I upvoted it, did not. I want to know. So what we're going to do is we're going to add some guardrails and have it tell the truth. Like, if you failed, tell me the truth. How might we do that? Well, we'll check it out here. So many things changed. Or not. I don't know. Hang on a second. So we run harness and we added a third argument here, which is a verify step and max attempts. Max attempts goes to our guardrails. So if you took more than three tries to do this, just give up. And if we go to the harness, we added a lot of things that are just manual code. This is not different prompt. This is my logic. The main logic is run harness no longer wraps over the code we moved. But we moved that to a different function called run harness attempt. So if we come to run harness, let's go here. I need to check the branch out. Sorry. Yeah. So now if we go to run harness attempt, I'll collapse this. I'll collapse this. I'll collapse all of these. And if we go to run harness attempt, now this is the same thing from our index. This is not a different prompt. This is my logic. The main logic is run harness no longer wraps over the code we moved. But we moved that to a different function called run harness attempt. So if we come to run harness, let's go here. I need to check the branch out. Sorry. Yeah. So now if we go to run harness attempt, I'll collapse this. I'll collapse this. I'll collapse all of these. And if we go to run harness attempt, now this is the same thing from our index. We just moved it into a function called run harness attempt because our main run harness is just a loop that runs no more than three times. Okay. Is this clear? So we're just enforcing the max steps but at the harness level for safety. Then we have run harness attempt that calls it. We have this function called verify successful upvote. I wrote this. This is deterministic. That's what I want to show you. What does this do? Well, we see if, remember we were tracing in the agent loop. We're just adding history events. So we reflect on that and we see if there was a browser click on the upvote. And if it's successful but really successful, then we say true. [SPEAKER_00] But there's a huge but here, which is we have now cases for failed login. If there's a tool named Harness Auto Login and if the message starts with failed, then we return early and we say no, this failed. We're removing the lie. Okay. Similarly, unrecovered login redirect. We look over our agent loops tools that we've been pushing into. And if we see that the harness auto login didn't run and now we're on the page that is the login URL, then again, we just fail. Okay. And so what we're doing is we're just adding if this happened, if this happened, just fail, return early. Is this clear? This is what a harness does. And so let's run this now with the harness. We're going to go to Hacker News. We're going to go to Hacker News. And we're going to repeat the same cycle. Okay. It's going to come here. And now it still failed. But look, it stopped lying because our harness checks the tool history and actually sees what happened. This is what a harness is supposed to do. Great. This is already half the battle won because step one to solving a problem is admitting you have one. Okay. Test driven development vibes. So now that we're failing correctly, we can succeed. And I'd like to show you that in the last diff and then we'll finish the talk here. So number four, we have a whole new function. It's called login handler. I'll add some syntax highlighting here so you don't go blind. But here, create login handler. This is all it does. It runs every agent loop just before we push to the traces. And this is what it does. It checks the browser session's current URL. And if we're not on a login page, it just says cool. I didn't return. I have nothing for you. This computationally is not costly at all, right? If you're not on the login page, but if you are on the login page, then we fill in a temporary, this can be an environment variable. It can be secure. You get the idea. But we fill in credentials and submit the button programmatically from the harness, not from the agent, deterministically and securely because this file has access to any secrets I want it to, right? And so this file, how is this called? Well, this is called in the agent loop. So if we go back to our agent loop, and notice we were pushing traces, yeah? This is where we push the traces. Just before, if we have a login handler, we call the login handler just before this, in the agent loop. What does the login handler do? Well, if we're not on a login page, it does nothing. If we are on a login page, then it quickly will inject credentials and submit the form and then take you back. It will also add, as we can see here, it pushes a message into the queue saying, hey, I'm the harness. I logged in. You're good now. Is this clear? Yeah? So the harness is literally harnessing the agent to something stable, something deterministic. That's what it's for. Okay? Let's run this now and see what happens. So npm run agent. It's going to open Hacker News. And when it gets to the login, now that harness step, it logged in and it upvoted the first one. And it closed. Amazing. So successfully upvoted, a little snitch for Nileks, rank two, succeeded after six iterations. And I can click this and go into Hacker News and actually see, indeed, it was upvoted. And I can unvote now, which means it was upvoted. And so the agent used the computer, logged in as me with my harness that I just made here on stage. That's the purpose. Is this clear so far? Do you understand the role of a harness? Look at you nodding. This is music to my ears. Fantastic. Something to my eyes. I don't know the beauty to my eyes. We don't have an expression for that. Let's land the plane. I'm done. I think my work here is done. What does this look like in practice? Why do I care so much about our harnesses? Because they run the world. Models are non-deterministic. And you want to do more with less. You want to use a cheap model. Use Qwen or something. Or even something smaller. Use GPT OSS. It's free. And with a great harness, you can go very far. That's why. Let's land the plane. I'm done. I think my work here is done. What does this look like in practice? Why do I care so much about our harnesses? Because they run the world. Models are non-deterministic. And you want to do more with less. You want to use a cheap model. Use Quen or something. Or even something smaller. Use GPT OSS. It's free. And with a great harness, you can go very far. That's why. At IBM, we create an open source project that we deploy in the enterprise. That allows very large companies, huge companies, in their private data sensitive areas, to perform RAG operations on all kinds of things. Teams calls and PDFs and invoices. We build. It's called OpenRAG. And it's RAG. I don't know if RAG is cool or not anymore. But OpenRAG has a hell of a harness that provides enterprise level security to asking questions with internal, very, very siloed data. And that's where the harness engineering comes in. So let's summarize. We covered a lot of content. Was it a deep dive? I think it was a deep dive. It was a deep dive in 18 minutes or so. We went pretty far. It should not be lost on you that I did not touch the prompt once. I did not change a system prompt. We just built a harness and the outcome radically changed. And of course, we can add secrets. We can add tokens. Yeah, we did a lot. In the end, I hope you understand what a harness is, the value it can present, and how you can use it. What's next? Look, I don't have a crystal ball like everyone else here. But it's not lost on me that 2025 was the year of agents. Yes?