Building Agents Is Trivial Now, Context Is the Next Frontier — Jeff Ng, Unblocked

Transcript

Hi, all. My name is Jeff. I'm a founding engineer at Unblock, and I'm here to talk to you about how building agents has actually gone pretty easy. But unfortunately, they still get things confidently wrong. So six months ago, it required a team's effort and a quarter to build out an agent. An agent is more than just models and tools. It's the models, the tools, and everything required to build out a production service. Here are some examples of the different systems necessary in order to build something out. Each one of these was its own company or at least a company function. I'm not going to go through each one of these, but a few stood out to me. First one, checkpoint and state persistence. Agent runs, they're typically long-lived and stateful. Unfortunately, infrastructure itself, though, that's ephemeral. Crashing without durability can actually lead to a lot of state loss, and that state includes things like message history, tool calls, as well as where you are in the loop. Without these things, you can't resume the session. One option is maybe you want to restart the session. Unfortunately, that's actually quite expensive as well. You lose out on all the tokens that you had originally used, as well as latency. From a user experience standpoint, you've already triggered that session. Now you have to wait for the whole thing to go again. And lastly, side effects. Your agent might have performed some side effects, and now there's a chance of those doubling up. So next thing, sandbox infrastructure, right? So as we all know, we're running more and more agent-generated code, as well as third-party code. This gets all run on your infrastructure, and due to that, there are some complexities. Because of that, we want to introduce isolated sandboxes, which help prevent unnecessary reads of environment secrets, unnecessary network access. In general, we don't want to take down the shared host. And then, observability. How do we answer the question, where did this fail? Typically, this includes tracking logs and traces from across half a dozen systems. Everything I've mentioned here, none of this actually improves an agent's capabilities. They're all taxes one has to pay in order to get an agent out there to play the game. Thankfully, things have changed quite a bit. The whole ecosystem has matured quite a bit, and cloud infrastructure players such as Cloudflare, Vercel, AWS, they've gone and taken some of that complexity away and built primitives that these frameworks, Flu, VercelEve, Mastra, with these together, they've taken a lot of complexity away, and you can focus more on building the actual agent itself, the core logic that actually helps you and your team and your customers. So, here's an example of one. I've played around with Flu and Cloudflare. And as you can see on the left-hand side, we handle everything as mentioned before. So the primitives plus the framework lead to a situation where it's actually not that much code to define an agent. One of the things I was shocked at when I first took a look at the documentation. To get into details, all you really have to do when defining an agent is, A, deciding which model you want to use, B, the instructions or the system prompt, C, the tools that you want the agent to have access to, skills, the things that I can do, as well as the sandbox location, where things are being run. So, to give you an example of this, I've actually gone and built out an issue enrichment system, specifically for Linear. So, what this does is, given a Linear ticket and access to your code repository, it'll go out, fetch a Linear ticket, determine whether or not it's a feature or a bug. From there, it'll do some code searching, provide all that context to the agent, and then come up with a plan of next steps. On the left-hand side here, this is an issue that one of my colleagues, a support engineer, had posted, I think, a month ago. To summarize it, what had happened was, we had some pretty serious degradation in our agentic QA pipeline. Time to first character was taking three to four seconds, when it should realistically be in the hundreds of milliseconds. So, let's see what happens when we put this through the system. So, as you'll see here, I've set up the agent to go fetch an agent. I've given it the skills and tools to actually go and fetch the code, search code, and query against that. That's being passed back to the agent, which is doing some reasoning against that right now. And then, just wait a little bit. At this point, we've updated the Linear issue ticket. The recommendation here is to re-enable our async dispatch, which makes sense. It allows us to run a lot more of our QA pipeline in parallel on a single machine. Sounds great, right? Unfortunately, this is wrong. This had actually caused an outage a few days ago, and one of our support engineers had explicitly disabled this before the ticket was shown. So, where did things go wrong? Why did it get it wrong? The agent I had written, it didn't have a full picture. It was missing the context from the Slack discussion that happened after the issue, where the engineers came together, went through the actual outage, what went wrong, what was the fix, and the next steps. It also was missing the post-mortem Linear ticket, which came as a result of that. In general, it had a narrow understanding of the problem. This concept of missing knowledge and intent that's sorted across an organization and different systems, it's something that comes back again and again. And since this was deployed as a background agent, this is going to make that mistake silently in the background, misinforming both my teammates and potentially other agents. So, I guess the next question is, why don't we run into this locally? We all use agents locally. We don't necessarily run into these issues. Well, you, the human, the engineers, we currently act as that context layer. When working with an agent, you're there to ask questions, catch any errors, and supply the missing facts on every single turn. A person knew why the code is the way it is, what broke last time, and what we've decided to do about it. The agent, though, it only has what's on the right-hand side, right? It has instructions, the tools and skills we specifically gave it, the code, as well as the ticket in front of it. So, when a human is in the loop with the agent, we're there to catch this fear. Ultimately, we're there to babysit the agent. But as agents have gone trivially easy to deploy, as I've shown earlier with Flu and Cloudflare, without the human in the loop, this issue becomes more and more prevalent. This missing context becomes a silent failure. All that intuition and knowledge that we've had as humans needs to be replaced. Something needs to carry the load. So, that thing, that's a context engine. A context engine is a system that provides task-relevant information based on who you are and what matters. It also resolves all the conflicts across multiple data sets. It understands your access roles or the agent's access roles and only respects that and only provides information that's relevant. And most importantly, it delivers a synthesized understanding that an agent can act on, not just a list of documents that it has to reason upon itself. So, how does this context engine work? Well, let's take a step back. What does an agent actually need? An agent needs context outside of the context. It's not just your source code. Think about everything that you need to work day to day. It's not just the code. It's the Slack discussions where decisions are made, the documentation where we show all the best practices. All that is important to your day-to-day process, and that's true for your agent as well. So, what we do here is we connect everything: the docs, code, tickets, conversations. We then build a model of your organization, of your system, and we piece together how all of these things work together and make it generally available to your agents. From that model, the agents are only provided a slice of that data which has been reconciled, ranked, and scoped to your permissions. Scattered context comes in, grounded context comes out. The obvious next question is, why can't we just do this with MCP, right? You could connect a Slack MCP, a Linear MCP, a GitHub MCP, and with that, all that data is accessible. MCP is great at access, but access isn't understanding. An MCP hands the agent the raw results, and you're now dependent on the agent to actually decide what to believe in. You end up flooding the agent with irrelevant data, filling up the context window, and overall context costs just go up. It also leaves a local agent to handle conflicts in data. Your Linear MCP and your Slack MCP may come back with different results. You're just leaving the agent to make that decision somewhat ad hoc at this moment. So, back to the original problem I had earlier. This is the same file, same agent, but now we've connected the context agent. What we do here is we're currently prompting Unblock to do some research on the ticket and provide that context to the agent. So, let's see that in action. Sorry about that. So, here we go. We're doing a very similar thing. We're fetching the Linear ticket. Well, you'll notice here that we're actually calling the Unblock context engine. And what it's done here is actually it's found the relevant Linear post-mortem, as well as a Slack conversation where we've had the entire discussion between the engineering teams. And as part of that, we've returned an understanding, and that's now been provided to the agent as a summary. So, the agent no longer has to actually reason from those documents. And at this point, you'll notice here, the agent now has been updated. The recommendation has gone from breaking and causing another issue to actually preventing another outage. So, the example I've shown here is issue ticket enrichment. But this context layer can actually go a lot further. For example, coding. Everyone here does coding with a cloud code or codex. Using an Unblock context engine to actually hydrate the agent plan goes a long way in terms of saving context and tokens. Code review. It makes the PRs look as if they've been reviewed by an expert-oriented team. Who doesn't like that? As well as surfacing the correct answers to your customer success team as well as sales. In general, there are many instances where you might want an agent to have institutional and tribal knowledge of your organization. I just want to leave you on this. I think this quote encapsulates what we're trying to solve at Unblocked. The gap isn't intelligence, it's context. So, thank you. I'll be at booth P16 along with the rest of my team if you guys have any questions. There will be additional breakout sessions later tomorrow, I believe, that go a lot more in-depth about actually how the context engine works and how you can benefit from that. Cheers. Cheers yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay So as we all know, we're running more and more agent-generated code, as well as third-party code. This gets all run on your infrastructure, and due to that, there are some complexities. Because of that, we want to introduce isolated sandboxes, which help prevent unnecessary reads of environment secrets, unnecessary network access. You know, just in general, we don't want to take down the shared host. And then, observability. How do we answer the question, where did this fail? Typically, this includes tracking logs and traces from across half a dozen systems. Everything I've mentioned here, none of this actually improves an agent's capabilities. They're all taxes one has to pay in order to get an agent out there to play the game. Thankfully, things have changed quite a bit. The whole ecosystem has matured quite a bit, and cloud infrastructure players such as Cloudflare, Vercel, AWS, they've gone and taken some of that complexity away and built primitives that these frameworks, Flu, VercelEve, Mastra, with these together, you know, they've taken a lot of complexity away, and you can focus more on building the actual agent itself. The core logic that actually helps you and your team and your customers. So, here's an example of one. I've played around with Flu and Cloudflare. And as you can see on the left-hand side, you know, we basically handle everything as mentioned before. So, the primitives plus the framework lead to a situation where it's actually not that much code to define an agent. One of the things I was shocked at when I first took a look at the documentation. To get into details, all you really have to do when defining an agent is, A, deciding which model you want to use, B, the instructions or, you know, the system prompt, C, the tools that you want the agent to have access to, skills, the things that I can do, as well as the sandbox location, where things are being run. So, to give you an example of this, I've actually gone and built out an issue enrichment system, specifically for linear. So, what this does is, given a linear ticket and access your code repository, it'll go out, you know, fetch a linear ticket, determine whether or not it's a feature or a bug. From there, it'll do some code searching, provide all that context to the agent, and then come up with a plan of next steps. On the left-hand side here, this is an issue that one of my colleagues, a support engineer, had posted, I think, a month ago. To summarize it, what had happened was, we had some pretty serious degradation in our agentic QA pipeline. Time to first character was taken three to four seconds, when it should realistically be in the hundreds of milliseconds. So, let's see what happens when, you know, we put this through the system. So, as you'll see here, I've set up the agent to go fetch an agent. I've given it the skills and tools to actually go and fetch the code, search a code, and query against that. That's being passed back to the agent, which is doing some reasoning against that right now. And then, just wait a little bit, at this point, we've updated the linear issue ticket. The recommendation here is to re-enable our async dispatch, which makes sense. It allows us to run a lot more of our QA pipeline in parallel on a single machine. Sounds great, right? Unfortunately, this is wrong. This had actually caused an outage a few days ago, and one of our support engineers had explicitly disabled this before the ticket was shown. So, where did things go wrong? Why was it, you know, why did it get it wrong? The agent I had written, it didn't have a full picture. It was missing the context from the Slack discussion that happened after the issue, where the engineers came together, went through the actual outage, what went wrong, what was the fix, and the next steps. It also was missing the post-mortem linear ticket, which came as a result of that. In general, it had a narrow understanding of the problem. This concept of missing knowledge and intent that's sorted across an organization and different systems, it's something that comes back in the back again. And since this was deployed as a background agent, this is going to make that mistake, silently in the background, misinforming both my teammates and potentially other agents. So, I guess the next question is, why don't we run into this locally? You know, we all use agents locally. We don't necessarily run into these issues. Well, you, the human, the engineers, we currently act as that context layer. When working with an agent, you know, you're there to ask questions, catch any errors, and supply the missing facts on every single turn. A person knew why the code is the way it is, what broke last time, and what we've decided to do about it. The agent, though, it only has what's on the right-hand side, right? It has instructions, the tools and skills we specifically gave it, the code, as well as the ticket in front of it. So, when an agent is in the loop, oh, sorry, when a human is in the loop with the agent, we're there to catch this fear. Ultimately, we're there to babysit the agent. But as agents have gone trivially easy to deploy, as I've shown earlier with Flu and Cloudflare, without the human in the loop, this issue becomes more and more prevalent. This missing context becomes a silent failure. You know, all that intuition and knowledge that we've had as humans needs to be replaced. Something needs to carry the load. So, that thing, that's a context engine. A context engine is a system that provides task-relevant information based on who you are and what matters. It also resolves all the conflicts across multiple data sets. It understands your access roles or the agent's access roles and only respects that and only provides information that's relevant. And most importantly, it delivers a synthesized understanding that an agent can act on, not just a list of documents that it has some reason upon itself. So, how does this context engine work? Well, let's take a step back. What does an agent actually need? An agent needs, clearly, it needs context outside of the context. It's not just your source code. Think about everything that you need to work day to day. It's not just the code. It's, you know, the Slack discussions where decisions are made, the documentation where we show all the best practices. All that is important to your day-to-day process, and that's true for your agent as well. So, what we do here is we connect everything. The docs, code, tickets, conversations. We then build a model of your organization, of your system, and we piece how all of these things work together and make it generally available to your agents. From that model, the agents are only provided a slice of that data which has been reconciled, ranked, and scoped to your permissions. Scattered context comes in, grounded context comes out. The obvious next question is, why can't we just do this with MCP, right? You could connect a Slack MCP, a Linear MCP, a GitHub MCP, and with that, all that data is accessible. MCP is great at access, but access isn't understanding. An MCP hands the agent the raw results, and you're now dependent on the agent to actually decide what to believe in. You end up flooding the agent with irrelevant data, filling up the context window, and overall context costs just go up. It also leaves a local agent to handle conflicts in data. Your Linear MCP and your Slack MCP may come back with different results. You're just leaving the agent to make that decision somewhat ad hoc at this moment. So, back to the original problem I had earlier. This is the same file, same agent, but now we've connected the context agent. What we do here is we're currently prompting Unblock to do some research on the ticket and provide that context to the agent. So, let's see that in action. Sorry about that. So, here we go. We're doing a very similar thing. We're fetching the Linear ticket. Well, you'll notice here that we're actually calling the Unblock context engine. And what it's done here is actually it's found the relevant linear post-mortem as well as a Slack conversation where we've had the entire discussion between the engineering teams. And as part of that, we've returned an understanding, and that's now been provided to the agent as a summary. So, the agent no longer has actually reason from those documents. And at this point, you'll notice here, the agent now has been updated. The recommendation has gone from breaking and causing another issue to actually preventing another outage. So, the example I've shown here is issue ticket enrichment. But this context layer can actually go a lot further. For example, coding. Everyone here does coding with a cloud code or codex. Using an unblocked context engine to actually hydrate the agent plan goes a long way in terms of saving context and tokens. Code review. It makes the PRs look as if they've been reviewed by an expert-oriented team. Who doesn't like that? As well as surfacing the correct answers to your customer success team as well as sales. In general, there are many instances where you might want an agent to have institutional and tribal knowledge of your organization. I just want to leave you on this. I think this quote encapsulates what we're trying to solve at Unblocked. The gap isn't intelligence, it's context. So, thank you. I'll be at booth P16 along with the rest of my team if you guys have any questions. There will be additional breakout sessions later tomorrow, I believe, that goes a lot more in-depth about actually how the context engine works and how you can benefit from that. Cheers. Cheers yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay yay