Continual Learning for AI Agents: From Failures to Durable Improvements - Soheil Feizi, RELAI

Transcript

Hi everyone, my name is Sohel Feizzi. I'm founder and CSO at RELi. I'm also an associate professor in the computer science department at University of Maryland. Today I'm going to talk about continual learning for AI agents. How we can go from failures to durable improvements. And if you're interested in any of the tools that I'll be talking in this presentation, you can visit our website RELi.ai. Let's get started. Humans learn mainly from experience by interacting with the world and getting feedback. The goal of continual learning is to imitate the same for agents so they can also learn from experience by acting, getting feedback and improving without forgetting. So here's a bigger picture of how continual learning for agents looks like. An agent interacts with the world, with diverse users, with complex tools, with various data policies. And as I mentioned, the goal is to continuously improve the agent from its experience without forgetting. And this learning can happen in different layers of the agent. It can happen in the model layer where potentially we can change weights of LLMs or other models used in the agent or use different types of models in the agent. It can happen in the harness layer where it brings the proper context to the LLM with components like prompts, skills, tools, code, workflow. And it can also happen in the memory, either in session memory or persistent memory of the agent. So there are two fundamental challenges in continual learning for agents. The first challenge is how to get feedback. How do we know if the agent did well? And if not, what should it have done instead? That's the first part, getting the feedback. And the second part is how we can act upon that feedback, how we can optimize and improve the agent and learn from that feedback. Which layer, which component do we need to change? And also how? I'll be talking about these two challenges, current approaches in order to deal with them and also provide some perspective of how we think about these two problems. So let's get started with the first problem. Where does the feedback come from? The easy case is when we have a benchmark and some evaluators on that benchmark. The agent can run tasks from the benchmark. Now we have the evaluators in order to score and we can get grades like pass, fail or reward, as well as potentially some feedback on the agent behavior and agent performance. This is usually what is happening during development time where different teams curate benchmarks in order to understand the performance of the agent in certain applications. But in production, we don't have such benchmarks. We have logs. Here's an example of a session log where a user is interacting with the agent. Maybe the user is not very happy with the way the agent is behaving, but we don't have any explicit feedback. So there are two ways of getting such feedback on such session logs. One is automatic using some other models or LLMs or code in order to analyze the log and provide feedback. In some cases, even the agent itself can look at its log and provide some criticism of it. It is automatic and scalable. And the second approach is where we have human experts to look at some handful of these logs and provide domain expert feedback on those agent outputs. This is lowering the volume, but it is critical because it provides expert knowledge on the behavior of the agent and its alignment with the way that we want the agent to behave in those applications. Either way, now we have session log plus some feedback on those logs. Is it enough? The answer is no, because it is still not testable. Here we have log and feedback, but what we really need is a replayable learning environment, a simulation that we can rerun with defined grading on what success looks like, not one instance of what happened and the feedback on top of it. So what is a learning environment? Here we are inferring a distribution from one observation that replaces what happened and what success means. The input is what we have, some session logs and feedback. This is one observation of what happened. And now we want to create a simulation and evaluation environment from that information. That involves, for example, understanding how tools in the agent behavior in the agent log should behave. Should we use real tools? Should we use mock tools? And if so, what kind of data should be brought to the mocking process of those tools? If the agent is interacting with some users, how we can infer synthetic users from that data, and also how success looks like in that learning environment, what are the evaluators that need to be inferred? So there are lots of technical challenges in any of these components. But if we could do this successfully, the good news is the output is executable. We can now run different candidates of the agents against such learning environments, understand the behavior and the performance of the agents in those scenarios and in those patterns. And we can fix the issues based on the feedback based on the information that we observe because not everything becomes testable and verifiable. All right, so the second problem is now we have this feedback, how we can act upon it, how we can optimize the agent. And from a high level point of view, there are three layers that we can improve the agent. There's a model layer where we can change the weights of the model and there are methods like SFT, supervised fine tuning, RL based post training in order to make those changes. And these are usually expensive because that requires more intensive compute in terms of changing the model weights. We can change the model weights. The second layer is the harness layer, harness engineering, context engineering, where we can potentially rewrite prompts, maybe learn some skills, change tools or add tools, maybe change code around the LLM. And there are different methods like GEPA, trace to harness, which provides a lot of flexibility in terms of learning from that feedback. And the last layer is the memory layer where we store facts and learn skills in order to not repeat those issues and failures in the future. But good learning is not going to be focusing on any of these components exclusively. A good learning engine should ask for the smallest durable change at the right layer of the agent. All right. So let me dig a little bit deeper into each of these layers. First, in terms of updating the model weights, there are various approaches in order to do that, including SFT supervised fine tuning, where we imitate correct trajectories. We often need labeled samples in order to fit the model to those samples. Other approaches are based on RL reinforcement learning post-training, like DPO, GRPO, RLHF, where we sample and score against reward or preference signals and reinforce what wins. And there are some categories based on LoRA and adaptation that limits the set of parameters that can potentially change. It makes learning in this layer cheaper and also safer in terms of the updates. But these methods, they usually need benchmarks and explicit evaluators. They cannot be directly applied on, for instance, if you have a log and feedback, unless we turn those into replayable learning environments. So let's look at the next step. We often need labeled samples in order to fit those samples, fit the model to those samples. Other approaches are based on RL reinforcement learning, post-training, like DPO, GRPO, RLVR, where we sample the score against the reward or preference signals and reinforce what wins. And there are some categories based on LoRa, LoRa, LoRa, and adaptation that limits the set of parameters that can potentially change. It makes this, the learning in this layer cheaper and also safer in terms of the updates. But these methods, they usually need benchmarks and explicit evaluators. They cannot be directly applied on, let's say, if you have a log and feedback, unless we turn those into replayable learning environments. So let's look at the next step. In terms of updating the harness, I would highlight the two categories in this domain, in this layer. One is trace to harness approaches. Let's say you observe a log, you have some feedback on top of it, you can effectively ask a coding agent in order to analyze the log and improve the agent. So this works on the case where we have log and feedback, but it is white box based. We don't know if even for that particular sample, if the change is effective, because it is not testable. And we don't know what is the impact of it on other samples and other scenarios, what might have been working previously, but these changes might not work properly and create some hidden regressions. The other category that I want to highlight is methods like GEPA and PROMPT SEARCH, where they mutate prompts, they score different candidates and they keep the winners using some search algorithms like evolutionary algorithms. These methods are testable, but they need benchmarks and explicit evaluators in order to have those scorings. And in the memory layer, we effectively write down facts and distill skills so the agent doesn't rediscover them. It can happen in the information memory layer, methods like Letta and Mem0, where they can effectively store a fact or correction. And we have also methods through skill distillation that compresses successful trajectory into reusable how-to scale for the agent. So this layer in terms of the update is cheapest and fastest. It works directly on the cases where you only have log and feedback, but usually it is unverified because you don't have a way in order to test whether or not writing in the memory will resolve the issues that you have dealt with and whether or not it can potentially create some regressions on some other cases. With that, let me introduce a new subcategory of continual learning called verifiable continual learning. In a verifiable continual learning, the goal is to improve an agent from its own experience where every fix is proven to help and proven to break nothing that already worked. And usually it involves three steps where we need to have an executable test where the failure becomes a task you can replay and create. Then we need to have a measure delta where the update is scored on the test before and after. And then we have a regression test. So prior tests still pass even after we make such changes to the agent. So let's think about what are the principles of a practical verifiable continual learning first. And I will argue there are four important principles that we need to keep in mind. So the first principle is replayability. We need to turn a one-off failure into a test that we can rerun. Here, as I had mentioned previously, many cases we have log and feedback, but that is not testable. We need to lift it in a learning environment to simulate and evaluate the agent on a similar pattern, on a similar scenario. So everything becomes testable based on that simulation and evaluation environment. So that's the first principle that we need to have. The second principle is holisticness. One failure may have several causes and several possible repairs. Let me give you an example. Let's say you have an agent that uses a stale policy and skips the required escalation. The issue might come from the memory where you have some stale fact. It can come from not optimized prompt. It might come from a tool that doesn't normalize the policy. It might come from the workflow that we need to add escalation gate before we find it. It might come from the model. Maybe the model is not a good model in order to have a strong reasoning. So here we need to route the fix to the right layers that explains the failure with the smallest durable change to the agent. And that is the principle of holisticness in verifiable continual learning. The third principle is lifelongness. A new fix must improve the new case without breaking the past. Next, let's consider this setup where we already optimize the agent on K past learning environments. And a new failure comes and we turn that into a learning environment, E K plus one. What do you want to change? So the first approach is, okay, just focus on this new learning environment. But that can create regression on the past behavior on the past learning environments that the agent was successful. A better approach is a regression-aware learning where the regression is not treated as a post-hoc approach, but as a mechanism within the optimization itself. So here we are fixing the recent failures subject to having no regression of the past learning environments. And obviously this needs to be done in an efficient manner. So it doesn't scale even linearly with K because K can grow and the complexity of this approach can be very high. And the last but not least principle is efficiency. This continual learning loop needs to run frequently. And we need to have efficiency in different layers in updates to the agent. So sometimes the change can be cheap, like writing something in the memory can be medium in terms of the complexity by changing the prompt or harness. And sometimes it can be very expensive by changing the weight of the model. Also efficiency should be in the optimization loop itself, especially when we have regression-aware optimization. And regression is treated within the loop, not as a post-hoc approach. To sum up, these are the four principles of a practical verifiable continual learning. Replayability, holisticness, lifelongness, and efficiency. And this is what we have been working on at Rely to create a verifiable continual learning engine for AI agents based on these four principles. In particular, here's how Rely's learning loop runs. You can start with some signals to this loop. It can be logs, feedback, or even instructions and prompts. We lift those signals to replayable learning environments. So that's based on the replayability principle. This makes everything that follows testable and verifiable. Then we do root cause analysis and route the fixes to the right layer of the agent. It can be memory, it can be model, or it can be harness. So that touches the holisticness principle that I described. We have regression-aware optimization. Regression is not being treated as a post-hoc approach. So that touches the lifelongness principle that I mentioned. And obviously, this loop should run efficiently. That touches the efficiency principle. So the output of this is a reviewable version update to the agent, explaining what changes in the agent during this loop and why those changes are improving the agent without creating regression. So the beauty of it is you can add a VCL, verifiable continual learning, to your current agent in just two comments. So the first one is a one-time setup. So that touches the holisticness principle that I described. We have regression-aware optimization. Regression is not being treated as a post-hoc approach. So that touches the lifelongness principle that I mentioned. And obviously, this loop should run efficiently. That touches the efficiency principle. So the output of this is a reviewable version update to the agent, explaining what changes in the agent during this loop and why those changes are improving the agent without creating regression. So the beauty of it is you can add a VCL, verifiable continual learning, to your current agent in just two comments. So the first one is a one-time setup. You create a learning harness in your agent. You can use your own LLM and your agent can be built on top of any of available major agent frameworks. And then after that, you need two commands in order to activate this learning group. You can create learning environments using various type signals that you can have either log feedback or some instructions. And then you can call RelyOptimize in order to use holistic lifelong optimizer to improve the agent. And the output is an optimized version pull request that you can review and you can use it in order to improve your agent. So let's look at how it actually works in practice. We build a continual learning benchmark on a fictional support agent case, where we have reproducible testbeds for continual learning in a tool using support agent. So we have a single source of truth and the policies are interacting for this agent to be handling. So we have deterministic evaluators and we also build this benchmark in a way that it has some regression traps. So if the optimizer focuses on overfitting on the latest fix, it can potentially break what the agent was previously successful on other tasks. So let's say we have an agent. We don't even have logs or anything. And we want to just see how the agent is behaving. Let's say when a caller is rude and adversarial. So simply we can create a learning environment using such instruction. And what it will do, it will create a learning environment to simulate and evaluate the agent. The simulator will include personas, intent, mock or real tools. And also the learning environment contain evaluators in order to define success metrics. All of these are produced from just one interactive comment. And after that, we can just simulate the agent using this learning environment. See how it behaves. Okay, the score is not too high. It is 78%. And in particular, there are two evaluators that show very low scores of agent in this environment. So these are some of the failures that we observe. How to improve such failures. We can do that by calling rely optimize with certain number of rollouts. And as you can see, the average improvement can be quite high. It is 10% improvement on average, just with one loop. And the score increases to 97% from 87%. Okay. Now, let's consider the case that the agent is in production. Now you have a log, you have an agent session that is not desired and you have feedback. For example, you can say keep fast eligible refunds, but do not generalize generosity beyond refund thresholds. So that's feedback on the agent behavior. Again, the flow is the same. So we lift it into a replayable learning environment and we can rely optimize in order to mitigate this issue. Use this feedback without creating regression of the agent behavior in past environments. [SPEAKER_00] And this is lifelong. So you can keep doing that to improve the agent without breaking what already works. And it is compounding. This is verifiable continual learning in practice, where each update is tested, every gain is measured, and nothing that already works breaks during this optimization. So that's it for today and for this talk. So there are three key takeaways that I want to highlight. The first one is agent continual learning is not necessarily model fine tuning. The updates and many useful updates can happen in the harness and memory layer. So the second takeaway is production logs are not learning environments. We need to transform them into replayable learning environments to simulate and evaluate the agent on the same patterns and scenarios. And the third takeaway is that the frontier is regression-aware continual improvement. Where when fixing the new failure, we verify that we don't forget the old ones. We don't create regression. So that's verifiable continual learning built on four principles, replayability, holisticness, lifelongness, and efficiency. And if you want to try VCL and apply to your agent, you can use it today at rely.ai. Thank you. And then after that, you need two commands in order to activate this learning group. You can create learning environments using various type signals that you can have either log, log feedback or some instructions. And then you can call RelyOptimize in order to use holistic lifelong optimizer to improve the agent. And the output is an optimized version, pull request that you can review and you can use it in order to improve your agent. So let's look at how it actually works in practice. We build a continual learning benchmark on a fictional support agent case, where we have reproducible testbeds for continual learning in a tool using support agent. So we have a single source of truth and the policies are interacting for this agent to be handling. So we have deterministic evaluators and we also build this benchmark in a way that it has some regression traps. So if the optimizer focuses on overfitting on the latest fix, it can potentially break what the agent was previously successful on other tasks. So let's say we have an agent. We don't even have logs or anything. And we want to just see how the agent is behaving. Let's say when a caller is rude and adversarial. So simply we can create a learning environment using such instruction. And what it will do, it will create a learning environment to simulate and evaluate the agent. The simulator will include personas, intent, mock or real tools. And also the learning environment contain evaluators in order to define success metrics. All of these are produced from just one interactive comment. And after that, we can just simulate the agent using this learning environment. See how it behaves. Okay, the score is not too high. It is 78%. And in particular, there are two evaluators that basically show very low scores of agent in this environment. So these are some of the failures that we observe. How to improve such failures. We can do that by calling rely optimize with certain number of rollouts. And as you can see, the average improvement can be quite high. It is 10% improvement on average, just with one loop. And the score increases to 97% from 87%. Okay. Now, let's consider the case that no, the agent is in production. Now you have a log, you have an agent session that is not desired and you have a feedback. For example, you can say keep fast eligible refunds, but do not generalize generosity beyond refund thresholds. So that's a feedback on the agent behavior. Again, the flow is the same. So we lift it into a replayable learning environment and we can rely optimize in order to mitigate this issue. Use this feedback without creating regression of the agent behavior in past environments. And this is lifelong. So you can keep doing that to improve the agent without breaking what already works. And it is compounding. This is verifiable continual learning in practice, where each update is tested, every gain is measured, and nothing that already works breaks during this optimization. So that's it for today and for this talk. So there are three key takeaways that I want to highlight. The first one is agent continual learning is not necessarily model fine tuning. The updates and many useful updates can happen in the harness and memory layer. So the second takeaway is production logs are not learning environments. We need to transform them into replayable learning environments to simulate and evaluate the agent on the same patterns and scenarios. And the third takeaway is that the frontier is regression-aware continual improvement. Where when fixing the new failure, we verify that we don't forget the old ones. We don't create regression. So that's verifiable continual learning built on four principles, replayability, holisticness, lifelongness, and efficiency. And if you want to try or VCL and apply to your agent, you can use it today at rely.ai. Thank you.