AI Engineer

Scaling up Continual Learning — Ronak Malde, Trajectory

2076 summary words 9 min summary Watch video

Start with the signal

9 min read

Summary

At-a-Glance

  • Verdict: Watch fully
  • Core thesis: On-policy self-distillation can turn production agent traces into dense, token-level learning signals without GRPO's costly parallel rollouts, but scaling it to long-horizon agents requires explicit controls for trajectory divergence and hint leakage.
  • Why it matters: This is directly relevant to building agent systems that improve from real-world usage: it frames a possible learning/control plane architecture, identifies why standard RL pipelines are mismatched to production, and surfaces concrete failure modes for long tool-use trajectories.
  • Best use: Use it as a technical and strategic briefing on continual-learning architecture—especially the tradeoffs among online task data, on-policy sampling, rollout parallelism, dense rewards, trace quality, and harness co-evolution.

Executive Summary

Ronak Malde argues that frontier post-training is over-invested in increasingly expensive benchmarks and RL environments even though deployed models already generate vast quantities of production evidence: traces of what agents did, where they failed, and where they succeeded. His proposed next step is continual learning, in which that online evidence becomes training data so software improves through use rather than only through periodic benchmark-driven retraining.

His central technical proposal is on-policy self-distillation policy optimization (OPSD). A model first generates its own rollout; a version of the same model given privileged information—such as environment feedback, a worked solution, or guidance derived from logs—acts as a stronger teacher. The training objective makes the unassisted student match the teacher's token probabilities. In theory, this preserves on-policy learning while replacing GRPO's multi-rollout, sequence-level reward setup with one rollout and dense, vocabulary-wide token-level supervision.

The talk is most valuable because it does not present short-horizon results as solved continual learning. Malde says OPSD becomes unstable at roughly 120B+ scale and 50–100 tool calls: teacher and student trajectories diverge, evaluation variance rises, and tool-call formatting degrades. He proposes step-level divergence weighting to train only reliable portions of a trajectory, and residual guidance plus hint filtering to prevent the teacher's privileged information from leaking an answer that the student could not realistically infer.

Trajectory's product thesis is a control plane that ingests production agent traces, optimizes both model and harness behavior, and redeploys improvements. However, Malde explicitly characterizes the present state as "pseudo-continual learning": updates are still batched offline and then redeployed, while truly online aggregation across thousands of simultaneous product rollouts remains unresolved.

Key Takeaways

  • Claim: Benchmark-centric post-training is increasingly expensive and poorly aligned with deployed AI behavior; production inference is the more relevant source of continual-learning signal. | Evidence: Malde says benchmark tasks now take four hours, six hours, 24 hours, or days, while labs spend heavily on RL environments and data. In contrast, he claims models consume hundreds of trillions of inference tokens daily, generating real-world traces of successes and failures. | Implication: Ken should treat trace capture, outcome attribution, and feedback-quality design as strategic assets—not merely observability infrastructure—for any agent product intended to improve over time. | Caveat: Production traces are not automatically usable training data; the rest of the talk shows that their learning signal can be biased, leaky, or unstable without careful algorithmic and systems controls.
  • Claim: GRPO's strengths—on-policy sampling and strong performance—come with structural drawbacks for continual learning: offline task distributions, multiple parallel rollouts, environment replication burden, and sequence-level rewards. | Evidence: The speaker contrasts SFT, DPO/RLHF, and GRPO across four criteria: task distribution, on-policy sampling, parallelism, and reward granularity. GRPO requires groups of rollouts such as four or eight per task, depends on robust copies of environments, and assigns an end-state reward across an entire sequence. | Implication: For tool-using agents, the cost of producing comparable environment rollouts may dominate model-training cost; reducing rollout multiplicity is an architectural advantage only if feedback remains trustworthy. | Caveat: The critique is a conceptual framing by the speaker rather than a controlled comparison across all RL variants; GRPO remains his stated baseline for powerful on-policy post-training.
  • Claim: OPSD replaces an external stronger teacher with the same model conditioned on privileged information, allowing a single on-policy rollout to yield dense token-level training targets. | Evidence: The student generates a rollout from the original prompt. The same model receives a hint—e.g., a golden solution, environment information, or guidance—and serves as the teacher; training matches the unhinted student's log probabilities to the hinted teacher's. Malde emphasizes optimization over the full vocabulary, illustrated as roughly 65K tokens, rather than only reinforcing sampled actions. | Implication: A production continual-learning stack needs a distinct privileged-information pipeline: it must transform post-hoc outcomes, logs, and human feedback into guidance that helps identify better next-token decisions without becoming an answer key. | Caveat: The teacher is only meaningfully better when the privileged information is useful and appropriately constructed; an invalid or overly revealing hint undermines the objective.
  • Claim: For short-horizon tasks, OPSD may outperform GRPO while reducing inference-time reasoning length and rollout infrastructure needs. | Evidence: Malde reports that on LiveCodeBench, GRPO saturated around Sonnet-level performance in his experiments, whereas OPSD reached new result territory across datasets. He also says difficult-task token counts "collapse" under OPSD, unlike RL's tendency to improve by spending more thinking tokens, and cites OpenClaw RL as a plug-and-play route for short-horizon chatbots learning from unstructured behavior data. | Implication: OPSD is worth evaluating first on bounded workflows with clear feedback and limited tool depth, rather than treating it as proven for full autonomous-agent operations. | Caveat: No quantitative charts, model configurations, training budgets, or independent results are provided in the transcript, and the favorable claims are from Trajectory's presentation.
  • Claim: Long-horizon OPSD fails when an unassisted student trajectory diverges from the hinted teacher, causing the teacher to repeatedly course-correct and pushing the model toward compromised, low-quality behavior. | Evidence: At 120B models and tasks involving 50–100 tool calls, Malde reports high run-to-run evaluation variance and tool-call format errors. He calls the resulting tendency to overemphasize hedging/correction tokens such as "but," "wait," and "maybe" the "butt weight problem." | Implication: Do not train every token of a long production trace equally. The learning system must identify where teacher/student alignment is credible and avoid backpropagating corrections far downstream from an already-invalid agent state. | Caveat: The failure description is qualitative, and the transcript does not specify thresholds, ablations, or whether it generalizes across model families and agent environments.
  • Claim: Step-level divergence weighting is proposed as a remedy for long-horizon instability: use student-teacher KL divergence at each tool-use step to decide which portions of a trace should influence training. | Evidence: Instead of applying KL only as a penalty, Trajectory uses divergence to multiply token weights by step. In a heavily diverged trajectory, the proposed approach may train the early recoverable step while downweighting later invalid steps; if an agent returns to a plausible trajectory later, that segment can receive moderate weight again. | Implication: A continual-learning control plane should persist step-level state, tool outcomes, and model-distribution diagnostics—not only final task success/failure—so it can segment traces and assign differentiated training eligibility. | Caveat: This is presented as Trajectory's solution rather than a broadly validated standard, and it introduces another weighting policy whose calibration could itself determine training quality.
  • Claim: Hint leakage is OPSD's analogue to reward hacking: if privileged guidance reveals the answer or a fact unavailable to the original agent, the model can learn unrealistic shortcuts rather than valid reasoning. | Evidence: In a last-three-digits example, a hint reveals that the answer is 000; the rollout then works backward from that answer. In a support example, an expired SSO token is the actual diagnosis; Malde proposes converting it into permissible guidance such as investigating logs rather than revealing the solution. He also describes residual guidance: compare partial-hint and full-hint teacher distributions, then combine them to temper shifts that are too far outside the student's original distribution. | Implication: For agents with access to sensitive production state, separate information that can legitimately guide a policy from information that only exists after resolution. Audit for hindsight leakage before accepting traces into a training loop. | Caveat: LLM-based hint filtering is described as a practical but only "decent" solution; robust construction and verification of non-leaky hints remain open technical work.

Detailed Brief

What "continual" currently means operationally

  • Claims: The speaker does not claim real-time, per-interaction model updating is solved.; Current practice is "pseudo-continual learning": batch updates occur offline, followed by model redeployment.; True continuous adaptation requires both better algorithms and infrastructure for merging learning signals from large volumes of concurrent product rollouts.
  • Evidence: In Q&A, Malde uses the example of 10,000 simultaneous product rollouts whose updates would need to be collected and merged.; Trajectory positions its platform as taking production traces in, optimizing them in a self-serve loop, deploying updated systems, and managing the process through a control plane.
  • Caveats: The presentation provides no latency, retraining cadence, rollback, evaluation-gating, or data-governance design for this operating loop.; Offline batch updates create a window in which agents continue making the same mistakes before an improved policy is deployed.
  • Implications: Model improvement should initially be designed as a gated release pipeline, not as autonomous instant adaptation.; The core systems problem is aggregation and safe deployment of learning updates at product scale, not just selecting an optimization objective.

Model–harness co-evolution is a separate frontier

  • Claims: Continual learning should improve not only weights but also the agent harness: prompts, tool interfaces, routing, policies, and execution scaffolding.; The interaction between a changing model and a changing harness is underexplored and may be more consequential than optimizing either in isolation.
  • Evidence: In Q&A, Malde calls the interplay of model and harness updates an especially underexplored question and says Trajectory is exploring it with current customers.; The product description explicitly targets "model improvement, harness improvement, and the entire agentic loop."
  • Caveats: Changing both model and harness creates attribution problems: a performance gain or regression may not be assignable to one change without disciplined experimentation.
  • Implications: Maintain versioned evaluations across model, prompt/harness, tools, and policies; avoid simultaneous uncontrolled changes if the goal is to learn which intervention improved the agent.

Notable Concepts & Terms

  • Continual learning: The envisioned shift from static benchmark/retraining cycles to systems that use deployed interactions to improve models and agent behavior over time.
  • GRPO: The speaker's baseline post-training approach: it uses multiple on-policy rollouts and relative outcome-based advantage, but is criticized for rollout/environment cost and sequence-level feedback.
  • On-policy self-distillation policy optimization (OPSD): Trajectory's proposed approach: train an unassisted model rollout to match the token distribution of the same model given privileged guidance.
  • Privileged information / hint: Teacher-only contextual information such as a worked solution, environment feedback, or diagnostic guidance that makes the model stronger during training but is unavailable at normal inference.
  • Step-level divergence weighting: A proposed long-horizon safeguard that weights training segments according to student-teacher KL divergence, rather than trusting or discarding an entire trace.
  • Hint leakage: The OPSD failure mode where privileged information exposes a final answer or unavailable fact, causing hindsight shortcuts rather than behavior feasible for the deployed agent.
  • Residual guidance: A proposed anti-leakage method combining partial-hint and full-hint teacher distributions to isolate and temper overly strong, out-of-distribution guidance.
  • Pseudo-continual learning: Malde's term for the current practical state: offline batched learning updates and redeployment rather than immediate online adaptation.

Operator Notes / Why Ken Should Care

  • Instrument agents to retain replayable, step-level traces: model/version, prompt and harness version, tool calls and results, final outcome, and enough context to reconstruct whether each step was valid.
  • Create a training-data policy that explicitly separates permissible diagnostic guidance from hindsight-only resolution data; require automated filtering plus sampled human audits for leakage.
  • For long-horizon agent learning experiments, gate token/step eligibility using trajectory-confidence or divergence signals rather than training uniformly on full traces.
  • Run ablations that change model weights and harness components separately; keep a joint-change experiment only after establishing attribution baselines.
  • Treat any vendor claim of continual learning as a release-management and governance question: request update cadence, trace isolation, evaluation gates, rollback behavior, and regression monitoring.
  • Evaluate OPSD-style approaches on bounded OpenClaw or tool-use workflows with reliable outcomes before applying them to 50+ tool-call autonomous tasks.

Source/Metadata

  • Title: Scaling up Continual Learning — Ronak Malde, Trajectory
  • Transcript words: 5169
  • Duration seconds: 1383
  • Timestamp note: No timestamps or chapter markers were present. The transcript contains a repeated closing segment covering residual guidance, Trajectory's product pitch, and Q&A.
Search this video Resume transcript at 0%
Full transcript 4008 words · 23 min read
0:00

All right. Hey, everyone. Thanks for coming by over here. Hope you enjoyed the chat by Parth on how to measure continual learning. Now I'm here to talk about how we scale it up.

0:12

So a bit of background about me. I went to this company called Windsurf where I was growing the research team over there. We trained this model called SWE1 that ended up leading to the $2 billion acquisition at DeepMind, and then I ended up giving up all the acquisition money to start Trajectory, where we're building the platform for continual learning.

0:17

So, to dive in, I think it's useful to talk a little bit about what AI progress has looked like for the past couple of years. We've been in this mode where we've been rapidly scaling up benchmarks that have saturated within first years and then months. And that has continued on for the past couple of years and will continue to grow as well. The problem is it's becoming more time-consuming and more expensive. We're seeing domains where it takes four hours, six hours, 24 hours, or even several days in order to scale up benchmarks. And obviously you guys have seen the massive amount of money that the labs are pouring into scaling up RL environments and data.

0:23

Now, the thing is, we are left with a bunch of benchmarks that are getting more time-consuming, more and more expensive, and perhaps more concerningly, they're not tied to real-world use cases where people are using AI. Now, the trillion token problem here is that we are actually spending hundreds of trillions of tokens every single day on inference. And we're generating great amounts of data on how models in the real world are failing, how they're doing well, and that should be a signal that we should be capturing and training on. And this is actually how humans learn, right? We're continuously updating in the real world and getting smarter every single day.

0:29

And we're seeing now the entire field talking about continual learning, from AI experts like Ilya, Karpathy, Sholto, and Demis, to also many industry leaders as well. Satya talking about how it helps companies. And there was actually a great video by Dwarkesh a couple days ago talking a lot about the intricacies of scaling up continual learning. And obviously the goal is, we started with more data training on Internet-scale pre-training, scaling up to benchmarks, but the real unlock moving forward will be continual learning. So what's the problem? Why aren't we here yet? The main thing is there's several problems right now with our current algorithms.

0:37

So first is we have a task distribution mismatch. We're scaling up these benchmarks, which might not even be tied to reality of what people are training on. And second, oftentimes some of these methods aren't even sampling truly on policy with what's happening online. Third is we have a paradigm right now where we're sending out multiple rollouts. And this requires huge amounts of infrastructure to make sure our environments are one-to-one copies of the real world. And it ends up being a bias that we're adding to our training paradigm. And then finally, third is we are shoving every single reward into one scaler in order to train on. When the real world is messy, it's noisy, and it has rich amounts of data that should be per-token signal.

0:44

So we have all of these broken criteria, and we're training on it. And I want to take a little bit of a step through memory lane to see how we've been dealing with algorithms in the past. So the first part, we started with SFT. These were the days of instruction fine-tuning, GPT-3.5, ChatGPT. And if you take a look at these four criteria, we had parallelism solved back then. It was just one use case or example that we needed to train on. So that was a solved problem. And then the reward for SFT is per token, which is great. But we are not sampling on policy, and the task distribution is just some sort of benchmark that we've curated or some sort of data set.

0:49

Then we moved on to DPO, RLHF. This is when ChatGPT really started taking off. And we finally got online task distributions that we were able to train on. But sampling, it was a little bit better because we were actually sampling from a model, but it was still off policy. But then we ended up losing some of the key infrastructure, easy infrastructure, that we had with SFT. Now we suddenly had pairs that we were training on. And then the reward went back to sequence level.

0:53

Then we moved on to GRPO, which is the mode that we're in right now. We took a Faustian bargain and wanted to max on-policy rollouts, which is extremely powerful. We now have models that are capable of amazing things because they are on policy and able to grow and not have all of the catastrophic forgetting problems that we've had with previous methods. But on the other hand, we're working with off-policy task distributions. Our parallelism has now exploded, meaning that we need really robust environment infrastructure. And then finally, for rewards, we're back to this paradigm of training on the entire sequence.

0:59

So can we get to a world where all of these are true, where we have an online task distribution, where sampling on policy, we have one parallelism, so we don't need any of this crazy infrastructure, and then finally, our reward is actually token level? Well, this is what I'm excited to share and see how we can scale this up.

1:06

So first, let's just go through what our best algorithm of post-training is today. Hopefully, you guys are all familiar with how GRPO works at a high level. You start off with some sort of task as an input. You do a bunch of parallel rollouts. So we'll call these 01 through 04. And then you have some sort of end-state reward. So you classify these, let's say, as a couple numbers. And then GRPO works on advantage, the idea that you have some sort of mean that you're calculating over. And you're trying to shift the distribution of the model to be better on the ones that you're better on, and then obviously move away from the ones that were worse than the mean. So this is how GRPO works.

1:13

Now, let's take a look at a new continual learning algorithm called self-distillation policy optimization. Sorry, first, we'll talk through why GRPO actually is still not enough. So the first part is, if you remember, the tasks. So these benchmarks are obviously very different. It requires a bunch of parallelism. And then it's almost like you're drinking through a straw in order to get the reward. The way to think about it is, imagine you were trying to write an essay, and your teacher just gave you a score of 87 out of 100. You'd have to run through so many different examples to get to the idea of what a good essay is. And it's very simple and efficient. And so these are the fundamental problems with RL.

1:19

Now, we're going to go through a new algorithm called on-policy self-distillation. So you guys are probably familiar with distillation in general. There's been obviously a lot of talk about mythos and blocking that from happening for a lot of open-source models. The high level here is that you start with some sort of data set. It's usually off-policy or not actually tied to reality, but some sort of fixed data set. And what you have is a student model and then a smarter teacher model, so usually a bigger model. And you're trying to basically fit the log probs of the student to the teacher. And both of these are passed in with the same data.

1:24

All right. So this is normal distillation. Now, there's a new innovation called on-policy distillation. The only thing we're doing is actually swapping out the fixed data for instead a rollout of what the student would have rolled out. All right. So we take a rollout of the student. That is our trajectory. And then we're trying to fit the log probs again of a smarter teacher model to the student. Great.

1:32

Now, the problem is when we're trying to push the frontier, we don't magically have some smarter model, right, like a teacher. And so now what do we do if we're already at the smartest model? Well, here's the final algorithm. And this is where the self-distillation part comes in. If you basically take the student model and give it some sort of what we call privileged information, a hint about the world, and put that into the prompt, suddenly that student is a little bit smarter. And that's essentially the key idea of on-policy self-distillation. You take what's called this hint, put it into the beginning of the prompt, and now you match the log probs of the student without that hint to the teacher with that hint.

1:39

And this is an extremely powerful algorithm. To visualize it a little bit more, let's say you have this student prompt, right? Find the derivative of this function. What a hint would be is if you had some sort of environment information or some sort of guidance on how you should actually solve the problem. The simplest form of this is literally just an example of a golden solution. You put that into the teacher prompt and then say, here is guidance on how you would solve the problem. If you take the student model and give it what we call privileged information, a hint about the world, and put that into the prompt, suddenly that student is a little bit smarter.

1:54

And that's essentially the key idea of on-policy self-distillation. You take what's called this hint, put it into the beginning of the prompt, and now you match the log probs of the student without that hint to the teacher with that hint. And this is an extremely powerful algorithm. To visualize it a little bit more, let's say you have this student prompt, right? Find the derivative of this function. What a hint would be is if you had some environment information or some guidance on how you should actually solve the problem.

2:11

The simplest form of this is literally just an example of a golden solution. You put that into the teacher prompt and then say, here is guidance on how you would solve the problem. And you can imagine now that solving that problem from the teacher's perspective becomes a little bit easier, and we're trying to shift to those log probs had it known the answer in the first place. And so by doing this, we've solved several key problems that existed with RL. So if you remember the task distribution, now we can suddenly take something that is truly from online without needing to have benchmarks that are created.

2:32

The second part is we're still on-policy sampling, which is great. But now there's no parallel rollouts. We don't need a group of eight in order to roll out, but just from a single example, we're able to get information. So that takes away the environment bottleneck and all of these other infrastructures. And then finally, the most exciting part is we're matching every single log prob of every token.

2:46

So there's massively rich feedback about what this algorithm is doing. To see it a little more in action, and this is a really exciting part of self-distillation, it's actually not just the top token that you sampled that you're making better, but instead the entire vocabulary. So for every single token, there's a vocabulary of, let's say, 65K tokens that you're optimizing over. And let's say the model generated some sort of for loop or a range in Python.

2:59

And essentially what we're doing is saying, hey, the teacher, for some of these areas where it's blue, it wasn't the top sampled token of the student, but instead we're pushing that distribution to sample a brand new token. And this is really exciting because we're not just taking now a distribution like RL and slightly sharpening it, but we're instead actually shifting entire distributions. So this is a really exciting part about OPSD. Now, we might be saying OPSD is awesome and we've solved this problem.

3:23

For a short horizon task, it works incredibly well. If you take a look at LiveCodeBench, actually we found that GRPO saturates around Sonnet-level performance and doesn't really push the frontier. But because we're actually shifting distributions, we're able to get to brand new territory of results with a lot of data sets. And then the really cool part is with RL, a fundamental limitation, and if you guys have ever trained RL models, is the models like to think a lot, right? The more tokens that you expend, it's just going to do better. But with OPSD, you don't have that problem.

3:48

And so the actual tokens to solve some of these really difficult challenges actually collapses, which is really exciting for token efficiency. And for some of these short horizon tasks, you can actually plug and play OPSD right now with several open source projects. So OpenClaw is a really good example of this, OpenClaw RL. And you can actually use it to do simple chatbots, learn from your behaviors with unstructured data, which is super, super awesome. All right. So we might be thinking, okay, continual learning is solved. We can all go home and be super happy. The thing is, this works really well for small models, short horizon tasks, like something like a chatbot.

4:24

But this is where academic papers kind of end and where you really need to scale things up to start to see the limitations. So at Trajectory, we've been scaling up this algorithm to 120Bs, to 500Bs, to 1 trillion parameter models. And as soon as you get to the 120B range with not just one or two tool calls, but 50 or 100, things start to break apart a little bit. So first of all, eval accuracy is all over the place, right? The range is going really high. Run-to-run variance is extremely high. And then also, we start to see a lot of tool call errors.

4:58

The model is not behaving according to the format that it was trained on in the first place with the instruction fine-tuning. All right, so what are the problems that we started to see with this algorithm as we were scaling it up? So the first part is actually really funny and what I call the butt weight problem. So on shorter tasks, you're fitting to a distribution, everything's great. But when you move on to longer tasks, what you get is this student model. It's going off and doing whatever it thinks is on-policy. And then at some point, because you're so divergent in a long task, the teacher is going to try to course-correct every single time it gets a chance.

5:36

And so what you end up with is the teacher model just continuously trying to improve this token of weight or maybe or some of these hedging words. And there's this really interesting word on the bottom left that you can see, or word cloud, that as steps go on with OPSD and you really scale it up, you start to see some of these words like weight and then but start to appear. And then you actually end up in this really interesting local suboptimal position where everything just turns into maybe. And on the right here, you can kind of see some visualization of this. As the task goes on, you get two different distributions that are really divergent.

6:09

And what you end up with is the model trying to be in the middle of both of those, which is obviously really suboptimal. All right, so there's a couple of ways to solve this. One potential solution is to define step-level divergence. So in a tool-calling trajectory, let's say you have 100 tool calls going on. What we can do is start to look at the KL divergence of the student model and the teacher model as time goes on. And now we can use this as a weighting factor. So not just like normal KL where we use that as a KL penalty, but instead we're actually multiplying the token weight of every single step based on this divergence property.

6:57

And now the cool part of this is that, on the left side, we have a normal trajectory that is pretty in distribution and everything's fine, right? Everything is a weight of one. On the second part, we have a trajectory that diverges pretty heavily. And so in here, we're only going to modify W1 as the first step so that we can train that, get that right, and then move on, which is awesome. And then finally on the third part, and this is really cool by having independent weighting for every step, we can actually have a scenario where the model might go off track. We don't want to heavily weight that in.

7:34

But then later on in the trajectory, it might get back on track again, and we're fine with that and we'll mildly shift the distribution. So this is one way that we've been able to overcome this for long-horizon tool calling. The second part, and this is actually really nefarious with OPSD. So with RL, the number one problem that people face is reward hacking, as you guys are all aware of. And that's a game that continuously RL researchers have had to play. Well, there is an equivalent for OPSD as well, and that is hint leakage. The way you can think about it is we're taking this hint, right, putting it in the beginning of a prompt.

8:25

Well, if the student had no way of knowing what that hint would be, then you're going to end up with some weird scenarios and skipping some steps along the way. So the way this manifests is, let's take this example here where you're trying to find the last three digits of this formula. And a normal hint might be actually giving the correct steps and then maybe a final answer and saying, hey, actually, hint, the last three digits are all zeros. Well, you can see on the right here that when you actually roll out the model, you end up with something that says, oh, actually, I know what the solution is. It's zero, zero, zero.

9:06

So let me go back and put that into my reasoning trace and then figure out what's going on. Well, you can imagine that this is not going to occur whatsoever in the real world, and so we've ended up in this really strange position. And so there's a lot of care that you have to put into how you design these hints and making sure there's not leakage of information. So there's one kind of trivial solution to this that you can imagine, and this is just literally using an LLM to filter out these hints. So an example of this is, let's say a user can't log in. You have some sort of problem like this.

10:01

Well, you can see on the right here that when you actually roll out the model, you end up with something that says, "Oh, actually, I know what the solution is." It's zero, zero, zero. So let me go back and put that into my reasoning trace and then figure out what's going on. Well, you can imagine that this is not going to occur whatsoever in the real world, and so we've ended up in this really strange position. And so there's a lot of care that you have to put into how you design these hints and making sure there's not leakage of information. So there's one trivial solution to this that you can imagine, and this is just literally using an LLM to filter out these hints.

11:11

So an example of this is, let's say a user can't log in. You have some sort of problem like this. And there might be some sort of information, like exactly the solution, right? You'll find their SSO token that is expired. And you can have an LLM translate that into what is something reasonable that they should have known. And that's the process of looking through the logs, but not actually giving it the solution that would shortcut some of its vital reasoning. So this is one trivial solution, and it works decently well. But there are some more satisfying algorithmic approaches, too. One of these is called residual guidance.

12:23

The general idea is, in a hint, most of the time you need to actually get through the entire hint in order to get the full information. So what if you, let's say, cut it in half? Then you have a partial hint. And then you say, okay, this is the partial teacher. We're able to get the log probs of a slightly smart teacher. And then we have the full teacher as well, and that's with the full hint. Now what we can do is actually take the linear combination of both of these. And this gives us a good idea of how strong the hint is and how out of distribution it is for the original model.

13:19

And what you end up with here is, on the left you have normal OPD, where you might be entirely shifting distributions and there's almost no overlap between your model and the original solution. To this cool world where, let's say, half the hint is actually quite close to your distribution, but the full hint is very off. And you take a linear combination, and so then you're not shifting the model into unknown territory. So these are just some of the solutions to a lot of the challenges that we've had with scaling up OPSD. But it is a very, very powerful algorithm.

13:51

And with a lot of these combinations together, we're actually able to scale this up to a 120b model on Mercore Apex agents, which often requires 100-plus tool calls in order to achieve. And it's a really powerful algorithm that has even surpassed RL as well. So now we've finally arrived at an algorithm. It's not necessarily the algorithm to solve continual learning, but definitely one that is a huge step forward, that keeps the on-policy nature that makes RL so powerful. But then it also has, finally, the task distribution that is online. It has parallelism that is singular, so we don't need all of this infrastructure. And then finally, it is per-token dense reward.

14:35

So the really exciting part for us, and what we're really focused on in NetTrajectory, is this just gives you one taste of the entire continual learning loop. And there is a really exciting world that is about to come where software, in general, just gets smarter every single time it's used. And that is the most exciting unlock that's going to happen in 2026, 2027, and as we scale up. So a little bit about Trajectory. We are building the platform that turns every interaction into model improvement, harness improvement, and the entire agentic loop just getting smarter over time.

15:12

And we're building this platform where we take in agent traces data from production, we're able to optimize that as a self-serve loop, and then deploy that as a continually learning system. And we have a control plane that goes over all of that. So very quickly, our team is super awesome. We're from DeepMind, Meta, Superintelligence, OpenAI, and a lot of great product builders as well. And we've given early access to a lot of companies, so Harvey, Dekagon, Rogo, and they're super excited about what we're doing. If you're interested in any of the research that we're doing, or any of the product that we're building, definitely let me know.

15:47

Keep in touch, and happy to answer any questions. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Is it continual in the sense of very little latency so that it can come into your time, or is this something that happens? Yeah, that's a great question. So what I would say is, as a research community right now, we're in this zone of what I call pseudo-continual learning, where there's still some level of batch updates offline, and then re-uploading the model. I think it's partly an infrastructure question.

16:57

It's partly still an algorithmic question as well, of how do you truly get, when you have 10,000 rollouts going out in a product, merging those together, the infrastructure to pull all of those together. So those are some of the problems that we are solving on Trajectory. But I wouldn't say we're anywhere close to the end-to-end solution. I have a question about harness. I have a study of different harnesses. I think it's not just a little bit of a challenge. Totally. I think that's actually one of the most underexplored and most exciting questions, not only just harness improvement, right? And I think there's some literature out there now starting to explore that.

17:43

But the really exciting part is, how does the model and the harness interplay with each other as you're both updating them? That's some of the stuff that we're now exploring with our current customers and really doing those things online. But it's completely underexplored territory, and there's some really exciting innovations to be made there as well. Take all these questions outside. Cool. All right. Thanks so much, guys. We're able to get the log probs of a slightly smart teacher. And then we have the full teacher as well, and that's with the full hint. Now what we can do is actually take the linear combination of both of these.

18:22

And this gives us a good idea of how strong the hint is and how out of distribution it is for the original model. And what you end up with here is, so on the left you have normal OPD, where you might be entirely shifting distributions and there's almost no overlap between your model and the original solution. To this kind of cool world where, let's say, half the hint is actually quite close to your distribution, but the full hint is very off. And you take a linear combination, and so then you're not shifting the model into unknown territory. So these are just some of the solutions to a lot of the challenges that we've had with scaling up OPSD.

18:59

But it is a very, very powerful algorithm. And with a lot of these combinations together, we're actually able to scale this up to a 120b model on Mercore Apex agents, which often requires 100 or plus tool calls in order to achieve. And it's a really powerful algorithm that has even surpassed RL as well. So now we've finally arrived at a algorithm. It's not necessarily the algorithm to solve continual learning, but definitely one that is a huge step forward, that keeps the on policy nature that makes RL so powerful. But then it also has, finally, the task distribution that is online. It has parallelism that is singular, so we don't need all of this infrastructure.

19:41

And then finally, it is per token dense reward. So the really exciting part for us, and what we're really focused on in NetTrajectory, is this just gives you one taste of the entire continual learning loop. And there is a really exciting world that is about to come where software, in general, just gets smarter every single time it's used. And that is the most exciting unlock that's going to happen in 2026, 2027, and as we scale up. So a little bit about trajectory. We are building the platform that turns every interaction into model improvement, harness improvement, and the entire agentic loop just getting smarter over time.

20:23

And we're building this platform where we take in agent traces data from production, we're able to optimize that as a self-serve loop, and then deploy that as a continually learning system. And we have a control plane that goes over all of that. So very quickly, our team is super awesome. We're from DeepMind, Meta, Superintelligence, OpenAI, and a lot of great product builders as well. And we've given early access to a lot of companies, so Harvey, Dekagon, Rogo, and they're super excited about what we're doing. If you're interested in any of the research that we're doing, or any of the product that we're building, definitely let me know.

21:01

Keep in touch, and happy to answer any questions. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Is it continual in the sense of very little latency so that it can come into your time, or is this something that happens? Yeah, that's a great question. So what I would say is, as a research community right now, we're in this zone of what I call pseudo-continual learning, where there's some still level of batch updates offline, and then re-uploading the model. I think it's partly an infrastructure question.

21:42

It's partly still an algorithmic question as well, of how do you truly get, when you have 10,000 rollouts going out in a product, merging those together, the infrastructure to pull all of those together. So those are some of the problems that we are solving on trajectory. But I wouldn't say we're anywhere close to the end-to-end solution. I have a question about harness. I have a study of different harnesses. I think it's not just a little bit of a challenge. Totally. I think that's actually one of the most underexplored and most exciting questions is not only just harness improvement, right? And I think there's some literature out there now starting to explore that.

22:22

But the really exciting part is, how does the model and the harness interplay with each other as you're both updating them? That's some of the stuff that we're now exploring with our current customers and really doing those things online. But it's completely underexplored territory, and there's some really exciting innovations to be made there as well. Take all these questions outside.

22:41

Cool. All right. Thanks so much, guys.

Reading tools

Type to find a passage

Appearance
Ask this transcript

Notebook

Export Markdown

Add a note