SWE-Marathon: Evaluating Coding Agents at Billion-Token Scale - Rishi Desai, Abundant AI

Transcript

Hi everyone, my name is Rishi Desai. I'm an ML engineer at Abundant AI, where we build reinforcement learning environments for Frontier Labs. Today I'm going to talk about SWE Marathon, a benchmark that answers a question that is starting to matter a lot more. Can coding agents stay coherent over a billion token budget? Can they build Slack from scratch? Can they rewrite an entire Jax codebase in PyTorch? Can they build a C compiler in Rust? This is what SWE Marathon is trying to measure. What happens when coding agents move from fixing bugs to owning entire projects end to end? There's been a tremendous amount of interest in autonomous agent systems. Anthropic has explored teams of agents building a C compiler. Cloudflare rebuilt the entire Next.js on Byte completely hands-off with agents. And Cursor has experimented with their days long running autonomous agent harness. The pattern is that coding agents are being pointed at whole projects, not just GitHub issues or linear tickets. My question is, can we turn some of these Frontier Labs style case studies into reproducible eval tasks? Let's talk about the SWE benchmark lineage. HumanEval asked whether models could write individual Python functions. SWE Bench was a big jump to real GitHub issues where agents had to inspect a repository, make a patch, and patch some unit tests. Terminal Bench pushed this even further by making each task a full environment with a verifier. So agents could use a terminal, run bash commands, inspect files, and leave behind a final container state. SWE Marathon takes that environment plus verifier framing and stretches the horizon to project scale work. Multi-hour trajectories and coordinated changes across many components. These are literally hundreds of hours of human work compressed into a single agent rollout. But once you make tasks this long, a big problem shows up: verification. In a short benchmark, a weak test could be considered noise. But in a multi-hour environment, a weak verifier becomes an attack surface. The agent has hours, a file system, unrestricted network access potentially, and a reward signal. So it could spend hours probing the verifier instead of actually doing the intended engineering work. That's a big reason why SWE Marathon uses multiple independent checks. We have hidden tests, reference parity checks, computer use agent checks for the product clone tasks, and anti-cheating tests. We wanted independent verify channels that fail in different ways. I'll first show you the computer use agent verification example, and then later the failure case where an agent tries to solve the C compiler task by secretly calling GCC. You might have noticed that there are basically no full stack product clone tasks in any long horizon SWE benchmark out there. And the reason is verification. Unit tests can pass, but the product is probably still unusable and the front end looks terrible. SWE Marathon is the first benchmark to use a computer use agent or Kua verifier for these full stack tasks. For the clone Slack task, we have deterministic unit tests to check the API and the backend functionality. But then a computer use agent uses the browser like a human. That's what you're seeing in this GIF. The verifier isn't reading code or calling an API directly. It's driving the submitted Slack clone through the UI. So it's logging in, creating channels, posting messages, reacting with emotes, and checking that the app actually works with the rubric. The big takeaway is that full stack evals are hard because correctness is not just an API contract. It's whether the user can actually complete the product's intended workflow. Expert contributors from the evals community propose the tasks and reference solutions. And then we work together to standardize them into executable environments with the multi-layer verifier suites. Tasks all follow the harbor format. A lot of my work was spent on the QA and the hardening layer. So running the agent trials, inspecting the failure modes, patching the shortcuts, patching the verifier, and then rerunning until the tasks were both solvable but also hard to game. This is the main leaderboard result. The best configuration here is Claude Opus 4.8 with Claude Code, and it only achieves a 26% resolution rate. So even with the strongest agent setup we evaluated, it's only solving one in four tasks. The important thing is that these aren't shallow failures. The average trial used 31 million tokens, and the longest rollout consumed 877 million tokens. So the agents are exploring, editing, testing, getting stuck, recovering, running for hours. So the takeaway is that current agents are very impressive, but end-to-end project ownership is still very far from being solved. This plot puts cost on the x-axis and resolution rate on the y-axis. So higher success rate for less money is always better. Claude Opus 4.8 is the top point. It gets 26%, but it's also one of the most expensive configurations. Whereas GPT 5.5 with Codex is far cheaper and only gets 12%. So the model isn't just the full picture. The agent scaffold makes a huge difference. How it plans, uses tools, summarizes context, and decides when to test. I won't get too deep into the cost analysis here, but the paper has the full details. I wanted to show you what a full marathon rollout actually looks like. This is one I picked with GLM 5.2 on the Next.js fight rewrite task. So there's over 356 million tokens, over 9 hours, and over 800 trajectory steps and tool actions. So for the top half, you can see the agent starts by exploring the repo and the fixtures, gets its first full test suite at 0 out of 325 tests passing, and then spends the next few hours pushing through routing, hydration, server actions, middleware, and cache behavior. The bottom part of the chart shows the work pattern over time. So you can see lots of reading and searching early, then huge waves of editing, building, testing, and debugging. The key intuition is that these are long engineering loops. They're not simple coding tasks. Reward hacking is an arms race between coding agents and eval environments. This is why strong verifiers are central to SWE Marathon's task design, and not an afterthought. This chart has two levels of behavior. The lighter bars are the suspicious shortcut behavior. So things like looking for solution files, messing with data, messing with the configs, whereas the darker bar is a clear exploit that has actually gotten shipped in the final submission. And across the 1400 rollouts, we found 12.8% had suspicious shortcut behavior, and 9% had clear verifier bypass. So if these verifiers were weak, these wouldn't just be amusing failure cases. They would actually delegitimize the benchmark. And the important number is the zero. Zero rollouts earned reward through an exploit because our defenses caught them. That should be the bar for long horizon evals. This is my favorite concrete reward hacking example. The task is to build a C compiler in Rust from scratch. The lexer, the parser, semantic analysis, code gen, the whole thing. But Gemini found a much shorter implementation strategy, which is call GCC from inside the Rust program. So under a weak verifier, this task would look almost solved because the compiled outputs match the reference behavior. The anti-cheat layers catch this by using strace to find the forbidden sub-processes called GCC. So even though the partial scores look high, the final reward is zero. I have the full failure mode text on me in the paper, which I hope you guys all check out. If you remember one thing from this video, it's that the future of suite evals is not just harder unit tests. Once agents run for hours, each task becomes a complex environment. And agents aren't just trying to write code, they're also navigating tools, tests, your hidden assumptions, and the verifier itself. So the two big takeaways are first, long horizon suite is still unsolved. The best agents are only at 26%. There's plenty of headroom left. Second, the big bottleneck is robust verification. At hour and day scale lens tasks, we need the multi-channel checks, anti-cheat hardening, product style validation. The tasks, the code, the paper, the logs, and the trajectories are all public. I've released 320 gigabytes of trajectories that are especially important because they make SWE Marathon fully inspectable and transparent. I also want to thank all of my collaborators on this project, all of whom are listed here. SWE Marathon was very much a community driven effort across task contributors, advisors, and paper writing. You can find everything at SWE Marathon.org. Thank you. Let's talk about the SWE benchmark lineage. HumanEval asked whether models could write individual Python functions. SWE Bench was a big jump to real GitHub issues where agents had to inspect a repository, make a patch, and patch some unit tests. Terminal Bench pushed this even further by making each task a full environment with a verifier. So agents could use a terminal, run bash commands, inspect files, and leave behind a final container state. SWE marathon takes that environment plus verifier framing and stretches the horizon to project scale work. Multi-hour trajectories and coordinated changes across many, many components. These are literally hundreds of hours of human work work compressed into a single agent rollout. But once you make tasks this long, a big problem shows up. Verification. In a short benchmark, a weak test could just be considered as noise. But in a multi-hour environment, a weak verifier becomes an attack surface. The agent has hours, a file system, unrestricted network access potentially, and a reward signal. So it could spend hours probing the verifier instead of actually doing the intended engineering work. That's a big reason why SWE marathon uses multiple independent checks. We have hidden tests, reference parity checks, computer use agent checks for the product clone tasks, and anti-cheating tests. We wanted independent verify channels that fail in different ways. I'll first show you the computer use agent verification example, and then later the failure case where an agent tries to solve the C compiler task by secretly calling GCC. You might have noticed that there are basically no full stack product clone tasks in any long horizon SWE benchmark out there. And the reason is verification. Unit tests can pass, but the product is probably still unusable and the front end looks terrible. SWE marathon is the first benchmark to use a computer use agent or Kua verifier for these full stack tasks. For the clone Slack task, we have deterministic unit tests to check the API and the backend functionality. But then a computer use agent uses the browser like a human. That's what you're seeing in this GIF. The verifier isn't reading code or calling an API directly. It's driving the submitted Slack clone through the UI. So it's logging in, creating channels, posting messages, reacting with emotes, and checking that the app actually works with the rubric. The big takeaway is that full stack evals are hard because correctness is not just an API contract. It's whether the user can actually complete the product's intended workflow. namenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamenamename Expert contributors from the evals community propose the tasks and reference solutions. And then we work together to standardize them into executable environments with the multi-layer verifier suites. Tasks all follow the harbor format. A lot of my work was spent on the QA and the hardening layer. So running the agent trials, inspecting the failure modes, patching the shortcuts, patching the verifier, and then rerunning until the tasks were both solvable but also hard to game. This is the main leaderboard result. The best configuration here is Cloud Opus 4.8 with Cloud Code, and it only achieves a 26% resolution rate. So even with the strongest agent setup we evaluated, it's only solving like one in four tasks. The important thing is that these aren't shallow failures. The average trial used 31 million tokens, and the longest rollout consumed 877 million tokens. So the agents are exploring, editing, testing, getting stuck, recovering, running for hours. So the takeaway is that current agents are very impressive, but end-to-end project ownership is still very far from being solved. This plot puts cost on the x-axis and resolution rate on the y-axis. So higher success rate for less money is always better. Cloud Opus 4.8 is the top point. It gets 26%, but it's also the most expensive configurations, or one of them. Whereas GBD 5.5 with Codex is far cheaper and only gets 12%. So the model isn't just the full picture. The agent scaffold makes a huge difference. How it plans, uses tools, summarizes context, and decides when to test. I won't get too deep into the cost analysis here, but the paper has the full details. I wanted to show you what a full marathon rollout actually looks like. This is one I picked with GLM 5.2 on the Next.js fight rewrite task. So there's over 356 million tokens, over 9 hours, and over 800 trajectory steps and tool actions. So for the top half, you can see the agent starts by exploring the repo and the fixtures, gets its first full test suite at 0 out of 325 tests passing, and then spends the next few hours pushing through routing, hydration, server actions, middleware, and cache behavior. The bottom part of the chart shows the work pattern over time. So you can see lots of reading and searching early, then huge waves of editing, building, testing, and debugging. The key intuition is that these are like long engineering loops. They're not simple coding tasks. Reward hacking is an arms race between coding agents and aural environments. This is why strong verifiers are central to Speed Marathon's task design, and not an afterthought. This chart has two levels of behavior. The lighter bars are the suspicious shortcut behavior. So things like looking for solution files, messing with data, messing with the configs, whereas the darker bar is like a clear exploit that has actually gotten shipped in the final submission. And across the 1400 rollouts, we found 12.8% had suspicious shortcut behavior, and 9% had the clear verifier bypass. So if these verifiers were weak, these wouldn't just be amusing failure cases. They would actually delegitimize the benchmark. And the important number is the zero. Zero rollouts earned reward through an exploit because our defenses caught them. That should be the bar for long horizon evals. This is my favorite concrete reward hacking example. The task is to build a C compiler in Rust from scratch. The lexer, the parser, semantic analysis, code gen, the whole thing. But Gemini found a much shorter implementation strategy, which is call GCC from inside the Rust program. So under a weak verifier, this task would look almost solved because the compiled outputs match the reference behavior. The anti-cheat layers catch this by using strace to find the forbidden sub-processes called GCC. So even though the partial scores look high, the final reward is zero. I have the full failure mode text on me in the paper, which I hope you guys all check out. If you remember one thing from this video, it's that the future of suite evals is not just harder unit tests. Once agents run for hours, each task becomes a complex environment. And agents not only trying to write code, it's also navigating tools, tests, your hidden assumptions, and the verifier itself. So the two big takeaways are first, long horizon suite is still unsolved. The best agents only at 26%. There's plenty of headroom left. Second, the big bottleneck is robust verification. At hour and day scale lens tasks, we need the multi-channel checks, anti-cheat hardening, product style validation. The tasks, the code, the paper, the logs, and the trajectories are all public. I've released 320 gigabytes of trajectories that are especially important because they make SWE Marathon fully inspectable and transparent. I also want to thank all of my collaborators on this project, all of whom are listed here. SWE Marathon was very much a community driven effort across task contributors, advisors, and paper writing. You can find everything at SWE Marathon.org. Thank you.