Hi everyone, I'm Pierluca Loro, and I'm the founder of Programma Labs. Today I'm going to talk about computer usage and evaluation, and most of the work and details about it are in a paper with this title. I did this work while at Manta Superintelligence Labs with the collaborators you see on this slide.
To start, I want to introduce this type of agent. It's a weird type of agent that I call a replay agent. Imagine we run this process: we run our frontier model, a good one, on a benchmark we like, and then for every task we collect a successful trace or a successful trajectory, and we have our recorded table of these types. The actions might be tapping, typing, scrolling, and we record this. Then we do this for all the tasks in the benchmark, and we compile these into a replay agent that, when the tasks arrive, replays that sequence of actions blindly.
If you do this for a common benchmark with hundreds of tasks, this is going to be a script that is less than a megabyte, and this is a completely valid type of agent that you can evaluate on the benchmark. If you try to evaluate this kind of agent on standard benchmarks such as OS Word or Mobile Word, you will see that the success rate of this agent, compared to the frontier model from which the agent was extracted, is actually the same or even better.
This is a weird but maybe trivial phenomenon, but I would argue that we shouldn't accept this kind of blind script beating the frontier models. The reason why this happens is the determinism of most existing benchmarks. If the benchmark is static and deterministic, then it is somehow gameable by this sort of strategy.
It goes even deeper than this. If you look at one of the metrics that people have been using in the past for evaluating computer use agents, it's pass at K. This metric is defined as the probability of at least one of K attempts succeeding, but if you look into the details of how this metric works on a deterministic environment, you will see that it's literally, and we prove it formally in the paper, evaluating the success rate of the replay agent that I've shown to you. If that replay agent felt weird to you, also pass at K on computer use tasks should somehow feel weird to you. In other words, pass at K is a metrification of that exploit of the replay agent.
These are two specific problems, but they point at two general classes of problems in Kua benchmarks. These problems are around environments, building environments, and evaluation, building good metrics to know if your agent is good or not. In particular, we want to have environments that don't have exploitable structure, and we want to have metrics that are not fragile or based on fragile statistics. I'm going to talk about both of the aspects now, so let's talk about building principled environments first.
The first aspect that I worked on while working on environments is to try to design a set of principles that could be guiding principles when building environments, so that we build robust environments and trustworthy environments. If you think about the problem that I was describing with replay agents, the first thing that you could think about as a solution, to not have a replay agent hack your benchmark, is to have your benchmark be multifactorial. That means generating variation for your benchmark, having stochasticity in the benchmark. For computer use environments, that means varying stuff like data or appearance or simply the initial state.
But if you want to make sure that all the combinations that you generate are valid, you want to have, as a design principle in your environment, a system for checking and verifying that everything is working as intended for every combination. Of course, you want the usual things for your environment, so you want your environment to be sandboxed, and you want your environment to support verifiers. You want your environment to provide privileged information, and you want your environment to be realistic. If it's a reproduction of a real system, you want that reproduction to be faithful so that the score that you get out is a good one.
If you sort them out, you can remember these principles as the PRISM principles for environment design. We tried that method to build a benchmark that would satisfy all of these principles. If you look at existing benchmarks, some of them do some things in a good way, some others do other things in a good way, but there is no unified benchmark that matches all these boxes. We built one that's called DigiWord.
The way DigiWord was built in practice is as a set of mobile apps for Android devices. It's 15 apps spanning different domains, with 387 verified scenarios and a number of configurations. These configurations are large in number, 3.2 million, but the important thing is that they are verified. The axes are the ones that I was mentioning before, so you can imagine for each one of the tasks you can vary things like the instance, what is the exact amount of money that you're sending, for instance, or the data profile, which kind of contacts or emails you have in the data for your tasks, or the theme, or the starting screen. Do you start from the login page, or do you start from another valid page?
If you do the math, even if you start from a relatively low number of base cases for each one of these variables, you end up having many combinations, so you can get to millions of combinations. If you scale this up, you can get easily to billions of combinations. All of these different axes can be manipulated by coding agents because, in the end, they are forms of software, so you can have a coding agent generate different instances, different themes, and such.
You might think maybe it's easy to build an environment. You just generate as much software as you can with a coding agent, and then you have a diverse environment. But it's a little trickier than that. Coding agents can generate a lot of software, but a lot of software is not the same as an effective KUA environment. The reason for this is that you need to verify the correctness of your combinations. The key to scaling this up is to have a verification strategy for the variations of your tasks.
The kind of verification strategy to follow is this one. You can generate many configs, all the combinations of the different factors that I've explained before, and you can then have a system that rejects the ones that are not valid and just keeps the valid configs. In the case of DigiWord, we did this by building a system that looks a little bit like a compiler and works in the following way.
You start from a parameterized task template, and this might look like something like this: send certain amount to certain recipient. Then you have a verifier that corresponds to that template, and then you have mock data for that task, data that you need for that specific task to happen. Then we have a system that is like a visual compiler that takes all of this and, given a base case of data and a base case of UI state, puts all of this together and creates a valid configuration. You can build systems like this in which the main craft is software engineering, to make sure that the combinations that you have are both diverse and valid.
If you do the same process we did before, you evaluate your frontier model and then you evaluate the corresponding replay agent, you will see that the replay agent doesn't get a lot of performance. It gets low performance, which is probably what you want. Sometimes some tasks may be repeatable by nature, but on average you shouldn't expect a replay agent to have good performance on the benchmark.
Once you build these diverse combinations, you also can do other things, like measuring the robustness of frontier models over different axes of variation. The axes of variation I described before are represented here, and you can see that, in the worst case, frontier models are actually pretty bad at being robust to these variations. If you have a model that seems to be good at a given task, you would expect that if you just vary which screen the task is starting from or what the theme of the app is, the models should pretty much have the same performance. But this is actually not the case for most frontier models. If you have infrastructure like this, you can actually measure that and tailor your expectations about this kind of robustness.
This was about the first aspect, building an environment that supports diversity and is robust enough to evaluate models. But the second aspect is as important as the first one, which is to measure uncertainty honestly. Once you have all of this variation, how do you handle computing the real performance of your agent?
There are two sources of stochasticity or variation, and they are not exactly the same, but they are equally important. The one that we usually think about is the one about the actions. You run your model multiple times, and in many cases you can have quite different trajectories out of it because the action on each step will be different. But if you have a benchmark like the one that I've described, with multiple combinations and multiple variations, then also the variability from the environment becomes important, and we want to capture that because that is what we are going to find in the real world.
We need a methodology that captures both of these types of variation, and in the paper there are the details, but basically we build a methodology that can accurately capture these two types of variation, taking into account the structure of the benchmark. In practice, it is useful to use this concept of coverage. When you compute a confidence interval, you have some confidence that the performance of the model is inside that range, and so you would expect that a 95% confidence interval would say that 95% of the time the performance of the model is in that range.
But if you only use rollouts, so you only use the base case and what people would use normally, actually in realistic cases you have something like 17% or 20% coverage. That means that basically only 20% of the time you guess the right performance of the agent, which can be pretty bad. But if you take into account the hierarchy and you use the proper way of computing confidence intervals, you can get to the full confidence interval and be 95% accurate.
If this seems quite abstract, in practice that means that if you want to make a decision about which models to deploy, maybe you have model A and model B, and you do an eval for those two models, you can have cases in which the confidence intervals seem really small, and so you make a decision based on those small confidence intervals, but they are actually overconfident. So this was the wrong decision. The orange bars are the real performance here.
If a mistake is pretty costly for you and you have many tasks, if you have one million tasks and there is a 4% mismatching performance for real in the models, and each mistake is $20, like $12 on average, it can cost you hundreds of thousands of dollars in a single month. It can be super costly as a mistake, just from confidence intervals being overconfident. But if you have a reliable way of computing the confidence interval, the method would tell you, I'm not confident enough to make an informed decision. Then you can choose to spend more money or more time evaluating models and avoid the costly mistake. We don't want to delude ourselves with wrong confidence intervals because there's money on the table, essentially.
This is a final checklist of the things that I've discussed so far. To recap, some of the things that are important in building a benchmark are about the environment, and some other things are about the metrics. On the environment, you can follow the principles that I described before, the PRISM principles. Some of these things are rather common, but some things, like varying initial state across runs, are pretty rare across existing benchmarks, but they are very important, so I would suggest you try to incorporate these into your evals.
On the metrics side, you can read the paper for the details, but essentially it's very important to avoid replayability as something that you can have in your benchmark, and also to focus on having accurate confidence intervals, so respecting the benchmark structure and trying to avoid underestimating the uncertainty overall.
I've heard many times sentences like, this benchmark can be gamed, but everybody's using it, or, there is no error bar, but I don't see people using them. These are things that we can think when we don't have enough time, but actually a non-rigorous benchmark can be misleading. It can be misleading for the field because everybody could be maximizing a score on a benchmark that maybe is not capturing what we can do. But especially, it can be misleading for your own decisions. If you are deluding yourself into thinking that a score is confident and that it is confidently telling you that your model is good, actually you are going to pay for those mistakes. I think it's usually very good to be honest with yourself and to try to be rigorous in the evaluations that you have.
As a last slide, I just started this company, Programma, and we are building the best infrastructure for QA and Ableware verification, and so we are hiring if you are interested or want to chat. This is our website. Thank you very much. Thank you very much.
and they're always describing with replay agents the first thing that you could think about as a solution not to have a replay agent to like hack your benchmark is to have your benchmark to be multifactorial so that means varying generating variation for your benchmark so having stochasticity into the benchmark and for computer use environment that means varying stuff like data or appearance or simply the initial
the initial state but if you want to make sure you want to make sure you want to make sure you want to make sure that all the combinations that you generate are valid and so you want to have as a design principle in your environment also a system for checking and verifying that everything is working as intended for every combination and of course you want the usual things for your environment so you want your environment to be sandboxed and you want your environment to support like verifiers so
you want your environment to be privileged information and you want your environment to be realistic so if it's a reproduction of a real system you want that reproduction to be faithful so that the score that you get out is a good one and so if you sort them out you can remember these sort of principles as the prism principles for environment design and we tried that method to build a benchmark that would be satisfying all of these principles
and if you look at existing benchmarks some of them do some things in a good way some others do other things in a good way but there is no unified benchmark that sort of matches all these boxes and we built one that's called DigiWord so the way DigiWord in practice was built is as a set of like mobile apps for Android devices so it's 15 apps spanning in different domains with 387 verified
scenarios and a number of scenarios and a number of configurations so these configurations they are in large numbers of 3.2 million but the important thing is that they are verified and indeed the axes are the ones that I was mentioning before so you can imagine for each one of the tasks you can vary things like the instance so what is the exact amount of money that you're sending for instance or the data profile like which kind of contacts or emails
you have in the data for your tasks or like the theme or the starting screen so do you start from the login page or do you start from another valid page so if you do the math even if you start from a relatively low number of base cases for each one of these variables you end up having many many combinations so you can get to like millions of combinations and if you scale this up you can get to easily to billions of combinations and all of these you know different axes can be manipulated by coding agents because in the end they are like forms of software so you can have a coding agent to generate different instances different teams and such
so you might think maybe it's easy to build an environment you just generate as much software as you can with a coding agent and then you have like a diverse environment but it's a little bit trickier than that and indeed coding agents can generate a lot of software but a lot of software is not the same as an effective KUA environment and the reason for this is that you need to verify the correctness of your combination right
and so the key to scale these up is to have a verification strategy for the variations of your tasks and so the kind of verification strategy to follow is this one so you can generate many configs all the combinations of the different factors that I've explained before and you can then have a system that rejects the ones that are not valid and just keeps the valid configs and so in the case of DigiWord we did this by building a system that looks a little bit like a compiler and that works in the following way
so you start from a parameterized task template and so this might look like something like this so you have send certain amounts or certain recipient and then you have a verifier that corresponds to that template and then you have mock data for that task so data that you need for that specific task to happen
and then you have a system and then you have a system and then you have a system and then we have a system that is like a visual compiler that takes all of this and given a base case of data base case of UI state puts all of this together and creates like a valid configuration and so you can build systems like this in which the main craft is software engineering to make sure that actually the combinations that you have are both diverse and valid
and so if you have a benchmark and so if you have a benchmark and so if you have a benchmark and so if you do the same process we did before you evaluate your frontier model and then you evaluate the corresponding replay agent you will see that the replay agent doesn't get a lot of performance it gets a lot of performance it gets a lot of performance it's probably what you want sometimes some tasks maybe are repeatable by nature but on average you shouldn't expect a replay agent to have good performance on the benchmark
once you build like these diverse combinations you also can do other things like measuring the robustness of frontier models over different axes of variation so the axis of variation I described before are here represented there and you can see that in the worst case frontier models are pretty bad actually at being robust to these variations and so for instance if you have a model that seems to be good at a given task you would expect that if you just vary you know which screen the task is starting from or like what is the theme of the app the models should pretty much have the same performance but this is
actually not the case for most frontier models and so if you have infrastructure like this you can actually measure measure that and like tailor your expectation about this kind of robustness so this was about the first aspect that was building an environment that supports diversity and that is robust enough to evaluate models but the second aspect is as important as the first one is to measure uncertainty
honestly so once you have all of this variation how do you handle like computing the real performance of your agent and basically there are two sources of stochasticity of variation and they are not exactly the same but they are equally important and so the one that we usually think about is the one about the actions right and so you you run your model multiple times and in many cases you can have even quite
different trajectories out of it because the action on each step will be different but if you have a benchmark like the one that I've described with multiple combinations with multiple variations then also the variability from the environment becomes important and we want to capture that because that is what we are going to find in the real world and so we need a methodology that captures both of these types of variation and in the paper there are the details but basically
we build a methodology that can accurately capture these two types of variation taking into account the structure of the benchmark and so if you start like in practice it is useful to use this concept of coverage so when you when you compute a confidence interval basically you have some confidence that the performance of the model is inside of that range and so you would expect that a 95% confidence
confidence interval would say that you know 95% of the time the performance of the model is on that range but if you only use rollout so you only use the base case and what people would use normally actually in realistic cases you have something like 17% or 20% coverage and that means that basically you only 20% of the time you guess the right performance of the agent which can be pretty bad but like if you take it into account the hierarchy and you use the proper
way of computing confidence intervals you can get to the full confidence interval and be 95% accurate and so if this seems quite abstract you know in practice that means that if you want to make a decision about which models to deploy maybe you have model A model B and you do an eval for those two models you can have cases in which the confidence intervals seem really really small
and so you make a decision based on those small confidence intervals but they are actually overconfident and so this was the wrong decision so that the orange bars are the real performance here so you make this decision and if a mistake is pretty costly for you and you have many tasks like if you have one million tasks and there is a 4% mismatching performance for real in the models and each mistake is like $20 like $12 on average
it can cost you like hundreds of thousands of thousands of dollars in a single month so it can be like super costly as a mistake just for confidence interval being overconfident but if you have like a reliable way of computing the confidence interval the method would tell to you I'm not confident enough to make an informed decision and so you can choose like to spend more money to spend more time on evaluating models and avoid the costly mistake so we don't want to like elude ourselves with like elude ourselves with like wrong confidence intervals because there's you know money on the table essentially
and so this is sort of a final checklist of the things that I've that I've discussed so far so again to recap some of the things that are important in building a benchmark are about the environment and some other things are about the metrics so on the environment you can follow the principles that I described before
like prism principles so some of these things are rather common but some things like varying initial state across runs they are pretty rare across existing benchmarks but they are very important and so I would suggest you to try to incorporate these into your evals and things about the metrics you can of course read the paper for the details but essentially it's very important to avoid replacing the ability as something that you can have in your benchmark and also to focus on having accurate confidence intervals so respecting the benchmark structure and trying to avoid underestimating the uncertainty overall
and so I've heard many times sentences like this benchmark can be game but everybody's using it or like there is no error bar but I don't see people using them so they are sort of things that we can think when we don't have enough time but actually a non-regurous benchmark is misleading you know it can be misleading for the field because everybody could be seeking you know maximizing a score on a benchmark that maybe is not capturing what we can do
but especially it can be misleading for you know your own decisions and so if you are deluding yourself on thinking that a score is like confident and that is confidently telling that your model is good actually you are going to pay for those mistakes and so I think it's very good usually to be honest with yourself and to try to be rigorous in the evaluations that you have as a last slide I just started this company program and we are building the best infrastructure for QA and Ableware verification and so we are hiring if you are interested or want to chat and this is our website thank you very much thank you very much