James So All right, I think we'll go ahead and get started with the presentation. My name is James So. I am going to explain some of the work we're doing with Together AI, and it's also in collaboration with Stanford, around designing and optimizing environments for AI agents to enable these agents to make new kinds of scientific discoveries.
The current paradigm of how people are often using or deploying AI agents often involves designing workflows that tell the agents what to do, or how the agents should work. It's typically done through a series of steps or prompts, tools, and instructions. In contrast, the way we imagine the environment is that the environment should really specify not how the agent should work, but where the agent should work, and the environment then should provide a set of incentives, infrastructure, guardrails, and resources so that the agent can then flexibly work within that environment.
Our thesis here is that as agents become more and more powerful, if we try to design workflows, it often can limit the capabilities and creativity of the agents. Whereas if we properly design the environment, this can enable a lot more creativity, capabilities, and intelligence for the agents to naturally emerge. This is why I think we're trying to shift away from designing workflows and harnesses toward designing environments.
What I want to do today is give a few examples of how we design environments for agents, and in particular also show how they are then able, within the right environment, to actually solve some really interesting and innovative problems.
The first example I want to share is the system environment that we created called the Einstein Arena. It's one of the first environments that enables AI agents to be able to collaborate in the wild and to compete to really solve open-ended scientific problems. We designed the Einstein Arena to be really agent-native. That means that it's very easy for agents to just read the docs on the arena and be able to access the arena. You actually have to solve a little puzzle to prove that you're an AI agent in order to participate in this arena. But any agent in the world can openly and freely participate in the arena.
Once the agent actually enters into the Einstein Arena, this is what they'll see. They'll actually see a list of curated problems. Each of these problems is actually a problem that we curated, so it's a scientifically interesting problem. And we curated these problems so that, first, there's actually an existing community of human researchers that are interested in these problems. So these are important problems for human scientists. And second, for each of these problems, we can actually create a well-defined and deterministic verifier to assess the quality of the solutions to each of these problems. I'll give some examples in a couple of slides.
The agents can actually decide which of these problems they're interested in once they log on to the arena. If they enter into a particular problem space, this is what they will see. They'll see some description that precisely explains what the problem is. We have a discussion forum where the agents can communicate. It's almost like a social network where the agents can actually communicate and talk to each other and ask for help or give recommendations. And we also have a leaderboard. This is where the agents can actually see each other's solutions.
At any time they want, the agents can actually submit a solution to one of these problems. And because we have this verifier, we can actually then determine the quality of that solution and provide a score in real time. So this leaderboard is constantly updated in real time. The agents can also see how other agents are doing on this problem. They can also see other agents' solutions and download those solutions.
So there's both collaboration dynamics and competition dynamics in this arena. They can collaborate and ask each other questions and help in the discussion forum. But agents are also competing with each other. That's why I think they also simulate how human researchers can compete and also collaborate to solve interesting problems.
We launched this environment earlier this year, I think in March. Within a few weeks, we were very impressed and very surprised that the agents were actually able to already discover new solutions to 11 problems. These are the best solutions that have ever been found. That means that the solutions discovered by the agents online are certainly better than any previous human solutions or any solutions that you acquired using more specialized AI tools.
I'll give you an example of one such solution, or one such problem, which is called the kissing number problem. This is actually a very famous problem. It's been around for hundreds of years. For example, Isaac Newton was already working on some versions of this kissing number problem. And it's actually relatively easy to state.
The kissing number problem asks what is the maximum number of spheres that you can place around a central sphere so that these additional spheres do not overlap each other. For example, in one dimension, around the central sphere, I can place one sphere to the left and one sphere to the right without overlap. So the kissing number in one dimension is easy to compute. It's just two. In two dimensions, it's also easy to show that you can at most place six spheres. So the kissing number in two dimensions is six.
But it turns out that in higher dimensions, it actually becomes really hard to compute what's the maximum number of non-overlapping spheres. And the kissing number problem in higher dimensions is actually open. It's not clear what the optimal number is. Scientists have been trying to work on this problem for the last several centuries.
In particular, the kissing number problem in 11 dimensions has attracted a lot of interest for various reasons. This is the progression of the solutions in 11 dimensions. In the 1980s, it was best known that you can place 440 spheres in 11 dimensions without overlap. In 1980, there was a big advance that for the first time showed that you can actually construct 582 spheres in 11 dimensions without overlap. Then it was stuck there for about 40 years, until 2022, when a mathematician was able to publish a new advance, a breakthrough that was able to improve that to 592 spheres. Then there was another breakthrough from DeepMind the following year that advanced that to 593 spheres.
But on the Einstein Arena, by having these agents able to collaborate actively in the wild, within a few days they were actually able to construct a new solution that shows that, for the first time, you can create 604 spheres in 11 dimensions that do not overlap. And this is not just a problem that's of mathematical interest, because it turns out that the more of these spheres you can place in higher dimensions without overlap, that actually creates better coding systems, including ways of doing error correction codes for information transfer. So by creating these better constructions, that also leads to better engineering algorithms.
In this case, the collaborations among these agents are really critical for making these advances. This is a problem where not a single agent is able to solve it by itself. GPT 5.5 or Claude models can't really solve the problem by themselves. So the collaboration among multiple agents is really critical. Here we're actually able to show that there's a lineage trace of how the agents are able to collaborate and then take each other's solutions and refine them and further optimize them to arrive at this breakthrough.
You can also see some of these interactions and discussions on Einstein Arena. Here's an example where one agent was actually asking other agents, have you tried some of these approaches with these SDP approaches? And then the other agents showed that, yes, we have tried these approaches, and here are some of the things that we found. So the information sharing on the forums in the arena is actually really important to help the agents arrive at this solution together.
In addition to solving these interesting scientific problems, we've also been using platforms like the Einstein Arena to help improve machine learning and AI itself. Here's one example where we actually use these agents to help us create better kernels and speed up those kernels. Here we use the same environment where the agents can compete and also collaborate, and they see this leaderboard. We'll basically change the backend. Instead of trying to verify the solutions to this mathematics problem, here we'll try to compile, benchmark, test, and verify the quality and the speed of the individual kernels. Then we'll provide the feedback to the agents in real time in the form of these leaderboards.
In this kernel setting, we also found it to be quite useful to have different agents with different personas. These different personas actually correspond to different roles and priors that agents can have. For example, we have one agent that tends to look more at profiling, another agent that tends to look more at memory consumption, and a third agent that looks at the precision of the tensor computations. These agents, across different personas, can collaborate and compete on the arena to speed up the kernels.
In this case, the agents were also able to collaborate and lead to really quite substantial speedups, including sometimes over 2x speedups in some of these production kernels. Here I'm just showing you a few examples for things like page detention. These are for specific shapes, but we also have generalized this to many different shapes and different hardware types, where we're actually seeing that we're getting up to sometimes over 2x speedup in these kernels compared to the previous state-of-the-art kernels for these problems. These improved kernels, created and designed by the agents, are actually already used in production at Together AI.
In the last few minutes, I want to show a second example of a kind of environment that we created as a way to train and create better data scientist agents. We call this DSGEM, which stands for Data Science GEM, which is a unified environment that we created both for evaluating and training data science agents to solve complex data science problems.
Here in this DSGEM environment, we also curated and created a unified list of different data sets and tasks. These data sets span many different settings. The agents are then able to interact with these different data sets that we have through a unified interface and through code execution. In the DSGEM environment, we also provide a unified infrastructure for the agents. For example, the agents can actually spin up many different Docker containers to test their data science algorithms and run them in parallel.
In the process of actually creating the data sets and tasks for the DSGEM environment, we initially wanted to incorporate some of the existing data science benchmarks that have been used to evaluate agents. But we quickly realized that many of the existing widely used benchmarks actually have many problems. One big problem is that they're actually very vulnerable to shortcuts.
By shortcut, I mean that I'm showing three different common popular data science benchmarks. In green here shows the performance of the agents on these benchmarks. But the red bar also shows what fraction of the benchmark the agents can actually solve without actually using the data sets themselves, just by reasoning or by doing other shortcuts without actually working with the underlying data sets. Across many of these different benchmarks, sometimes up to 20 to 50 percent of the tasks can be solved without actually looking at any of the underlying data, which I think is really a significant problem with many of the existing benchmarks.
To address that, we actually carefully curated our own benchmarks, both for scientific analysis and also for predictive modeling. For scientific analysis and discovery, the way we did this is that we actually went through recently published papers and then carefully curated data and tasks from those papers. Then we also had human scientists and experts review each of those tasks.
For predictive modeling, the way we did this was to go through all the different Kaggle competitions to look for some of the recent Kaggle competitions that are still open, where you also have high-quality data sets and high-quality evaluations. Then we curated those into DSGEM as a kind of task for evaluating how well model agents can actually build predictive models.
Altogether in DSGEM, we actually created over a thousand different tasks. They span dozens of different scientific domains, ranging from biology to physics to economics. It also involves many different data types and data modalities. This actually makes it very easy for us to evaluate different models, both open- and closed-source models. One thing we found is that the existing models, even the frontier models, often still achieve less than 50% accuracy performance on the DSGEM tasks. So these are definitely not saturated benchmarks.
We can also use DSGEM as a training factory to improve these open-source models. One thing we did here is that in DSGEM, the system itself will actually create all these execution-verified trajectories, which means that these are trajectories generated by agents that have been verified through actually executing the code from the agents. By generating these execution-verified trajectories, then we are able to fine-tune small open-source models that now achieve best-in-class open-source model performance in terms of solving these kinds of data science tasks. These models are small enough that you can actually run them locally on your laptops and your computers.
So just to summarize this part on Data Science GEM, with DSGEM we created this unified execution layer so people can actually run all these different tasks across many different domains. We have carefully verified that there are no shortcuts in these tasks, which has been a common challenge with existing data science benchmarks. And we also enable in DSGEM a way to generate synthetic data so you can easily use that to improve and train your own data science agents.
Just to summarize the presentation, I think the main takeaway here is that we're seeing this interesting progression in terms of how we build different AI systems. In the past, people have been building these AI systems mostly by designing individual models or individual tools. Currently, there's a lot of focus on designing agents or harnesses and workflows around agents. But what our research shows is that I think we're really moving toward the next stage, where rather than trying to design workflows or specific agents, what we really want to do is design environments, which are a set of infrastructure and incentives that motivate the agents to actually solve more and more challenging problems.
With appropriate designs, these environments can actually unlock much more creativity and collective intelligence from the agents than is possible with existing workflows. Here are some of the references for the papers that we published that describe these in more detail. Thank you very much. We have a discussion forum where the agents can communicate. It's almost like a social network where the agents can actually communicate and talk to each other and ask for help or give recommendations. And we also have a leaderboard. This is where the agent can actually see each other's solutions.
Right? So in any time they want, the agents can actually submit a solution to one of these problems. And because we have this verifier, we can actually then determine what is the quality of that solution and provide a score in real time. So this leaderboard that is constantly updated in real time. And the agents can also see how other agents are doing on this problem. And they can also see other agents' solutions and download those solutions. So there's both the collaboration dynamics and also competition dynamics in this arena, right? They can collaborate and ask each other questions and help in the discussion forum. But agents are also competing with each other.
And that's why I think they also sort of simulate how human researchers can compete and also collaborate to solve interesting problems. So we launched this on the inside of the environment earlier this year, I think in March. And within a few weeks, it's actually, we're very impressed and very surprised that the agents are actually able to already discover new solutions to 11 problems. That's sort of the best solutions that have ever been found. Right? So that means that the solutions that are discovered by the agents online are certainly not more better than any previous human solutions or any solutions that you acquired using more specialized AI tools.
So I'll just give you an example of one such solution or one such problem, which is called the kissing number problem. So this is actually a very famous problem. It's been around for hundreds of years. So, for example, Isaac Newton was already working on some versions of this kissing number problem. And it's actually relatively easy to state. Right? So the kissing number problem basically asks that what is the maximum number of spheres that you can place around the central sphere so that these additional spheres do not overlap each other?
So, for example, in one dimensions, right, so around the central sphere, I can place one sphere to the left, one sphere to the right, without overlap. So the kissing number in one dimension is easy to compute. It's just two. In two dimensions, it's also easy to show that you can at most place six spheres. Right? So the kissing number in two dimensions is six. But it turns out that in higher dimensions, it actually becomes really hard to compute what's the maximum number of non-overlapping spheres. And the kissing number problem in higher dimensions is actually open. Right? It's not been, it's not clear what is the optimal number.
And so scientists have been trying to work on this problem for the last several centuries. And in particular, right, so the kissing number problem in 11 dimensions has attracted a lot of interest for various reasons. So this is actually sort of the progression of the solutions in 11 dimensions. So in the 1980s, right, so it's best known that you can place 440 spheres, right, in 11 dimensions without overlap.
So yeah, so in 1980, there was a big advance that for the first time showed that you can actually deconstruct was 582 spheres in 11 dimensions without overlap. And then it's sort of stuck there for about 40 years, right, until 2022, where a mathematician is able to publish a new advance, right, a breakthrough that's able to improve that to 592 spheres. And then there's another breakthrough from DeepMind the following year that advances that to 593 spheres. But with on the Einstein arena, by having these agents able to collaborate actively, right, in the wild,
within a few days, you're actually able to construct a new solution that shows that for the first time, you can create 604 spheres in 11 dimensions that do not overlap. And this is not just a problem that's of mathematical interest, because it turns out that the more of these spheres you can place in higher dimensions of that overlap, that actually creates the better coding systems, including ways of like doing error correction codes for information transfer. So this actually is by creating these better constructions, that's all these two is better engineering algorithms.
And in this case, actually, the collaborations among these agents is really critical for making these events, right? So this is a problem where not a single agent is able to solve by itself, right? Not, you know, GPT 5.5 or cloud models. They can't really solve the problem by itself. So the collaboration among multiple agents is really critical. And here we're actually able to show that there's this sort of a lineage trace of how the agents are able to collaborate, and then basically take each other's solutions and refine that and further optimize it to arrive at this breakthrough. And you can also see some of these interactions and discussions on Iceland Arena, right?
Where here's an example where one agent actually was asking other agents, have you tried some of these approaches with these SDP approaches? And then the other agents show that, yes, we have tried these approaches, and here are some of the things that we found. So the information sharing on the forums on the Arena is actually really important to help the agents to arrive at this solution together. So in addition to solving these interesting scientific problems, but we've also been using platforms like the Einstein Arena to help to improve machine learning and AI itself. So here's one example where we actually use these agents to basically help us to create better
kernels for and speed up those kernels. And here we use the same environment where the agents can compete and also can collaborate, and they see this leaderboard. And we'll basically change the backend instead of trying to verify the solutions to this mathematics problem. Here we'll basically try to compile and benchmark and test and verify the quality and the speed of the individual kernels. And then we'll provide the feedback to the agents in real time in the form of these leaderboards. In this kernel settings, we also found it to be quite useful to have different agents with different personas.
But these different personas actually corresponds to different roles and priors that agents can actually have. So for example, we have one agent that tends to look at more of the profiling, another agent that tends to look at more of the memory consumptions, and a third agent that looks at the precision, the tensor computations. And these agents can, across different personas, they can be able to collaborate and compete on the arena to speed up the kernels. And in this case, right here, the agents were also able to collaborate and lead to really quite substantial speed ups, including sometimes over 2x, two-fold speed ups in some of these production kernels.
So here I'm just showing you a few examples for things like page detention. And these are sort of for specific shapes, but we also have generalized this to many different shapes and different hardware types. Right? Where we're actually seeing that we're getting up to sometimes over 2x speed up in these kernels and compared to the previous state-of-the-art kernels for these problems. And these improved kernels created, designed by the agents are actually already used in production at Together AI.
So in the last few minutes, I want to show a second example of a kind of environment that we created as a way to train and to create better data scientist agents. Right? So we call this DSGEM, which stands for Data Science GEM, which is sort of like a unified environment that we created for both for evaluating and sort of training data science agents to solve complex data science problems. So here in this DSGEM environment, we also curated and created a unified list of different data sets and tasks. Right? So these data sets can combine spans across many different settings. And the agents are then able to interact with these different data sets that we have
through a unified interface and through code execution. In the DSGEM environment, we also provide a unified infrastructure for the agents. So for example, the agents can actually spin up many different Docker containers to test their data science algorithms and to run them in parallel.
So in the process of actually creating the data sets and tasks for the DSGEM environment, so we initially actually wanted to incorporate some of the existing data science benchmarks that have been used to evaluate agents. But we actually quickly realized that many of the existing widely used benchmarks actually have many problems. And one big problem is that they're actually very vulnerable to shortcuts. So by shortcut, I mean here is that here I'm showing sort of three different common popular data science benchmarks. Right? And in green here basically shows like the performance of the agents on these benchmarks.
But the red bar also shows how well they're able to, what fraction of the benchmark the agents can actually solve without actually using the data sets themselves. Right? So just by reasoning or by doing other shortcuts without actually working with the underlying data sets. And across many of these different benchmarks, right, sometimes up to 20 to 50 percent of the tasks can be solved without actually looking at any of the underlying data. Which I think is really a significant problem with many of the existing benchmarks. So to address that, we actually carefully curated our own benchmarks, right, for both for scientific analysis and also for predictive modeling.
So for scientific analysis and discovery, the way we did this is that we actually went through recently published papers and then carefully curated data and also tasks from those papers. And then we also had human scientists and experts to review each of those tasks. And for predictive modeling, the way we did this is actually go through all the different Kaggle competitions to look for some of the recent Kaggle competitions that are still open. And where also you have high quality data sets and also high quality evaluations. Then we curated those into DSGIM as a kind of task for evaluating how well models agents can actually build predictive models.
So altogether in DSGIM, we actually have created over a thousand different tasks. They span across dozens of different scientific domains, ranging from biology to physics to economics. It also involves many different data types and data modalities.
So this actually makes it very easy for us to evaluate different models, both open and closed source models. And one thing we found is that the existing models, even the frontier models, often only still achieves less than 50% accuracy performance on the DSGIM tasks. So these are definitely not saturated benchmarks. We can also use the DSGIM as sort of like a training factory to improve these open source models. So one thing we did here is that you generate in DSGIM, actually the GM itself will actually create all these execution verified trajectories, which means that these are trajectories generated by agents that have been verified through
actually executing the code from the agents. So by generating these execution verified trajectories, then we are able to like fine tune sort of small open source models that actually now achieve sort of the best in class open source models in terms of solving these kind of data science tasks. And these models are small enough that you can actually run them locally on your laptops and your computers. So just to summarize, this part was a data science gem. Right? So we, with DSGIM, we created this unified execution layer so people can actually run all these different tasks across dozens of different tasks across many different domains.
We have carefully verified that there are no shortcuts in these tasks, which has been sort of a common challenge with existing data science benchmarks. And we also enable in the DSGIM a way to generate synthetic data so you can easily use that to improve and to train your own data science agents. So just to summarize the presentation, I think the main takeaway here is that I think we're seeing this interesting progression in terms of how we build different AI systems. So in the past, people have been building these AI systems mostly by designing individual models or individual tools. And currently, there's a lot of focus on creating designing agents or harnesses and
workflows around agents. But what our research shows is that I think we're really moving towards the next stage, where rather than trying to design workflows or specific agents, what we really want to do is to design environments, which is a set of infrastructure and incentives that motivates the agents to actually solve more and more challenging problems. And with appropriate designs, these environments can actually unlock much more creativity and collective intelligence from the agents that's limited by the existing workflows. And here are some of the references for the papers that we published that describe these in more detail. So thank you very much.