A Genius With Amnesia - Victor Savkin, Nx

Transcript

Imagine you find a magic lamp in an antique store. You rub it, a genie appears and asks how it can help. You bury the lead, so you say I need the best engineer to help with an impossible project at work. And the genie grants your wish. For me the best engineers probably joined Carmack from his id days, so you get Carmack. But the genie had a sense of humor and imposes restrictions, maybe for safety. Carmack can only see one small part of your codebase, maybe one thousandth of it. And he remembers nothing he did before. Every conversation starts fresh. That would be maddening, right? You would know there is a standard way to do stuff and Carmack wouldn't. You would have to explain the same thing over and over and over again. You would have a genius on one side and something deeply deficient on the other. And that's what agents are. Now let me walk you through an example of how many times we explain things in a simple interaction. We have four repos: UI, module one, module two, and platform. I want to change the UI and propagate the change through the system. First we change the UI library, say we change the button or whatever. That's the first explanation. Unavoidable. We have to express intent, okay? Then we publish it. We go to module one and we have to explain what just happened in the UI library so it can consume the package here. We know that that's often a different person, right? Every box in this diagram can be done by a different person. Then we discover that the published UI library doesn't work with module one. So we go back to UI and we have to re-explain the original change and the issue, right? Because it's a new agent. It doesn't know the original change. It obviously doesn't know about the issue. Let's say we fix it, right? And publish it again. We go and again we explain the new change in the context of module one, same ordeal. And we do the same for module two again. And then we go to the platform repo and we explain how everything fits together and we implement the change there. Let's imagine a week after release, a bug appears in the UI component and we have to fix it. So we start an agent in the UI repo and we have to explain again the original change from a week ago and this production issue we've seen. So we have seven explanations for what essentially is one change. And also it may not be one person making all these seven explanations, but they still occurred, right? So that's very, very typical with agents. How do we solve it? Well, there are many problems in here that contribute to this experience, but they roughly fall into two categories. The first one is that an agent essentially is repo bound. The agent sees and changes generally one repo at a time. It never sees the whole system, which can be hundreds or thousands of repos. So that's the space component of the problem. Second is amnesia. The agents forget the work. Every session starts with a blank slate. The human becomes a memory in this case. That's the time component of the problem. Let me look at the two closer. Take the repo boundary first. Without a model of how repos fit together, the agent leans on the human to do the research. It can't align the code with the rest of the system. It couldn't align the UI change with module one. The human didn't explain it. So a bad version shipped. It can't reliably reference best practices and standards either because those often live in other repos. Writing is even worse. The agent writes to one repo at a time. It means it can't validate changes downstream. Module one CI should have failed on the UI change, but it didn't. The agent can't update consumers at the same time, even though while making the UI change it has perfect information to do so. It knows exactly what it's doing. So the user has to explain stuff imperfectly to each consumer. Changing something across 20 repos means explaining things 20 times. A lot of developer time spent, but also a lot of talk and burn. The second category is that the agent forgets. The agent has no episodic memory. Every session is a blank slate. And the human in this case becomes the memory. Here's what the graph of your work actually looks like. At the bottom there is a repository graph. The artifacts your organization produces, plus every open source repo you depend on. Maybe a thousand repos you own and tens of thousands of open source repos. At the top there are all agentic sessions that create and modify that code. Sessions relate to each other. Repos relate to each other. So this graph is a faithful picture of the work in your organization. It describes what's there at the bottom and how it came to be at the top. That's what you want your agent to see. Here's what it actually sees: one session, one small fraction of your codebase, no memory, okay? Because it sees so little, it leans on the one who understands the system, the developer. Every developer has a part of that graph, right? In their hand, at least in the domain they know. The agent generally speaking doesn't. If this doesn't sound crazy, right? Imagine an agent that could see one file at a time maximum and can only look five messages back. Constraints again, both in space, what it can see, and time, how far in the past it could see. You would say that's impossible to work in. What we have now is similar to that crazy picture. And the more complex the organization is, the more apparent it becomes. I will show you how we solved it. Other organizations I talk to have similar solutions. So look at the problem and the solution conceptually, not a specific tool, although the tool is pretty cool. We built an agent agnostic meta harness called Polygraph. Let me show you what it does and how it fixes the issues we just discussed. The first idea that we arrived at is that if a GitHub user, any user, has access to thousands of repos, some of them they own, many of them are open source, we can analyze them and extract a lot of metadata out of them to build a unified dependency graph. No line of code changes in those repos. So that all happens on a side, right? And then we can get this metadata and feed it to the meta harness and create an illusion of one big code base the agent can read and write anywhere. This is my personal graph. I only have about 300 repos I own, right? And thousands of open source repos my projects depend on. Polygraph computes what each one produces, each repo, each project in each repo. What each project in each repo consumes, package-wise. What APIs they produce and consume and lots of other stuff, right? And it stitches this together into this one big body of code that your agent can work with. So let's see what it does, right? The first thing it does is it lets you start the session to print the relevant repositories in, right? So what it needs to do, it needs to set up the source code, install dependencies, set up an agent for each repo, wire them up so they can work together, and provide a clean, beautiful TUI to make non-trivial changes without getting lost. I will show you how it all works in a second. So that's pulling information in. Pulling information in is only one part of the story, right? Honestly, it's an easy part. Making changes is harder. If you have 10 repos in one session, it means you can have 10 pull requests, right? You need to run CI, you need to coordinate all of it, right? You need to do all this stuff, right? What if one of them fails, right? Polygraph treats all the CI as one vector. I will show you how it all works in a second. So that's pulling information in. Pulling information in is only one part of the story, right? Honestly, it's an easy part. Making changes is harder. If you have 10 repos in one session, it means you can have 10 pull requests, right? You need to run CI, you need to coordinate all of it, right? You need to do all this stuff, right? What if one of them fails, right? Polygraph treats all the CI as one vector. If you look at the earlier example, when we run CI for UI module 1 and module 2, if module 1 fails within a polygraph session, it will figure out who fixes it. Whether module 1 needs a patch or the UI component itself is wrong and incompatible with module 1, at which point everyone will need a patch, right? Polygraph lets you treat complex multi-repo change as if it was a single repo change. The same machinery, by the way, fixes episodic memory. Because we capture your work, no matter how many repos are involved, we know your intent, the repository is involved, PRs, we also capture all agent traces. Because we capture all of this stuff, we can relate it. So now we can say your work in one repo connects to another work in another repo, right? And all of that lets us restore any session, any piece of work on any machine or reference it from anywhere. And I will show you again how it works in a second. What you get is an agent with identical photographic memory of your entire organization. It understands how repos are written, how they relate, how they put together, and remembers every session from every repo by every developer, right? And that creates a completely different development experience. Let me show you. First, let's look at how we create a session. Something simple. You run a command and you pick some repositories from a list. Here's a tiny GitHub work with only three repos because it's a demo. I pick a backend and a frontend. Let's say I need to make a change that changes the API and has to update both the API and how stuff is being displayed. I need to give my session a name. I need to pick an agent from the ones I have installed. I picked Claude, but any install agent works the same way. Remember, Polygraph isn't an agent. It's a meta harness around an agent that makes them more capable. And in a second, the agent boots. And here I could interact with it as if I was in a single repo, even though multiple repos are involved, right? I could give it instructions. It's going to plan out the change. There are some cool animations in the UI as well. Eventually, it figures out how the two repos relate and what the change is. I can ask it to implement the change. My interaction with this is exactly the same as if I was working on a single repo. The fact that there are multiple repos involved is not really important, right? The only part where it becomes important is that I have multiple pull requests, right? But I also get a polygraph session where those pull requests are. If I look at the session, I will see I have a description. That description of the session, it describes the work conceptually, bypassing the repo boundary, saying we have to change stuff in this repo and change stuff in that repo. It gives me a good view of which repos are involved, pull requests involved, CI in those repos, everything I need to know. A lot of the stuff is what I would have in a single repo but many, right? And I also have all the agent logs captured as well, which is important for resuming, which I'm going to show you in a second. [SPEAKER_00] Now it gets interesting. [SPEAKER_00] I already saved one re-explanation. I didn't re-explain the backend change in the frontend repo, right? I explained the change once and I got it implemented in both repos and it's all in agreement. Now let's resume a session. Say I want a coworker to finish the backend change. Perhaps they own the backend repo. I send them the session. They resume it on their machine, right? So this time I'm sending them a session. They could run the command. Different machine, different everything. They use different terminal, right? They would reconstruct it on their machine. They don't have the session, right? They've never worked on it. They can pick an agent. The agent they pick could be a different agent, right? I use Claude in the original session. Say I want a coworker to finish the backend change. Perhaps they own the backend repo. I send them the session. They resume it on their machine, right? So this time I'm sending them a session. They could run the command. Different machine, different everything. They use different terminal, right? They would reconstruct it on their machine. They don't have the session, right? They've never worked on it. They can pick an agent. The agent they pick could be a different agent, right? I use Claude in the original session. Let's say they're using a different one, Codex. The same setup happens on their machine. Same repos, same SHAs, everything set up correctly. Agent starts in each repo like in mine, right? They all connect it again, so they work together. They all primed with a trace captured from my machine. So the backend repo agent on their machine has the same SHA and the same history. The frontend repo situation is the same. It's checked out at the correct SHA, has the agent running with the correct history. So my agent was Claude. They're Claude, but they share memory. And they could actually make changes in here as shown in a small video. But the memory sharing part is key, right? I can work, they can work, and we can share our memories, although we use two different agents and different machines. The full state of my session gets materialized on their machine. It's more about the state, right? The state of the world attached to the session is what enables them to continue my session, even though they didn't do anything with it originally. It's close to the transporter in Star Trek. A whole copy of my session is materialized on their machine so they can continue. And that's how I often work. When there is a pull request for me to review and I have questions, I usually don't ask the person. I resume their session on my machine. I can work, they can work, and we can share our memories, although we use two different agents and different machines. The full state of my session gets materialized on their machine. It's more about the state, right? The state of the world attached to the session is what enables them to continue my session, even though they didn't do anything with it originally. It's close to the transporter in Star Trek. A whole copy of my session is materialized on their machine so they can continue. And that's how I often work. When there is a pull request for me to review and I have questions, I usually don't ask the person. I resume their session on my machine. I get the exact state, fully functional, zero setup. And then I just talk to my agent about the decisions we made, right? Because all the decisions are in the traces captured. So my agent knows exactly what the other person talked to their agent. Side note, this is also useful when I want to switch from, say, Claude to Codex mid-session, when something goes down, okay? Take the earlier case I talked about where a bug lands in production. Here, I'm going to reference this session and say it's broken. Can you figure out what's wrong and fix it? The agent will look it up. We'll download what it needs. If description and high level information is enough, that's great. If not, it's going to pull relevant repos, relevant chats, agent logs, right? It's going to get all this information from the original session to reconstruct that state such that it can do the necessary fixes as shown here. Here, actually, it provided a fix, right? I only had to say this happened, there is a bug, that's it. No extra information was required for me to provide. So far, we have manually selected repos in sessions, but we don't have to, right? Instead of selecting the repos by hand, I can also tell the agent what I want. Remember, that graph has all this intelligence, right, about how repos relate. I could tell my agent, find every repo that depends on a particular version of a library and update it, right? I didn't have to select them. It knows a lot of metadata about what's going on. I can also ask loose questions. Things like, what if I want to write a blog post or an article? I could describe it and it will figure out which repo is the most relevant based on relationships between repos and what's in them. Another example. Let's say I want to add vector index into the PR collection. And I want to know if anyone at any point did something relevant in any repo that I can draw from. So in this case, if I do it, I'll see that it will find several sessions that appear to be relevant. And I can load one of them or both of them, right? It's useful for many reasons. Just one small example. It helps with best practices and consistency. Instead of doing stuff from scratch, where every single permutation is bespoke, I can make it replicate the approach used in a session by an engineer I respect. Now our code across repos is consistent. That's a big deal. There is a lot more to it, of course. If you are in a repo, I can ask for sessions, it will prioritize sessions that are relevant to the repo and vice versa. If I'm asking for repos, it will look at my session and see what similar sessions tend to bring in. There is a lot of interesting intelligence that makes it a lot more useful than it appears at first glance. Lastly, so far I've used the polygraph CLI, the meta-harness CLI to start it and then you can start Claude or Codex or whatever from within it. But you don't have to use it this way. So in this case, I'm already in a Claude session, but it works with anything. And I could just say, I actually think a separate repo would be useful. Maybe I'm working on a VTest plugin in this Annex repo and I could say, can you add the VTest repository to this session so I know what's going on? In this case, we'll engage polygraph and we'll set it up, configure everything and we'll bring the VTest library, which is the VTest repo, the open source repo to my session. So now my agent can explore it. It could figure out how it works and maybe resolve an issue I have in my repo. I much prefer this to context windows, because if I have the real code, the agent can go really deep. So deep problems I discover this way. All right. So agents are constrained in space and time. They only see a small fraction of the code base as they don't know the past. And both limits could be lifted. Polygraph gives agents access to the entire code your organization can reach. So now my agent can explore it. It could figure out how it works and maybe resolve an issue I have in my repo. I much prefer this to context windows, because if I have the real code, the agent can go really deep. So the deep problems I discover this way. All right. So agents are constrained in space and time. They only see a small fraction of the code base as they don't know the past. And both limits could be lifted. Polygraph gives agents access to the entire code your organization can reach, the one you own in open source. So it's no longer constrained in space. Any agent can bring all of it. Right. And it gives your agent a perfect memory of what happened. Every session, every decision made is within reach. Because it crosses developer boundaries, it's not per developer. The agent can have more context than any single developer. Like a thousand engineers have an organization, create all the sessions. They're all accessible to each of them. Almost like the Borg. Every agent can run, but every developer contributes to one big hive mind. So if it's interesting, my name is Victor. You can follow me on Twitter. If you want to check it out, go to trypolygraph.com and see if it works for you. Thank you. They don't have the session, right? They've never worked on it. They can pick an agent. The agent they pick could be a different agent, right? I use Claude in the original session. Let's say they're using a different one, Codex. The same setup happens on their machine. Same repos, same SHAs, everything set up correctly. Agent starts in each repo like in mine, right? They all connect it again, so they work together. They all primed with a trace captured from my machine. Thank you. They don't have the session, right? They've never worked on it. They can pick an agent. The agent they pick could be a different agent, right? I use Claude in the original session. Let's say they're using a different one, Codex. The same setup happens on their machine. Same repos, same SHAs, everything set up correctly. Agent starts in each repo like in mine, right? They all connect it again, so they work together. They all primed with a trace captured from my machine. So the backend repo agent on their machine has the same SHA and the same history. The frontend repo situation is the same. It's checked out at the same, at the correct SHA, has the agent running with the correct history. So my agent was Claude. They're Claude, but they share memory. And they could actually make changes in here as shown in a small video. But the memory sharing part is key, right? I can work, they can work, and we can share our memories, although we use two different agents and different machines. The full state of my session gets materialized on their machine. It left memory about the state, right? The state of the world attached to the session is what enables them to continue my session, even though they didn't do anything with it originally. It's close to the transporter in Star Trek. A whole copy of my session is always materialized on their machine so they can continue. And that's how I often work. When there is a pull request for me to review and I have questions, I usually don't ask the person. I resume their session on my machine. I get the exact state, fully functional, zero setup. And then I just talk to my agent about the decisions we made, right? Because all the decisions are in the traces captured. So my agent knows exactly what the other person talked to their agent. Side note, this is also useful when I want to switch from, say, Claude to Codex mid-session when something goes down, okay? Take the earlier case I talked about where a bug lands in production. Here, I'm going to reference this session and say it's broken. Can you figure out what's wrong and fix it? The agent will look it up. It will download what it needs. If the description, high level information is enough, that's great. If not, it's going to pull relevant repos, relevant chats, agent logs, right? [SPEAKER_00] It's going to get all this information from the original session to reconstruct that state such that it can do the necessary fixes as shown here. Here, actually, it provided a fix, right? I only had to say this happened, there is a bug, that's it. No extra information was required for me to provide. So far, we have manually selected repos in sessions, but we don't have to, right? Instead of selecting the repos by hand, I can also tell the agent what I want. Remember, that graph has all this intelligence, right, about how repos relate. I could tell my agent, find every repo that depends on a particular version of a library and update it, right? I didn't have to select them. It knows a lot of metadata about what's going on. I can also ask loose questions. Things like, what if I want to write a blog post, right, or an article? I could describe it and it will figure out which repo is the most relevant based on relationships between repos and what's in them. Another example. Let's say I want to add vector index into the PR collection. And I want to know if anyone at any point did something relevant in any repo that I can draw from. So in this case, if I do it, I'll see that it will find several sessions that appear to be relevant. And I can load one of them or both of them, right? It's useful for many reasons. Just one small example. It helps with best practices and consistency. Instead of doing stuff from scratch where every single permutation is bespoke, I can replicate the approach used in a session by an engineer I respect. Now our code across repos is consistent. That's a big deal. There is a lot more to it, of course. If you are in a repo, I can ask for sessions, it will prioritize sessions that are relevant to the repo and vice versa. If I'm asking for repos, it will look at my session and see what similar sessions tend to bring in. There is a lot of interesting intelligence that makes it a lot more useful than appears at first glance. Lastly, so far I've shown, I used the Polygraph CLI, the kind of meta-harness CLI to start it and then you can start Claude or Codex or whatever from within it. But you don't have to use it this way. So in this case, I'm already in a Claude session, but it works with anything. And I could just say, hey, I actually think a separate repo would be useful. Maybe I'm working on a VTest plugin in this Annex repo and I could say, can you add the VTest repository to this session so I know what's going on? In this case, we'll engage Polygraph and we'll set it up, configure everything and we'll bring the VTest library, which is the VTest repo, the open source repo to my session. So now my agent can explore it. It could figure out how it works and resolve an issue I have in my repo. I much prefer this to context windows, because if I have the real code, the agent can go really deep. I discover deep problems this way. So agents are constrained in space and time. They only see a small fraction of the code base as they don't know the past. And both limits could be lifted. Polygraph gives agents access to the entire code your organization can reach, the one you own in open source. So it's no longer constrained in space. Any agent can bring all of it. And it gives your agent a perfect memory of what happened. Every session, every decision made is within reach. Because it crosses developer boundaries, it's not per developer. The agent can have more context than any single developer. So agents are constrained in space and time. They only see a small fraction of the code base as they don't know the past. And both limits could be lifted. Polygraph gives agents access to the entire code your organization can reach, the one you own in open source. So it's no longer constrained in space. Any agent can bring all of it. Right. And it gives your agent a perfect memory of what happened. Every session, every decision made is within reach. Because it crosses developer boundaries, it's not per developer. The agent can have more context than any single developer. A thousand engineers in an organization create all the sessions. They're all accessible to each of them. Almost like the Borg. Every agent can run, but every developer contributes to one big hive mind. If you're interested, my name is Victor. You can follow me on Twitter. If you want to check it out, go to trypolygraph.com and see if it works for you. Thank you. Every session, every decision made is within reach. Because it crosses developer boundaries, it's not per developer. The agent can have more context than any single developer. Like a thousand engineers have an organization, create all the sessions. They all accessible to each of them. Almost like sort of the Borg. Every agent can run, but every developer contributes to kind of one big hive mind. So if it's interesting, my name is Victor. You can follow me on Twitter. If you want to check it out, go to trypolygraph.com and see if it works for you. Thank you. You can follow me on Twitter. You can follow me on Twitter. You can follow me on Twitter. You can follow me on Twitter.