Lessons from Studying Every Memory System — Shlok Khemani, Independent

Transcript

Shlok Sulek Reviewer Okay. Hi, everyone. I'm Shlok, and I've spent the past year studying different memory systems. Now, before I get started, one thing I've realized, speaking to people over the last two days, is that memory is a very overloaded term now. It can mean a lot of different things. So when I talk about memory today, it is going to be in the context of personalization, especially for consumer AI applications. Now, a little bit about me. My claim to fame, the reason I get to speak to you here, is that I've spent the past year trying to reverse engineer how products like ChatGPT, Claude, Gemini, and Poke implement their memory systems. And I've then worked with multiple teams across different domains in helping them design their memory. I'm going to break the talk down into two parts. First, we're going to look at how memory has evolved over the past three years, especially in the context of ChatGPT and Claude. And then in part two, I'm going to discuss some of the lessons I've learned, maybe a rant, and where I think all of this is going. To kick things off, we go back to ancient times, which in our industry is 2023. This is ChatGPT just after the launch of GPT-4. Now, you could have back-and-forth conversations within a single thread, and context was maintained inside that thread. But as soon as it started a new conversation, nothing was carried over. Now, for early adopters, this wasn't a problem. GPT-4 was such an amazing model that if we ever had the need to carry context, we would do so by hand. But as ChatGPT started becoming more popular, as regular people started using it for things like learning, cooking, as a companion, the need for some sort of memory system became really apparent. So in February of 2024, we got ChatGPT Memory V1. And what you could do is ask ChatGPT to remember things about you. So you could say things like, "Hey, remember that I'm vegetarian." And ChatGPT would extract what it thought was a fact, which is that the user is vegetarian, store it in a list of memories, and this list was then added to the context window for every single conversation. You could also then go into settings and view this list of memories. And if you thought that something didn't apply anymore, you could delete a memory. Now, as the first serious memory implementation within our industry, I think this was a really decent effort, but there were also some fundamental flaws with it. The biggest one was that, as a user, because you could see every time a memory was created, it felt like you were responsible for both creating memories while you were just trying to have a conversation. So the burden of memory management fell to the user. Also, if you notice this list of memories here, these held true at the time they were being created, but that doesn't necessarily hold true over time. For example, it says that Shlok is going to Bengaluru. Now, I obviously am in SF right now. I am not going to Bengaluru, but this fact, this memory, is still added to my context window today. So staleness was another huge problem with this version of ChatGPT's memory. A little more than a year later, in April of 2025, ChatGPT released V2 of its memory, and this was a little more sophisticated. The most important addition was this thing called user knowledge memories. I am just going to call it a running profile for the rest of this talk. And what a running profile really is, is that every few days, ChatGPT looks at all the conversations you have had with it, extracts anything it thinks is important for it to know about you, and updates this profile that it maintains on you. Now, this update process is also what a bunch of folks now call dreaming. How many of you all were there for Lance Martin's talk yesterday? Okay, not many, but he did a great talk on this. So every few days, ChatGPT looks at the new conversations you're having, updates your profile, and then this updated profile is added to the context window for every single new conversation. These are two excerpts from my running profile. I want you to notice a few things. First, these are extremely dense memories. So ChatGPT tries to pack in as much context as it can within every single memory. What's essentially happening here is that they're trying to put in keywords almost like clues, and because LLMs, especially the frontier models today, are so good at inferring context from limited information, when you're having a conversation, it connects these clues to what you're talking about. Also, these are just two of 16 different sections in my profile. Other sections include my personal life and things I'm working on. In total, my profile is almost 4,000 tokens long. And because these updates are happening asynchronously, they're happening in the background, this new version does away with the flaw we discussed in V1, which is the burden of memory management was taken away from the user. But I want you to notice the highlighted memory. This is about places I traveled to in 2025. But if you pay attention, it says Thailand and Turkey, but the dates are overlapping. And that's because the source of this memory was conversations I was having with ChatGPT, deciding between where to go among these two places. Now, I did end up going to Thailand. I've never been to Turkey. But ChatGPT still says that I've been to Turkey in 2025. So the staleness problem with V2 didn't completely go away. Another very important thing is that if you go to your settings, ChatGPT doesn't let you view this raw profile. So you could view your memories from V1. Your raw profile is not visible to you. Now you may ask, "Shlok, how did you see your profile then?" And that's because this prompt works really well if you want to jailbreak ChatGPT and view your raw profile. You might have to attempt it a few times, try different thinking modes. But Claude enough and you shall receive. In August of 2025, Claude released its first version of memory. This surprised me a bit because if you compare ChatGPT and Claude, they are very similar applications. You have a chat box, you have back-and-forth conversations, you have a list of previous conversations, you can start a new conversation. My assumption going into studying Claude was that the memory systems would also be similarly designed. Not the case, at least for V1. So in V1 of Claude, you had no user profile, you had no list of facts. Instead, the model was given two tools. It was given a tool to search over previous conversations by keyword or topic, and it was given another tool to search over conversations by time period. So queries like, "What did we discuss last week?" or "What did we discuss at the start of November of 2025?" So in V1, every single conversation starts fresh, with no context on the user. And when the model thinks that it needs to retrieve something, it can do so on demand. On September 11 of last year, I released a blog post saying, "Claude's memory architecture is the opposite of ChatGPT's." This hit the Hacker News front page. Funnily, on that very day, Claude released V2 of its memory, and they added a running profile similar to ChatGPT, but with a few differences. First, Claude made this profile visible to users. So you could go to settings, and you could view your raw profile. Second, this profile was 1,000 tokens. So it was much smaller than ChatGPT's 4,000 tokens. And also, if you notice, these are complete sentences rather than the dense keyword approach of ChatGPT. So less dense and smaller. Claude's profile updates every 24 hours, versus ChatGPT's every few days. And Claude also let the user make explicit edits to this profile. So you could request an edit, and that edit would lead to a re-synthesis of the profile. It gave you an interface to manage previous edits, and you could delete the things that no longer held true. And this is how Claude's memory works even today. So it hasn't changed since September of last year. We have seen two updates within ChatGPT's memory this year, though. The first was that it added a tool to look over past conversations, like we just saw with Claude. So the model can retrieve summarized context based on queries it makes. And then a month ago, at the start of June, ChatGPT finally made the user profile visible to users, somewhat. So what you can see is an LLM-generated summary of your profile, which is weird because your profile is already an LLM-generated summary of your conversations. It's all a bit confusing. I've written about it, but it is visible in some sense. You could also request explicit edits to your profile. And with this update, ChatGPT deprecated the V1 fact list from its memory system. So what we've seen here is a convergence after three years of each of these products evolving independently, where they both now have a running profile. This profile is visible and editable, again, somewhat. And the model has tools to look over past conversations. Okay, so what can we learn from this evolution, and where are things going? I think the biggest lesson for me is that there is no single way to do memory. It wasn't too long ago that everyone, including me, assumed that RAG was the way to go about memory, where you would take conversations, chunk them, create embeddings, put them in a vector store, and then, as user queries came in, do some sort of semantic search. But as we saw, neither ChatGPT nor Claude really do this. Instead, they both evolved independently using different approaches. And while the general architectures have converged, the specific implementation details are still very different. And then if you look at Gemini, it also has a running profile, but each memory comes with detailed timing logs. So when was it created? When was it last updated? And then if you look at agents like ClaudeCode, OpenClaw, Hermes, they have completely different memory systems, with markdown files, heartbeat, knowledge bases, skills. The point being that there is no one way to do memory. The implication of this is that memory cannot be outsourced. If you're a serious team, you do not outsource memory. It is something that you build alongside your product. Your memory system evolves with your product, and it cannot be thought of as an afterthought. And there is plenty of evidence for this. So if you look at all of the top consumer products today across different categories, each of these has some form of memory, yet none of them outsource it. All of them build memory in-house. Lesson two: memory is a function of compute. What does that mean? Let's look at the costs associated with a running profile. So there are two types of costs. There is a cost to maintain a profile. And that depends on how frequently you update it and how much compute you apply to each update. And then, because these profiles are part of the context window for every single conversation, there's a cost of serving, which is the longer the profile, the more it costs to serve. Now, thought experiment: if you were to design the ideal memory system with no restraints, what would you do? You might want to update your profile every hour, or maybe after every conversation. You might want to task a bunch of Opus sub-agents with the update itself. And why stop at 4,000 tokens? Why not make it 400,000 tokens, store every single thing you would want about the user? Unfortunately, we live in a GPU-constrained world, and tradeoffs have to be made. And you can see that happening here. So ChatGPT, the profile length is 4,000 tokens. It updates every few days. So they have a higher serving cost for a lower update cost. And for Claude, it's 1,000 tokens, updates every 24 hours, so they make the exact opposite tradeoff. And this is what I mean by memory as a function of compute. You have to really think about how much compute you want to put into memory. Third, we had a bunch of talks about continual learning today. I'm not an expert here, but what I would say is that continual learning is already here. Going back to running profiles, what exactly is happening here? Your running profile starts with something that the model knows about you. This is then applied to every single conversation. Each of these conversations brings in new information. This new information is then synthesized through the dreaming process back into the profile. And then this profile dictates future conversations. And this loop keeps repeating itself again and again and again. And what you have is a continual learning process. Now, obviously, this learning loop is happening outside the weights. And a big question, particularly for consumer AI, is: will this process ever make its way into the weights? Now, obviously, updating weights, training models, is an expensive process. Continual learning does make sense at an enterprise level because the costs of these models are amortized across different employees and different customers. But that's not the case at an individual level. So big, open questions that I don't yet know the answers to are: will each of us get our own self-learning model? What data do we need to kick the CL process off? And how do we generate it? And finally, who's going to pay for this? How would the economics for this work? Gwern recently wrote an essay called Guardian Angels, where he explores this topic in beautiful detail. And if you're interested in what the future for one model per person looks like, I would recommend reading this. Finally, my rant is that we have a massive context problem. You could have the best memory architecture in the world. You could pour infinite amounts of compute into it. You could have continuous learning working at an individual level, where every single data point you bring up is somehow perfectly integrated into the model weights. Yet your memory system is capped by how much context it can gather about you. Let's go back to the example we discussed earlier, which was the conflict between where I traveled in the summer of 2025. These are the two source conversations. Again, I was trying to use ChatGPT to decide between which of these two countries to go to. Now, the decision to go to Thailand was actually made in a conversation I had with my partner in person. And ChatGPT couldn't reason over this or listen to this. But there were also traces of this conversation in my emails because I had flight and hotel bookings for Thailand. But even if ChatGPT is connected to my email, it doesn't reason over my email and it doesn't update my profile from my email, so it couldn't resolve this conflict. And I think that's okay. It's understandable. But what really bothers me is that ChatGPT today doesn't realize that there is a conflict. It's not curious about trying to fill in gaps in the information it knows about me. And this is particularly interesting and also infuriating because it's not a technology problem. It's a product problem. There is no fundamental reason at an LLM level that these things can't be solved. It's just that our products today are not designed to help us with this. So my personal stack today is a bunch of chatbots, assistants, vertical-specific applications, agents, and even hardware devices. Each of these products is trying to build its own memory of me. None of these memories are shared with each other. So I have to rebuild context within every single product from scratch every time. When something in my life changes, I have to individually update all of them. And then I have a bunch of very rich existing context sources, like my email, calendar, and photos. None of these products are able to reason over my existing, very rich context sources. So for me, none of this feels like 2026. And what I keep asking myself every day is, when will personal AI feel like personal AI? All of that frustration aside, I still think we're very early. Memory for AI is just a three-year-old field. Memory is also foundational to how humans interact with AI. And because I hope to be talking to AI all my life, and I know that's going to be the case for every single one of us here today, memory is something that's going to be important for the rest of human history. And there's so much left to build. That's it from me. Thank you so much. You can find my website. You can find me on Twitter. Have a great rest of the conference. Thank you. Thanks. and it cannot be thought of as an afterthought. And there is plenty of evidence for this. So if you look at all of the top consumer products today, across different categories, each of these has some form of memory, yet none of them outsource it. All of them build memory in-house. Lesson two, memory is a function of compute. What does that mean? Let's look at the costs associated with a running profile. So there are two types of costs. There is a cost to maintain a profile. And that depends on how frequently you update it, and how much compute you apply to each update. And then because these profiles are part of the context window for every single conversation, there's a cost of serving, which is the longer the profile, the more it costs to serve. Now, third experiment. If you were to design the ideal memory system with no restraints, what would you do? You might want to update your profile every hour, or maybe after every conversation. You might want to task Fable with a bunch of Opus sub-agents for the update itself. And why stop at 4,000 tokens? Why not make it 400,000 tokens, store every single thing you would want about the user? Unfortunately, we live in a GPU-constrained world, and tradeoffs have to be made. And you can see that happening here. So chat GPT, the profile length is 4,000 tokens. It updates every few days. So they have a higher serving cost for a lower update cost. And for Claude, it's 1,000 tokens, updates every 24 hours, so they make the exact opposite tradeoff. And this is what I mean by memory as a function of compute. You have to really think about how much compute you want to put into memory. Third, we had a bunch of talks about continual learning today. I'm not an expert here, but what I would say is that continual learning is already here. Going back to running profiles, what exactly is happening here? Your running profile starts with something that the model knows about you. This is then applied to every single conversation. Each of these conversations bring in new information. This new information is then synthesized through the dreaming process back into the profile. And then this profile dictates for the conversations. And this loop keeps repeating itself again and again and again. And what you have is a continual learning process. Now, obviously, this learning loop is happening outside the weights. And a big question, particularly for consumer AI, is will this process ever make its way into the weights? Now, obviously, updating weights, training models is an expensive process. Continuous learning does make sense at an enterprise level because the costs of these models are amortized across different employees, different customers. But that's not the case at an individual level. So big, big open questions that I don't yet know the answers to, which is, will each of us get our own self-learning model? What data do we need to kick the CL process off? And how do we generate it? And finally, who's going to pay for this? How would the economics for this work? Gwern recently wrote an essay called Guardian Angels, where he explores this topic in beautiful detail. And if you're interested in what the future for one model a person looks like, I would recommend reading this. Finally, my rant is that we have a massive context problem. You could have the best memory architecture in the world. You could pour infinite amounts of compute into it. You could have continuous learning working at an individual level where every single data point you bring up is somehow perfectly integrated into the model weights. Yet, your memory system is capped by how much context it can gather about you. Let's go back to the example we discussed earlier, which was the conflict between where I traveled to in the summer of 2025. These are the two source conversations. Again, I was trying to use ChatGPT to decide between which of these two countries to go to. Now, the decision to go to Thailand was actually made in a conversation I had with my partner in person. And ChatGPT couldn't reason over this or couldn't listen to this. But there were also traces of this conversation in my emails because I had flight and hotel bookings for Thailand. But because even if ChatGPT is connected to my email, it doesn't reason over my email and it doesn't update my profile over my email, it couldn't resolve this conflict. And I think that's okay. It's understandable. But what really bothers me is that ChatGPT today doesn't realize that there is a conflict. It's not curious about trying to fill in gaps in the information it knows about me. And this is particularly interesting and also infuriating because it's not a technology problem. It's a product problem. There is no fundamental reason from an LLM level that these things can't be solved. It's just that our products today are not designed to help us with this. So my personal stack today is a bunch of chatbots, assistants, vertical-specific applications, agents, and even hardware devices. Each of these products is trying to build its own memory of me. None of these memories are shared with each other. So I have to rebuild context within every single product from scratch every time. When something in my life changes, I have to individually update all of them. And then I have a bunch of very rich existing context sources. Like my email, calendar, photos. None of these products are able to reason over my existing very rich context sources. So for me, none of this feels like 2026. And what I keep asking myself every day is when will personal AI feel like personal AI? All of that frustration aside, I still think we're very early. Memory for AI is just a three-year-old field. Memory is also foundational to how humans interact with AI. And because I hope to be talking to AI all my life, and I know that's going to be the case for every single one of us here today, memory is something that's going to be important for the rest of human history. And there's so much left to build. That's it from me. Thank you so much. You can find my website. You can find me on Twitter. Have a great rest of the conference. Thank you. Thanks.