AI Engineer

No Memory, No Harness: Why the Database Is the Last Line of Defense — Kay Malcolm, Oracle

1964 summary words 9 min summary Watch video

Start with the signal

9 min read

Summary

At-a-Glance

  • Verdict: Watch fully
  • Core thesis: An enterprise agent is not just an LLM plus workflow: it needs a harness with durable, shared, governed memory, and Malcolm argues that a unified multimodal database is the most reliable control point for that memory.
  • Why it matters: This directly addresses a core agent-systems failure mode: AI can accelerate individual contributors while reducing team-level throughput when context, decisions, branch history, and tool traces do not persist across people, sessions, and agents.
  • Best use: Use it as an architecture and operating-model prompt for designing agent memory, especially for multi-agent or human-agent workflows; discount the Oracle-specific product conclusion but retain the shared-memory and data-governance design principles.

Executive Summary

Kay Malcolm frames the problem through her Oracle product team: Codex made individual developers faster, but the team did not become proportionally more productive because code checked in by one geography arrived without the AI context, decisions, and procedural history that produced it. Git tracked code changes but not intent, leaving downstream developers to repeat validation, reconstruct rationale, and resolve divergence. Her proposed missing layer is shared agent memory.

She defines an enterprise agent as an LLM and workflow plus tools, in-window context, retrieval, persistent memory, and security guardrails—the “harness” that lets the model act safely and consistently. Memory is presented as the central nervous system of that harness, carrying operational context between the model and its tools, users, workflows, and other agents.

The central technical argument is that enterprise memory spans several forms—short-term, long-term, episodic, procedural, and semantic—and will often require relational, JSON/document, graph, vector, text, and immutable/auditable storage capabilities. Malcolm argues that spreading these forms across disconnected specialty databases creates integration overhead, ambiguity over the source of truth, token waste, and unreliable agent retrieval.

Oracle is the solution being sold: Malcolm says Oracle AI Database 26AI can store multiple data types and workloads in one database and that Oracle’s Agent Memory SDK, used in a “Poly” memory-broker example, can associate context with commits, branches, and forks while allowing developers to remain in control. The broadly useful lesson is not that Oracle must be the store, but that agent memory requires explicit ownership, cross-session sharing, retrieval discipline, provenance, and a governed data plane.

Key Takeaways

  • Claim: AI coding tools can increase individual output without increasing team throughput when the context behind generated code is not transferred with the code. | Evidence: Malcolm describes an EMEA team checking in Codex-assisted code at 4 a.m.; when the U.S. team began work, it had the code but not the Codex context. The result was continued testing, validation, and repository divergence despite increased AI token spend. | Implication: Treat context capture and handoff as a first-class delivery artifact alongside code, tickets, tests, and documentation; otherwise AI may shift rather than remove coordination costs. | Caveat: This is an internal Oracle team anecdote rather than a quantified productivity study, but it identifies a credible coordination failure mode in distributed AI-assisted engineering.
  • Claim: Git alone is insufficient as an agent-era collaboration system because it records code changes, not the human or agent intent behind them. | Evidence: Malcolm explicitly identifies missing rationales, unresolved questions, conflict history, next steps, and the decisions an agent made as information Git does not inherently track. | Implication: A production agent platform should connect memory to operational objects such as tasks, commits, branches, tool calls, approvals, and outcomes rather than treating conversational logs as the only memory source.
  • Claim: A real enterprise agent requires a harness beyond the model and workflow: tools, context, memory, retrieval, and security guardrails are essential components. | Evidence: Malcolm’s model of an enterprise agent includes the LLM/workflow as the “brain,” with the harness as the body that enables action; she calls memory the central nervous system connecting the model to the rest of that system. | Implication: Evaluate agent systems by their control plane and state-management design—not merely model quality or prompt orchestration—because durable action depends on those surrounding components.
  • Claim: Enterprise agent memory is not one data type; it includes short-term session state, long-term persistent facts, episodic interaction history, procedural tool/step knowledge, and semantic knowledge. | Evidence: Malcolm enumerates these five memory categories and maps examples such as session memory to short-term memory, prior interactions to episodic memory, and tools/steps to procedural memory. | Implication: Design memory schemas and retrieval policies by memory function and lifecycle—for example, session state, durable user/org facts, execution traces, and knowledge retrieval—rather than putting all agent state into an undifferentiated vector store. | Caveat: The categories overlap in practice, and the transcript does not provide a formal schema or retention policy for separating them.
  • Claim: Fragmenting agent memory across separate relational, document, graph, and vector systems creates a source-of-truth and retrieval problem that agents may resolve incorrectly and expensively. | Evidence: Using a volunteer exercise, Malcolm illustrates that separate stores cannot easily coordinate who owns the truth for a shared fact. She ties this to her Southern Company experience, where adding separate unstructured and graph databases multiplied operational/security/patching obligations, and says an agent querying scattered data can choose wrongly and consume excess tokens. | Implication: Before adding specialized memory stores, define canonical ownership, synchronization rules, cross-store retrieval behavior, provenance, and operational accountability; favor consolidation where it materially reduces these costs. | Caveat: A unified database is not the only viable pattern: a federated architecture can work if it has strong metadata, identity, lineage, synchronization, and retrieval controls. Malcolm’s conclusion is naturally shaped by Oracle’s product position.
  • Claim: Shared memory must remain owned and governed by the enterprise rather than being delegated entirely to a model provider’s local or file-based memory mechanism. | Evidence: Malcolm cites Harrison Chase’s formulation, “your harness, your memory,” and argues that file-system-style memory such as memory.md may work for one agent but becomes problematic when scaled across enterprise participants. She also references an OpenAI in-house data-agent paper as identifying memory as crucial for correct filtering rather than indiscriminate streaming. | Implication: Keep durable agent state, authorization boundaries, retention, auditability, and retrieval logic in infrastructure the organization controls; use model-provider memory only as a constrained convenience layer where appropriate. | Caveat: The transcript names these references but does not explain their full architectures or establish that all hosted-model memory features are unsuitable.
  • Claim: A memory broker can turn isolated human and agent work into coordinated work by preserving and routing context across branches, forks, sessions, and participants. | Evidence: Oracle’s internal “Poly” example uses an Oracle Agent Memory SDK and Autonomous Database to retain procedural, episodic, and long-term context, identify the relevant fork, branch, and commit, and let one developer continue work with another developer’s context while developers retain control. | Implication: For multi-agent engineering workflows, consider a memory-broker service that links state to work artifacts and exposes scoped retrieval, rather than relying on each agent’s isolated transcript or repository access. | Caveat: Poly is presented as a simplified internal example, without details on conflict resolution, permissions, evaluation metrics, retrieval accuracy, or the cost of maintaining the broker.

Detailed Brief

Multimodal storage argument and Oracle’s proposed mapping

  • Claims: Malcolm positions Oracle AI Database 26AI as a single platform capable of handling the heterogeneous storage needs of agent memory.; Her proposed storage mapping puts long-term and procedural memory in relational storage; short-term and long-term memory in JSON; procedural relationships in graph; and episodic and semantic memory in vector storage alongside text.; She further claims the same Oracle database can natively support JSON, graph, vector, spatial, and blockchain/immutable-data capabilities, including at the same table or partition level.
  • Evidence: Her Southern Company story is used to contrast a consolidated platform with an estate of separate specialized systems: an unstructured database and Neo4j each introduced additional security and patching meetings.; She states that Oracle can be deployed across AWS, GCP, Azure, OCI, and on-premises environments.
  • Caveats: The talk does not demonstrate benchmarks, data-model implementation details, interoperability boundaries, or comparative evidence against Postgres-based, lakehouse, graph-plus-vector, or federated alternatives.; The suggested one-to-one mapping of memory types to storage models is a heuristic, not a universal design requirement.
  • Implications: The useful architecture question is whether the memory plane can serve multiple retrieval and provenance needs without forcing agents to reconcile inconsistent copies.; Consolidating storage can reduce operational burden, but only if it does not compromise independent scaling, access isolation, latency, or specialized-query performance.

Implementation entry points presented by Oracle

  • Claims: Oracle offers an Agent Memory package available through “pip install Oracle agent memory.”; Malcolm says the SDK can retain live conversations, memories, and facts while deciding what should be retained.; The proposed deployment lets teams pair the memory layer with a chosen LLM or a local model through Oracle Private AI Services Container.
  • Evidence: In the Poly example, the memory SDK runs against Oracle Autonomous Database and is used to transfer context between developers Kevin and Linda.; Malcolm points viewers to Oracle AI Developer Hub, LiveLabs, and OCI Always Free for experimentation; she states the free tier includes a database, compute, 3,000 emails per month, and 200 GB of storage.
  • Caveats: No technical demonstration of the SDK’s retention heuristics, authorization model, evaluation methodology, exportability, or failure handling appears in the transcript.; The stated free-tier and product capabilities should be confirmed against current Oracle documentation before making implementation decisions.
  • Implications: A practical evaluation should test the SDK against a representative workflow involving branch handoffs, permission-scoped retrieval, stale-memory handling, conflicting context, audit logs, and model-provider portability.

Notable Concepts & Terms

  • Agent harness: The operational system around an LLM—tools, context, memory, retrieval, and guardrails—that enables the agent to act in an enterprise setting.
  • Memory broker: A coordination layer, exemplified by Poly, that captures, associates, and retrieves context across people, agents, branches, forks, and work sessions.
  • Procedural memory: Memory of tools, steps, and operational relationships; Malcolm treats it as important for preserving how work was performed, not only what outcome was reached.
  • Episodic memory: Memory of prior interactions or events, used to preserve what happened the last time an agent interacted with a user, system, or task.
  • Semantic memory: Persisted knowledge or facts relevant to enterprise work; Malcolm associates it with vector retrieval and text storage.
  • Source of truth: The canonical owner of memory data; the talk argues that disconnected specialist stores leave agents uncertain about which answer is authoritative.
  • Oracle Agent Memory SDK: Oracle’s productized memory layer, presented as an SDK for managing conversations, facts, retained memories, and retrieval in an Oracle-backed agent architecture.
  • Oracle AI Database 26AI: Oracle’s proposed unified data plane for agent memory, claimed to support relational, JSON, graph, vector, spatial, and immutable/blockchain-oriented data capabilities.

Operator Notes / Why Ken Should Care

  • Audit current agent workflows for handoffs where code, task state, approvals, tool traces, or decision rationale are lost between people, sessions, and agents.
  • Define a memory contract that specifies memory classes, canonical sources, retention/deletion rules, provenance requirements, access controls, and conflict-resolution behavior before selecting a vector store or framework.
  • Prototype a memory broker against one high-friction workflow—such as PR review, incident response, or research-to-execution handoff—and measure rework, retrieval precision, stale-context incidents, and token consumption.
  • Require any memory platform evaluation to demonstrate artifact-linked context retrieval across commits/tasks, tenant and role isolation, auditability, exportability, and behavior when memory conflicts with current source-of-truth data.
  • Treat provider-native chat/file memory as non-authoritative until it is proven to meet enterprise persistence, sharing, authorization, and portability requirements.

Source/Metadata

  • Title: No Memory, No Harness: Why the Database Is the Last Line of Defense — Kay Malcolm, Oracle
  • Transcript words: 3180
  • Duration seconds: 1297
  • Timestamp note: No timestamps or chapters were present in the supplied transcript.
Search this video Resume transcript at 0%
Full transcript 2974 words · 14 min read
0:00

Everyone, are we having fun?

0:12

Oh, you've got to give me way more than that. So let me tell you, my name is Kay Malcolm. I am a retired hip-hop instructor. So if I don't get more energy than that, we will start. We'll start. Are you having fun? Okay, all right. So here's what we're going to talk about today. Now you guys have heard a lot about two letters. Does anyone want to guess what those two letters are that I'm going to talk about today? That was pretty good, DB.

0:45

I'm going to talk about AI, but I'm specifically going to talk about agent harnesses. But before I do that, I want to introduce you all to a few people. Is that okay? Yes or yes? Is that okay? I gave choices. Yes? Anyway. All right. Okay. All right. This is my team. I run an outbound database product management team at Oracle. I've been at Oracle a really long time, 20 years. Funny story. I started when I was 12. So don't do the math and don't start adding in your head. And we've got a problem. That problem is I've got one group that does platform development. And then I have another group that does content development for Live Labs, a platform that I wrote myself.

1:33

So, yes, I'm an engineer, but I'm also a developer. And then I've got another group who does QA. And then I have another group who does my front end development. With AI, here's what I found out as a leader. Because in the token maxing era of 2025, because we're not token maxing anymore, we are responsible AI-ing now. But in the token maxing era, the thing that I found out was while AI was making the individuals on my team faster, there was another problem it was creating. It wasn't making my team more productive.

2:08

And the reason was when one team from the Netherlands checked in code at 4 a.m. in the morning—because I've got half of my team that's in EMEA and I have half of my team that are here in the United States—they checked in the code, but they didn't check in their context from Codex. We use Codex at Oracle.

2:22

So then when the US team woke up, they got the code, but no information about the context. So we used AI to solve a problem that AI created. Here's what we did. Oh, well, let me talk about this first. So some of the issues. The context, like I said, wasn't shared. GitHub wasn't tracking that. I had repositories that were diverging. And I was asking the managers who work for me, what's happening to your teams? Why are we not going faster? We're spending all of this money on tokens. We're spending all this money on AI. Yet something is missing because we're still spending time doing testing and validation. So our net net wasn't really working for us.

3:26

Because Git records the code and not human intent. So it's a problem. And even though code creation was no longer our problem, we still had a bottleneck. We needed a collaboration layer. Now, I do have members of my team in the audience. So don't judge me. And you know who you are. I'm not saying that you all didn't collaborate. But now we've got a new team member. And that new team member is AI. So we needed to figure out how to track our progress and our next steps. How to rationalize decisions that the agent was making. We needed to figure out how to resolve questions and conflicts. Okay. Hold my problem. Will you all hold my problem for me? Right here.

4:38

We're going to just tuck that in a little box. Let me define what an enterprise agent actually is. Now, most people think that an enterprise agent is the model and workflow. How many people agree with me? Man, this is a tough crowd. Okay, one person. Okay, the rest of you think it's a little bit more. Okay. Let's see what. Could it be that a real enterprise agent has tools? Tools are how it does things. Context. The context, that's a context window. That's what's in the actual prompt. Memory. Huh. And if you're thinking, wait, Kay, memory, you just said that the model is the brain of the operation. Hold tight. We're going to talk a little bit more about memory. Retrieval.

5:40

Because you don't want to get everything back. So that's being able to retrieve the right information back. And then, I know that there are a lot of developers here, and you all don't care about security. I care about security. Because I work for the most secure database company. And I used to work for an agency that has no name. But guardrails is also important. This is the harness. I speak in analogies, and I speak in stories. Because if I tell you this and Marvel, you know exactly what I'm talking about. So the agent, think of it as the model, little brain floating in a glass jar, plus this harness. This harness is the body.

6:21

So it's how the agent can actually do things and get things done. That memory, that's the part of the central nervous system. And you remember, the central nervous system connects the brain to the rest of the body. Legs, arms. That's the part of the central nervous system that carries context. So you remember my problem with Git? What I needed was memory. Okay, so there are a number of memory types. I chose five, the five most common ones that people talk about. And these are the ones that I want you to remember. The first one is short-term memory. That's the session, right? And so if you're storing memory of an AI process, that is the short-term memory.

7:21

If you're with chat, cloud code, Codex, pick your poison. The long-term memory is what persists across sessions. Episodic memory. What happened the last time I interacted with fill-in-the-blank? That's your episodic memory. Procedural memory. Tools. Steps that were taken. And then finally, semantic memory. And semantic memory, because we're talking enterprise agents. We're not talking the agent that I built, Sasha Fierce. Because remember, I told you guys that I'm a dancer. So of course my chief of staff is going to be called Sasha Fierce because that was Beyoncé. Any Beyoncé fans? Okay, I'm sorry. All right. We've got to focus. Okay. So these are the memory types.

8:25

Now, when you're defining this real enterprise agent and this memory, there's something you need to consider. Where to store it. And so I'm going to tell you guys a story. But when I tell you the story, you have to promise me that you're not going to judge me. Do you promise? Do you promise? Yes. Yes. You're not recording me, right? Because this doesn't paint me in a good light. Okay. All right. The world of data was once simple. I've been at Oracle a long time, but I came from a customer.

9:05

That customer's name was Southern Company. It was a power company. I'm based out of Atlanta. And I was hired at Southern Company because I was a rock star performance tuner. You had a SQL query. I mean, I'm dating myself, but whatever. You had a SQL query. I knew all of the init.ora parameters. Even the ones when you called support and they said, don't remember these. Don't write them down. I wrote them down in my little notebook. I could tune a query within one inch of its life. Then, one of you came to my desk because, I mean, the world was rows and columns. It was a great time back in my Al Bundy days. And said, hey, I need to store data unstructured. Why?

9:20

Why do you do that? And so, me being Kay, the diligent DBA, I was, let me figure it out and get back to you. Did I get back to him? I didn't get back to him. Now, the thing you have to know about Southern Company was for every database system that a DBA managed, I had to attend two meetings. Today, when I hear Sarbanes-Oxley, I still throw up a little bit in the back of my throat. So, I had to attend a security meeting and a patching meeting every week. Never failed. Now, because this developer installed a database that was specialized for unstructured. Okay, there were really smart people in the room. How many meetings am I going to now? Four.

9:55

Okay, I'm a little annoyed, but I'm, okay, we can do this. Then they said, okay, since you're such a good tuner, I need you to figure out this relationship. Now, the way that Southern Company worked, there was this people could die application. And it was like a Nokia phone that people who were climbing the towers, right? So, you guys have been in a storm and the power goes out, right? And then you're pretty sure that within maybe an hour or two the power will go on. Well, that system that would tell the people who were climbing those trees and risking their lives to turn the power back on, sometimes would have false positives or false negatives.

10:18

So, they wanted to look at all of the other polls in the area to try to get away from the false positive or the false negative. And so, I did that in a SQL query. And it was amazing. It was a five-nested union all statement. It was some of my best work. Now, it might have taken 20 minutes to work, but it was a predecessor to graph. Yeah, they installed Neo4j.

10:53

So, now, how many meetings am I going to? Six. That's a problem. So, I, oh, let me, I got ahead of myself. So, you know what I did? I quit. I left and I came to Oracle because I was, this is a problem and maybe I can go to Oracle to help solve it. So, then Joe Mundy called me and he said, hey, we are installing Redis. Oracle is late to the game. We've got a vector database. Okay. But here's your problem, Joe. Agents now need access to all of this data. So, if data is in an Oracle database, if it's also in an unstructured JSON database, if it's in a graph database and it's in a vector database, where is your single source of truth? The agent has to figure that out.

12:06

Sometimes it'll get it right. Most times it'll get it wrong and it's going to burn up a whole bunch of tokens. And so, now, if you want to store your memory somewhere, you can store it in a file system. You can store it in Claude or ChatGPT because we all know about the memory.md file. But that's going to be a problem. Now, I want to illustrate this. I need four volunteers. I can see you. Raise your hand. One. Two. Okay, I can't. Maybe I can't.

12:45

Three. I need a fourth. Ah, fourth in the back. Okay, fourth in the back. You are going to be our old reliable. You're going to be a relational database. Yes or yes? You got your assignment? Okay, and then there was someone here. You're going to be my unstructured database. Okay? And then where was my other, ah, very good. You're going to be my graph database. You good? Relationship guy. You look like a relationship guy. All right, very good. Fourth, where was my fourth? Was it you? Yes. Yeah. You are my vector database. Okay? Now, everybody be really, really quiet. For my four volunteers, I need you all.

13:54

I'm going to say something to you, and I need you all to decide how you're going to store it, and who's going to have the single source of truth. You can't get up from your seats, and you have to whisper, because if you talk loud, that's five extra tokens for you. Yes? Yes? Okay, are we ready? All right. The cow jumped over the moon. Go. Hmm. Doesn't really work, does it? That's a problem. Okay. Oracle, and if you don't forget one, if you forget everything I say, and you remember one thing, Oracle is not the Oracle that you think. That is why I am here today.

14:15

How many of you knew that Oracle could natively, in the same table, down to the same partition, store JSON, graph, vector? My vector friend over there. My JSON friend. Spatial. You want your memory to be immutable? Blockchain. In the same database. Raise your hand. Yeah. Yeah. We have a marketing problem. So, any data type can be stored in a 26AI database. Any workload. Anywhere. AWS, GCP, Azure, OCI, on-prem. Choice and flexibility. So now when we take this and we talk about the agent, I want to be able to store my long-term and procedural memory in relational. In JSON, I want to store my short-term and long-term memory.

15:11

Graph, I want to store procedural because procedural, that's how I figure out the relationships, right? The steps. My episodic and semantic memory, I need to do some vector and then store it also as text. Now, if I have four different databases, you all saw, they can't talk to each other. It's going to be a problem. And so what I'm saying to you today is the Oracle AI database is the best place to store this agent memory that's going to power your harness. Remember, your harness is your body and that memory is your central nervous system. Okay, back to Poly. So, the problem that I had, we solved it with a memory broker named Poly. We used agent memory.

15:35

We got out of that automatic continuity.

15:39

So, with my team, they were able to share not just their code, but Poly also kept track of the context. So, if one context window had procedural memory, episodic memory, information about the long-term memory, that was then shared with the other folks on the team. You could call them agents, if you will, they're just human agents, shared across forks. The developers on the team remained in control while Poly was able to create the context, figure out which fork and branch it belonged to, and which commit it belonged to. Now, this is a very simplistic example, but when you take this to the enterprise, here's what happens.

15:54

Memory is the thing that becomes non-negotiable in an agent's harness. Now, these are three papers that I read on the airplane. This first one is from OpenAI, and it's about its in-house data agent. And the thing that it says is, its in-house data agent actually needs memory. Memory was crucially important to ensure that its agent was able to filter correctly instead of trying to stream it. And it's not a green match. Harrison Chase said, your harness, your memory. And if you don't own your harness, you don't own your memory, which is key. And then, I'm sure you all are wondering, well, Claude has memory, why can't I use that?

16:44

Well, it's kind of file system memory, and it works with one. But just like in my example, when you scale past one, and you're going to scale past one in the enterprise, it creates a problem. So, Oracle has an Oracle agent memory package. Pip install Oracle agent memory. You get access to it. And this memory, this SDK that we have, is the thing that will hold your live conversations, your memories, your facts, and figure out what is worth keeping. So, if we look at Poly, now Kevin can share his context with Poly, our memory broker. We use the Oracle agent memory SDK. It's stored in our Oracle autonomous database.

17:49

We can use the LLM of our choice, or we can use a local model through the Oracle private AI services container. And then, Linda, who's actually sitting right here, can interact and work with Kevin, no issues. So, yes, AI makes individuals faster. Shared memory on an Oracle AI database makes teams faster. So, I don't want you all to compromise. In the age of AI, what 26AI does is you can choose and pick what's best. For agent memory, file systems stored in a database file system or in the database. If you need to do data modeling, you've got JSON, you've got relational. We've got choice. Okay. I've got some goodies for you.

18:45

The Oracle AI developer hub, that's where you can, guys, you can get coding materials, the applications, what I talked about today. LiveLabs.oracle.com, if you've done any of our workshops today, that happens to be something that I wrote myself about six years ago. And 40 million users ago. Spend my OCI tenancy money. Kick the tires on any Oracle technology for six hours, 12 hours, however long you need. And then, I'm giving you all a Mac mini. No, I'm just kidding. I'm giving you an OCI mini. So, I don't know if you knew, but there is an always free OCI. It is the most generous of any of the hyperscalers where you can get a free Oracle database, free compute.

19:42

You can send 3,000 emails a month, 200 gig in storage. And if you click on that, you can get access to it. Or just search Google for Oracle Cloud, always free. Connect with me. If you build something, will you all message me and let me know? Yes or yes? Yes. Thank you. I would like to receive you now. LiveLabs.oracle.com, if you've done any of our workshops today, that happens to be something that I wrote myself about six years ago. And 40 million users ago. Spend my OCI tenancy money. Kick the tires on any Oracle technology for six hours, 12 hours, however long you need. And then, I'm giving you all a Mac mini. No, I'm just kidding. I'm giving you an OCI mini.

20:47

So, I don't know if you knew, but there is an always free OCI. It is the most generous of any of the hyperscalers where you can get a free Oracle database, free compute. You can send 3,000 emails a month, 200 gig in storage. And if you click on that, you can get access to it. Or just search Google for Oracle Cloud, always free. Connect with me. If you build something, will you all message me and let me know? Yes or yes? Yes. Thank you.

21:34

I would like to receive you now. I would like to receive you now. I would like receive you now.

Reading tools

Type to find a passage

Appearance
Ask this transcript

Notebook

Export Markdown

Add a note