AI Agent Memory
- Authors
- Name
- Amit Shekhar
- Published on
I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.
I teach AI and Machine Learning, and Android at Outcome School.
Join Outcome School and get a high-paying tech job:
In this blog, we will learn about AI Agent Memory - why agents need it, the memory stack, the four core operations (write, read, update, forget), how memory flows at runtime, and the common mistakes.
We will cover the following:
- The Big Picture
- Why AI Agents Need Memory
- The Memory Stack
- The Four Core Operations
- How Memory Flows at Runtime
- What to Store and What Not to Store
- Common Mistakes and How to Fix Them
- Quick Summary
Let's get started.
The Big Picture
Before we go into the details, let's understand the big picture.
An LLM by itself is stateless. Every time we call it, it is like talking to someone who has never met us before. It does not remember our name, our last conversation, or what task it helped us with yesterday. AI Agent Memory is everything we build around the LLM to give it the feeling of remembering. The LLM does not actually remember. The agent remembers on its behalf.
In simple words:
AI Agent Memory = The system that lets a stateless LLM act like it remembers across turns, sessions, and users.
Think of an AI agent like a brilliant assistant who forgets everything the moment every meeting ends. Without memory, they cannot learn our preferences, recall past decisions, or build on yesterday's work. Memory is the notebook, the filing cabinet, and the personal diary that keeps them useful from one day to the next.
Why AI Agents Need Memory
Before we go deeper, let's understand why memory matters so much. Without memory, an agent has four big problems:
No continuity. The agent cannot remember what the user said 30 seconds ago once the current call ends - the next call starts fresh, with nothing carried over unless we explicitly pass it in.
No personalization. The agent cannot remember that the user prefers short answers, writes in Python, or lives in Bangalore. It treats every user the same.
No learning. The agent cannot remember what worked and what did not. It will make the same mistakes again and again.
No long tasks. Multi-step tasks that take hours or days become impossible. The agent forgets the plan halfway through and has to start over.
So, here comes AI Agent Memory to the rescue. It solves all four problems by giving the agent a place to store information and a way to bring it back when needed.
Now, let's understand what memory actually looks like inside an agent.
The Memory Stack
AI Agent Memory is not a single thing. It is a stack of layers, each with a different lifespan and a different purpose. Let's decode each layer.
+--------------------------------------------+
| AI Agent Memory |
| |
| Layer 1: Context Window (in the prompt) |
| - In the current prompt |
| - Lost when context overflows |
| |
| Layer 2: Short-Term Memory (session) |
| - Scratchpad, current task state |
| - Lost when the session ends |
| |
| Layer 3: Long-Term Memory (persistent) |
| - Facts about the user |
| - Past conversations |
| - Learned preferences |
| |
| Layer 4: External Knowledge (tools/RAG) |
| - Documents, databases, APIs |
| - Queried, not maintained by agent |
+--------------------------------------------+
Layer 1 - Context Window. This is the text the LLM sees in a single call. It is the fastest memory because the LLM reads it directly. But it is also the smallest. Once it is full, older messages have to be dropped or summarized.
Layer 2 - Short-Term Memory. This is the scratchpad the agent uses during one task or session. It holds the plan, the current step, and the intermediate results. When the session ends, this memory is usually gone.
Layer 3 - Long-Term Memory. This is the persistent memory that survives across conversations and sessions. It stores user facts, preferences, past conversation summaries, and learned behaviors. This is what makes the agent feel like it "knows" the user.
Layer 4 - External Knowledge. This is everything the agent can query but does not maintain - documents, databases, APIs, the web. The agent reads from these sources but does not write back to them as its own memory, even when the sources belong to the same company. Some frameworks classify this as a type of long-term memory; others keep it separate. From the agent's point of view, it looks the same either way - the agent asks, and information comes back.
All four layers work together. The context window is the LLM's desk. Short-term memory is the notepad next to the desk. Long-term memory is the personal diary in the drawer. External knowledge is the library across the street. We want to keep our desk clean, our notepad organized, our diary trustworthy, and the library indexed.
Note on terminology: Some frameworks use "short-term memory" to mean the context window itself, not a separate layer. Here we keep them as two layers because the context window is what the LLM literally reads in one call, while the short-term scratchpad is the session-scoped store the agent maintains between calls and loads back into the context window each turn.
Note: Within Long-Term Memory, there are further sub-types - episodic (past experiences), semantic (facts and concepts), and procedural (how to do things). We will cover these in future posts.
The Four Core Operations
Now that we have seen the layers, let's look at what we actually do with them. No matter how many layers we have, every memory system does only four things. Let's decode each one.
1. Write. Save new information. When something important happens - a user preference, a task result, a key fact - we write it to memory. Example: after the user says "I prefer code in Go", we write "preferred language: Go" to long-term memory.
2. Read. Pull relevant information back when needed. This is usually the hardest part. We do not want everything, only what matters right now. Example: if the user asks "Write me a function to reverse a string", we pull up "preferred language: Go" before calling the LLM - but we do not pull up their shipping address.
3. Update. Modify existing memories when new information contradicts them. If the user's favorite language was Python last month and is now Go, the memory must be updated.
4. Forget. Remove memories that are stale, wrong, or no longer relevant. Without forgetting, the memory store grows forever and retrieval gets worse. Example: a note like "user is debugging login right now" should be dropped once the session ends.
Memory System = Write + Read + Update + Forget.
If any one of these is broken, the whole memory system breaks. An agent that writes but never reads is a waste of disk space. An agent that reads but never updates will keep acting on outdated facts. An agent that never forgets will eventually drown in its own history. All four operations must work together.
How Memory Flows at Runtime
Now, let's put all the parts together to see how memory flows through an agent at runtime.
Here is the flow as a picture:
User Message
|
v
+----------------------------+
| READ Memory |
| - Long-Term (user facts, |
| preferences, summaries) |
| - Short-Term (plan, |
| recent steps) |
+-------------+--------------+
|
v
+----------------------------+
| Build Prompt |
| (system prompt + |
| retrieved memories + |
| user message) |
+-------------+--------------+
|
v
+----------------------------+ +---------------------+
| LLM |<-->| External Knowledge |
| (reasons, replies) | | (tools, RAG, APIs) |
+-------------+--------------+ +---------------------+
|
v
+----------------------------+
| WRITE Memory |
| - Short-Term (new step) |
| - Long-Term (if worth |
| keeping) |
+-------------+--------------+
|
v
Response
Now, let's walk through each step on every user turn:
Step 1: The user sends a message.
Step 2: The agent searches long-term memory for anything relevant to this message. It pulls out a handful of useful facts - maybe the user's name, their preferred language, and a past conversation summary.
Step 3: The agent also reads its short-term memory for the current session - the plan so far, recent actions, and intermediate results.
Step 4: The agent builds the prompt for the LLM by combining: the system prompt, the retrieved long-term memories, the short-term scratchpad, and the current user message. This all goes into the context window.
Step 5: The LLM reasons and responds. The agent may call tools, which use external knowledge (like RAG on documents or a database query). The agent can call tools multiple times in a loop before producing its final response.
Step 6: The agent writes new information to memory. Short-term memory gets the new step. If something important was learned, it gets written to long-term memory too.
Step 7: The response is sent to the user.
The same flow in pseudocode:
while user_active:
user_message = receive_message()
# READ
long_term_context = search_long_term_memory(user_id, user_message)
short_term_context = read_short_term_memory(session_id)
# BUILD PROMPT (retrieved memories are formatted into text and
# injected into the system prompt, not sent as separate messages)
messages = [system_prompt_with(long_term_context, short_term_context), user_message]
# LLM LOOP (may call tools multiple times before the final answer)
while True:
response = call_llm(messages, tools)
if response.is_tool_call:
tool_result = call_tool(response.tool_name, response.tool_args)
messages.append(response)
messages.append(tool_result)
else:
break
# WRITE
write_short_term_memory(session_id, user_message, response)
if is_worth_keeping(user_message, response):
write_long_term_memory(user_id, user_message, response)
send_to_user(response)
This full flow happens on every single turn. Read -> Build prompt -> Respond -> Write, forever. The agent is constantly pulling memory in and pushing new memory out. From the user's point of view, it just feels like the agent remembers.
Note: In production, the long-term memory write is often done asynchronously after the response is sent, so the user does not pay the latency cost.
The best way to learn this is by taking an example. Let's see this with a short one.
User turn 1: "My name is Priya and I code in Go."
- Long-term memory is empty. Nothing to retrieve.
- The LLM responds: "Nice to meet you, Priya."
- The agent writes to long-term memory: "User's name is Priya. Preferred language: Go."
User turn 2 (a week later): "Help me write a function that reverses a string."
- The agent searches long-term memory for context about this user. It retrieves: "Priya, prefers Go."
- The prompt becomes: System prompt + "User is Priya, prefers Go" + "Help me write a function that reverses a string."
- The LLM responds with a Go function, addressed to Priya.
That's the beauty of AI Agent Memory. The second turn feels personal even though it happened a week later. The LLM itself remembers nothing. The agent did all the remembering for it.
What to Store and What Not to Store
Now, a natural question arises - if we have a memory system, should we store everything? The answer is no. One of the biggest mistakes is storing too much. Memory is valuable only if retrieval works well, and retrieval works well only if the memory store is not full of junk.
Store this:
- Stable user facts (name, role, language, timezone).
- Strong preferences (prefers short answers, wants code in Python).
- Outcomes of past tasks (what worked, what failed).
- Decisions that shape future work (architecture choices, policies, constraints).
Do not store this:
- Every single message. The context window already has the live conversation.
- Low-signal chit-chat ("hi", "thanks", "ok").
- Temporary state that is already in short-term memory.
- Sensitive data we do not have consent to keep.
In simple words:
Store what will matter next week. Skip what only matters for the next minute.
Very important: Before writing anything to long-term memory, ask one simple question: "Will this be useful in a future conversation?" If the answer is no, do not write it.
Common Mistakes and How to Fix Them
Even a well-designed memory system can fail in specific ways. Let's decode each one.
1. Context window overflow. The prompt becomes too long and either fails or gets truncated. The agent loses the oldest (and sometimes most important) information.
How to fix: Summarize old turns into a compact summary. Drop tool outputs that are no longer needed. Put only the most relevant retrieved memories in the prompt, not everything.
2. Retrieval misses the important memory. The agent has the memory stored, but the retrieval step does not find it, so the LLM answers as if it never knew.
How to fix: Use semantic search (embeddings) instead of keyword match. Store memories with good titles and descriptions. Retrieve more candidates than we need and let the LLM pick.
3. Outdated memories. The agent keeps acting on old facts. The user's job changed, but the agent still thinks they are a student.
How to fix: On every write, check if an existing memory contradicts it. If yes, update or replace it. Do not just append.
4. Memory bloat. The memory store grows forever and retrieval gets slow and noisy.
How to fix: Add a forget policy. Delete memories that have not been used in a long time, or that are superseded by newer ones.
5. Privacy leaks. The agent remembers sensitive information and surfaces it in the wrong context - for example, mentioning one user's data while helping another user.
How to fix: Scope every memory to a user ID. Never mix memories across users. For sensitive data, encrypt both at rest and in transit, and require explicit consent before writing.
6. Confusing short-term and long-term. The agent writes ephemeral state to long-term memory, or keeps stable facts only in short-term memory and loses them on session end.
How to fix: Be deliberate about which layer each write goes to. If it is useful only for this task, keep it short-term. If it will matter later, promote it to long-term. When in doubt, prefer short-term - we can always promote later.
A production memory system is mostly about handling these failure modes well. The happy path is easy. The edge cases are where the real engineering happens.
Very important: Most of these mistakes are silent. The agent does not crash - it just gives subtly wrong answers. That is why we must add logging, monitoring, and regular audits of the memory store. Without that, we will only find out when a user complains.
Now, we have understood how AI Agent Memory works end to end. Let's wrap up with a quick recap.
Quick Summary
Let's recap what we have decoded:
- AI Agent Memory is the system that lets a stateless LLM act like it remembers across turns, sessions, and users.
- The memory stack has four layers: the context window, short-term memory, long-term memory, and external knowledge. Each has a different lifespan and purpose.
- The four core operations are Write, Read, Update, and Forget. Break any one of them, and the whole memory system breaks.
- The runtime flow on every turn is: retrieve relevant memories, build the prompt, call the LLM, write new memories, respond. Read -> Build prompt -> Respond -> Write, forever.
- Store what will matter next week. Skip what only matters for the next minute. Most memory systems fail from storing too much, not too little.
- Common mistakes include context overflow, retrieval misses, outdated memories, memory bloat, privacy leaks, and confusing the memory layers. Handling these is the real work.
- Memory is what turns an LLM from a clever but forgetful answerer into an agent that truly knows us and improves over time. Every serious AI agent, from coding assistants to customer support bots to personal assistants, has memory at its core.
Prepare yourself for AI Engineering Interview: AI Engineering Interview Questions
That's it for now.
Thanks
Amit Shekhar
Founder @ Outcome School
You can connect with me on:
Follow Outcome School on: