By the end of this lesson you can explain why a language model has no memory of its own, what "memory" really means in an agent, and which problems it is meant to solve.
Watch
IBM walks through four ways the context an agent works from can go wrong, which explains many cases of agents forgetting or getting confused.
The model is stateless
A language model call is a pure function: tokens in, tokens out. Nothing from the last call survives unless you put it back into the next prompt. Everything we call "memory" in an agent is code around the model that decides what to store, and what to put back into the context window next time.
That gives you three constraints to design around:
- The context window is finite. Even with very large windows, you pay for every token on every call, and latency grows with it.
- More context is not always better. Irrelevant or contradictory text dilutes the signal. Selecting what goes in matters as much as fitting it.
- Sessions end. Without storage, the agent greets a returning user as a stranger.
What goes wrong without memory
- The user repeats their preferences every session.
- A long task loses its early decisions once the history is truncated.
- The agent repeats a mistake it made yesterday.
- Two agents working on one project cannot share what they learned.
What goes wrong with bad memory
This is the part tutorials skip. A naive "store everything and retrieve the top matches" design causes its own failures:
- Stale facts: "I live in Austin" is still retrieved after the user moved.
- Contradictions: both versions get retrieved and the model picks one at random.
- Noise: trivia crowds out the three facts that matter.
- Privacy leaks: sensitive details are remembered, and surface in the wrong place.
- Poisoning: an attacker plants an instruction in a message and it is stored as a "fact".
Memory is a data engineering problem with a model in the loop. Treat it like any other store: decide what is written, how it is indexed, when it is updated or deleted, and how you measure whether retrieval was right.
Context engineering is the umbrella
Memory is one piece of a larger discipline: curating the set of tokens the model sees on each step. Anthropic's engineering post frames this well, and covers techniques such as compaction and structured note-taking that we use later.
Resources
The best single starting point. Read it fully before lesson 3.
How a popular agent framework splits short-term and long-term memory.
Read the abstract and introduction. It frames the context window as a scarce resource to be managed.
Check yourself
- Write down five things an assistant for your own work should remember across sessions, and five it should deliberately forget.
- For each of the ten, say what could go wrong if it were stored and retrieved naively.
- In one paragraph, explain to a colleague why "just use a bigger context window" does not solve this.