Memory & Context

Think of the context window as the agent's desk. Whatever you lay on the desk, the model can see and use this turn; whatever is not on the desk might as well be in another building. The desk holds a mixture of things: the standing instructions (who the agent is, how it should behave, what it must not do), the running conversation or task history, the results of tools it has called, and any documents or data you have fed in. Each turn, the scaffolding assembles this desk, hands it to the model, gets back the model's next piece of text, and then must decide what the desk should look like on the next turn. The window is measured in tokens and is finite - large in modern models, but never unlimited - and that finiteness is the source of much of what follows.

The context window is best understood as short-term, working memory: fast, immediate, and where all the actual thinking happens, but bounded and transient. It is analogous to the few things you can hold in your head at once while working on a problem - vivid and usable right now, but limited in number and gone when you turn to the next thing. Two properties matter for practice. First, capacity is finite: you cannot simply pour every document, every prior message and every tool result onto the desk forever; at some point it is full, and something must give. Second, position and quantity affect quality: a model attends better to a focused, well-ordered context than to an enormous, cluttered one, and important details buried in the middle of a very long context can be effectively overlooked even when technically present. More context is not always better; the *right* context, well arranged, is better.

For a designer this reframes what 'prompting' even is. Getting good work from an agent on a real task is less about a clever opening sentence and more about context management - making sure the information the agent needs for the step at hand is on the desk, clearly, and the clutter is not. When an agent gives a disappointing answer, the first question is often not 'is the model not smart enough' but 'did the right information make it onto the desk this turn, and was it findable among everything else there'. That is a question about the scaffolding and about how you supplied information - and it is a lever you control.

If the desk is finite but real work involves far more information than fits on it - a whole project's documents, a practice's standards, months of prior conversation - the agent needs somewhere to keep things that is bigger than the desk, and a way to bring the right item back onto the desk when it is needed. That is long-term memory, and in agents it is usually implemented as retrieval. The pattern is worth understanding because it is everywhere. Instead of trying to hold everything in the context window, the system stores information outside it - in files, a database, or a specialised vector store that indexes text by meaning - and, when a step needs some of it, fetches only the relevant pieces and places them on the desk for that turn. The rest stays filed away, out of the window, until it too is relevant.

The most common form of this is retrieval-augmented generation (RAG): rather than relying on what the model absorbed in training, or cramming an entire corpus into the context, the system searches a knowledge base for the passages relevant to the current question and supplies just those, so the model reasons over fetched, specific, up-to-date material. This is how an agent can 'know' your practice's specification library, a particular building code, or the full history of a project far larger than any window - it does not know it all at once; it fetches what it needs, when it needs it. Retrieval is, in effect, memory implemented as search: the store is the long-term memory, and the retrieval step is the act of remembering. (Module 2.3 treats grounding and retrieval as a craft in its own right; here the point is that this *is* how agents extend memory beyond the window.)

Seeing retrieval as memory clarifies both its power and its fragility. Its power: it lets an agent work against enormous, specific, current bodies of knowledge - yours - rather than only the general knowledge baked into the model, and it grounds answers in real sources you can check. Its fragility: the agent only ever reasons over *what was retrieved*, so if the retrieval fetches the wrong passages, an outdated version, or misses the crucial one, the model will reason confidently over the wrong material and you may never see the gap. A retrieval-backed agent that cites a superseded code clause is not lying; it faithfully reasoned over what it was handed, which was stale. This is why, for anything that matters, you verify not just the model's reasoning but whether it was working from the right, current source - the remembering can fail as surely as the thinking.

A single question needs only the question on the desk; a multi-step job needs the *state of the job* held together across many turns, and this is one of the harder things an agent's memory has to do. Consider a goal that runs for forty steps: the agent has a plan, it has completed some steps and not others, it has learned things along the way (this supplier is out of stock; that dimension is fixed at 3.2 metres; the client rejected the first option), and each new step should build on all of it. But the model is stateless and the desk is finite - it cannot simply keep every prior turn in full view forever, because the context would overflow. So the scaffolding must continuously curate the job's state: keep the goal and the key decisions in view, carry forward what has been learned, track what is done and what remains, and compress or set aside the rest.

The usual techniques are worth naming because their limits are where long tasks fail. Summarisation: as the history grows, the scaffolding replaces older, verbose turns with a compact summary, keeping the gist while freeing desk space - efficient, but lossy, because whatever the summary omitted is now gone. Scratchpads and state files: the agent writes key facts, decisions and the running plan to a durable note it can re-read, so the working state survives even as the raw conversation is trimmed. Structured task tracking: an explicit list of steps, ticked off as they complete, so the agent knows where it is. Together these let an agent hold a long job together far better than raw conversation would - but every one of them is a managed approximation of perfect memory, not the real thing.

The practical upshot is that the longer and more complex the task, the more the quality depends on memory management rather than on raw model intelligence, and the more a professional should watch for state to drift. In a long run, an agent can lose track of a constraint set early ('but I told it the ceiling is fixed at 3.2 metres' - twenty steps and one summary ago, and it fell out of view), contradict an earlier decision, or forget a step it was meant to do. These are not the model being stupid; they are the working state failing to survive the trip across many turns and a finite desk. The defence is partly the scaffolding's design and partly your practice: keep long tasks scoped, re-state the constraints that must not be lost, checkpoint at intervals to confirm the agent still holds the key facts, and never assume that because you said something once, it is still on the desk.

Because memory in an agent is a managed approximation, it fails in characteristic ways, and knowing the catalogue lets you spot trouble before it costs you. Context overflow: the desk fills up, and as older material is trimmed or summarised, information you assumed the agent still had quietly disappears - the classic 'it forgot what I told it at the start.' Lost-in-the-middle: even within a very long context, details buried in the middle get less attention than those at the beginning or end, so a crucial fact can be technically present yet effectively ignored. Stale or wrong retrieval: the long-term memory returns an outdated document or the wrong passage, and the agent reasons faithfully over bad source material. Context confusion: too much irrelevant material on the desk dilutes the signal, and the agent's answers get vaguer or drift off-task. Cross-contamination: in a long session, information from an earlier, unrelated part of the task lingers and bleeds into a later one it should not touch.

There is also a quieter risk that matters especially in professional practice: what goes into the context, and into long-term memory, is data you are handling. If you feed a client's confidential drawings, a project's commercial terms, or personal data into an agent, that information sits in the context and may be stored in the agent's memory or sent to a model provider. Where it goes, how long it is retained, and who can see it are real questions of confidentiality and data protection - not abstractions but duties you owe your clients. Memory is not only a capability to exploit; it is a surface to govern. What you are willing to put on the desk, and into the store, is a decision to make deliberately, and Module 8 returns to it as a matter of ethics and law.

The throughline is that memory is a designed, imperfect system you must work *with*, not a perfect recall you can lean on. So the professional habits are concrete. Do not assume the agent remembers something because you said it earlier - if a constraint is critical, keep re-stating it or put it in a durable instruction that stays on the desk. On long tasks, checkpoint: pause and ask the agent to confirm the key facts and the plan, and catch drift early. Prefer scoping a big job into smaller, well-defined pieces over one enormous run that strains the window. Verify that retrieval fetched the right, current source, not just that the reasoning sounds good. And govern what enters the context and the store as the confidential data it often is. Handle memory as the fallible mechanism it is, and you get the benefit of a long-running agent without being blindsided by the way it forgets.