A long context window is not memory

Context windows grew by two orders of magnitude in a few years, and a lot of architecture diagrams quietly stopped drawing a retrieval layer. The reasoning goes: if everything fits, why fetch anything.

The failure this produces

A system with no retrieval layer has no notion of what is relevant. It has a notion of what was recent. Those coincide often enough in testing to look like the same thing, and diverge exactly when a conversation gets long or a document set gets heterogeneous.

The bug reports read the same way every time. The system was right yesterday and wrong today, on the same question, and nobody changed the prompt. What changed was position: the relevant passage moved, and attention did not follow it.

What to keep

Retrieval is not only a way to make things fit. It is the component that decides what deserves the model’s attention, and it is testable in isolation. A retrieval step you can evaluate on its own is worth keeping even when the whole corpus would fit in the window, because it turns one opaque failure into two legible ones.

Cost is the second argument, and the less interesting one. Filling a large window on every request is expensive and slow. But the reason we teach the retrieval layer as non-optional is diagnostic, not financial.

The distinction worth holding

A window is what the model can see this turn. Memory is what the system chooses to keep and why. The first is a capacity, the second is a design decision. No increase in the first will make the second unnecessary.