Context Quality, Not Context Volume, Is the New Bottleneck
Context quality, not window size, now limits agent performance. Here's what quality means in measurable terms — relevance, recency, non-redundancy, structure — and how to optimize for it.
For two years, the default answer to "the agent doesn't know enough" was to feed it more: more retrieved documents, more conversation history, a bigger context window. That instinct is now the problem. Context quality — not context volume — is the new bottleneck for LLM agents in 2026, and it changes what counts as good engineering practice. Datadog's 2026 State of AI Engineering puts it directly: the core engineering challenge has shifted from managing token counts to understanding which information actually drives a model's decisions. This post makes the case for why, and what building for quality instead of volume actually looks like.
The Volume Instinct, and Why It Made Sense Until It Didn't
Between 2023 and 2024, adding more context was a reasonable default. Context windows were small, retrieval pipelines were immature, and the binding constraint on most agents was simple: did the model even have the fact anywhere in its input? Under that constraint, volume was a decent proxy for quality. Teams responded predictably — retrieval systems widened from top-5 to top-20 results, conversation history got kept in full rather than summarized, and vendors marketed 100K, 200K, and 1M-token windows as headline features. None of that was wrong at the time. The problem is that the constraint has moved, and a lot of production systems are still optimizing for the old one.
What "Quality" Actually Means, in Measurable Terms
Quality is not a vibe. It breaks into four dimensions that can each be measured and optimized independently.
- Relevance — not "is this token topically related," but does this specific token change the model's decision. A document can be thematically on-topic and still contribute nothing to the answer.
- Recency and currency — is this still true. A fact retrieved from three weeks ago competes with a fact from yesterday, and stale information that contradicts current state is worse than no information at all. This is why recency needs to be a scoring input, not an afterthought — Feather DB's adaptive decay, for instance, weights memory stickiness as
1 + log(1 + recall_count)against a default 30-day half-life and a 0.3 time-weight, so frequently and recently accessed facts outrank stale ones automatically instead of relying on the model to sort it out mid-generation. - Non-redundancy — does this duplicate something already in context. Every duplicate document consumes tokens without adding information, and under a fixed context budget, duplicated facts crowd out distinct ones that might actually matter.
- Structure — is the information organized so the model can use it, or just technically present. Forty unranked search results dumped into a prompt is not the same input as a synthesized, deduplicated, ranked context block, even if the underlying facts are identical.
None of these four are captured by "how many tokens are in the context window." That's the core of the argument: volume measures capacity, not usefulness.
Why More Context Now Actively Backfires
The 2023-2024 instinct doesn't just stop helping past a certain point — it starts hurting. Four mechanisms explain why.
Context rot. Research from Chroma across 18 frontier models shows accuracy dropping 30-50% as input length grows, well before models hit their documented context limits. That's direct evidence that adding tokens can degrade output quality, not merely fail to improve it.
Cost. Token cost scales with context length, and every irrelevant token gets paid for on every call in a session, not once. Padding context "just in case" is a recurring tax with no corresponding benefit.
Latency. Prefill time scales with context length. Marginal documents added for safety margin add latency without adding answer quality — a pure loss.
Signal dilution. Salesforce's research on enterprise AI agent trends finds that for large organizations, hallucinations, output consistency, and managing context at scale rank as the top challenges in agent quality — above raw model capability. The model isn't the bottleneck; what's being handed to it is. LangChain's State of Agent Engineering makes the same point more bluntly: most agent failures today aren't model failures, they're context failures.
Evidence: What Optimizing for Quality Actually Buys You
This isn't just theoretical. On the LongMemEval_S benchmark, a full-context GPT-4o baseline — every historical message stuffed into the prompt, maximum volume — scored 0.640. Feather DB paired with GPT-4o, retrieving only the messages selected for relevance and recency instead of surfacing everything, scored 0.693. Feather DB with Gemini-2.5-Flash scored 0.657, for roughly $2.40 across the full benchmark run. The retrieval-based approach won using a fraction of the tokens per query, because it was optimizing for which messages would change the answer rather than how many messages it could fit. Filtering for quality doesn't have to cost you latency either — p50 ANN retrieval lands at 0.19ms at 500K vectors with 97.2% recall@10, so the selection step itself is not where the volume-era instinct's overhead was hiding.
The Metric Shift: From Tokens Available to Tokens That Changed the Answer
If volume is the wrong thing to optimize, the practical question is what to measure instead. The useful metric is something like "tokens that changed the answer," and the mindset shift is the same one marketing teams made years ago when they stopped reporting raw traffic and started reporting conversion rate. Traffic is easy to grow and easy to feel good about. It's also not the number that pays the bills.
Concretely, this means:
- Running ablation tests on eval sets — remove a candidate context item and check whether the answer changes. If it doesn't, that item was noise, regardless of how relevant it looked at retrieval time.
- Scoring retrieval precision against downstream task correctness, not embedding similarity. A chunk can be a strong vector match and still contribute nothing to the correct answer.
- Plotting answer accuracy against context length directly, per query type, to find the point where additional context starts costing you accuracy instead of buying it — the context-rot curve, measured on your own data rather than assumed.
- Tracking redundancy and staleness rates in what actually gets retrieved, not just recall@k.
None of this requires perfect instrumentation to be useful. Even a rough ablation pass on a representative eval set will usually surface which parts of a pipeline are padding the prompt versus earning their place in it.
Frequently Asked Questions
Does this mean smaller context windows are better?
No. Window size and context quality are separate axes. A larger window gives you more room to work with, but only helps if what you put in that room is selected well. The context-rot findings above show that raw size can hurt when it's filled indiscriminately — the window isn't the problem, indiscriminate filling is.
How do I start measuring "tokens that changed the answer" without a research team?
Start with ablation on a small, representative eval set: for each query, remove one candidate context item at a time and re-score the answer. Items whose removal never changes the score are candidates for cutting from the pipeline entirely. This is cheap to run and directionally useful long before it's rigorous.
Is this only relevant to RAG pipelines, or does it apply to agent memory too?
It applies to memory just as much as retrieval. Conversation history, prior tool outputs, and task summaries that get carried into context are subject to the same four failure modes — stale facts, duplicated information, and unranked dumps of history. Recency-weighted retrieval, like decay-based scoring on stored memories, is one concrete mechanism for keeping memory quality high instead of just keeping memory.
Won't next-generation context windows just solve this?
A bigger window raises the ceiling; it doesn't change what gets admitted below it. Without quality filtering, a bigger window just gives context rot more room to happen — the Chroma research measured degradation well within existing documented limits, not at the edges of them.
Feather DB ships as a free, MIT-licensed, single-file embedded vector database built around exactly this problem — try pip install feather-db if you want to see relevance-and-recency-scored retrieval on your own agent's context.