RAG vs long context: is stuffing the whole corpus cheaper?

When context windows reached a million tokens, a popular take followed: retrieval is obsolete, just put everything in the prompt. RAG is dead.

It isn't. Long context changed how RAG is built, but the arithmetic on tokens, the way models read long prompts, and the size of real document sets all still favor retrieval once the corpus grows. Here is the reasoning, and where long context is the right call.

Cost: about 200 times the input per request

Two setups:

• Stuff everything: every request carries the full 1M tokens of source material.
• RAG: retrieval narrows it down and the request carries 5,000 tokens (a few passages plus the question).

Input is billed per token on every model, so the ratio is the same whichever model you pick: 1,000,000 / 5,000 = 200. Stuffing costs about 200 times the input of RAG, per request.

The volume matters more than the ratio. At 100 questions a day, stuffing sends 100 million input tokens a day; RAG sends 500,000. Multiply by any model's input price on the pricing page, or run your own numbers through the cost calculator.

Doesn't prompt caching fix that?

It's a fair objection, and caching does help. Models that support prompt caching bill cached input tokens at a much lower rate than fresh input; each model's cached rate is on the pricing page.

With caching, the gap shrinks a lot. It doesn't go away: reading a million cached tokens per request still costs more than sending 5,000 fresh ones. And caching only works under conditions:

• The documents have to stay fixed. Change one character and the cache is invalid; you pay full price again.
• Caches expire. Let traffic lapse and the cache has to be rebuilt, and writing a cache costs more than reading it.
• It doesn't help when each user has different documents. Customer A's contracts and customer B's contracts are separate caches.

Everyone in the company asking questions about one shared handbook is the best case for caching. Answering questions about each customer's own contract is close to the worst.

Accuracy: longer isn't better

Accuracy matters more than cost. Models are known to miss information placed in the middle of a long context, a pattern usually called "lost in the middle": the start and end of the prompt get attention, the middle gets less.

So passing a million tokens doesn't mean the model understands a million tokens. The 995,000 tokens unrelated to the question act as noise that dilutes the answer.

This is where RAG gains on accuracy. Retrieval is a cost control, and it's also a way of not handing the model irrelevant text.

Scale: it often doesn't fit

There's a simpler limit. A million tokens is roughly 750,000 words of English. That sounds like a lot until you list what a company actually has:

• every page of an internal wiki: doesn't fit
• several years of support tickets: doesn't fit
• full product documentation plus past meeting notes: doesn't fit

Context windows have grown tenfold, and company document sets are bigger still. Stuffing only works when the material is small to begin with.

Latency

A million-token prompt also takes longer to return its first token. In an interactive UI the difference is obvious, and caching reduces the cost of reading the input but not to zero.

When long context wins

Stuffing is the right answer in real cases:

• The material is small. If it fits in a few tens of thousands of tokens, building a retrieval stack costs more effort than it saves.
• It's a one-off. One contract, one spec. There's no reason to build a pipeline.
• Nothing can be missed. "List every condition in this document" needs full coverage. Retrieval can miss a passage; the full document can't.
• You need the whole structure. Summaries, or following an argument across chapters.

A practical rule: small, one-off, or coverage-critical means long context.

The realistic answer is a hybrid

Most current systems combine the two:

1. Retrieve broadly (20 to 50 candidates).
2. Rerank down to 5 to 10.
3. Pass those to a long-context model and let it reason over them.

The value of long context isn't that everything fits. It's that retrieval can be sloppier and the answer still holds up. You used to have to cut to the top three passages to fit; now twenty fit comfortably. Long context lowered the precision retrieval needs.

Long context didn't kill RAG; it made RAG easier. The retrieval step is still there, as the first stage.

How to decide

1. Measure the material in tokens. If it fits in a few tens of thousands, pass it whole and skip RAG.
2. Ask whether the documents differ per user. If they do, caching won't help; budget for the full 200x.
3. Ask whether you need full coverage. If you do, retrieval is risky. Pass everything, or measure retrieval recall.
4. If either would work, build the stuffing version first. Get something running, then add retrieval once cost or accuracy actually becomes a problem.

The fourth step matters most in practice. Starting with a RAG pipeline is often over-engineering. Prototype by passing everything, and move to retrieval when you hit a real cost or accuracy wall.

FAQ

Now that long-context models exist, do I still need RAG? Yes, once the material grows. If it fits in a few tens of thousands of tokens, passing it whole is simpler. Beyond that, stuffing loses on cost (up to about 200 times the input per request) and on accuracy (information in the middle gets missed).

Does prompt caching remove the cost difference? No. It narrows it a lot, but stuffing stays more expensive than RAG. Caching also needs fixed documents and doesn't help when each user's documents differ.

If I pass a million tokens, does the model understand all of it? Not evenly. Information in the middle of a long context is more likely to be missed, and the irrelevant parts act as noise.

Which should I build first? The stuffing version. It's enough while the material is small, and adding retrieval once you hit a cost or accuracy wall is faster than starting with it.

Summary

"Just pass everything" is right while the material is small. As it grows, it costs about 200 times the input tokens of RAG (caching narrows that, but a gap remains) and loses on accuracy too. Long context didn't replace RAG; it made retrieval more forgiving.

Start by measuring your material in tokens. Long-context models and their prices are in the model catalog and on the pricing page.