The question is not whether RAG is dead. The question is what your system should do when the answer is missing, stale, or buried in a document it never retrieved.
These are two ways to supply context
Long context means putting a large chunk of source material directly into a model request. Retrieval augmented generation searches a collection first, then places selected passages beside the question. Both try to give a model evidence it did not learn during training. Neither guarantees that the model will use that evidence correctly.
A bigger context window reduces the need to build an indexing pipeline for some tasks. A small changing set of documents can be simpler to pass directly. RAG can lower repeated token costs, search a much larger corpus, and return specific source locations. The trade is that retrieval introduces its own chance to miss the one passage the answer depends on.
Pick based on your main failure mode
If a user needs a complete comparison across four short contracts, passing all four may be more reliable than hoping a retriever finds every relevant clause. If there are millions of support articles that change every day, searching and then supplying a few sources is usually more practical than sending the entire library on every request.
A paper from Liu and colleagues found that model performance can drop when relevant information appears in the middle of long inputs. The finding cautions against assuming a large context window is uniformly usable. It does not prove that every current model fails in exactly the same way. Test with your model and your documents.
| Need | Consider | Test for |
|---|---|---|
| A few stable documents in full | Long context | Missed or contradictory detail across files |
| A growing searchable corpus | RAG | Recall of the exact supporting passage |
| Fresh data with precise citations | RAG with source metadata | Citation-to-source correctness |
| Cross-document reasoning | Context packing or hybrid retrieval | Whether all necessary context survives packing |
| Low setup overhead | Direct context first | Prompt cost and answer reliability |
Hybrid systems are normal, not a failure
You can retrieve candidate documents, rerank them, and then include full text from the few documents that need deep comparison. You can use long context for an individual session and retrieval across many sessions. The architecture should respond to evidence about failures, rather than ideology about which approach has won.
Do not start by creating embeddings because a tutorial did. Take twenty real questions and mark which source passages an expert would need. See whether direct context works. Build retrieval only when the corpus or measured quality requires it. Keep the retrieval results visible so a reviewer can tell whether the model had its evidence.
An evaluation small enough to start today
Write twelve questions from the tasks people actually ask. Include three questions whose answer is absent, three where a newer policy contradicts an older one, and three requiring information from two sources. Record the acceptable answer and exact source passages before testing.
For each architecture, record whether the correct passage reached the model, whether the answer was supported by that passage, whether citations resolved to the right place, and how long the request took. If both approaches work, choose the simpler one. If retrieval misses, improve chunking or search before asking the model to hallucinate around absent evidence.
The right architecture is the one whose important failures you can see, reproduce, and fix.
FOLLOW THE EVIDENCE
Sources & further reading
Process note: AI tools assisted with research and drafting. Sources are linked so you can inspect the evidence. Unless an essay says otherwise, it does not claim hands-on testing or independent reproduction.
