RAG (Retrieval-Augmented Generation) and CAG (Context-Augmented Generation) are often pitched as rival architectures, but in practice they’re two different answers to the same question: how do you make an LLM reliably use your information, not just its own trained data set? In 2025, the debate has sharpened because context windows have grown dramatically and prompt caching has become mainstream, changing the performance and cost maths for “load it all in” systems. This article explains the core differences, where each approach shines, where it falls over, and how organisations are using them in the real world.
1) The core concept: retrieval vs preloaded context
Retrieval-Augmented Generation (RAG) is an approach where the model consults an external knowledge base at question time, pulling in relevant passages before generating an answer. AWS describes RAG as a way to optimise LLM outputs by having the model reference an authoritative knowledge base outside its training data, then generate a response grounded in that retrieved content.[1] In other words, your system doesn’t “know” your documents by default; it fetches what it thinks it needs, and only then responds.
Context-Augmented Generation (CAG) flips the timing. Instead of searching at query time, you pre-load the relevant material into the model’s context so it can reason over the full body of content in one go. The promise is simple: if the model can “see” the whole manual, whole contract set, or whole codebase, you can ask global questions that are hard for retrieval-based pipelines, such as contradictions across chapters or trends across a portfolio.
That’s the idea, but it comes with a very real constraint: context is finite. Even with modern long-context models, you are bounded by the maximum context window available to you. Anthropic’s documentation, for example, describes a 1-million token context window for Claude Sonnet 4 and 4.5 (in specific tiers), which is large enough for very long documents or substantial codebases, but still not “infinite”.[2]
2) Why “cache” keeps coming up: the enabler behind practical CAG
If you simply shove a 1,000-page manual into an LLM for every single question, you’ll pay for it repeatedly in both latency and input tokens. That’s why the term “cache” appears in serious CAG conversations: not because CAG equals caching, but because caching is what makes CAG economically and operationally viable at volume.
At the infrastructure level, LLMs can reuse computation through key-value (KV) caching, which stores intermediate key/value tensors during inference so the system doesn’t recompute attention over the same prefix again and again. Sebastian Raschka’s technical explainer summarises this cleanly: KV caches store intermediate key and value computations for reuse during inference, delivering substantial speed-ups, at the cost of more memory and some complexity.[3]
At the product/API level, “prompt caching” is the developer-facing version of that idea. OpenAI documents prompt caching as a system that routes requests to servers that recently processed the same prompt, making it cheaper and faster than processing from scratch, and notes it can reduce latency and input token costs significantly for repeated prefixes.[4] OpenAI’s own launch post frames it as a production scaling tool for balancing performance, cost, and latency.[5]
Put plainly: CAG is the architectural strategy (preload context); caching is the optimisation technique that stops you paying the “read the whole book” cost every time. When you ask many questions against the same static corpus, caching turns “painful” into “practical”. Without caching, CAG can still work, but the economics may be brutal.
3) The 2025 showdown: what changes in day-to-day behaviour
Here are the differences that tend to show up in production.
Data size and coverage
RAG scales to very large corpora because it only retrieves a small slice at a time. If you have millions of documents, you cannot realistically preload them into context, but you can index them and retrieve the few passages most likely to matter. AWS’s prescriptive guidance frames RAG as augmenting an LLM with external data (like internal documents) so it has the context needed for a specific use case, implying the corpus can live outside the model and be pulled in as needed.[6]
CAG is limited by context windows, even when those windows are enormous. A million tokens feels like magic until you try to ingest a decade of email, a full enterprise SharePoint, or every customer ticket you’ve ever had. You can sometimes compress, summarise, or prioritise, but you’re still choosing what “makes the cut”.
Global questions vs local questions
RAG is good at “local” questions where the answer lives in a small number of passages, such as “What’s the return policy for item X?” or “Which endpoint handles password resets?” It can fail on global questions that require scanning the entire corpus for patterns, shifts, or contradictions. Why? Because retrieval typically supplies a handful of chunks, and any pattern outside those chunks might as well not exist.
CAG can excel at global questions because the model can consider the entire preloaded dataset at once. If you preload a full contract history, you can ask, “How does the indemnity language in earlier templates contradict the current one?” and the model has a fighting chance of answering without hoping retrieval surfaced the right comparisons.
Reasoning style: fragmented vs holistic
RAG’s reasoning is often fragmented because the model sees isolated slices of a larger truth. Even when you add reranking, multi-query retrieval, or iterative retrieval loops, the model is still reasoning through a keyhole.
CAG’s reasoning can be holistic when the corpus genuinely fits and is coherent. It can connect page 5 to page 500, or compare a design decision in an old architecture document with a later incident report. The trade off is that you may need disciplined document hygiene (deduplication, clear versions, “single source of truth” practices), otherwise the model will see everything,including conflicts you forgot you had.
Speed and cost profile
RAG has a recurring per-question pipeline: retrieve, optionally re-rank, then generate. That overhead can be totally fine for low query volumes, or when freshness matters more than raw speed.
CAG can be extremely fast at query time once the corpus is preloaded and cached. This is where caching becomes decisive: OpenAI’s prompt caching documentation explicitly positions caching as reducing latency and cost for repeated prompt prefixes.[4] In high-volume scenarios (think internal teams hammering the same handbook all day), the “load once, query many” approach can produce a very different unit economics curve than “retrieve every time”.
4) Real-world examples: where RAG wins, where CAG wins
Example A: Customer support for a fast-moving product catalogue
Imagine an ecommerce business where product specs, shipping timelines, and promotions change weekly. A RAG-based assistant can retrieve the latest policy and product pages at the moment of the question. This plays to RAG’s strength: the knowledge base can be updated without retraining the model, and retrieval can point the LLM at the current truth.[1]
CAG struggles here if you don’t refresh the preloaded context constantly. If your cache contains last month’s promotion rules, you’ll confidently answer with the wrong discount. You can rebuild the cache frequently, but then you’re back to paying the “load the world” tax over and over.
Example B: Legal teams comparing contracts across an entire portfolio
Now switch to a legal team that wants to compare a new contract against a history of prior contracts to spot subtle clause drift. RAG can work if your retrieval is excellent and your chunking strategy captures the right boundaries, but it’s easy to miss the one clause that matters if it’s worded differently or buried in a schedule. The cost of a miss is not “the answer is slightly off”; it can be material risk.
CAG is attractive when the portfolio fits in context because the model can compare across the whole set without relying on retrieval to surface the right clauses. If you can preload, deduplicate, and keep a stable “contract pack” in memory, you can ask comparative questions that are inherently global. This is precisely the kind of workload long-context capabilities are designed to support, where massive windows allow processing far larger documents and longer conversations in one request.[2]
Example C: Software engineering — refactoring across a repository
Developers don’t just want answers; they want changes that compile. A RAG coding assistant might retrieve the authentication module and a couple of related files, then propose a refactor that ignores a dependency hidden elsewhere. It’s not that RAG can’t do code; it’s that retrieval can miss the “thin thread” that makes the system actually work.
CAG shines when you can preload the whole repository (or the relevant subset) into context, because cross-file dependencies are visible. If the model can see the middleware, the config, the tests, and the service calls, it’s far less likely to propose a refactor that breaks everything. The limitation is practical: many repos are huge, and even a million tokens can be consumed faster than you’d think, especially when you include generated files or vendored dependencies. A disciplined “what counts as relevant code” policy becomes non-negotiable.
Example D: Finance and communications detecting tone shifts over time
Consider an analyst asking: “Across the last 20 quarterly reports, has the CEO’s tone changed on margin pressure?” A typical RAG system may retrieve the last couple of reports and some mentions of “margin” and “headwinds”, then generate a plausible answer that feels analytical but is anchored to too few samples. The global question is the trap.
CAG works better if you can preload all 20 reports, because the model can compare language patterns across the entire set. It won’t be perfect LLMs can still overinterpret, but the information access problem is reduced: you’re no longer hoping retrieval guessed which quarters matter most.
5) Pros and cons
RAG: what you gain
Freshness and scale are RAG’s headline advantages. Your corpus can be enormous, you can update it continuously, and the model can reference new materials without retraining. That is why RAG remains a pragmatic enterprise default: it’s relatively modular, and it aligns well with how organisations already store knowledge (wikis, ticket systems, document stores). AWS’s description of RAG emphasises referencing an authoritative external knowledge base before generating a response, which is exactly what you want when “authoritative” changes frequently.[1]
RAG: what bites you
Retrieval failures are the core weakness. If the retriever doesn’t find the right passage, the model can’t use it. Even worse, it may produce a confident answer anyway. In production, you end up obsessing over embedding models, chunk sizes, metadata filters, re-rankers, and evaluation harnesses. RAG is not one technique; it’s a small ecosystem you have to keep healthy.
RAG can also struggle with systemic questions, the ones that require reading everything, not just the top-k hits. You can simulate global analysis with repeated retrieval and aggregation, but that’s more engineering and more failure modes.
CAG: what you gain
Holistic reasoning and speed at volume are the big wins. When the entire dataset is truly present, the model can connect distant dots. And if you’re asking lots of questions over the same static corpus, caching changes the cost curve: OpenAI’s prompt caching docs explicitly position caching as a way to reduce latency and cost for repetitive prompt content.[4]
CAG also removes an entire class of operational work. There’s no retriever to tune, no index freshness to monitor, no reranker outages. You trade it for different concerns (context budgeting, refresh cadence, and memory constraints), but the pipeline is conceptually simpler at query time.
CAG: what bites you
Context limits are the obvious constraint, even with 1M-token systems.[2] If your business reality is “tens of millions of documents”, you can’t CAG your way out of that without aggressive filtering or summarisation, and at that point you’re making retrieval-like choices anyway just earlier.
Staleness is the quieter problem. The more you rely on caching and preloaded context, the more you need a strategy for updates: what triggers a refresh, how you handle versioning, and how you prevent users from getting last week’s truth with this week’s confidence.
Finally, caching itself has trade offs. KV caching improves inference speed but increases memory requirements; it’s not free magic.[3]
6) When should you use CAG, when should you use RAG, and when should you use both?
If your information is large, messy, and constantly changing, RAG is usually the safer default. Your model can stay grounded in the latest policy pages, and your system can scale beyond any context window. If you need citations back to source documents for governance, RAG’s retrieval step also makes attribution more natural, because you know exactly what text you fed the model.
If your information is stable, bounded, and you need deep cross-document reasoning, CAG becomes compelling. This is especially true when users ask many questions over the same corpus, because caching rewards repetition by avoiding reprocessing the same long prefix.[4]
In practice, many mature teams land on a hybrid. They keep a stable “core pack” preloaded (handbook, canonical architecture docs, standard contract templates), then use retrieval to pull in recent updates, edge cases, or long-tail materials. The hybrid is less philosophically pure, but it reflects organisational reality: some knowledge is stable and heavily reused, other knowledge is sprawling and volatile.
7) Alternatives and examples you can point to in the wild
Alternatives: If you can’t fit everything into context and retrieval feels brittle, one practical alternative is a staged approach: retrieve broadly, summarise or compress into a structured “brief”, then run a second pass for reasoning. This doesn’t eliminate retrieval, but it reduces the fragmentation problem by giving the model a more coherent intermediate representation. Another alternative is to treat long-context as a scarce resource and preload only the “high-leverage” materials, while retrieval handles the rest, effectively a tiered knowledge strategy.
Examples: On the RAG side, AWS’s RAG guidance and explainers are closely aligned with how many enterprises implement grounded assistants over internal documentation and knowledge bases.[1][6] On the CAG-enabler side, prompt caching is now a first-class feature in major LLM platforms, explicitly marketed for latency and cost reduction in repeated-prefix workloads.[5]
My view: the “winner” depends on what you’re optimising
If you’re building a demo, CAG can feel like a superpower: preload a big document, ask clever questions, and the model seems to “understand everything”. In production, though, the decision is less about vibes and more about operational truth. If your corpus changes daily and your users demand the latest answer with an audit trail, you will probably end up with RAG as your backbone. If your corpus is stable and users repeatedly interrogate the same materials, CAG plus caching is a serious efficiency play, and it can unlock genuinely better global reasoning.
The most honest takeaway is that RAG and CAG aren’t enemies; they are different placements of the same bet. RAG bets your retriever will find the right pages at the right moment. CAG bets you can decide what matters upfront, fit it into context, and keep it fresh enough to trust. In 2025, the smart teams are not choosing dogma. They’re choosing a portfolio: preload the stable stuff, retrieve the volatile stuff, and evaluate relentlessly.
Summary:
RAG augments an LLM by retrieving relevant external passages at query time, which scales to massive, frequently updated corpora but can fail when retrieval misses key information.[1] CAG preloads a bounded corpus into long context for holistic reasoning and fast repeat Q&A, but is constrained by context limits and needs a refresh strategy.[2] Prompt/KV caching is a major enabler for CAG-style workloads by reducing repeated-prefix latency and cost, making “load once, query many” economically realistic.[4][1] What is Retrieval-Augmented Generation (RAG)? — link
[2] Context windows — Claude Docs — link
[3] Understanding and Coding the KV Cache in LLMs from Scratch — Sebastian Raschka — link
[4] Prompt caching — OpenAI API Docs — link
[5] Prompt Caching in the API — OpenAI — link
[6] Understanding Retrieval Augmented Generation — AWS Prescriptive Guidance — link
beFirstComment