RAG + Agents: The Difference Nobody Explains Well

"We built a RAG system" and "we built an agentic RAG system" get said like they mean the same thing, and the gap between them explains a lot of why some RAG demos feel magical and others feel brittle. The difference isn't the retrieval step, or the vector database, or the embedding model — it's whether the system can decide, on its own, that the first answer wasn't good enough and go look again.

Plain RAG: one lookup, no judgment

Classic retrieval-augmented generation is a fixed pipeline. Every query follows the same three steps, no matter what:

def plain_rag(query: str) -> str:
    # 1. Embed the query and retrieve the top matches — always, no exceptions
    results = vector_db.search(embed(query), top_k=3)
 
    # 2. Stuff them into the prompt
    context = "\n\n".join(r.text for r in results)
 
    # 3. Ask Claude to answer using only that context
    response = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": f"Context:\n{context}\n\nQuestion: {query}\n\nAnswer using only the context above.",
        }],
    )
    return response.content[0].text

This works well when the first retrieval is actually relevant. It falls over the moment it isn't — if the top 3 chunks miss the point, the model has no way to say "these aren't helpful, let me search differently." It just answers with whatever it was handed, confidently, even if the context was wrong.

Agentic RAG: retrieval becomes a decision, not a step

The fix is to stop treating retrieval as a fixed pipeline stage and turn it into a tool — something Claude can choose to call, evaluate the results of, and call again with a better query if needed. This is exactly the tool-use loop from earlier in this series, with search as the tool.

tools = [{
    "name": "search_knowledge_base",
    "description": "Search the knowledge base for relevant information. Can be called more than once with different queries if earlier results weren't sufficient.",
    "input_schema": {
        "type": "object",
        "properties": {"query": {"type": "string"}},
        "required": ["query"],
    },
}]
 
def search_knowledge_base(query: str) -> str:
    results = vector_db.search(embed(query), top_k=3)
    if not results or results[0].score < 0.5:
        return "No sufficiently relevant results found for this query."
    return "\n\n".join(r.text for r in results)

Run this through the agent loop, and something genuinely different happens: if the first search comes back weak, Claude sees "No sufficiently relevant results found" and — because it's reasoning about the result, not just receiving fixed context — tries a reformulated query instead of answering badly.

  → calling search_knowledge_base({'query': 'refund policy'})
  ← "No sufficiently relevant results found for this query."
  → calling search_knowledge_base({'query': 'return and refund eligibility timeframe'})
  ← [3 relevant chunks]
  → "Based on our policy, refunds are available within 30 days..."

That second call — reformulating instead of giving up — is the entire difference. Plain RAG can't do that because retrieval isn't a decision it gets to make; it's a step that already happened before the model saw the question.

The tradeoff, honestly

Agentic RAG isn't strictly better — it's slower and more expensive, because it can make multiple retrieval calls and multiple model calls per question instead of one clean pass. For simple, well-indexed knowledge bases where the first search is usually right, plain RAG is faster, cheaper, and just as accurate. Agentic RAG earns its cost specifically when queries are ambiguous, the knowledge base is large and heterogeneous, or a wrong-context answer is expensive to get wrong — customer support, legal, medical-adjacent use cases, anything where "confidently wrong" is worse than "took two seconds longer."

The one-sentence version

Plain RAG retrieves once and hopes. Agentic RAG retrieves, judges whether that was good enough, and tries again if it wasn't — which is really just the same "plan, act, observe, adjust" loop from the rest of this series, applied specifically to search.

Reference links

• Claude embeddings guide: platform.claude.com/docs/en/build-with-claude/embeddings
• Claude tool use overview: platform.claude.com/docs/en/agents-and-tools/tool-use/overview
• Anthropic's engineering write-up on contextual retrieval: anthropic.com/engineering

Next in this series: "How Much Does an Agentic Claude Workflow Actually Cost? (Real Numbers)"