Retrieval-augmented generation (RAG)

Retrieval-augmented generation (RAG) lets a large language model (LLM) answer questions using information it was never trained on, such as your company’s policies, product documentation or codebase. When a question arrives, the system first retrieves the most relevant passages from your content, then asks the model to generate an answer using those passages.

It’s the most common architecture for enterprise AI assistants, because it keeps answers current and traceable to sources without retraining a model.

Diagram

flowchart LR
    subgraph ingest [Indexing, ahead of time]
        docs[Documents] --> chunk[Split into chunks]
        chunk --> embed1[Embedding model]
        embed1 --> vdb[(Vector index)]
    end
    subgraph query [At question time]
        q[User question] --> embed2[Embedding model]
        embed2 --> search[Similarity search]
        vdb --> search
        search --> prompt[Prompt with retrieved passages]
        q --> prompt
        prompt --> llm[LLM]
        llm --> answer[Answer with citations]
    end

How it works

• Chunking. Documents are split into passages small enough to be relevant and large enough to carry meaning.
• Embeddings. An embedding model turns each chunk into a vector, a list of numbers representing its meaning. Similar meanings produce similar vectors.
• Vector search. The question is embedded the same way, and the closest chunks are retrieved. Many systems combine this with keyword search (hybrid search) and a re-ranker that reorders results.
• Prompt assembly. Retrieved passages are placed in the prompt with instructions to answer only from them and cite sources.
• Generation. The LLM writes the answer.

Pros

• Uses private, current knowledge without retraining the model.
• Traceable answers: responses can cite the passages they came from.
• Fewer hallucinations than asking the model from memory, when retrieval is good.
• Access control: retrieval can respect who is allowed to see which documents.

Cons

• Retrieval quality decides answer quality: if the right passage isn’t retrieved, the answer will be wrong.
• Many tuning knobs: chunk size, embedding model, ranking and prompt all interact.
• Not good at whole-corpus questions such as “summarize every contract”, which need different approaches.
• Data freshness and security: indexes must be kept in sync with sources and their permissions.

When to use it

• Question answering over internal documentation, policies, tickets or code.
• Customer support assistants grounded in product knowledge.
• Any use where answers must be current and cite sources.

When to avoid it

• Teaching the model a style, format or skill, where fine-tuning or better prompting fits better.
• Precise numerical or transactional questions, which should query a database or API directly through tool use.

Common pitfalls

• No evaluation set. Without a list of real questions and expected answers, changes can’t be measured, only guessed at.
• Bad chunking. Splitting mid-table or mid-section separates facts from their context.
• Ignoring permissions. Retrieval that ignores document access rights leaks information.
• Stuffing too much context, which raises cost and can bury the relevant passage.

Related patterns

• AI agents
• Hybrid search and re-ranking
• Model gateway
• Human-in-the-loop review

Further reading

• Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (NeurIPS 2020)
• Chip Huyen, AI Engineering