RAG for Knowledge-Grounded Agents: Retrieval Architecture and an Internal Knowledge Base Agent

When Prompt Engineering Hits Its Ceiling

In Part 2, we built a production-grade billing mediation agent using prompt engineering alone — five layers of structured prompting, circuit breakers, and a strict JSON output schema. That architecture works well when the agent's knowledge is stable, bounded, and small enough to fit in a system prompt.

But most enterprise agents operate in a different reality. The knowledge they need is large, frequently updated, and distributed across documents, wikis, databases, and codebases. You cannot fit a 50,000-page internal knowledge base into a system prompt. You cannot update a fine-tuned model every time a policy document changes. This is the problem RAG was built to solve.

Retrieval-Augmented Generation (RAG) is the pattern of dynamically fetching relevant context from an external knowledge store at inference time and injecting it into the model's context window before generation. The model never "learns" the knowledge — it reads it fresh on every request, from a source you control and can update in real time.

This article builds an Internal Knowledge Base Agent — a system that can answer questions about a company's internal documentation, policies, and procedures with citations. Along the way, we cover every architectural decision that separates production RAG from the toy demos that fail in the real world.

Why Most RAG Systems Fail

Before building the right system, it helps to understand why the wrong ones fail. The failure modes cluster into four categories:

Retrieval failures — the right document exists in the knowledge base but the retrieval step does not surface it. This happens when chunking destroys semantic coherence, when the embedding model does not capture domain-specific vocabulary, or when the query and the document use different terminology for the same concept.

Context window failures — too many retrieved chunks are injected into the context, diluting the signal. The model attends to irrelevant passages and produces answers that are technically grounded in the retrieved context but wrong in practice.

Hallucination under retrieval failure — when retrieval returns nothing useful, a poorly designed system falls back to the model's parametric knowledge and generates a confident, plausible-sounding answer that is not grounded in the actual knowledge base. This is worse than returning no answer.

Staleness failures — the knowledge base is not updated when source documents change. The agent answers based on outdated information while the user assumes they are getting the current state.

Each of these has a specific architectural solution. We will address all four.

The RAG Architecture Stack

A production RAG system has six distinct components. Each is a design decision, not a commodity:

1. 1Document ingestion pipeline — how source documents enter the system
2. 2Chunking strategy — how documents are split into retrievable units
3. 3Embedding model — how chunks are converted to vector representations
4. 4Vector store — where embeddings are stored and queried
5. 5Retrieval strategy — how queries are matched to chunks
6. 6Context assembly and generation — how retrieved chunks are presented to the model

Let's build each one.

Component 1: Document Ingestion Pipeline

The ingestion pipeline is the least glamorous part of RAG and the most frequently underbuilt. It has three jobs: extract text from source documents, track document provenance, and detect when source documents change.

Text Extraction

Different document types require different extraction strategies:

• PDF: Use a layout-aware extractor (PyMuPDF, pdfplumber) rather than a simple text stripper. Layout-aware extraction preserves table structure, column order, and heading hierarchy — all of which matter for chunking quality.
• HTML/web pages: Strip navigation, footers, and boilerplate before extraction. The main content is what matters; the surrounding chrome adds noise.
• DOCX/Office formats: Extract with python-docx or similar. Preserve heading levels — they are critical for hierarchical chunking.
• Markdown: Treat heading markers (#, ##, ###) as structural signals, not just formatting.

Provenance Tracking

Every chunk in the vector store must carry metadata that traces it back to its source:

{
  "chunk_id": "policy-handbook-v3-chunk-047",
  "source_document": "employee-policy-handbook-v3.pdf",
  "source_url": "https://internal.company.com/docs/policy-handbook-v3.pdf",
  "section_title": "Remote Work Policy",
  "page_number": 12,
  "last_updated": "2026-06-01T00:00:00Z",
  "document_version": "3.2"
}

This metadata serves two purposes: it enables citation in the agent's responses ("According to the Remote Work Policy, section 3.2..."), and it enables targeted re-ingestion when a specific document changes.

Change Detection

Implement a document fingerprint (SHA-256 hash of the source content) stored alongside the ingestion timestamp. On each ingestion run, compare the current fingerprint against the stored one. If they differ, delete all chunks from that document and re-ingest. This ensures the knowledge base reflects the current state of source documents without requiring a full re-index.

Component 2: Chunking Strategy

Chunking is where most RAG implementations make their first critical mistake. The naive approach — split every document into fixed-size chunks of N tokens with M tokens of overlap — is simple to implement and reliably mediocre in production.

The problem is that fixed-size chunking ignores document structure. A 512-token chunk might start mid-sentence, cut across a table, or split a code example in half. The resulting chunks are semantically incoherent, and incoherent chunks produce poor embeddings.

Hierarchical Chunking

The correct approach for structured documents is hierarchical chunking: split first by document structure (sections, subsections), then by semantic coherence within each structural unit.

The algorithm:

1. 1Parse the document into a structural tree using heading levels
2. 2For each leaf node (the smallest structural unit), check its token count
3. 3If the leaf is under the target chunk size (typically 256–512 tokens), keep it as a single chunk
4. 4If the leaf exceeds the target, split it at sentence boundaries, not token boundaries
5. 5For each chunk, prepend the full heading path as context: "Employee Policy Handbook > Remote Work Policy > Equipment Allowance"

The heading path prepend is critical. Without it, a chunk about "Equipment Allowance" has no context about what kind of equipment, for what purpose, or in what policy framework. With it, the embedding captures the full semantic context.

Sentence-Window Chunking

For documents without clear structural hierarchy (transcripts, emails, support tickets), use sentence-window chunking:

• Embed at the sentence level for precise retrieval
• At generation time, expand each retrieved sentence to its surrounding window (typically ±2–3 sentences) before injecting into context

This gives you the precision of sentence-level retrieval with the coherence of paragraph-level context.

Chunk Size Guidelines

Component 3: Embedding Model Selection

The embedding model converts text into a dense vector representation. The quality of this representation determines the quality of semantic search. Choosing the wrong model is the second most common RAG failure mode.

The Embedding Model Decision Matrix

Asymmetric Embedding

Most production RAG systems use asymmetric embedding: the query and the document chunks are embedded differently. The query is typically short and question-like; the document chunks are longer and declarative. Models like E5 and Instructor-XL are explicitly trained for this asymmetry — you prefix queries with "query: " and documents with "passage: " to activate the correct embedding mode.

Using a symmetric model (same embedding for queries and documents) on an asymmetric retrieval task degrades recall by 15–30% in practice.

Embedding Dimensionality

Higher-dimensional embeddings capture more semantic nuance but cost more to store and query. The practical sweet spot for most enterprise RAG systems is 1,536 dimensions (text-embedding-3-large) or 1,024 dimensions (embed-english-v3.0). Going above 3,072 dimensions rarely produces meaningful retrieval improvements for document-level tasks.

Component 4: Vector Store Architecture

The vector store is the database that holds your embeddings and supports approximate nearest-neighbour (ANN) search. The choice of vector store matters less than most people think — the retrieval quality is dominated by chunking and embedding quality, not the ANN algorithm. What matters is operational fit.

Vector Store Selection Guide

For most enterprise internal knowledge base use cases — corpora under 10 million chunks, latency requirements above 100ms — pgvector is the pragmatic choice. It runs in your existing PostgreSQL instance, supports hybrid search via tsvector, and eliminates a separate infrastructure dependency. The ANN performance gap versus dedicated vector stores is irrelevant at this scale.

Index Configuration

For pgvector, use the HNSW index (available since pgvector 0.5.0) rather than IVFFlat:

CREATE INDEX ON document_chunks
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);

The m parameter controls the number of connections per node (higher = better recall, more memory). ef_construction controls index build quality (higher = better recall, slower build). For a knowledge base that is re-indexed infrequently, ef_construction = 64 is a reasonable default.

Component 5: Retrieval Strategy

This is where the architecture diverges most sharply from the toy demos. Naive RAG uses a single vector similarity search. Production RAG uses a multi-stage retrieval pipeline.

Stage 1: Hybrid Search

Hybrid search combines dense vector search (semantic similarity) with sparse keyword search (BM25 or TF-IDF). The two signals are complementary:

• Dense search excels at semantic matching: "What is the policy on working from home?" retrieves chunks about "remote work" even if the query never uses that phrase.
• Sparse search excels at exact matching: "What is the GDPR Article 17 procedure?" retrieves chunks containing that exact string, which dense search may miss if the embedding space does not preserve legal citation strings precisely.

The standard fusion approach is Reciprocal Rank Fusion (RRF):

RRF_score(chunk) = Σ 1 / (k + rank_in_list_i)

Where k is a smoothing constant (typically 60) and the sum is over all retrieval lists. RRF is parameter-free, robust to score scale differences between dense and sparse systems, and consistently outperforms weighted linear combination in practice.

Stage 2: Metadata Filtering

Before or alongside vector search, apply metadata filters to restrict the search space:

{
  "query": "equipment allowance for remote employees",
  "filters": {
    "document_type": "policy",
    "department": "HR",
    "last_updated_after": "2026-01-01"
  }
}

Metadata filtering dramatically improves precision in large knowledge bases. Without it, a query about HR policy might surface results from engineering documentation that happen to use similar vocabulary.

Stage 3: Reranking

The initial retrieval (hybrid search + metadata filtering) returns a candidate set — typically 20–50 chunks. Reranking applies a more expensive cross-encoder model to score each candidate against the query and reorder them by relevance.

The key distinction:

• Bi-encoder (embedding model): encodes query and document independently, computes similarity via dot product. Fast, scalable, but misses fine-grained query-document interactions.
• Cross-encoder (reranker): encodes query and document together, attends to their interaction. Slower (cannot pre-compute), but significantly more accurate.

Recommended rerankers: Cohere Rerank, bge-reranker-large, ms-marco-MiniLM-L-12-v2.

The typical pipeline: retrieve top-50 via hybrid search → rerank → take top-5 for context injection. The reranker adds 50–200ms of latency but reduces hallucination by ensuring only the most relevant chunks reach the model.

Stage 4: Maximal Marginal Relevance (MMR)

After reranking, apply MMR to the final selection to balance relevance with diversity:

MMR(chunk) = λ · relevance(chunk, query) - (1-λ) · max_similarity(chunk, selected_chunks)

MMR penalises chunks that are highly similar to already-selected chunks. Without it, the top-5 chunks often cover the same passage from slightly different angles, wasting context window space. With MMR (λ = 0.7 is a good default), the selected chunks cover different aspects of the answer.

Component 6: Context Assembly and Generation

The final stage assembles the retrieved chunks into a context block and injects it into the model's prompt. This is where the retrieval pipeline connects back to the prompt engineering principles from Part 2.

Context Block Structure

[RETRIEVED CONTEXT]
Source: Employee Policy Handbook v3.2, Section: Remote Work Policy (updated 2026-06-01)
---
Equipment Allowance: Remote employees are eligible for a one-time equipment allowance of $1,500 upon joining, and an annual refresh allowance of $500 thereafter. Allowances must be claimed within 90 days of eligibility. Approved categories include monitors, keyboards, webcams, and ergonomic accessories.
---
Source: IT Procurement Guidelines v2.1, Section: Approved Vendors (updated 2026-03-15)
---
Equipment purchases must be made through approved vendors listed in the IT procurement portal. Personal purchases for reimbursement are not permitted for items above $200.
---
[END RETRIEVED CONTEXT]

Using only the information in the RETRIEVED CONTEXT above, answer the following question. If the context does not contain sufficient information to answer, state that explicitly — do not use knowledge outside the provided context.

Question: What is the equipment allowance for new remote employees, and where can they buy equipment?

The Grounding Instruction

The instruction "Using only the information in the RETRIEVED CONTEXT above... do not use knowledge outside the provided context" is the single most important line in a RAG system prompt. Without it, the model will seamlessly blend retrieved context with parametric knowledge, producing answers that are partially grounded and partially hallucinated — and indistinguishable from each other.

This instruction does not eliminate hallucination entirely, but it dramatically reduces it and makes failures more detectable: when the model violates the instruction, it tends to do so in ways that are easier to catch (citing specific facts not present in the context).

Citation Format

Require the model to cite its sources in a structured format:

Answer: New remote employees are eligible for a one-time equipment allowance of $1,500 [1]. Equipment must be purchased through approved vendors in the IT procurement portal; personal purchases for reimbursement are not permitted for items above $200 [2].

Sources:
[1] Employee Policy Handbook v3.2, Remote Work Policy (updated 2026-06-01)
[2] IT Procurement Guidelines v2.1, Approved Vendors (updated 2026-03-15)

Structured citations serve two purposes: they allow users to verify the answer against the source, and they create an audit trail for compliance-sensitive use cases.

Handling Retrieval Failure

When retrieval returns no relevant chunks (similarity scores below threshold), the system must not fall back to parametric generation. The correct response:

{
  "answer": null,
  "retrieval_status": "no_relevant_context",
  "message": "I could not find relevant information in the knowledge base to answer this question. Please consult [specific resource] or contact [specific team].",
  "suggested_escalation": "[email protected]"
}

Define a similarity threshold (typically cosine similarity > 0.75 for the top result) below which the system returns the no-context response rather than attempting generation. This threshold should be tuned on a held-out evaluation set, not set arbitrarily.

Building the Internal Knowledge Base Agent

With the six components defined, here is how they compose into a complete agent:

System Architecture

User Query
    ↓
Query Analysis (intent classification, entity extraction)
    ↓
Metadata Filter Construction
    ↓
Hybrid Search (dense + sparse, top-50)
    ↓
Reranking (cross-encoder, top-10)
    ↓
MMR Selection (top-5)
    ↓
Similarity Threshold Check
    ↓ (above threshold)         ↓ (below threshold)
Context Assembly              No-Context Response
    ↓
Grounded Generation
    ↓
Citation Extraction
    ↓
Structured Response

Query Analysis Layer

Before retrieval, run a lightweight query analysis step:

Intent classification: Is this a factual lookup ("What is the equipment allowance?"), a procedural question ("How do I submit an expense claim?"), or a policy interpretation ("Does the remote work policy apply to contractors?")? Different intents may warrant different retrieval strategies — procedural questions benefit from retrieving step-by-step documents; policy interpretation questions benefit from retrieving the policy document plus any related FAQs.

Entity extraction: Extract named entities from the query (department names, policy names, product names, regulation identifiers) and use them to construct metadata filters. "What is the GDPR data retention policy?" should filter to documents tagged with regulation: GDPR and document_type: policy.

Query rewriting: For conversational agents, rewrite the current query to be self-contained using conversation history. "What about contractors?" in the context of a previous question about remote work policy should be rewritten to "Does the remote work policy apply to contractors?" before retrieval.

Evaluation Framework

A RAG system without an evaluation framework is a system you cannot improve. The minimum viable evaluation suite has three metrics:

Retrieval recall@k: For a set of test questions with known relevant documents, what fraction of the relevant documents appear in the top-k retrieved chunks? Target: recall@5 > 0.85.

Answer faithfulness: For a set of test questions with generated answers, what fraction of the claims in the answer are supported by the retrieved context? Use an LLM-as-judge approach with a strict faithfulness rubric. Target: > 0.90.

Answer relevance: Does the generated answer actually address the question asked? Again, LLM-as-judge. Target: > 0.90.

When RAG Is Not Enough

RAG solves the knowledge currency problem — the agent always reads from the current state of the knowledge base. But it has its own ceiling:

Highly specialised output format requirements — if the agent needs to produce outputs in a very specific style, tone, or structure that the base model does not naturally produce, RAG cannot fix this. The model's generation behaviour is determined by its weights, not its context. This is where fine-tuning (Parts 5 and 6) becomes relevant.

Domain vocabulary gaps — if the knowledge base uses highly specialised terminology that the base model has never encountered (rare medical terms, proprietary product names, niche regulatory frameworks), the model may misinterpret retrieved context even when retrieval is accurate. Continued pretraining (Part 6) addresses this.

Reasoning over large retrieved sets — if answering a question requires synthesising information from 20+ documents simultaneously, RAG struggles. The context window fills up, attention dilutes, and the model loses track of the full picture. This is an active research area; current mitigations include map-reduce patterns and multi-hop retrieval chains.

Latency-sensitive applications — the multi-stage retrieval pipeline (hybrid search + reranking + MMR) adds 200–500ms of latency before generation begins. For real-time applications with sub-200ms requirements, this pipeline is too slow. Distillation into a fine-tuned model (Part 5) may be the right trade-off.

Frequently Asked Questions

Q: How large does a knowledge base need to be before RAG is worth the complexity?

RAG adds significant architectural complexity compared to prompt engineering. The break-even point is roughly when your knowledge base exceeds what fits in a single large context window (currently ~128K–200K tokens for frontier models). Below that threshold, consider context stuffing — injecting the entire knowledge base into every request. It is inelegant but operationally simple, and it works surprisingly well for small, stable corpora. Above the threshold, RAG is the right tool.

Q: Should I use OpenAI embeddings or open-source models?

For most enterprise use cases, text-embedding-3-large (OpenAI) is the pragmatic choice: strong performance, simple API, no infrastructure to manage. The main reasons to use open-source models are data residency requirements (you cannot send documents to an external API), cost at very high volume (millions of embeddings per day), or specialised domains where fine-tuned open-source models outperform general-purpose commercial ones. For code-heavy knowledge bases, voyage-code-2 consistently outperforms text-embedding-3-large.

Q: How often should I re-index the knowledge base?

It depends on how frequently your source documents change. For a policy handbook that is updated quarterly, a weekly re-index is more than sufficient. For a knowledge base fed by a live ticketing system or a frequently updated wiki, consider event-driven re-indexing: trigger re-ingestion whenever a source document is modified, rather than on a fixed schedule. The change detection fingerprint approach described above makes this straightforward.

Q: What is the right chunk size?

There is no universal answer, but the most common mistake is chunking too large. Large chunks (1,000+ tokens) reduce retrieval precision — the retrieved chunk contains the relevant passage plus a lot of irrelevant surrounding text, which dilutes the model's attention. Start with 256–512 tokens and measure retrieval recall on your evaluation set. Increase chunk size only if you find that relevant information is being split across chunk boundaries.

Q: How do I handle tables and structured data in documents?

Tables are the hardest document element to chunk correctly. The naive approach (treating table rows as text) loses the column header context. The correct approach: convert each table to a set of natural language statements, one per row, with the column headers embedded in each statement. "The equipment allowance for new remote employees is $1,500 (one-time) and $500 (annual refresh)." This produces chunks that embed and retrieve correctly.

Q: Can RAG work with private, on-premise models?

Yes. The RAG architecture is model-agnostic. Replace the OpenAI embedding API with a locally hosted model (bge-large-en-v1.5 via Ollama or a dedicated inference server), replace the generation model with a locally hosted LLM (Llama 3, Mistral, Qwen), and replace the managed vector store with pgvector or Qdrant running on your own infrastructure. The architecture is identical; only the infrastructure changes. This is the standard approach for regulated industries with strict data residency requirements.

Q: How do I prevent the agent from answering questions outside the knowledge base scope?

Two mechanisms work together. First, the similarity threshold check: if no retrieved chunk exceeds the relevance threshold, return the no-context response rather than generating. Second, the grounding instruction in the system prompt: "Using only the information in the RETRIEVED CONTEXT above... do not use knowledge outside the provided context." The threshold handles the case where retrieval fails; the grounding instruction handles the case where retrieval returns marginally relevant results that the model might otherwise supplement with parametric knowledge.