RAG Isn't a Vector Database. It's a Retrieval Problem.

Every team building with RAG right now has roughly the same setup. Documents get chunked, chunks get embedded, embeddings go into a vector database, and at query time you run a similarity search, grab the top-k chunks, and stuff them into the prompt.

It works well enough to demo. It falls apart once you move past a handful of short documents and into real enterprise content: hundreds of pages, multiple documents, sections that reference each other, information buried on page 40 that your top-5 similarity search will never surface.

I want to talk about why that happens and what actually fixes it. Not "what is RAG," you've read that post already, from ten other people. I want to talk about retrieval strategy, because that's the part almost nobody gets right, and it's the part that decides whether your RAG system is useful or just expensive.

What RAG actually solves

Strip away the acronym and RAG solves one problem: your model doesn't know your data, and you can't fit all your data into the context window. So instead of hoping the model already knows the answer, you retrieve the relevant pieces of your data and hand them to the model alongside the question.

That's it. Retrieval, then generation, grounded in what was retrieved.

The generation part, the LLM producing an answer from the retrieved context, is the easy part now. Every frontier model is good at synthesising an answer from provided context. The hard part, the part that decides whether your system actually works, is retrieval. Did you hand the model the right pieces of information? If you didn't, it doesn't matter how good the model is. It will confidently answer using incomplete or wrong context, and you'll get a wrong answer that reads as if it's right.

When RAG isn't the answer

Before getting into strategy, it's worth being honest about when RAG is the wrong tool. I've seen teams reach for it by default, and that's as much of a mistake as reaching for multi-agent architecture by default.

When the data fits in the context window. If you're working with a handful of documents that comfortably fit in a modern model's context window, just put them in the prompt. Skip the vector database, skip the chunking, skip the retrieval step. You'll get better answers because the model sees everything, with no risk of the retrieval step excluding something relevant.

When you need exact, structured data. If the question is "what's the current balance for account 4471," that's a database query, not a retrieval problem. RAG is for unstructured or semi-structured knowledge. Don't put structured, queryable data into a vector store and hope similarity search finds the right row. Query your database directly.

When you need real-time or transactional data. RAG retrieves from a corpus that was indexed at some point in the past. If the answer depends on the current state of the world, today's inventory count, the live status of a shipment- that's a tool call or an API call, not a retrieval problem. Trying to keep a vector index in sync with real-time state is fighting the wrong battle.

When the task requires reasoning over the entire document, not fragments. Some tasks genuinely need the whole document: summarising an entire contract, checking a document for internal consistency, comparing two full policies clause by clause. Chunked retrieval fundamentally can't do this well, because it's designed to surface relevant fragments, not preserve the whole. For these tasks, either fit the whole document in context or use a different pattern entirely, like a map-reduce summarisation pass over the full document.

When the answer requires an action, not a lookup. If the task is "cancel this order" or "update this record," that's a tool call routed through an agent, not a retrieval task. RAG answers questions. It doesn't take actions.

If none of those apply and you genuinely have a large, evolving, unstructured knowledge base that needs to answer questions, RAG is the right call. The question then becomes: how do you retrieve well?

Chunking is where most systems fail first

Before you even get to retrieval strategy, you have to get chunking right, because a weak chunking strategy makes every retrieval strategy built on top of it weaker too.

Fixed-size chunking is the default, and it's usually wrong. Split every document into 500-token chunks with some overlap, embed each chunk, done. The problem is that fixed-size chunks don't respect the actual structure of the document. A chunk boundary lands in the middle of a paragraph, splitting a complete thought across two chunks. Neither chunk retrieves well on its own because neither contains the full idea.

Structure-aware chunking respects the document's own organisation. Split on headings, sections, and natural boundaries, not an arbitrary token count. A section on "Payment Terms" becomes one chunk (or a few, if it's long), not an arbitrary slice that starts mid-sentence and ends mid-sentence. This preserves semantic coherence, which directly improves retrieval quality, because now each chunk is actually about one coherent thing.

Parent-child chunking gets you both precision and context. Embed small, precise chunks for the similarity search, a paragraph or a bullet point, but store a reference to the larger parent section each one belongs to. When a small chunk matches the query, retrieve the small chunk for matching precision, but return the parent section to the LLM for full context. This solves a real problem: small chunks match better on similarity search because they're focused, but small chunks alone often lack enough surrounding context for the LLM to answer well.

# Conceptual structure, not tied to any specific library

class Chunk:
    chunk_id: str
    text: str
    parent_section_id: str
    embedding: list[float]

class ParentSection:
    section_id: str
    full_text: str
    heading: str
    document_id: str

# At retrieval time:
matched_chunks = vector_search(query_embedding, top_k=8)
parent_sections = {get_parent(c.parent_section_id) for c in matched_chunks}
context = "\n\n".join(p.full_text for p in parent_sections)

Get the chunking strategy wrong, and no amount of clever retrieval logic downstream will fully make up for it. Structure-aware, parent-child chunking is the baseline I'd start from for any document set with real internal structure: reports, contracts, technical documentation, anything with headings.

Similarity search alone is not enough

Here's the honest limitation of naive vector similarity search: it finds chunks that are semantically close to your query. It does not guarantee it finds the chunks that actually answer your query, and it has no concept of completeness.

Hybrid search closes part of the gap. Pure vector search misses exact matches (specific terms, product codes, names, numbers) because embeddings capture semantic meaning, not exact lexical matches. Hybrid search combines vector similarity with keyword search, BM25 or similar, then merges the results. If someone asks about "clause 7.2," a keyword search will find it reliably even if the vector search's semantic match is weak.

vector_results = vector_search(query_embedding, top_k=20)
keyword_results = bm25_search(query_text, top_k=20)

combined = reciprocal_rank_fusion(vector_results, keyword_results)
top_results = combined[:10]

Reranking fixes the "top-k is a guess" problem. Vector similarity is a fast, approximate signal. It's good at narrowing thousands of chunks down to a shortlist, but it's not precise about which of those actually answers the query best. A reranker re-scores that shortlist with much higher precision. Typically this is a cross-encoder model that looks at the query and each candidate chunk together, instead of comparing pre-computed embeddings the way vector search does.

The pattern: retrieve a wide net with vector search, say the top 30 to 50 candidates, then rerank that shortlist down to the 5 to 10 you actually send to the LLM. This two-stage approach, cheap wide retrieval followed by expensive precise reranking, consistently outperforms single-stage similarity search, because the reranker catches relevant chunks that ranked lower on raw similarity but are actually more useful for answering the question.

candidates = vector_search(query_embedding, top_k=40)
reranked = rerank_model.score(query_text, [c.text for c in candidates])
top_results = sorted(reranked, reverse=True)[:8]

Query decomposition handles multi-part questions. If someone asks "what are the payment terms and what happens if we miss a deadline," that's really two separate retrieval questions bundled into one query. A single similarity search against the combined query often retrieves chunks relevant to neither part well. Decompose the query into sub-questions, retrieve separately for each, then merge the results before generation.

HyDE helps when the query and the answer don't share vocabulary. Hypothetical Document Embeddings work like this: you ask the LLM to generate a hypothetical answer to the query first, then embed that hypothetical answer and use it for the similarity search, instead of embedding the raw query. This works because the hypothetical answer is closer in phrasing and vocabulary to what the actual answer chunk looks like, compared to the original question. It's a useful trick when your users ask questions in casual language, but your source documents are written formally.

The real problem: large documents and the top-k ceiling

Here's the failure mode that matters most, and it's the one most RAG tutorials don't address.

Say you have ten documents, each fifty pages. You've chunked them properly, embedded everything, and you're retrieving the top 8 or 10 chunks per query. That works fine when the answer lives in one or two places in the corpus. It breaks down when the answer is spread across the document, or when something important sits in a section that just doesn't rank in the top 10 for that specific query's phrasing.

I ran into this directly, feeding large documents into vector storage and then asking questions where the answer depended on something near the end of a fifty-page document. The top-k similarity search, focused on the query's most semantically similar chunks, simply never surfaced it. Not because the retrieval was broken. Because top-k similarity search is fundamentally a "most similar" mechanism, not a "complete coverage" mechanism. It has no concept of an entire relevant section going unchecked.

My first instinct was to fix this after the fact. Run the normal vector search, then check whether anything structurally important got left out, and pull it in as a correction. That works, but it's reactive. You're paying for a full similarity search across the whole corpus and then patching the gaps it left behind. It also doesn't make the original retrieval any more targeted. You're still searching everything, blindly, and hoping the top-k happens to land on the right sections.

The better version flips the order. Build a lightweight structural map of each document at ingestion time: its sections, headings, and a short description of what each one covers, separate from the chunk-level embeddings. Think of it as an index of the document's shape, not its content.

At query time, use that structural map first, as a router. Run a quick relevance pass over the section summaries, not the full chunk corpus, to work out which sections are actually likely to matter for this query. That pass is small and cheap, because you're comparing the query against a handful of section summaries per document, not thousands of chunks. Once you know which sections are relevant, scope your vector and hybrid search to chunks inside those sections only, and rerank from there.

This gives you targeted retrieval instead of a blind sweep across the entire corpus. The search space shrinks before the expensive part even starts, and because you started from the document's actual structure, you're far less likely to silently skip a whole section that never happened to rank in a global top-k.

# Conceptual sketch: structural index used as an upfront router

class DocumentOutline:
    document_id: str
    sections: list[SectionSummary]  # heading, one-line summary, chunk_ids

def retrieve_with_structural_routing(query, document_ids, top_k=10):
    relevant_sections = []
    for doc_id in document_ids:
        outline = get_outline(doc_id)
        relevant_sections.extend(
            filter_sections_by_relevance(outline.sections, query)
        )

    candidate_chunk_ids = [
        chunk_id for section in relevant_sections for chunk_id in section.chunk_ids
    ]

    scoped_vector_hits = vector_search(
        embed(query), top_k=40, restrict_to=candidate_chunk_ids
    )
    scoped_keyword_hits = bm25_search(query, restrict_to=candidate_chunk_ids)

    combined = reciprocal_rank_fusion(scoped_vector_hits, scoped_keyword_hits)
    return rerank(query, combined)[:top_k]

I still keep a lightweight coverage check as a safety net on top of this, mainly for ambiguous queries where relevance across sections isn't obvious upfront. But the core mechanism should be the router, not the correction. Route first, search within scope, rerank. That's what actually solves the large-document problem, rather than papering over it after a broad search has already missed things.

The structural map itself is cheap to build. It's not another embedding pass; it's a lightweight extraction of headings, section boundaries, and a short description of what each section contains. Build it once at ingestion time and reuse it for every query against that document.

This pattern matters most exactly in the scenario you'd expect: large documents, many of them, where information relevant to a single query might be scattered across sections that don't share vocabulary. Ten fifty-page documents is precisely where naive top-k similarity search starts quietly dropping things, and where structural routing earns its cost.

Putting it together

For a real production RAG system working with large, structured documents, here's the combination I'd actually use.

Structure-aware, parent-child chunking at ingestion, respecting document boundaries, embedding small precise chunks, retaining larger parent context. A structural map built at ingestion time, used to route queries to the right sections before the expensive search even runs. A hybrid retrieval pass, vector similarity plus keyword search, scoped to those relevant sections, to catch both semantic matches and exact terms. Reranking on the combined candidate set, so the final chunks sent to the LLM are precision-ranked, not just similarity-ranked.

On the infrastructure side, Qdrant and pgvector both handle the vector layer well. pgvector is a strong choice if you want to keep everything in Postgres alongside your existing data rather than running a separate vector database. OpenAI's embedding models are a reasonable default for the semantic layer. None of this is exotic. What matters is the retrieval logic wrapped around the storage layer, not the storage layer itself.

The takeaway

Nobody's RAG system fails because the vector database was the wrong choice. It fails because the retrieval strategy was naive: one similarity search, top-k, done, applied to a problem that needed chunking discipline, hybrid retrieval, reranking, and, for large documents specifically, a structural router that scopes the search before it even runs.

The generation step gets all the attention because it's the part you see, the answer in prose that looks confident either way. The retrieval step is where the real engineering work is, and it's invisible right up until the system gives a wrong answer with complete confidence because it never looked in the right place to begin with.

Build the retrieval strategy like you mean it. The model was never the bottleneck.

I write about agentic systems, LLM infrastructure, and what actually works in production AI. Subscribe to get new posts in your inbox.