Beyond Naive RAG: Context Engineering for AI Products That Actually Answer

Every team that shipped an AI feature in the last three years has built some version of the same pipeline: split the docs into chunks, embed them, stick them in a vector database, retrieve the top five by cosine similarity, and paste them into a prompt. It demos beautifully. Then real users show up, ask real questions, and the thing confidently answers wrong — or worse, answers right while citing a document the user was never supposed to see.

We build docs assistants and support bots for clients, and the pattern is consistent: the naive RAG recipe gets you to a demo, not to a product. What gets you to a product is a discipline that's come to be called context engineering — treating everything that decides what the model sees as a system to be designed, measured, and maintained. Retrieval is one component of that system. It is not the system.

This post is the playbook we wish someone had handed us in 2023: why the naive recipe breaks, what replaces each piece, when to skip retrieval entirely, and how to know whether any of it is working.

Why the 2023 recipe disappoints

The chunk → embed → top-k → stuff pipeline fails in production for three structural reasons, not because you picked the wrong embedding model.

Chunks lose the context that made them meaningful

Split a document every 500 tokens and you get fragments like "It defaults to 30 days and can be extended via the API." What defaults to 30 days? The retention window? The trial period? The refund policy? The sentence was unambiguous under its heading, three levels deep in a page about session tokens. The chunk, floating free in a vector index, is ambiguous — so its embedding is mushy, retrieval ranks it poorly, and when it is retrieved, the model misreads it.

Naive splitting also cuts through structure: half a code sample without its imports, a table body without its header row, step 4 of a procedure without steps 1–3. The model gets shrapnel and does its best, which is exactly the failure mode you don't want — plausible answers assembled from decontextualized fragments.

Similar is not the same as relevant

Embeddings measure semantic similarity, and similarity is a proxy for relevance that breaks in predictable places. Ask "how do I delete my account" and the nearest neighbors are about creating accounts — the vocabulary overlaps almost entirely, and the one word that matters is the one the embedding smooths over. Exact identifiers are worse: error codes, flag names, version strings, ERR_CONN_RESET. Vector search treats these as noise; your users treat them as the whole question.

Some questions need synthesis, not lookup

"What changed in the billing flow between v2 and v3?" has no single chunk containing the answer. It requires retrieving the v2 docs and the v3 docs and comparing them. "Which of our SDKs support streaming?" requires a sweep across many pages, not the top five neighbors of the question. One-shot top-k retrieval structurally cannot answer these — no ranking function fixes a question whose answer must be assembled.

Context engineering: the actual discipline

The reframe that helped us most: stop asking "how do we improve retrieval?" and start asking "what should the model see for this request, and how do we construct it?" That's context engineering. It includes retrieval, but also:

• Corpus preparation — what gets indexed, how it's chunked, what metadata rides along
• Query understanding — rewriting, decomposition, routing to the right source
• Assembly — ordering, deduplication, citation scaffolding, instructions about how to use (and when to distrust) the provided material
• Access control — making sure the context is scoped to what this user may see
• The decision not to retrieve at all — sometimes the right context is the whole corpus, or a SQL result, or nothing

Every one of these is a design decision you're making whether you know it or not. Naive RAG is just the version where every decision defaults to whatever the tutorial did.

Chunking that respects structure

Documents have structure — headings, code fences, tables, lists — and your chunker should parse it, not bulldoze it. The rules we apply on nearly every project:

1. Split at structural boundaries (headings, then paragraphs), never mid-sentence, never mid-code-block, never mid-table.
2. Keep atomic units atomic. A code sample or a table is one chunk even if it's oversized. A truncated table is worse than a long chunk.
3. Prepend a contextual header to every chunk: document title plus the heading path that leads to it. This is the cheapest, highest-leverage fix in this whole post — it repairs both the embedding (the vector now encodes what the fragment is about) and the generation (the model sees where the text came from).

A sketch of the shape:

interface Chunk {
  content: string;      // header + body, what actually gets embedded
  headingPath: string;  // "Billing > Refunds > Annual plans"
  sourcePath: string;   // "docs/billing/refunds.md"
  tokenCount: number;
}

function chunkMarkdown(doc: ParsedDoc, maxTokens = 512): Chunk[] {
  const chunks: Chunk[] = [];

  for (const section of doc.sections) {          // split on heading boundaries
    const units = splitIntoUnits(section.body);  // paragraphs, code fences,
                                                 // tables — each kept whole
    let buf: Unit[] = [];
    const flush = () => {
      if (buf.length === 0) return;
      const header = `${doc.title} — ${section.headingPath.join(" > ")}`;
      chunks.push({
        content: `${header}\n\n${buf.map(u => u.text).join("\n\n")}`,
        headingPath: section.headingPath.join(" > "),
        sourcePath: doc.path,
        tokenCount: countTokens(header) + sum(buf.map(u => u.tokens)),
      });
      buf = [];
    };

    for (const unit of units) {
      const pending = sum(buf.map(u => u.tokens)) + unit.tokens;
      // Oversized atomic unit (big table, long code block): own chunk, intact.
      if (unit.atomic && unit.tokens > maxTokens) { flush(); buf = [unit]; flush(); continue; }
      if (pending > maxTokens) flush();
      buf.push(unit);
    }
    flush();
  }
  return chunks;
}

A stronger variant of the header trick is to have a cheap model write a one-sentence summary of the surrounding document and prepend that — Anthropic published this as "contextual retrieval." Start with heading paths; they're free and get you most of the way.

Hybrid retrieval and re-ranking

Vectors miss exact terms; keyword search misses paraphrases. Run both and fuse. BM25 (or Postgres full-text search, which is close enough in practice) catches the error codes and flag names; the vector index catches "my payment didn't go through" → the page titled "failed charges." Reciprocal rank fusion combines the two ranked lists without needing their scores to be comparable — each result contributes 1 / (k + rank) from each list it appears in, with k conventionally set to 60.

If you're on Postgres, you don't need a second datastore for this. One table, one vector column, one tsvector column:

// chunks(id, tenant_id, acl_groups text[], content, heading_path,
//        source_path, embedding vector(1024), tsv tsvector)

export async function hybridSearch(
  queryText: string,
  queryEmbedding: number[],
  tenantId: string,
  userGroups: string[],
  limit = 12,
) {
  const embedding = JSON.stringify(queryEmbedding);
  return db.query(
    `
    WITH semantic AS (
      SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> $1::vector) AS rank
      FROM chunks
      WHERE tenant_id = $2 AND acl_groups && $3
      ORDER BY embedding <=> $1::vector
      LIMIT 50
    ),
    lexical AS (
      SELECT id, ROW_NUMBER() OVER (
        ORDER BY ts_rank_cd(tsv, websearch_to_tsquery('english', $4)) DESC
      ) AS rank
      FROM chunks
      WHERE tenant_id = $2 AND acl_groups && $3
        AND tsv @@ websearch_to_tsquery('english', $4)
      LIMIT 50
    )
    SELECT c.id, c.content, c.heading_path, c.source_path,
           COALESCE(1.0 / (60 + s.rank), 0)
         + COALESCE(1.0 / (60 + l.rank), 0) AS rrf_score
    FROM chunks c
    LEFT JOIN semantic s ON s.id = c.id
    LEFT JOIN lexical  l ON l.id = c.id
    WHERE s.id IS NOT NULL OR l.id IS NOT NULL
    ORDER BY rrf_score DESC
    LIMIT $5
    `,
    [embedding, tenantId, userGroups, queryText, limit],
  );
}

Note the tenant_id and acl_groups predicates inside both branches — we'll come back to why.

On top of fusion, add a re-ranker: a cross-encoder that scores each (query, chunk) pair jointly instead of comparing precomputed vectors. Retrieve 50 candidates cheaply, re-rank, keep the best 10. Cross-encoders read the query and the document together, so they catch exactly the distinctions embeddings blur — negation, "delete" vs "create," which entity the question is actually about. Hosted re-rank endpoints (Cohere and others) make this an afternoon of work, and it's routinely the single largest retrieval-quality jump after contextual headers.

Query transformation

Users don't write queries; they write messages. "it still doesn't work" is a terrible retrieval query and a completely normal support message. Put a cheap LLM pass in front of retrieval:

• Rewriting. Fold conversation history into a standalone query ("it" → "webhook signature verification"), expand abbreviations, strip the pleasantries.
• Decomposition. Split multi-part questions into sub-queries, retrieve for each, and merge. This is how you make "compare X and Y" answerable by a lookup system: two lookups.
• HyDE (hypothetical document embeddings), briefly: instead of embedding the question, have the model write a hypothetical answer and embed that — answers live closer to documents in embedding space than questions do. It's clever and it works on some corpora, but it adds a generation step to every query. Try rewriting and hybrid search first; reach for HyDE when short queries against long documents are demonstrably your bottleneck.

Metadata filtering and access control — the hole everyone ships

Here's the uncomfortable one. Most teams index everything into one collection, retrieve by similarity, and enforce permissions... nowhere. The vector store doesn't know your org chart. If the HR policies and the customer contracts and the public docs share an index, a well-phrased question will surface any of them, and the model will happily summarize a document into an answer for a user who could never have opened it. This is not hypothetical; it's the default behavior of the naive pipeline, and we have found it in real systems during handovers.

The rules:

• Filter at query time, inside the database, before ranking. Attach tenant_id, ACL groups, and visibility metadata to every chunk at index time, and make every retrieval query carry the caller's identity as a hard predicate — as in the SQL above. Post-filtering ("retrieve 10, drop the forbidden ones") leaks information through what remains and sometimes returns zero results after filtering.
• The retriever runs with the user's permissions, not the service's. If your bot's database role can read everything, every prompt-injection attack inherits that power.
• Metadata is also a relevance tool. Product version, locale, doc type, recency — letting "v3 API" questions filter to version = 3 chunks fixes a whole category of subtly-wrong answers that no amount of embedding quality will.

If you take one thing from this post: your retrieval layer is part of your authorization surface. Review it like one.

Long-context models vs RAG — and how caching changed the math

Current frontier models take hundreds of thousands to a million tokens of context. So for a lot of products the honest question is: do you need retrieval at all? If your entire corpus is a few hundred pages — one product's docs, a policy handbook, a single codebase's guides — you can put all of it in the prompt, skip the index, and delete the failure modes that come with it. No chunking bugs, no stale index, no retrieval misses.

The objection used to be cost: re-sending a 200K-token corpus on every request is absurd. Prompt caching changed that. Mark the corpus as a cached prefix (on Anthropic's API, cache_control: {type: "ephemeral"}; other providers have equivalents) and after the first request it's read from cache — billed at roughly a tenth of the normal input price on Anthropic's pricing, with a small premium on the initial write. The corpus has to sit at the front of the prompt and stay byte-stable, with the volatile parts (the question, the history) after it — but that's an assembly-order decision, which is to say: context engineering.

The two compose, too: retrieve at the document level instead of the chunk level, and cache the shared system prompt and instructions. Long context made retrieval optional in more cases; it didn't make context construction optional in any of them.

Agentic retrieval: let the model search

One-shot retrieval bets everything on the first query. Agentic retrieval gives the model a search tool and lets it iterate: search, read, notice what's missing, reformulate, search again. This is how a person uses documentation, and it straightforwardly handles the synthesis questions that top-k can't — the model retrieves the v2 page, then the v3 page, then answers the comparison.

The trade is latency and cost: several model turns instead of one. Our split in practice: one-shot hybrid retrieval for the fast path (most support questions are lookups, and users want answers in two seconds), agentic search for the escalation path — the "research" mode, the complex ticket, the question the fast path failed on. You don't have to pick one; route between them.

Two implementation notes. Give the agent the same permission-scoped search function as the fast path — a tool call is not a reason to bypass ACLs. And cap the loop (search budget, turn limit); an agent that can search forever occasionally will.

Structured knowledge: stop embedding facts

Embeddings are for prose. If the answer lives in a database, query the database. "What's my current plan?" "How many seats do we have left?" "Is the EU region GA yet?" — these have exact answers in Postgres or a feature-flag service, and running them through an embedding index converts precise facts into approximate retrieval for no benefit. Same for relationships: "which services depend on the auth module" is a graph traversal, not a similarity search.

The pattern that works is routing: classify the question (a fast model call, or the agent's tool choice), send factual/entity queries to SQL or an API through safe parameterized tools, send conceptual/how-to queries to document retrieval, and let hard cases use both. A support bot that can look up the user's actual subscription and read the relevant docs page beats one that does either alone — and the SQL half is the half that never hallucinates.

Evaluation: golden questions or you're guessing

Every retrieval change you make — new chunker, re-ranker, query rewriting — either helps or hurts, and without evals you find out from users. The core asset is a golden set: 50–200 real questions (mine your support tickets and search logs, don't invent them) with expected facts and expected sources:

{
  "id": "billing-refund-window-annual",
  "question": "How long do I have to get a refund on an annual plan?",
  "expected_facts": [
    "refund window is 30 days from purchase",
    "applies to annual plans",
    "prorated credit after the window"
  ],
  "forbidden": ["14 days", "no refunds"],
  "gold_sources": ["docs/billing/refunds.md#annual-plans"],
  "persona": { "tenant": "acme", "groups": ["customer"] }
}

Measure two layers separately, because they fail separately:

• Retrieval metrics — did the gold sources appear in the retrieved set (recall@k), and how high (MRR)? These are cheap, deterministic, and run on every change. When an answer is wrong, they tell you whether retrieval or generation is to blame — the first question you'll ask in every debugging session.
• Answer quality — does the generated answer contain the expected facts, avoid the forbidden ones, and cite the right sources? Use an LLM as judge with a tight rubric, and spot-check the judge against human labels before you trust it.

Good retrieval metrics with bad answers means a generation/assembly problem. Bad retrieval metrics means nothing downstream matters yet. Then wire it into CI: the golden set runs on every prompt change, chunker change, and model upgrade, exactly like a test suite, with thresholds that fail the build. It's also your only honest way to evaluate vendor and model swaps — "the new embedding model feels better" is not a finding.

Failure modes to monitor in production

• Hallucinated citations. The model cites refunds.md for a claim that isn't in it — the most trust-destroying failure, because citations are your credibility signal. Mitigate by making citation IDs structured (the model must reference chunk IDs you provided, not invent URLs), and verify server-side that every cited ID was actually in the context. It's a set-membership check; do it.
• Stale indexes. The docs changed, the index didn't, and the bot confidently describes last quarter's pricing. Index freshness is a data-pipeline SLO: re-index on publish (webhook, not nightly cron, if docs change often), and alert on index-age drift. A "when was this last synced?" admin view pays for itself the first week.
• Embedding drift. Query vectors and document vectors must come from the same model version. Upgrade the embedding model without re-indexing and similarity scores become quietly meaningless — retrieval degrades with no errors anywhere. Version-stamp every vector with its model ID, and treat an embedding upgrade as a full re-index plus a golden-set run, never a config flip.

A pragmatic architecture for a docs assistant or support bot

What we'd actually build today for a typical client — boring on purpose:

1. Ingestion: pull from the source of truth (Markdown repo, CMS, help center) on publish events. Structure-aware chunking with heading-path headers; metadata (tenant_id, ACL groups, product version, doc type, updated_at, embedding model version) on every chunk.
2. Storage: Postgres with pgvector and tsvector. One database for chunks, metadata, and your app. Add a dedicated vector store only when scale forces the question — for most products it never does.
3. Query path: LLM rewrite of the conversational query → hybrid search (vectors + FTS, RRF) with permission and metadata filters in the SQL → cross-encoder re-rank of ~50 candidates down to ~10.
4. Routing: account/entity questions to parameterized SQL tools; doc questions to retrieval; an agentic search loop as the escalation path when the fast path scores low confidence.
5. Assembly: stable cached system prompt first, then retrieved chunks with IDs and source paths, then history, then the question. Instructions: answer only from provided context, cite chunk IDs, say "I don't know" when the context doesn't cover it.
6. Verification: server-side citation check before the answer ships.
7. Evals: golden set in CI; production sampling of live Q&A pairs into a review queue that feeds the golden set.

Nothing exotic. Every piece is replaceable. Every piece is measurable. That's the point.

Takeaways

• Naive RAG fails structurally: chunks lose context, similarity isn't relevance, and top-k can't answer synthesis questions. Tuning the embedding model fixes none of these.
• Think "context engineering" — the whole system that decides what the model sees — not "retrieval tuning."
• Chunk along document structure and prepend heading-path headers. Cheapest big win in the stack.
• Hybrid search (BM25/FTS + vectors, fused with RRF) plus a cross-encoder re-ranker should be your retrieval baseline, not your stretch goal.
• Rewrite and decompose user messages before retrieving; keep HyDE in your back pocket.
• Enforce permissions inside the retrieval query, with the user's identity. Your vector index is part of your authorization surface.
• With prompt caching, "put the whole corpus in the prompt" is a legitimate architecture for small, stable, single-tenant corpora. Know when you're in that regime.
• Route: SQL for facts, retrieval for prose, agentic search for hard questions. Don't embed what you can query.
• Build a golden set from real questions, measure retrieval and answer quality separately, and run it in CI. Without evals, every improvement is a guess.
• Monitor the quiet failures: verify citations server-side, alert on index staleness, and treat embedding upgrades as full re-index events.