RAG Beyond the Demo: Shipping Retrieval Over Messy Internal Documents

Chunking, hybrid search, and why you build the golden-question set before you touch a vector store

By Muhammad Zia | Full Stack AI Engineer

---

The demo that lies to you

Every RAG demo looks the same: drop a clean PDF into a notebook, embed the chunks, ask a question the author just wrote, get a perfect citation. Then you point the same pipeline at a real corpus (mixed Word docs, scanned datasheets, HTML exports, product codes that look like noise) and accuracy falls off a cliff.

This post is about the parts that matter once you leave the demo: chunking for structured vs prose documents, why cosine similarity alone fails on product-code queries, hybrid search, evaluation discipline, and latency budgets when the client is a browser extension.

I built ragbench-lite so the evaluation piece is a CLI you can run in CI instead of a vibes check before launch.

---

Know your documents

Internal corpora are not one distribution. Treat them as families:

| Family | Example | Chunking instinct | |---|---|---| | Prose | policies, wiki pages | Semantic / recursive text splitters, ~400-800 tokens, overlap | | Structured | datasheets, spec tables | Preserve row/section boundaries; attach header context to every chunk | | Semi-structured | tickets, emails | Keep thread metadata; do not merge unrelated messages | | Code-like IDs | SKUs, part numbers | Need keyword/BM25 path; embeddings alone will miss them |

At Novacom, the Chrome extension had to answer questions over product documentation that mixed all of the above. Spec sheets were the failure mode. A table cell saying 85°C under "Max operating temperature" would get sliced mid-row by a naive character splitter, and the retrieved chunk would lose the field name.

---

Chunking that respects structure

For datasheets we stopped using a single splitter:

function chunkDatasheet(doc: ParsedDoc): Chunk[] {
  const chunks: Chunk[] = [];
  for (const section of doc.sections) {
    const header = section.title; // e.g. "Thermal characteristics"
    for (const row of section.rows) {
      chunks.push({
        source: `${doc.path}#${section.anchor}`,
        text: `${header}\n${row.label}: ${row.value}`,
        metadata: { productCode: doc.productCode, section: header },
      });
    }
  }
  return chunks;
}

Ugly? Yes. Accurate? Much more than 512-token sliding windows. Prose pages kept a recursive character splitter. One pipeline, two strategies, routed by document type rather than hope.

---

Why naive cosine similarity fails on product codes

Ask: "What is the max operating temperature of product X-200?"

The embedding for X-200 is rarely distinctive. Similar product codes cluster. The model retrieves the wrong family's datasheet with a confident score. Classic dense-only failure.

Fix: hybrid search. Combine BM25 (or another lexical scorer) with vector search, then fuse with reciprocal rank fusion (RRF) or a learned ranker if you have labels.

flowchart LR
  Q[Query] --> Dense[Vector search]
  Q --> Lex[BM25 / keyword]
  Dense --> Fuse[RRF fusion]
  Lex --> Fuse
  Fuse --> TopK[Top-k chunks]
  TopK --> Gen[Answer generation]

Lexical search nails exact product codes and part numbers. Dense search handles paraphrases ("highest safe temp" vs "max operating temperature"). Together they cover the queries users actually type.

---

Evaluation before infrastructure

Do not buy a vector database until you have a golden-question set.

Build 30-100 questions with:

• the question
• expected source paths (retrieval ground truth)
• optional expected answer substrings
• a flag for expensive LLM-as-judge checks

Example (questions.yaml):

- id: q1
  question: "What is the max operating temperature of product X-200?"
  expected_sources: ["datasheets/x200.pdf#thermal"]
  expected_answer_contains: ["85°C"]
  judge: true

Run the same set against every pipeline change. Track:

• Hit-rate@k: expected source appears in top-k
• MRR: how high the first relevant hit sits
• Answer-contains: cheap string assertions
• Faithfulness (LLM judge): does the answer stick to retrieved context?
• Latency p50/p95: especially brutal in a browser extension

ragbench-lite turns that into a CI gate: fail the build when hit-rate drops. That single habit stops the "we swapped embedders and nothing looked worse in the chat demo" regression.

---

Latency in a browser-extension context

Extensions have a harder budget than a backend chat app:

• Cold start of the side panel
• Network to your retrieval API
• Generation stream back to the UI

Budget we targeted: p95 under 2.5s to first useful token for cached-index queries on a warm session. That forced:

• Precompute embeddings offline; never embed the whole corpus in the extension
• Keep the extension as a thin client over an edge/API retrieval service
• Stream tokens early; show citations as a second paint
• Cache frequent product-code lookups aggressively

If your architecture requires the extension to talk to three regions and an LLM before anything appears, users will close the panel. Retrieval quality that arrives too late is still a product failure.

---

Trade-offs / what I'd do differently

• I'd build the golden set in week one, even with 20 questions. Infrastructure without eval is decoration.
• I'd invest earlier in document-type routing. One splitter for everything is how datasheets die.
• I'd treat product codes as first-class query features, not hope the embedder notices them.
• I'd keep the judge optional and versioned. LLM judges drift; pin the prompt and score distribution.

---

Closing

RAG in production is information retrieval with a language model bolted on, not the other way around. Chunk for structure, search hybrid, evaluate with golden questions, and respect the latency envelope of the client you actually ship.

If you only take one thing: write the questions before you write the index.