Stop Using Top-K Retrieval. Try This Instead.

Everyone talks about RAG like the hard part is the generation. It’s not. The hard part is getting the right chunks in front of the model in the first place. I’ve written before about why RAG retrieval is really a filtering problem, not a search problem, and this experiment confirmed it.

I learned this the hard way. After three weeks of testing five different retrieval strategies on 12,000 chunks of production data, I found out that the default approach — naive top-k similarity search — was giving engineers useless answers 40% of the time. They stopped trusting the bot.

Two strategies improved answer quality measurably. Three made things worse while burning more tokens.

Here’s what I tested, what worked, and why the obvious choice failed.

The Setup

Knowledge base: 12,000 chunks (512 tokens each) – Internal engineering docs (40%) – Slack incident threads (35%) – Post-mortems and runbooks (25%)

Embedding model: nomic-embed-text-v1.5 (768 dimensions) Vector DB: Qdrant with HNSW index LLM: Claude Sonnet 5 via Ollama Cloud Evaluation: 50 real questions from engineers, graded 1-5 on relevance and actionability

Baseline: Naive top-k similarity search (k=5) – Average score: 3.2/5 – Token usage per query: ~2,800 tokens (context + response) – Latency: 340ms (p50)

The baseline was… fine. But “fine” meant engineers got useless answers 40% of the time and stopped trusting the bot. I needed to do better.

Strategy 1: Hybrid Search (BM25 + Dense)

Hypothesis: Dense embeddings miss exact keyword matches. BM25 catches them. Combine both.

Implementation: – Dense search: top 10 chunks by cosine similarity – BM25 search: top 10 chunks by keyword match (Elasticsearch) – Reciprocal Rank Fusion to merge results – Final k=5 after re-ranking

Result: Average score 3.8/5 (+19%) – Token usage: ~3,200 tokens (+14%) – Latency: 520ms (+53%)

Verdict: Worth it.

The hybrid approach caught questions that pure dense search missed — especially when engineers used exact error codes, function names, or config keys. Example: “error code 0x80070005” returned the exact Windows permission troubleshooting doc, not just semantically similar “permission denied” threads.

The latency hit was real but acceptable for the quality gain. We cached frequent queries, which brought p50 back down to 380ms.

Strategy 2: Parent-Child Retrieval

Hypothesis: Small chunks retrieve well but lack context. Large chunks have context but retrieve poorly. Use small chunks for retrieval, large chunks for generation.

Implementation: – Child chunks: 512 tokens (for embedding + retrieval) – Parent chunks: 2,048 tokens (the full section the child came from) – Retrieve top 10 child chunks by similarity – Return their parent chunks to the LLM – Final context: 3-5 parent chunks (~6,000-10,000 tokens)

Result: Average score 4.1/5 (+28%) – Token usage: ~7,500 tokens (+168%) – Latency: 410ms (+21%)

Verdict: Best quality, but expensive.

Parent-child retrieval gave the LLM enough context to actually answer questions instead of just pattern-matching on keywords. Engineers got full procedures, not just fragmented snippets.

The token cost was brutal though. At our query volume (~800 queries/day), this would have tripled our Ollama Cloud bill. I’ve written about how I cut AI agent costs by 60% with quantized models — and this is exactly the kind of cost analysis that matters.

Strategy 3: Query Expansion with LLM

Hypothesis: Engineers ask vague questions. Expand the query with an LLM before retrieval to catch more relevant chunks.

Implementation: – User asks: “deployment failed” – LLM expands to: “deployment failed production kubernetes rollout error troubleshooting rollback” – Use expanded query for dense retrieval – Top k=5 chunks

Result: Average score 2.9/5 (-9%) – Token usage: ~3,400 tokens (+21%) – Latency: 890ms (+162%)

Verdict: Don’t do this.

The LLM kept expanding queries in directions that felt relevant but actually retrieved worse chunks. “Deployment failed” became a generic list of deployment best practices, not the specific error the engineer was hitting.

The latency hit was the real killer. Adding an LLM call before retrieval made every query feel sluggish. Engineers noticed and stopped using the bot.

Strategy 4: Metadata Filtering

Hypothesis: Not all chunks are equal. Filter by recency, doc type, and confidence score before retrieval.

Implementation: – Pre-filter: chunks from last 18 months only – Pre-filter: exclude Slack threads with <3 reactions (low signal) - Pre-filter: exclude chunks with embedding confidence <0.7 - Then: naive top-k similarity search (k=5)

Result: Average score 3.6/5 (+13%) – Token usage: ~2,600 tokens (-7%) – Latency: 310ms (-9%)

Verdict: Free upgrade.

This was the easiest win. Just filtering out old docs and low-quality Slack threads improved results without any architectural changes. The token usage actually went down because we were retrieving more relevant chunks on the first try.

One catch: we had to maintain the metadata carefully. When docs got updated, the old chunks needed their “last modified” timestamp updated too. Otherwise they’d get filtered out incorrectly.

Strategy 5: Multi-Hop Retrieval

Hypothesis: Some questions need multiple retrieval steps. First find the concept, then find the procedure.

Implementation: – Step 1: Retrieve chunks for the core concept (k=3) – Step 2: Extract key terms from step 1 results – Step 3: Retrieve chunks for those terms (k=5) – Step 4: Deduplicate and return top 5 unique chunks

Result: Average score 3.1/5 (-3%) – Token usage: ~4,200 tokens (+50%) – Latency: 720ms (+112%)

Verdict: Overengineered.

Multi-hop retrieval sounded smart in theory. In practice, it compounded errors. If step 1 retrieved the wrong concept, step 2 went further off-track.

The only time this worked was for very specific technical questions like “how does the rate limiter interact with the circuit breaker” — where you genuinely need to retrieve two separate concepts and combine them. But that was maybe 5% of our queries. Not worth the complexity for the other 95%.

What We Actually Shipped

After three weeks of testing, here’s what made it to production:

Default queries (90% of traffic): – Metadata filtering + hybrid search – k=5 chunks – Token usage: ~3,200 tokens – Latency: 520ms (cached: 380ms)

High-stakes queries (10% of traffic): – Metadata filtering + parent-child retrieval – k=3 parent chunks – Token usage: ~7,500 tokens – Latency: 410ms

We route queries to parent-child retrieval based on intent classification. If the question contains “how do I”, “procedure”, “steps”, or “runbook”, we use parent-child. Everything else gets hybrid search.

The Real Lesson

The retrieval strategy matters more than the embedding model. I see teams obsessing over switching from nomic-embed-text to text-embedding-3-large or m3e-base, but they’re still using naive top-k search on unfiltered chunks.

That’s like upgrading your car engine while keeping square wheels.

Start with metadata filtering. It’s free and easy. Then add hybrid search if you have exact keyword queries. Only consider parent-child retrieval if you can afford the token cost — or if you’re answering questions where wrong answers have real consequences (incident response, medical, legal).

Skip query expansion and multi-hop retrieval unless you have a very specific use case that demands them. They add latency and complexity without moving the needle on quality.

The Numbers

Here’s the full comparison:

Strategy	Avg Score	Token Usage	Latency (p50)	Shipped?
Baseline (top-k)	3.2/5	2,800	340ms	No
Hybrid search	3.8/5	3,200	520ms	Yes (default)
Parent-child	4.1/5	7,500	410ms	Yes (high-stakes)
Query expansion	2.9/5	3,400	890ms	No
Metadata filtering	3.6/5	2,600	310ms	Yes (all queries)
Multi-hop	3.1/5	4,200	720ms	No

The two winners — hybrid search and parent-child retrieval — both have one thing in common: they retrieve chunks differently than they rank them. Hybrid uses two different algorithms. Parent-child uses two different chunk sizes.

The losers all tried to do everything in one pass. One embedding, one search, one result. That’s simpler, but simplicity doesn’t help if the answers are wrong.

What I'd Test Next

If I had another three weeks, I’d test:

1. Query routing by intent. Instead of one retrieval strategy for all queries, classify the query first (how-to vs. troubleshooting vs. conceptual) and route to different retrieval pipelines. We’re already doing a primitive version of this with the parent-child routing, but I’d make it more granular.

2. Learned re-ranking. Train a small model to re-rank retrieved chunks based on past click-through or thumbs-up data. The retrieval gets you 10 candidates; the re-ranker picks the best 5. This is what the big RAG platforms do, but I wanted to see if we could get 80% of the way there with simpler techniques.

3. Chunk size tuning. We used 512 tokens because that’s what everyone uses. But maybe 256 or 1,024 works better for our specific docs. This is a cheap experiment — just re-embed and re-test.

But for now, the two-solution setup is working. Engineers are getting better answers, the bot is cheaper to run than the parent-child-only approach, and I’m done tweaking retrieval for a while.

Sometimes the answer isn’t one perfect strategy. It’s knowing which of two imperfect strategies to use when.