Stop Using Top-K Retrieval. Try This Instead.
Everyone talks about RAG like the hard part is the generation. It’s not. The hard part is getting the right chunks in front of the model in the first place. I’ve written before about why RAG retrieval is really a filtering problem, not a search problem, and this experiment confirmed it.
I learned this the hard way. After three weeks of testing five different retrieval strategies on 12,000 chunks of production data, I found out that the default approach — naive top-k similarity search — was giving engineers useless answers 40% of the time. They stopped trusting the bot.