88Z AI, 16 November — When RAG apps have the right data but give wrong answers, the issue often lies in retrieval. OpenRouter tests show vector search alone achieves only 81% accuracy, while adding a reranker boosts it to over 97%. This briefing details how to optimize retrieval pipelines using cross-encoder architectures, compares the performance and cost of Voyage, Cohere, and LLM-based rerankers, and provides implementation strategies with benchmark data.
At a glance
- Vector search often confuses details of different plans due to semantic similarity, leading to retrieval errors
- Adding reranking can boost retrieval accuracy from 81% to over 97%
- Dedicated reranking models outperform LLM rerankers in speed (~1.3s) and cost
- Properly setting embedding input types prevents significant precision loss

Why Vector Search Retrieves Wrong Chunks
In Retrieval-Augmented Generation (RAG) applications, systems may return incorrect information even when the correct answer exists in the documents, often due to retrieval limitations. Standard vector search excels at finding topically relevant chunks but is prone to "near misses." For instance, when a user asks about limits for the "Growth" plan, vector search might incorrectly return content for the "Enterprise" plan. This occurs because embedding models compress entire text blocks into single vectors without seeing the user's question. In a 130-word chunk explaining bucket limits, the specific plan name constitutes only a small fraction, causing vectors for different plans (Starter, Growth, Scale, Enterprise) to cluster closely in vector space. Consequently, the retriever struggles to distinguish subtle differences and may rank the wrong plan first.
How Reranking Works and Implementation
Reranking addresses these issues by introducing a cross-encoder architecture. Instead of evaluating documents in isolation, the cross-encoder processes the user's question alongside each candidate chunk, rescoring them directly against the query. Since running a cross-encoder across thousands of documents is too slow for real-time queries, the best practice is to use it as a second stage: first, use vector search to quickly narrow down the field by retrieving the top 30 candidates; then, pass these candidates through a reranker model for fine-grained sorting; finally, send the top 3 highest-scoring chunks to the Large Language Model (LLM). This two-stage approach retains the speed of vector search while improving the relevance of content fed to the LLM. Implementation does not require replacing existing embedding search; simply increase the initial retrieval count and add a reranking call before passing results to the LLM.
Model Performance Comparison and Cost-Benefit Analysis
OpenRouter tested 11 rerankers on a synthetic dataset of 274 questions. Results showed that using google/gemini-embedding-2 for vector search alone yielded an accuracy of 81.0%. Adding voyageai/rerank-3-lite boosted accuracy to 97.1% at a cost of approximately $0.10 per 1,000 queries. While using an LLM (such as openai/gpt-6-luna) as a reranker increased accuracy to 99.6%, its latency was significantly higher at 4.0 seconds, compared to around 1.3 seconds for dedicated reranking models. Regarding cost, although reranking added about $0.10 per query, it improved retrieval precision, allowing the number of context chunks sent to the LLM to be reduced from five to three, thereby lowering LLM processing costs. Overall, in this test setup, the net cost increase was only $0.04 per 1,000 queries, while significantly reducing "I don't know" responses caused by missing key information.
88Z Perspective
Reranking is not a silver bullet but a high-return, low-cost supplement to existing RAG pipelines. For developers, blindly pursuing the highest accuracy with LLM-based reranking may compromise user experience due to excessive latency; dedicated models like Voyage offer a better balance between speed and precision. Notably, the source mentions that failing to set input_type in embedding calls can drop precision by 7 to 12 percentage points, suggesting that basic configuration checks should precede algorithmic optimizations. Additionally, some models (e.g., cohere/rerank-v3.5) have token reading limits (first 4,096 tokens), posing truncation risks for long documents. We recommend teams establish their own benchmark datasets, as real-world data distributions may differ from synthetic ones. Only through actual measurement can you determine the reranking strategy best suited for your specific business scenario.
Sources & editorial note
This article is an AI-assisted synthesis by 88Z AI based on the public sources below. Product claims and availability are attributed to their publishers. It does not represent independent testing or first-hand reporting.
- OpenRouter BlogSource published
Source dates above refer to the original announcements; the date under the headline is this article’s publication on 88Z AI.