Semantic search fails in predictable ways. Knowing them tells you what to test before users find them, and which fix to reach for. Four failure modes come up again and again, and each has a different cause.
Exact terms#
This is the biggest one. Error codes, product IDs, version numbers, proper names and rare jargon all embed poorly. The embedding model has no strong signal about what "E1047" means, so its vector does not reliably land near other content about that error.
The fix is hybrid search: add BM25 keyword matching and fuse the results with RRF, as described in keyword vs semantic search.
Negation#
"Not refundable" may embed close to "refundable", because almost all the surrounding words are the same. The model captures the neighborhood of refund-related content, not the logical flip that "not" causes.
This is a real gap with no clean fix. Find it through careful evaluation: write test queries that use negation and check what comes back.
Short queries, long documents#
A two-word query may not land close to a long paragraph that answers it. The two vectors represent very different amounts of content, so their directions can differ even when the paragraph is the right answer.
Small-to-big retrieval helps. Index small chunks so matching is precise, but when a chunk matches, give the model the larger parent section so it has enough context.
Stale index#
This failure is operational, not a modeling problem. If source documents change or get deleted but the index is not updated, the system retrieves outdated or removed content.
Two practices prevent it, and both are requirements, not extras:
- Re-index a document incrementally whenever it changes.
- Delete a document's chunks when the source is deleted.
Failure modes at a glance#
| Failure | Cause | Fix |
|---|---|---|
| Exact terms | No strong signal for rare tokens like IDs | Hybrid search with BM25 |
| Negation | "Not X" embeds near "X" | Evaluation and testing only |
| Short query, long document | Vectors cover very different amounts of text | Small-to-big retrieval |
| Stale index | Index not updated when sources change | Incremental re-index, delete on source delete |
The trade-off#
Semantic search captures meaning but misses exact matches, handles negation poorly and gets worse when the embedding model has not seen your domain's vocabulary. Hybrid search with reranking covers most of the gaps, but it adds latency, cost and infrastructure. You now build, monitor and tune four parts: BM25 search, vector search, RRF fusion and cross-encoder reranking.
So add complexity only when you see the need:
- Start with semantic search alone for early exploration.
- Move to hybrid search when users report missing exact matches.
- Add a reranker when precision matters more than speed.
Retrieval quality is the biggest lever in a RAG system. A better reranker or smarter chunking improves answers more than a bigger language model, because the model can only work with what retrieval gives it.
This ties back to reducing hallucinations: grounding only helps when the right documents reach the prompt.
Short line#
Semantic search misses exact terms, handles negation poorly, struggles when a short query meets a long document, and serves stale results if the index lags its sources. Hybrid search and reranking fix most of this at the cost of more parts, and retrieval quality matters more than model size.
Key takeaways#
- Exact terms such as error codes, IDs and names are the biggest gap. Hybrid search with BM25 fixes it.
- Negation embeds close to its opposite and has no clean fix beyond evaluation.
- Small-to-big retrieval matches on small chunks and returns the larger parent section.
- Keep the index in sync: re-index on change, delete chunks when their source is deleted.
- Start with semantic search, add hybrid for exact matches, add a reranker when precision matters.
- Better retrieval improves answers more than a bigger model.