Keyword vs Semantic Search

Semantic search finds matches in meaning. Keyword search finds matches in exact terms. They fail in different places, which is why real systems often run both and then add a second pass to sort the results.

Semantic search#

Semantic search is the similarity search over embeddings you have already seen. It handles paraphrases and synonyms well. "Fast car" matches "quick vehicle" even though no words overlap.

It fails on exact terms: error codes, product IDs, names, jargon and rare words that did not appear often in the embedding model's training data. The vector for "E1047" may not land near other content about that specific error, because the model has no strong signal about what E1047 means.

Keyword search with BM25#

Keyword search, usually BM25, matches on exact terms. It scores each document with three ideas:

• Term frequency: documents that contain the query terms more often score higher.
• Rarity: rare terms count for more than common ones, so "E1047" outweighs "error".
• Length normalization: long documents are scaled down, so they do not win on length alone.

BM25 is strong exactly where semantic search is weak: identifiers, names and technical jargon. It is weak where semantic search is strong, because "quick vehicle" contains neither "fast" nor "car".

Hybrid search and RRF#

Hybrid search runs both searches and combines the results. You cannot just add the scores, because they are on different scales. A BM25 score of 12 and a cosine similarity of 0.87 mean nothing relative to each other.

The standard solution is reciprocal rank fusion (RRF). It ignores the scores and combines the ranks instead. In each result list, a document gets $\frac{1}{k + \text{rank}}$, where $k$ is usually 60. Then you add up its scores from both lists.

A document ranked high in both lists scores much higher than one ranked first in only one. A document that appears in only one list still counts, just lower.

def rrf(result_lists: list[list[str]], k: int = 60) -> list[str]:
    scores: dict[str, float] = {}
    for results in result_lists:
        for rank, doc_id in enumerate(results, start=1):
            # Ranks, not raw scores, so BM25 and cosine can be mixed.
            scores[doc_id] = scores.get(doc_id, 0.0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

print(rrf([["A", "B", "C"], ["A", "C", "D"]]))  # ['A', 'C', 'B', 'D']

Reranking#

After hybrid retrieval you might have 50 to 100 candidate chunks, far too many for the prompt. Reranking is a second pass that scores those candidates more precisely.

Bi-encoders#

Your embedding model is a bi-encoder. It encodes the query and each document separately into vectors. Because document vectors are computed ahead of time and stored, search is fast. But the model never looks at the query and a document together, which limits how precise it can be.

Cross-encoders#

A cross-encoder reads the query and one document together in a single pass and outputs one relevance score. Because it sees both at once, it is far more accurate. It is also far too slow to run on every document, since nothing can be computed ahead of time.

The standard pipeline#

Use each where it is strong: the bi-encoder to narrow down fast, the cross-encoder to sort a short list well.

1. Run BM25 and vector search on the query.
2. Fuse the two lists with RRF into 50 to 100 candidates.
3. Rerank the candidates with a cross-encoder.
4. Keep the top 5 or 10 for the prompt.

Short line#

Semantic search matches meaning but misses exact terms like error codes, and BM25 keyword search is the opposite. Hybrid search runs both and fuses the ranks with RRF, then a cross-encoder reranks the top candidates before they go into the prompt.

Key takeaways#

• Semantic search handles paraphrases but misses exact terms, IDs, names and rare jargon.
• BM25 scores exact terms by frequency, weights rare terms higher and normalizes for document length.
• Scores from the two searches are on different scales, so RRF combines ranks: $\frac{1}{k + \text{rank}}$ per list, $k$ usually 60, summed.
• A bi-encoder embeds query and documents separately: fast, precomputable, less precise.
• A cross-encoder reads query and document together: precise, but too slow for everything.
• Retrieve 50 to 100 candidates fast, rerank with a cross-encoder, keep the top 5 to 10.