Ask which relation the query names

Consider two requests for a machine part:

1. XR-417-B;
2. “the part that stops the rattling near the fan.”

The first asks for symbol identity. Normalized exact matching, prefix lookup, or a rare-token inverted index can answer deterministically. A learned semantic representation may blur the identifier with visually or linguistically nearby strings and add no useful evidence.

The second asks for paraphrase and function. Its useful document may contain “vibration-damping bracket” without sharing a query term. Lexical evidence can miss it; a representation trained to place related descriptions nearby may recover it.

Choose the cheapest relation that is exact enough:

BEIR measured a robust BM25 baseline across its heterogeneous benchmark. That is source evidence for those datasets, not proof that lexical retrieval wins a new task. It is enough to reject the caricature that a deterministic baseline is obsolete before local evaluation begins.

A vector index searches a chosen geometry

A feature extractor maps each item $x$ to a vector

$$f(x)\in\mathbb{R}^d.$$

A distance or similarity then defines neighbors in that geometry. The index does not discover meaning independently; it accelerates neighbor search for the representation it was given.

Three errors must remain separable:

• representation error: relevant items are far apart or irrelevant items are close under $f$;
• neighbor-search error: the approximate index misses a true neighbor under the chosen geometry; and
• decision error: a retrieved neighbor is admitted, fused, or grounded under the wrong policy.

Changing index parameters cannot repair a representation that puts the wrong items together. Retraining or replacing the representation cannot prove an approximate index reached the nearest vectors. Evaluation needs an exhaustive vector-search oracle on a bounded corpus plus task relevance judgments.

Representation identity includes model, preprocessing, tokenizer, numeric precision, dimension, normalization, and corpus revision. Mixing vectors from two identities creates a geometry neither model defined.

Approximate neighbors spend a recall budget

Exact nearest-neighbor search compares the query with every vector. At scale, graph indexes such as HNSW build layers of proximity links. Search begins in a sparse upper layer and descends through denser neighborhoods, exploring a bounded candidate frontier.

The important controls are not magic constants:

• graph degree changes memory, construction, and navigability;
• construction effort changes index quality and build cost;
• query exploration width changes latency and neighbor recall; and
• filters, deletions, and corpus updates alter reachability and maintenance.

Every approximate result receipt carries these identities and the exploration budget. “Nearest” without the metric, representation, and recall regime is an overclaim.

The evaluator sweeps recall@k against latency, memory, build time, and update behavior. It also separates vector-neighbor recall from task relevance: finding the exact nearest vectors perfectly does not prove those vectors answer the query.

The middle is a spectrum, not a binary

“Lexical or vector” hides several useful representations and interaction points. Put them on two axes: what each document stores, and when the query may interact with it.

Top-k of top-k is exact in one homogeneous case

Let every shard use the same scorer $s(x)$, query interpretation, corpus revision, stable identity relation, and total tie-break order. Define

$$A\otimes B=TopK_k(A\cup B),$$

after deduplicating equal identities.

Then $\otimes$ is associative, commutative, and idempotent:

$$(A\otimes B)\otimes C=A\otimes(B\otimes C),$$

$$A\otimes B=B\otimes A,$$

$$A\otimes A=A.$$

More importantly,

$$TopK_k\left(\bigcup_i S_i\right)= TopK_k\left(\bigcup_i TopK_k(S_i)\right).$$

Proof: if candidate $x$ is omitted from shard $i$'s local top-k, at least $k$ candidates in that shard precede $x$ under the same total order. Those same candidates precede $x$ globally, so $x$ cannot enter global top-k. Thus local omission removes no possible global winner.

The proof fails when shards disagree on scorer, tie-break, query analysis, index revision, or identity. Overlapping shards must deduplicate before the idempotence claim. Heterogeneous providers do not inherit this law merely because each returns a list named “top-k.”

MERGE-TOP-K(LISTS, LIMIT, CONTRACT)
Input:  bounded LISTS sharing one scored total-order CONTRACT
Output: deterministic global top LIMIT

heap ← EMPTY-MIN-HEAP(LIMIT)
seen ← EMPTY-IDENTITY-SET
for each list in STABLE-SHARD-ORDER(LISTS)
    REQUIRE-SAME-CONTRACT(list, CONTRACT)
    for each candidate in TAKE(list, LIMIT)
        if not CONTAINS(seen, candidate.identity)
            seen ← ADD(seen, candidate.identity)
            heap ← KEEP-BEST(heap, candidate, LIMIT)
return SORT-BY-CONTRACT(heap, CONTRACT)

The stable shard order does not decide relevance; it only makes diagnostics and resource use reproducible. The total score/tie contract decides the result.

Raw heterogeneous scores have no common arithmetic

Suppose lexical scores range from 0 to 25 while vector cosine similarities lie between -1 and 1. Adding them gives the lexical scale more influence, not necessarily more evidence. Min-max normalization depends on the observed candidate set; z-scores depend on distribution shape; both can move when an unrelated candidate arrives.

Calibrated relevance probabilities may be combined under a justified model, but calibration itself is task-, corpus-, and policy-specific. Without it, raw score fusion is unit arithmetic without shared units.

Reciprocal rank fusion discards score scales. For document $d$ across ranked lists $R$, define

$$RRF(d)=\sum_{r\in R}\frac{1}{c+rank_r(d)},$$

omitting a term when $d$ is absent. The constant $c>0$ controls how sharply the contribution falls with rank.

RRF is invariant under any strictly increasing transformation of a source's raw scores because ranks do not change. It rewards agreement across systems and high placement without asking score 8.2 to mean the same thing as cosine 0.74.

The constant is not irrelevant. Large $c$ flattens the difference between nearby ranks; small $c$ emphasizes the head. Cormack, Clarke, and Büttcher reported results for their experimental settings. A new system pins $c$ and tests it across held-out tasks rather than calling the source's value universal.

Segmentation is not fusion

A search surface may show pages, passages, commands, and settings in separate groups. Each group has a correct local order, but the layout order between groups is not a relevance comparison.

This can be honest. Label groups and state that group placement is a product decision. The failure is to imply one global “best result” while concatenating heterogeneous lists with no cross-group algebra.

Segmentation is fusion by fiat when the surface inherits a global guarantee from a ranker that never compared the groups. Every local test can pass while the composed surface has no stated property.

Choose one:

• preserve groups and explicitly deny cross-group relevance order;
• fuse shared identities with a declared heterogeneous method such as RRF;
• route the query to one eligible provider before ranking; or
• define a calibrated common objective and prove its bounds.

Do not use color, section order, or independent top-k caps as an unstated aggregation function.

Tail latency compounds in two different ways

Let one retrieval call have latency distribution $F$.

For $n$ sequential calls with finite mean,

$$\mathbb{E}[T_{serial}]=n\mathbb{E}[T].$$

For $n$ independent parallel calls, completion waits for the maximum:

$$P(T_{max}\leq t)=F(t)^n.$$

The median parallel completion time is therefore

$$F^{-1}(0.5^{1/n}).$$

At $n=69$, $0.5^{1/69}$ is approximately $0.99$, so the median of the maximum lies near one call's p99 under the independence model. Correlation can make the tail worse or different; the equation is a diagnostic model, not a production promise.

An agentic loop can pay both costs: several providers fan out within a turn, then several turns occur serially. Optimize tail settlement, bounded fan-out, cache identity, and the number of retrieval calls—not only single-call p50. Dean and Barroso's tail-at-scale analysis motivates this question in large services; local traces must establish the actual distribution and dependence.

Sometimes the book is the wrong implementation plan

For a few thousand static documents, a database full-text index, an embedded library, or even a bounded scan may meet every relevance and latency need. For an exact codebase lookup, a line-oriented search tool may be clearer. For a large organization with operational staff and a suitable privacy contract, a hosted service may be cheaper than owning indexing and evaluation.

Use the cost model:

• corpus and update size;
• query volume and latency envelope;
• required relations and languages;
• privacy and authorization boundaries;
• evaluation and operations capacity;
• offline and deterministic requirements; and
• total maintenance cost.

Building an inverted index, vector pipeline, approximate graph, fusion layer, and evaluator for a tiny stable corpus is not mastery of the book. Recognizing that the simple tool already satisfies the contract is.

Negative results

Lessons

• Choose retrieval from the relation named by the query.
• Exact identifiers and rare tokens deserve a deterministic lexical floor.
• Vector retrieval searches one versioned representation geometry.
• Learned sparse retrieval trades vocabulary expansion against inverted-index sparsity and work.
• Late interaction retains token vectors; PLAID accelerates ColBERTv2's retrieval rather than defining a separate relevance model.
• A reranker cannot recover a relevant item omitted by candidate generation.
• QuickScorer evaluates tree ensembles; it does not choose their learning objective or evidence.
• GPT reranking is another bounded, versioned ranker with context, order, latency, and reproducibility costs.
• Separate representation, approximate-neighbor, and decision errors.
• Homogeneous top-k is associative, commutative, and idempotent under one scorer, total order, identity relation, query, and revision.
• That law exactly licenses top-k of top-k and exactly excludes heterogeneous providers.
• RRF combines heterogeneous rankings without assuming score-scale agreement.
• RRF's constant changes head-versus-tail weighting and must be evaluated.
• Grouped results are segmented; cross-group order is a layout decision unless a fusion algebra says otherwise.
• Sequential calls add means; parallel fan-out exposes high quantiles through the maximum.
• A database, scan, local tool, or hosted service may be the correct endpoint.

Practice

1. Classify ten queries by the exact, lexical-ranked, or semantic relation they request.
2. Give one representation error and one approximate-neighbor error that produce the same missing result.
3. Prove the top-k-of-top-k law and construct a counterexample with two incompatible scorers.
4. Show that RRF is invariant under a strictly increasing transformation of one source's scores.
5. Calculate RRF for three documents at two values of $c$ and explain the changed head weighting.
6. Specify the honest contract for a grouped pages/commands/settings surface.
7. Derive the median maximum quantile for 10 and 100 independent providers.
8. Choose a corpus where a bounded scan is preferable to every index in this book and defend the cost model.
9. Place BM25, SPLADE, dense retrieval, ColBERTv2/PLAID, LambdaMART with QuickScorer, and a GPT reranker into candidate-generation and reranking stages; name each stage's recall ceiling.

References

1. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. “BEIR: A Heterogeneous Benchmark for Zero-Shot Evaluation of Information Retrieval Models.” NeurIPS Datasets and Benchmarks, 2021.
2. Yu. A. Malkov and D. A. Yashunin. “Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs.” IEEE TPAMI 42.4, 2020.
3. Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. “SPLADE: Sparse Lexical and Expansion Model for First-Stage Ranking.” SIGIR, 2021.
4. Omar Khattab and Matei Zaharia. “ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT.” SIGIR, 2020.
5. Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. “ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction.” NAACL, 2022.
6. Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. “PLAID: An Efficient Engine for Late Interaction Retrieval.” CIKM, 2022.
7. Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, Nicola Tonellotto, and Rossano Venturini. “QuickScorer: A Fast Algorithm to Rank Documents with Additive Ensembles of Regression Trees.” SIGIR, 2015.
8. Weiwei Sun et al. “Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents.” EMNLP, 2023.
9. Gordon V. Cormack, Charles L. A. Clarke, and Stefan Büttcher. “Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods.” SIGIR, 2009.
10. Ronald Fagin, Amnon Lotem, and Moni Naor. “Optimal Aggregation Algorithms for Middleware.” Journal of Computer and System Sciences 66.4, 2003.
11. Jeffrey Dean and Luiz André Barroso. “The Tail at Scale.” Communications of the ACM 56.2, 2013.