Four routes to one clip

A reader may request the same clip in four ways:

• its authored title;
• a sentence spoken in it;
• a cropped logo visible in a frame; or
• “a train entering a station at night.”

The corresponding providers implement different relations: lexical metadata, transcript evidence, visual instance recurrence, and cross-modal semantic proximity. No one provider is defective because it cannot answer every form.

Every candidate carries a typed receipt:

Eligibility precedes cooperation

For principal (p), provider (i) has an eligible universe (U_i(p)). Its ranked candidates must satisfy

$$L_i(p)\subseteq U_i(p)$$

before any candidate, score, count, or latency-dependent behavior enters a shared fusion stage. A final post-filter is too late: forbidden candidates may have consumed top-k capacity, changed fused ranks, warmed caches, or altered timing.

The fused universe is the union of eligible provider evidence, not the intersection of all provider corpora:

$$U_F(p)=\bigcup_i L_i(p).$$

Each candidate retains which providers admitted it. Absence from one list means “no evidence from this provider under this budget,” not a zero-valued negative vote. A video with no transcript should not be punished as if a transcript ranker examined and rejected its speech.

Two honest fusion families

If provider outputs can be calibrated to a shared event—such as probability of relevance under one judgment policy—then a trained fusion model may combine them. Calibration must be evaluated by provider, query slice, missingness pattern, and time period. Correct ordering alone is insufficient.

When scores remain incomparable, ranks provide a weaker common object. Reciprocal rank fusion assigns

$$RRF(d)=\sum_{i:d\in L_i}\frac{w_i}{c+r_i(d)},$$

where (r_i(d)) is the one-based rank in provider (i), (w_i\geq0), and (c>0) controls how rapidly rank influence decays.

If a provider's scores undergo any strictly increasing transformation, its rank order is unchanged, so its RRF contribution is unchanged. This invariance is exactly what raw addition lacks. RRF also discards meaningful score gaps: a near tie and a cliff between ranks one and two look identical. That is the price of using only the shared ordinal structure.

Identity and diversity come after relation

The same media item may appear through several providers or segment identities. Fusion needs a canonical source identity plus relation-specific evidence. It deduplicates presentation without erasing the independent reasons the item was found.

Several top results may be near-identical copies. A diversity policy can spend some rank utility to cover different evidence, sources, or scenes, but it is a separate objective. Do not hide it inside score normalization. The receipt states the relevance fusion and the diversity selection separately.

FUSE-ELIGIBLE(PRINCIPAL, LISTS, WEIGHTS, OFFSET, LIMIT)
Input:  typed bounded LISTS, nonnegative WEIGHTS, positive OFFSET and LIMIT
Output: fused candidates with provider receipts

scores ← EMPTY-IDENTITY-MAP
evidence ← EMPTY-IDENTITY-MAP
for each provider in STABLE-PROVIDER-ORDER(LISTS)
    REQUIRE-ELIGIBLE-RECEIPT(provider, PRINCIPAL)
    rank ← 1
    for each candidate in provider.candidates
        REQUIRE-ELIGIBLE(candidate, PRINCIPAL, provider)
        scores ← ADD(scores, candidate.identity, WEIGHTS[provider] / (OFFSET + rank))
        evidence ← ATTACH(evidence, candidate.identity, candidate.receipt)
        rank ← rank + 1
return TAKE(SORT-BY-SCORE-THEN-IDENTITY(scores, evidence), LIMIT)

The stable provider order controls reproducible accumulation and diagnostics, not relevance. Exact rational accumulation or a declared numeric rule prevents provider traversal order from changing tied results.

Wrong turns

The top video is still too large

Fusion may place a two-hour lecture first because several providers agree. Yet the answer may occupy twenty seconds. Returning only the file hands temporal search back to the reader.

The next chapter changes the answer unit from media identity to a half-open time interval with evidence.

Lessons

• Providers expose typed relations and evidence, not one universal confidence.
• Eligibility holds before candidate generation and fusion become observable.
• Missing evidence is not a zero-valued negative judgment.
• Calibrated fusion requires a shared predicted event and local calibration.
• RRF is invariant under strictly increasing transformations of provider scores.
• Rank-only fusion discards within-provider score gaps.
• Deduplication preserves every provider receipt.
• Diversity is a separate, declared objective.

Practice

1. Compute RRF scores for two three-item lists with (c=10).
2. Prove a strictly increasing score transform preserves RRF.
3. Construct a case where min-max normalization changes after adding one item.
4. Distinguish missing transcript, failed transcript, and transcript rejection.
5. Design a fused receipt for one item found by title and visual instance.
6. Give a post-filter timing leak.
7. Explain when calibrated fusion retains information RRF discards.

References

1. Gordon V. Cormack, Charles L. A. Clarke, and Stefan Büttcher. “Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods.” Proceedings of SIGIR, 2009.
2. Josef Sivic and Andrew Zisserman. “Video Google: A Text Retrieval Approach to Object Matching in Videos.” Proceedings of ICCV, 2003.
3. National Institute of Standards and Technology. “TREC Video Retrieval Evaluation.”