Look for the words already present

Imagine a two-hour lecture whose title is “A Tour of Engines.” At 2:17 the speaker says, “A cache is a promise about the future.” A reader remembers the sentence, not the title. A transcript field makes the lecture searchable by the ordinary analyzer, inverted index, and phrase matcher already built in this book.

Now imagine a photograph titled IMG_1042. A sign in the background reads “Ferry closed after sunset.” Optical character recognition can expose those words to the same index. Neither question required a visual embedding.

The useful text fields have different authority:

Do not merge these into one anonymous text field. “The author wrote this” and “a model inferred this” are different claims even when their tokens coincide.

Extraction is a partial labeled relation

Let (M) be media items, (S_m) the spatial or temporal spans of item (m), and (T^*) finite token sequences. An extractor is not merely a function from a file to a string. Model it as a partial relation

$$E \subseteq M \times S \times T^* \times L \times C,$$

where (L) is a provenance label and (C) records confidence or another calibrated uncertainty statement. Partiality matters: silence, unreadable type, an unsupported language, and an exhausted budget may produce no token claim.

A row such as

$$(m,[137,141),\;\text{“cache promise”},\;\text{ASR-v4},\;0.82)$$

says that one extractor proposed those words during seconds 137 through 141. It does not say that the speaker, author, or curator asserted the words. The span is half-open so adjacent intervals meet without sharing an instant.

The text analyzer runs after extraction. It may normalize case, segment words, or stem them, but every resulting posting retains a path back to the extraction row and source span. A highlighted answer can then show the evidence rather than merely naming the media file.

Compose the two opportunities to miss

Fix a query and let (R) be the set of relevant media items. Let (A(m)) mean the extractor exposed sufficient correct text for item (m), and (B(m)) mean the retriever returned (m) given that exposed text. End-to-end recall is

$$Recall_{end}=\frac{|\{m\in R:A(m)\land B(m)\}|}{|R|}.$$

If extractor recall is (r_E=P(A\mid m\in R)) and conditional retriever recall is (r_R=P(B\mid A,m\in R)), then by the multiplication rule

$$r_{end}=r_Er_R.$$

This is not an independence assumption: (r_R) is explicitly conditional on successful extraction. With (r_E=0.80) and (r_R=0.90), the composed recall is (0.72), not (0.90). Perfect downstream retrieval cannot recover a word the extractor omitted.

Precision has two failure sources too. An invented token may cause an exact phrase match against something never said or shown. A ranking evaluation that judges only the returned file can hide the extraction error. Judge the source span and the token claim as well as the item.

Keep the evidence through indexing

An indexable projection needs a small receipt:

• source identity and revision;
• field kind: authored, observed, or inferred;
• extractor, model, language, and preprocessing identity;
• spatial rectangle or half-open time interval;
• original candidate sequence and analyzer identity;
• confidence semantics, if calibrated; and
• authorization label inherited from the source.

Confidence is not universal currency. A recognizer's token probability, a sequence score, and an OCR engine's heuristic rating need not share a scale. Use them only under the calibration contract that produced them. Never add an OCR confidence directly to a BM25 score because both happen to be numbers.

PROJECT-MEDIA(SOURCE, EXTRACTORS, ANALYZER, LIMIT)
Input:  authorized SOURCE, ordered EXTRACTORS, shared ANALYZER, positive LIMIT
Output: bounded postings with evidence receipts

postings ← EMPTY-SEQUENCE
for each extractor in EXTRACTORS
    claims ← EXTRACT-BOUNDED(extractor, SOURCE, LIMIT)
    for each claim in claims
        if VALID-SPAN(claim.span, SOURCE) and SAME-LABEL(claim, SOURCE)
            terms ← ANALYZE-WITH-OFFSETS(ANALYZER, claim.tokens)
            for each term in terms
                receipt ← EVIDENCE(SOURCE, extractor, claim, term)
                postings ← APPEND-BOUNDED(postings, POSTING(term, receipt), LIMIT)
return FREEZE(postings)

The bounds precede extraction and publication. A malformed hour-long file does not earn unbounded decoding, and a model that emits forever cannot grow the index forever. A typed exhausted result is more truthful than silent truncation.

Wrong turns

What text cannot say

No transcript answers “show another view of this chair” when nobody names the chair. No OCR string proves that two photographs depict the same physical object. Text projection is a powerful first tier, not a claim that media are only words.

That opens the next question: when words run out, what relation do we actually want—identical bytes, a transformed copy, the same photographed instance, or related meaning?

Lessons

• Authored metadata, OCR, transcripts, and generated captions have different authority.
• Extraction is a partial labeled relation from a media span to candidate text.
• Every analyzed posting retains its source span and provenance path.
• End-to-end recall equals extractor recall times conditional retriever recall.
• Exact downstream matching cannot restore an omitted token or correct an invented one.
• Component evaluation does not replace end-to-end judgment.
• A text projection inherits the source's authorization label.
• Purely visual questions require a different relation.

Practice

1. Define a receipt for one OCR token found in a rotated street sign.
2. With extractor recall (0.7) and conditional retrieval recall (0.95), calculate end-to-end recall.
3. Construct a hallucinated transcript token that creates a false exact match.
4. Explain why ASR confidence and BM25 score cannot be added without a model.
5. Design judgments that distinguish correct file, correct span, and correct extracted words.
6. Name the smallest bounded result for an unsupported language.
7. Give a media query no textual projection can answer faithfully.

References

1. George Awad et al. “TRECVID 2024 — Evaluating Video Search, Captioning, and Activity Recognition.” NIST, 2025.
2. Ray Smith. “An Overview of the Tesseract OCR Engine.” Proceedings of ICDAR, 2007.
3. National Institute of Standards and Technology. “TREC Video Retrieval Evaluation.”