A file is a container, not an answer

A two-hour lecture contains the requested definition from 2:17 through 2:37. Returning the lecture proves only that the container is relevant. The reader must still search 7,200 seconds to find 20 seconds of evidence.

Represent a temporal hit as a half-open interval

$$I=[s,e),\qquad 0\leq s<e\leq D,$$

where (D) is media duration. Half-open intervals make adjacency exact: ([10,20)) and ([20,30)) share no instant, and their union is ([10,30)).

A result also names source revision, segmentation generation, provider, evidence spans, score contract, and authorization label. “Starts at 2:17” without the referenced media revision can drift after editing.

Measure overlap, not merely co-residence

For predicted interval (P) and judged interval (G), temporal intersection over union is

$$tIoU(P,G)=\frac{|P\cap G|}{|P\cup G|}.$$

For half-open intervals,

$$|P\cap G|=\max(0,\min(e_P,e_G)-\max(s_P,s_G))$$

and

$$|P\cup G|=|P|+|G|-|P\cap G|.$$

If (G=[137,157)), exact prediction has tIoU 1. The whole two-hour lecture has tIoU (20/7200\approx0.0028), despite perfect file-level relevance. A one- second interval inside (G) has file correctness and precision-like overlap, but tIoU (1/20): it omits most of the event. The metric makes both bloated and truncated answers visible.

Boundary error should also be reported directly when user experience depends on seeking accuracy:

$$BE(P,G)=|s_P-s_G|+|e_P-e_G|.$$

No one threshold is universal. NIST's MedVidQA evaluation, for example, reports ranked retrieval and interval overlap under declared thresholds. Those choices belong to that task, not every video search product.

Segmentation is a hypothesis

A video index can expose several candidate segmentations:

• fixed windows are simple and bounded but split events arbitrarily;
• overlapping windows reduce boundary misses but multiply candidates;
• shots follow visual cuts but may split continuous speech or join several actions within one take;
• transcript spans follow speech but miss silent events;
• learned event proposals add model and domain error; and
• keyframes summarize appearance but miss motion-defined evidence.

For fixed window length (w) and stride (h\leq w), an interior instant is covered by roughly (w/h) windows. Reducing (h) improves boundary coverage but increases index entries, duplicate hits, fusion work, and preview choices by the same overlap factor. That is a measured storage–recall bargain.

A result always names the segmentation. Otherwise a boundary change can alter answers while the index appears to retain one identity.

Merge evidence under an explicit gap

Transcript terms or adjacent windows may produce several hits for one event. Let sorted intervals (I_1,I_2,\ldots) be mergeable only when they share source, provider relation, compatible evidence, and

$$s_{j+1}-e_j\leq g$$

for declared gap (g\geq0). A large (g) reduces duplicate rows but can join distinct events. A small (g) preserves separation but may fragment one answer. The gap belongs in the policy and receipt.

Temporal non-maximum suppression is another selection policy: keep a strong interval, then suppress weaker candidates whose tIoU with it exceeds a chosen threshold. It removes redundant proposals; it does not prove the retained boundary is correct.

MERGE-INTERVALS(HITS, GAP, LIMIT)
Input:  bounded HITS in source-start order, nonnegative GAP, positive LIMIT
Output: compatible merged intervals

merged ← EMPTY-SEQUENCE
for each hit in HITS
    REQUIRE-VALID-HALF-OPEN(hit.interval)
    previous ← LAST-OR-NONE(merged)
    if previous ≠ NONE and MERGEABLE(previous, hit, GAP)
        joined ← JOIN-EVIDENCE(previous, hit, previous.start, MAX(previous.end, hit.end))
        merged ← REPLACE-LAST(merged, joined)
    else
        merged ← APPEND-BOUNDED(merged, hit, LIMIT)
return FREEZE(merged)

Sorting is a precondition rather than hidden work. The algorithm never merges across source revisions, authorization labels, or incompatible relations.

Wrong turns

The metric waits for the question

tIoU is appropriate only when the answer is a temporal interval. Duplicate detection, instance retrieval, and semantic ranking need different judgments and metrics. Combining them into one “video search score” hides which relation improved.

The next chapter starts evaluation by naming the question and answer unit.

Lessons

• A video result is a source identity plus a half-open evidence interval.
• File relevance cannot measure temporal boundary quality.
• tIoU penalizes both oversized and truncated predictions.
• Fixed-window overlap trades boundary opportunities for index and query work.
• Shots, transcripts, keyframes, and event proposals have different blind spots.
• Adjacent evidence merges only under compatibility and a declared gap.
• Suppressing duplicate intervals does not validate the surviving boundary.
• Every hit names its source and segmentation generation.

Practice

1. Compute tIoU for ([10,30)) and ([20,40)).
2. Compare boundary error for an oversized and a shifted prediction.
3. Calculate the overlap factor for (w=30) seconds and (h=5) seconds.
4. Construct an event missed by keyframes but found through motion.
5. Give two adjacent hits that must not merge.
6. Design a receipt for a transcript-derived preview.
7. Explain why temporal NMS is not a relevance proof.

References

1. George Awad et al. “MedVidQA at TRECVID 2023.” NIST, 2023.
2. Ranjay Krishna et al. “Dense-Captioning Events in Videos.” Proceedings of ICCV, 2017.
3. Josef Sivic and Andrew Zisserman. “Video Google: A Text Retrieval Approach to Object Matching in Videos.” Proceedings of ICCV, 2003.