“Image search” is not a task

Consider four requests:

1. find a recompressed copy of this photograph;
2. find another view of this same mural;
3. find images answering “quiet places to read”; and
4. find when a speaker defines a monoid.

They require different positive judgments and answer units:

A system that succeeds on row one has not thereby succeeded on row two. A perceptual hash can be excellent at recompression and useless for a different viewpoint. The result is not contradictory; the relations differ.

Average precision rewards early relevant evidence

For one query, let (rel(i)\in\{0,1\}) indicate whether the item at rank (i) is judged relevant, and let (R) be the total number of relevant items in the judged corpus. Precision at rank (i) is

$$P(i)=\frac{\sum_{j=1}^{i}rel(j)}{i}.$$

Average precision is

$$AP=\frac{1}{R}\sum_{i=1}^{n}P(i)rel(i),$$

with an explicit policy for incomplete judgments and (R=0). Only ranks containing relevant items contribute, but their precision depends on earlier irrelevant items.

Suppose (R=2) and compare relevance sequences at depth four:

• A: (1,1,0,0), so (AP=(1+1)/2=1);
• B: (0,0,1,1), so (AP=(1/3+2/4)/2=5/12).

Both have recall@4 equal to one. Recall at a cutoff says both relevant items were present; AP says one system made the reader cross irrelevant items first. Neither metric is universally superior. Choose from the use: candidate generation may emphasize recall; a visible ranked list also values order.

Three recalls answer three different questions

Keep the pipeline's stages separate:

1. representation recall: are task-relevant items within exact top-k under the feature map?
2. neighbor recall: does ANN recover the exact vector neighbors?
3. final retrieval quality: after fusion, eligibility, and policy, are judged answers returned and ordered well?

A miss can occur at any stage. If a relevant item is outside exact vector top-k, the representation failed for this task. If it is exact neighbor five but ANN omits it, execution spent too much recall. If fusion removes it despite provider evidence, final policy failed. One aggregate number makes diagnosis impossible.

The same discipline applies to extraction: report word or span quality, component retrieval, and end-to-end answers without substituting one for another.

Preserve slices and uncertainty

Aggregate metrics weight the benchmark's query mixture. A duplicate-heavy test can make a system look strong while instance or multilingual semantic queries fail. Declare strata before inspecting results:

• task relation and query form;
• source domain, language, and media type;
• compression, crop, viewpoint, duration, and event length;
• missing modality and extractor settlement;
• authorization regime; and
• seen, held-out, and shifted collections.

Report per-slice estimates with query-level resampling or another justified uncertainty method. A confidence interval describes sampling under its model; it does not cover annotation mistakes, leakage, domain shift, or a metric chosen after seeing the outcome.

Hard negatives mirror the relation. For transformed-copy detection, use visually confusable noncopies. For instance search, use another object of the same type. For semantic retrieval, use topically nearby but nonanswering items. For localization, use the correct file with the wrong interval.

EVALUATE-TASK(TASK, QUERIES, RUN, LIMIT)
Input:  declared TASK contract, frozen QUERIES and RUN, positive LIMIT
Output: per-query and per-slice evidence receipt

REQUIRE-NAMED-RELATION(TASK)
REQUIRE-NAMED-ANSWER-UNIT(TASK)
results ← EMPTY-SEQUENCE
for each query in QUERIES
    judgments ← JUDGMENTS-FOR(TASK, query)
    score ← APPLY-DECLARED-METRIC(TASK.metric, TAKE(RUN[query], LIMIT), judgments)
    results ← APPEND(results, RECORD(query.identity, query.slices, score))
return SUMMARIZE-WITH-UNCERTAINTY(results, TASK.slices, TASK.resampling)

The task contract is an input, not inferred from the output file. That prevents metric shopping after a run's strengths are visible.

Wrong turns

Accuracy is not the final law

A system can score well on every declared task and still expose a private video's caption, thumbnail, existence, or timing. Evaluation has established usefulness, not authorization.

The final chapter follows every derived artifact through the index and proves that representation changes cannot downgrade its confidentiality label.

Lessons

• A retrieval task names a relation, query form, judgment, answer unit, and metric.
• Duplicate, instance, semantic, and temporal tasks are not interchangeable.
• AP rewards early relevant results; recall@k only records recovery by a cutoff.
• Representation, neighbor execution, and final policy require separate oracles.
• Aggregate quality inherits the benchmark's query mixture.
• Task and transformation slices expose hidden failures.
• Hard negatives are relation-specific.
• Confidence intervals do not cover leakage, annotation error, or domain shift.

Practice

1. Compute AP for relevance sequence (0,1,0,1) when (R=2).
2. Construct two lists with equal AP but different recall@2.
3. Design hard negatives for mural-instance search.
4. Separate representation, ANN, and final metrics for one query.
5. Name slices for a multilingual video-search evaluation.
6. Explain how repeated benchmark tuning creates leakage.
7. Define the answer unit for query-by-example audio retrieval.

References

1. Cyril W. Cleverdon, Jack Mills, and Michael Keen. “Factors Determining the Performance of Indexing Systems.” Cranfield Project, 1966.
2. National Institute of Standards and Technology. “TREC Video Retrieval Evaluation.”
3. Ellen M. Voorhees. “TREC: Continuing Information Retrieval's Tradition of Experimentation.” NIST, 2007.