Learn · Pictures Are Not Words

budding

A Picture Becomes a Point

The feature map decides what “near” means before the index does any work.

search, images, embeddings, cosine-similarity, clip, representation-learning, domain-shift, learn

“Learning directly from raw text about images is a promising alternative.”

— Alec Radford et al., Learning Transferable Visual Models From Natural Language Supervision

This is the thirtieth chapter in a book about search from first principles. You will define a feature map before defining neighbors; compare color, local-word, and text-aligned representations; derive the relation between cosine similarity and inner product after normalization; separate representation error from search error; and make model and preprocessing identity part of the index contract. The next chapter accelerates neighbor search without changing this geometry.

Three honest arrangements

Place six postcards on a table: a red bus, a blue bus, a red chair, a blue chair, a sunset, and a night sky. Sort them three times:

by dominant color;
by depicted object; and
by which would answer “transport at night.”

The nearest neighbor of the red bus might be the red chair under color, the blue bus under object identity, and a night train under a text-aligned representation. The neighbor algorithm did not become confused. The feature map changed the question.

Prediction — identify the missing definition.

Define the map before the neighborhood

Let (X) be media items. A feature extractor is a map

$f:X\rightarrow\mathbb{R}^{d}.$

Only after (f) is fixed can a distance (delta) define the nearest items to a query (q):

$N_k(q)=\operatorname*{arg\,top}_{x\in X}^{k} \big(-\delta(f(q),f(x)),\;id(x)\big).$

The stable identity term resolves equal distances. Without it, “exact top-k” is underspecified at ties.

The map may be transparent. A color histogram counts pixels in fixed bins. A visual-word representation quantizes local descriptors, then reuses inverted index ideas. Sivic and Zisserman's “Video Google” made that analogy explicit: image-region descriptors play the role of words, and frames or shots play the role of documents.

The map may instead be learned. A contrastive text-image model trains image and text encoders so paired examples receive stronger similarity than alternatives under its objective. This permits a text query and an image to inhabit a comparable feature space. It does not make the space a universal ontology.

An embedding maps an item into coordinates chosen so useful relations become geometric. The coordinates inherit the training objective, data, and preprocessing; they are not direct measurements of meaning.

Normalize before calling cosine an inner product

For nonzero vectors (u) and (v), cosine similarity is

$\cos(u,v)=\frac{u\cdot v}{\|u\|_2\|v\|_2}.$

Define normalized vectors (hat u=u/\|u\|_2) and (hat v=v/\|v\|_2). Then

$\cos(u,v)=\hat u\cdot\hat v.$

For unit vectors, squared Euclidean distance is

$\|\hat u-\hat v\|_2^2 =2-2\hat u\cdot\hat v.$

Therefore maximizing cosine similarity, maximizing inner product of normalized vectors, and minimizing Euclidean distance between normalized vectors induce the same order. This equivalence fails if normalization is omitted or zero vectors are silently admitted.

Normalization is part of representation identity, not a harmless query-time detail. So are resize rule, crop policy, color conversion, tokenizer, model weights, numeric precision, dimension, and output pooling.

Reveal — derive the equivalent order.

Two errors, two oracles

Suppose a relevant chair image ranks poorly. There are two fundamentally different explanations:

representation error: under (f) and (delta), the chair is not among the exact nearest vectors;
neighbor-search error: it is an exact neighbor, but an approximate index fails to return it.

The first needs task judgments and perhaps a different feature map or training regime. The second needs an exhaustive neighbor oracle over frozen vectors and perhaps a wider search frontier. More graph exploration cannot repair the first error. Retraining cannot prove the index found the exact neighbors of the new geometry.

Maintain two evaluations:

Question	Oracle	Typical metric
Did the geometry place relevant items nearby?	human or task judgments	recall@k, average precision, calibrated loss
Did the index recover the geometry's neighbors?	exhaustive vector scan	neighbor recall@k

Reporting only task quality after changing both model and index cannot attribute the gain. Reporting only neighbor recall can certify faithful execution of a bad representation.

Algorithm — versioned exact vector baseline

EXACT-NEIGHBORS(QUERY, ITEMS, REPRESENTATION, LIMIT)
Input:  bounded QUERY and ITEMS, complete REPRESENTATION identity, positive LIMIT
Output: exact top LIMIT receipt under one geometry

queryVector ← NORMALIZE-OR-REFUSE(ENCODE(REPRESENTATION, QUERY))
best ← EMPTY-MIN-HEAP(LIMIT)
for each item in ITEMS
    REQUIRE-SAME-REPRESENTATION(item, REPRESENTATION)
    score ← DOT(queryVector, item.normalizedVector)
    best ← KEEP-BEST(best, CANDIDATE(item.identity, score), LIMIT)
return RECEIPT(SORT-BY-SCORE-THEN-IDENTITY(best), REPRESENTATION)

This exhaustive algorithm is intentionally simple. It is the semantic oracle against which the next chapter's approximate execution is measured.

Training data defines possible shortcuts

CLIP demonstrated that predicting which caption goes with which image could learn broadly transferable representations from a large web-derived corpus. That measured result belongs to the studied models and benchmarks. The paper also analyzes limitations and social consequences; natural-language supervision imports the patterns, absences, and biases of its data.

A model can exploit background, typography, watermark, geographic, or collection artifacts instead of the intended object. Domain shift occurs when deployment changes those regularities: product photographs replace web images, medical scans replace everyday photographs, or a language was rare in training.

Evaluation therefore slices by domain, language, image source, transformation, and requested relation. A single aggregate score can rise while a critical slice collapses.

Wrong turns

Call embeddings semantic coordinates

Coordinates are meaningful only through the feature map, objective, and evaluation. “Semantic” without a task relation conceals those choices.

Mix vectors from two model revisions

Their coordinates need not share a geometry even if dimensions match. Rebuild or maintain explicitly separated generations; do not compare them as if one map produced both.

Tune the ANN index to repair relevance

Wider search can approach exact neighbors only. It cannot move a judged relevant item that exact search already places far away.

The next bargain

The geometry is now frozen and an exhaustive oracle can recover its exact neighbors. But exhaustive search computes a distance to every item. Must a million-image collection spend a million comparisons per query?

The next chapter replaces complete inspection with graph navigation. Its speed is bought by spending a measured recall budget against this exact baseline.

Lessons

A feature map (f:X\to\mathbb{R}^d) precedes every vector neighborhood.
Different coherent representations produce different coherent neighbors.
For normalized nonzero vectors, cosine, inner product, and Euclidean orders are equivalent in the stated directions.
Model, preprocessing, precision, dimension, normalization, and corpus revision form representation identity.
Representation error and neighbor-search error require different oracles.
Approximation cannot repair a relevant item placed far away by the map.
Cross-modal proximity is not factual or semantic interchangeability.
Domain shift and slice quality belong in every representation evaluation.

Practice

Give three feature maps that order the same postcard collection differently.
Derive (|u-v|_2^2=2-2u\cdot v) for unit vectors.
Construct vectors where raw inner product and cosine induce different orders.
Write the complete representation identity for an image index.
Design separate tests for representation recall and neighbor recall.
Name a plausible shortcut in a product-image collection.
Explain why equal vector dimensions do not license cross-model comparison.

References

Josef Sivic and Andrew Zisserman. “Video Google: A Text Retrieval Approach to Object Matching in Videos.” Proceedings of ICCV, 2003.
Alec Radford et al. “Learning Transferable Visual Models From Natural Language Supervision.” Proceedings of ICML 139, 2021.
Alec Radford et al. “CLIP Model Card.” 2021.

Arrange postcards by dominant color, depicted object, or the sentence someone would use to request one. Each arrangement is coherent. None is simply “the” meaning of the pictures.