The same word in different clothes

Two documents each contain apple pie once:

document A
title: Apple Pie
body:  A short recipe.

document B
title: Notes from October
body:  ... a footnote mentions apple pie among forty other desserts ...

The term counts are equal. Their authored locations are not. A title is a compact declaration of identity; a footnote is incidental body evidence.

A field-aware ranker needs two kinds of information:

1. authority — a product or editorial judgment that a match in one field should contribute at a different rate from a match in another;
2. discrimination — a measured corpus fact that a value occurring in few documents distinguishes more than one shared widely.

Authority is encoded by field weights. Discrimination still comes from the document-frequency argument in Chapter 6. Mixing those ideas into one handwritten priority ladder makes both hard to inspect.

First make field counts comparable

Let $F$ be the finite set of fields. For term $t$, document $d$, and field $f\in F$, record:

• $c_{t,d,f}$ — the term frequency in the field;
• $|d_f|$ — this document's analyzed field length;
• $\overline{|d_f|}$ — the corpus average length of that field;
• $b_f\in[0,1]$ — the field's length-normalization strength;
• $w_f\ge0$ — the field's authority weight.

Normalize the field frequency before weighting it:

$$g_{t,d,f}= \frac{c_{t,d,f}} {1-b_f+b_f|d_f|/\overline{|d_f|}}.$$

At $b_f=0$, field length is ignored. At $b_f=1$, the count is divided by the field's relative length. Comparing a 12-token title with the average title is different from comparing it with the average body; each field needs its own length statistic.

Now form one weighted pseudo-frequency:

$$\widehat c_{t,d}=\sum_{f\in F}w_f g_{t,d,f}.$$

It is called a pseudo-frequency because it need not be an integer. A title weight of 4 makes one normalized title occurrence contribute like four units to the shared term-evidence curve. The interpretation is a modeling choice, not a claim that the title literally repeated the word four times.

The nonlinear step cannot commute with addition

Use the BM25 saturation from Chapter 6:

$$S(x)=\frac{(k_1+1)x}{k_1+x},$$

for $x\ge0$ and $k_1>0$.

The correct field composition applies it once:

$$S\left(\sum_f w_fg_{t,d,f}\right).$$

A tempting alternative applies it independently to each already-weighted field and then adds:

$$\sum_f S(w_fg_{t,d,f}).$$

They are not equal. For two positive pieces $x$ and $y$,

$$S(x)+S(y)-S(x+y)= (k_1+1) \frac{xy(2k_1+x+y)} {(k_1+x)(k_1+y)(k_1+x+y)}.$$

Every factor is positive, so

$$S(x)+S(y)>S(x+y).$$

Independent saturation awards an excess solely because the same evidence was partitioned into two fields. With more nonempty fields, it can collect more copies of the steep first-occurrence region.

The same fact follows from concavity and $S(0)=0$, but the explicit difference shows the exact defect. This inequality applies when field weights have already scaled the inputs. A different legacy formula such as $\sum_f w_fS(g_f)$ has no universal comparison direction for arbitrary weights; it still fails to commute, and a concrete ranking test must expose the effect.

The proof is the positive rational difference above. Equality occurs only when one piece is zero or in a degenerate limit that removes the nonlinear behavior.

A two-document inversion

Set $k_1=1.2$ and suppose field normalization and weights have already produced the following evidence for one term:

The shared curve preserves the total-evidence order:

$$S(2.2)>S(2).$$

Independent saturation gives

$$S(2.2)<S(1)+S(1).$$

The scattered document wins by obtaining two first-occurrence bonuses despite having less total weighted evidence. Nothing about relevance justified the promotion; the schema partition did it.

BM25F is BM25 over the pseudo-frequency

Once $\widehat c_{t,d}$ is available, apply one saturation and the same document-frequency policy used for the term across documents:

$$\operatorname{BM25F}(q,d)= \sum_{t\in q} \operatorname{IDF}(t) \frac{(k_1+1)\widehat c_{t,d}} {k_1+\widehat c_{t,d}}.$$

Document frequency normally asks whether $t$ occurs in the document description being searched, not how many fields contain it. One document must not count several times merely because the same term occurs in title and body. The exact IDF corpus and variant remain part of the scoring contract.

BM25F-CONTRIBUTION(term, document, corpus, fieldPolicies, k1, idfPolicy)
Input:  term and document field statistics, corpus statistics, field policies
Output: term contribution and field-level explanation

pseudoFrequency ← 0
fields ← empty sequence
for each policy in fieldPolicies
    field ← policy.field
    count ← occurrences of term in document[field]
    if count = 0
        continue
    average ← corpus.averageLength[field]
    ratio ← document.length[field] / average
    divisor ← 1 - policy.b + policy.b × ratio
    normalized ← count / divisor
    weighted ← policy.weight × normalized
    pseudoFrequency ← pseudoFrequency + weighted
    append (field, count, ratio, normalized, weighted) to fields
n ← documents in corpus containing term in any searchable field
idf ← IDF(idfPolicy, corpus.documentCount, n)
saturated ← (k1 + 1) × pseudoFrequency / (k1 + pseudoFrequency)
return (idf × saturated, idf, pseudoFrequency, saturated, fields)

For a fixed finite field set, the arithmetic is $O(|F|)$ per matched query term and document. The index still pays to retrieve field frequencies. The working space is $O(1)$ without explanation and $O(|F|)$ when the decomposition is retained.

An empty field contributes zero. An average field length of zero is a schema error: no ratio can be defined for a field absent from the entire indexed corpus. Reject or remove that field policy rather than manufacture a divisor.

Authority is authored; discrimination is measured

A field weight $w_f$ states an authority judgment. It can say that a title occurrence should enter the pseudo-frequency four times as quickly as a body occurrence. It cannot say whether a particular title value is rare.

Consider two metadata families:

• an identity-like field often takes a distinct value per document: title, canonical path, product identifier;
• a membership-like field assigns many documents to one group: tag, series, section, author in a collected corpus.

Matching the exact title “Adaptive Set Intersection” may identify one document. Matching the exact tag “search” may identify one hundred. The string match class is equally exact; its discrimination is not.

For a value appearing in $n$ of $N$ documents, the same rarity logic gives a quantity like

$$\log\frac{N}{n}.$$

The distinction is corpus-dependent. author may identify one document in a collection of one paper per author and hundreds in a single author's archive. Do not permanently label the schema column “identity” or “membership” without measuring its value distribution in the active corpus.

The exact-match ladder that forgets rarity

Navigation boxes often use a hand-built order:

whole field equals query
field starts with query
field contains query
fuzzy match

That ladder is a scoring function even if it contains no floating-point numbers. If “whole field” applies equally to titles and tags, a query search can promote every document tagged search above a uniquely titled page that contains the phrase. The tag group then ties and falls back to alphabetical or insertion order. To a reader it looks as though ranking stopped working; in fact, the ranker obeyed an incomplete model.

There are two principled repairs, depending on intent:

1. If the tag is a requested filter, treat it as an eligibility predicate and rank only inside that group. Chapter 8 derives this separation.
2. If the tag is free-text evidence, combine match quality, field authority, and measured value rarity. An exact shared label need not receive the same evidence as an exact unique identifier.

A universal rule that membership equality must always rank below title containment would merely replace one rigid ladder with another. The relation and task decide: filter facts constrain; ranking evidence accumulates.

Field boundaries are extraction contracts

BM25F assumes that title, body, path, and metadata were populated correctly. A large title weight magnifies extraction mistakes. If a listing page accidentally inherits the first listed article's title, the scorer becomes confidently wrong.

Each indexed field therefore needs:

• an extraction rule over the source representation;
• provenance identifying the source span or authored fact;
• its analyzer contract;
• length and missing-value semantics;
• a field-weight and normalization policy version.

The query explanation should show the field contribution before saturation and the shared saturated result. Showing only a final total hides both extraction errors and multiple-saturation errors.

What the weights cannot tell you

BM25F introduces at least one $w_f$ and one $b_f$ per field, plus shared $k_1$ and an IDF policy. A five-field schema can easily expose eleven knobs. Values chosen because they “feel right” are undocumented opinions with decimal points.

Useful evidence includes:

• held-out relevance judgments by query class;
• ablations that set one field weight to zero;
• sensitivity curves rather than one winning parameter point;
• per-field coverage, value cardinality, and length distributions;
• regression queries where a field is misleading or missing;
• latency and index-byte changes caused by storing field statistics.

The evaluation chapter will make these obligations precise. Until then, use a small understandable parameter set, record why it exists, and keep a structural baseline that can be restored.

Engineering reality

Field-aware scoring multiplies the statistics that must remain consistent:

• per-field average lengths change as segments arrive or documents are deleted;
• document frequency must use one declared searchable-field universe;
• sparse fields need compact storage rather than dense zeros;
• field weights and analyzers belong to the index/scorer version;
• shard-local averages can make otherwise identical documents incomparable;
• an upper bound must include every field's maximum possible weighted contribution before pruning is safe.

Keep the arithmetic pure and retain a direct oracle. Generate small documents with random field counts and verify:

• collapsing to one field reproduces the corresponding BM25 score;
• permuting field iteration does not change the result;
• splitting one weighted pseudo-frequency across fields does not change the shared-saturation result when normalization is held fixed;
• the legacy independently saturated model differs on a pinned two-field counterexample;
• explanation contributions sum exactly to the reported score.

The field that should not be scored

A reader asks for troubleshooting documents for product Fiber, platform Android, and version current. Those facts are not weak hints that a result might be relevant. A document for another product or an obsolete platform is ineligible.

Giving these values large field weights still lets enough body-text evidence overcome them. Filtering only after top-k can return too few results and may reveal aggregate facts about documents the reader cannot access.

The next chapter treats structured metadata as secondary indexes and set operations. Eligibility chooses the universe; BM25F ranks only within it.

Lessons

• Field authority and corpus discrimination are separate dimensions.
• Normalize a term frequency against the average length of its own field.
• BM25F forms one weighted pseudo-frequency and applies the term's saturation once.
• For positive $x,y$, $S(x)+S(y)>S(x+y)$: independent field saturation overrewards evidence for being partitioned.
• A document frequency counts documents, not matching fields.
• Exact field equality measures string relation, not uniqueness or relevance.
• Membership-like values may be filters or ranking evidence; the task decides, and their corpus frequency remains observable.
• Field weights need judgments, ablations, and sensitivity evidence. They are not self-validating.

Practice

1. Prove the displayed formula for $S(x)+S(y)-S(x+y)$ by bringing the fractions to a common denominator.
2. At $k_1=1.2$, compute shared and independent saturation for evidence pairs $(2.2,0)$ and $(1,1)$. Identify the ranking inversion.
3. Build a two-field example in which changing only $b_{body}$ reverses two documents. Explain the corpus-length assumption encoded by the reversal.
4. For a title, author, tag, and body corpus, measure the document frequency of every exact field value. Which fields behave identity-like, and where does that classification fail?
5. Design a property test showing that splitting a pseudo-frequency across fields leaves correct shared saturation unchanged but changes independent saturation.

References

1. Stephen Robertson, Hugo Zaragoza, and Michael Taylor. “Simple BM25 Extension to Multiple Weighted Fields.” Proceedings of the Thirteenth ACM International Conference on Information and Knowledge Management, 2004.
2. Stephen Robertson and Hugo Zaragoza. “The Probabilistic Relevance Framework: BM25 and Beyond.” Foundations and Trends in Information Retrieval 3, no. 4, 2009.
3. Karen Spärck Jones. “A Statistical Interpretation of Term Specificity and Its Application in Retrieval.” Journal of Documentation 28, no. 1, 1972.