Expansion meets rarity

Suppose a query term work expands to works, working, workhorse, and a likely typo. Chapter 6 gave each indexed term a rarity weight derived from its document frequency. Applying that rule independently to expansions produces a trap:

A page mentioning the obscure variant can outrank the page titled with the literal query. The formula behaved exactly as designed: rare terms are stronger evidence. The mistake was declaring each spelling independent evidence.

Let $G_t$ be the set of variants that stand for source query term $t$. Define its matching document set

$$D(G_t)=\bigcup_{v\in G_t}D(v)$$

and its blended document frequency

$$df(G_t)=|D(G_t)|.$$

Every same-word variant uses the group's rarity. The union, not the sum, is the honest cardinality because a document containing two variants is still one document.

Expansion meets accumulation

The second failure survives blended rarity. If a document matches script, script1, script2, and scripts, summing every variant contribution rewards it four times for one query word. A page whose title is exactly script can lose to a page listing many variants.

Within one group, choose one contribution:

1. if the literal term matches, use its contribution;
2. otherwise use the best admitted alternative; and
3. if no variant matches, use the additive identity zero.

Across distinct query groups, add as before. Expansion is therefore choice inside a word and accumulation across words.

Represent a contribution as a pair

$$(e,s)$$

where $e$ is 1 for an exact literal match and 0 otherwise, and $s$ is the numeric contribution. Order pairs lexicographically: exactness first, score second, followed by a stable identity tie-break outside the numeric value. Define

$$a\sqcup b=\max(a,b)$$

under that order. The group fold is $\bigsqcup$ over matching variants.

Choice has the laws expansion needs

The maximum of a total order is associative, commutative, and idempotent:

$$(a\sqcup b)\sqcup c=a\sqcup(b\sqcup c),$$

$$a\sqcup b=b\sqcup a,$$

and

$$a\sqcup a=a.$$

Associativity allows streaming or tree reduction. Commutativity means expansion enumeration order cannot change a document's group score. Idempotence means a duplicate variant cannot vote twice. The identity is the no-match contribution.

These laws cover combination over a given admitted set. They do not cover an earlier cap. If a scanner enumerates corpus order and an automaton enumerates lexical order, “keep the first 64 expansions” can select different sets before the commutative fold begins. Reimpose the promised order before the cap and test with inputs exceeding it.

Rarity has two equivalent faces

Common IDF forms are antitone in document frequency: when $df$ rises, $idf$ falls. For same-word variants, one implementation can blend counts first:

$$idf\left(\left|\bigcup_v D(v)\right|\right).$$

If only a safe upper approximation to union frequency is available, another can conservatively take the minimum permitted rarity weight. These are the same direction of law—more group coverage cannot yield greater rarity—though exact equality requires the actual union count and the same IDF policy.

Summing document frequencies is not the union. It double-counts overlap and can exceed corpus size. Taking maximum individual frequency is a lower bound on the union and can still overstate rarity. The artifact receipt records which count was used and whether it was exact or conservative.

GROUP-CONTRIBUTION(SOURCE, VARIANTS, DOCUMENT, CORPUS)
Input:  SOURCE term, admitted VARIANTS, DOCUMENT, and CORPUS statistics
Output: one exact-else-best contribution

groupDocuments ← empty set
for each variant in VARIANTS
    groupDocuments ← UNION(groupDocuments, DOCUMENTS-MATCHING(CORPUS, variant))
rarity ← IDF(size(groupDocuments), DOCUMENT-COUNT(CORPUS))
best ← NO-MATCH
for each variant in VARIANTS
    if MATCHES(DOCUMENT, variant)
        exact ← variant = SOURCE
        value ← LOCAL-SCORE(DOCUMENT, variant) × rarity × EXPANSION-WEIGHT(variant)
        best ← CHOOSE(best, PAIR(exact, value))
return best

In a production evaluator, the union may be computed from postings rather than materialized as a mutable set. The pseudocode states semantics; representation follows workload and preserves the same cardinality.

Synonyms are not spelling variants

A typo, case variant, inflection, or corpus-approved compounding variant can stand for the same lexical evidence. A synonym is a different word carrying an editorial claim of related meaning. Treating it as identical can erase useful rarity and admit the wrong sense.

For synonyms:

• keep their own authored expansion weight;
• cap their imported rarity at the source term's anchor so a rare synonym cannot create a jackpot;
• prefer matches in identity fields—title, slug, tags, command name—over incidental prose membership; and
• record which curated relation admitted the term.

The exception is deliberate, not a loophole. The group receipt says whether a relation is same-word equivalence or a semantic alternative.

Where the vocabulary comes from

Three sources look attractive.

Learn from the local corpus. Distributional similarity needs repeated contexts. Count first: vocabulary size, token count, and the fraction of terms appearing in exactly one document. When much of a small technical corpus is singleton vocabulary, its geometry describes accidents rather than stable relations. A large outside model solves sample size by importing outside senses.

Import a general thesaurus. Coverage is usually highest for common words, where expansion adds ambiguity, and weakest for product names, abbreviations, identifiers, and domain terms where vocabulary mismatch hurts. index has book, finger, database, and economic senses; the correct one depends on the corpus and task.

Curate a small corpus-local table. This is the recommendation, not a fallback. Include abbreviation/expansion pairs, alternate terminology, spelling and compounding variants, and known reader-versus-author vocabulary. Keep it small enough for a domain owner to read in one sitting.

Voorhees's experiments found that automatically selected lexical-semantic expansions could degrade retrieval even when hand-selected concepts provided an upper-bound setting. The durable lesson is not “never expand.” It is that sense selection is part of the algorithm and must be evaluated on the target task.

Explain the group

For each query term, the receipt contains:

The explanation is emitted by the scoring pass. Reconstructing it afterward risks choosing a different winning variant or forgetting the cap.

Negative results

Lessons

• Expansion creates two independent jackpots: rarity and accumulation.
• Same-word variants share the union document frequency of their group.
• A document contributes once per query word: exact when present, otherwise the best admitted alternative.
• Choice by maximum is associative, commutative, and idempotent.
• Those laws do not protect order-sensitive admission caps.
• Synonyms are editorial semantic alternatives, not automatically equivalent spellings.
• Count corpus evidence before claiming locally learned semantic relations.
• A small curated vocabulary is often safer and more useful than a general thesaurus.

Practice

1. For document sets $A$, $B$, and $C$, compare $|A\cup B\cup C|$ with $|A|+|B|+|C|$. State when they are equal.
2. Prove maximum under a total order is idempotent and commutative.
3. Construct a corpus in which per-variant IDF promotes an obscure variant above an exact title match.
4. Construct a separate corpus in which summation fails even after IDF is blended.
5. Design a curated entry for an abbreviation with two domain-dependent senses. Which fields may it expand into?
6. Test two expanders that enumerate identical variants differently, both below and above a cap. Which fixture exposes semantic drift?

References

1. Apache Lucene. “=BlendedTermQuery=.” Versioned API documentation describing blended term statistics.
2. Stephen Robertson and Hugo Zaragoza. “The Probabilistic Relevance Framework: BM25 and Beyond.” Foundations and Trends in Information Retrieval 3(4), 2009.
3. Ellen M. Voorhees. “Query Expansion Using Lexical-Semantic Relations.” SIGIR, 1994.