The reader changes the loss

Suppose a research assistant reads every passage placed in a folder and treats each as evidence. Adding an irrelevant passage is no longer a harmless extra row. It consumes context, competes with useful evidence, and may support a wrong statement.

The evaluator in Chapter 24 already warned that recall can be gamed by widening acceptance. The machine-reader boundary sharpens the consequence: a false positive can cost more than a false negative.

Let $p=P(relevant\mid x,q)$ be a calibrated probability for candidate $x$ under query and task $q$. Let $C_{FP}$ be the loss of including an irrelevant candidate, and $C_{FN}$ the loss of excluding a relevant one.

If we include $x$, expected error loss is

$$L(include)=C_{FP}(1-p).$$

If we exclude it,

$$L(exclude)=C_{FN}p.$$

Include when the first is no greater than the second:

$$p\geq \frac{C_{FP}}{C_{FP}+C_{FN}}.$$

Equal losses recover the familiar $1/2$ threshold. If false inclusion costs nine times false exclusion, the threshold becomes $9/10$. The direction is not a matter of taste; it follows from the declared loss ratio.

A ranking score is not automatically a probability

BM25 scores, vector similarities, edit distances, and reciprocal-rank fusion scores order candidates. Their numeric values do not inherently equal $P(relevant\mid x,q)$, and scores from different indexes do not share a probability scale.

To use the loss threshold, estimate calibration on held-out judgments from the same task and regime. Report reliability by probability bucket and relevant strata. A calibrated model can drift when corpus, query population, analyzer, or retrieval policy changes, so the calibration identity belongs in the query receipt.

Without defensible calibration, choose thresholds empirically against the cost-weighted evaluator and describe them as policy thresholds, not posterior probabilities. The formula still clarifies which errors matter; it does not magically transform a score.

Abstention is an answer

Add a third action: reject or abstain. Let $C_R$ be the cost of asking for clarification, returning no evidence, or routing to review. Choose the action with least expected loss:

$$\min\{C_{FP}(1-p),\;C_{FN}p,\;C_R\}.$$

When neither inclusion nor exclusion is sufficiently safe, abstention occupies the middle region. Chow's reject-option analysis and later selective classification make the trade explicit: lower coverage can buy lower error on the accepted set.

An empty result therefore has several legitimate meanings:

• no eligible evidence met the acceptance policy;
• the system abstained because uncertainty was too costly;
• relevant evidence existed but did not fit the context budget;
• a provider was unavailable or refused work; or
• an access boundary withheld evidence.

Keep those settlements distinct. “No results” cannot safely summarize them all.

Filtering after top-k starves predictably

Suppose retrieval takes the top $k$ documents and then keeps only those with an eligibility predicate: document kind, language, access label, date, or source. Let each returned document survive with probability $s$, under a simplified independent-selectivity model. If $X$ is the surviving count,

$$X\sim Binomial(k,s)$$

and

$$\mathbb{E}[X]=ks.$$

Retrieve ten and filter at selectivity $0.2$; expect two survivors. The empty-looking result is not surprising. It is the predicted output of the pipeline.

Overretrieving roughly $k/s$ candidates reaches $k$ survivors only in expectation, and correlation or rank-dependent selectivity can make that model poor. It also spends work on known-ineligible candidates.

The correct semantic operation is top-k within the eligible universe:

$$TopK_k(\{x\in C\mid eligible(x)\}).$$

Apply eligibility during candidate traversal and continue until $k$ eligible results settle or the eligible universe, query budget, or source is exhausted. The predicate participates in safe pruning: an ineligible high scorer cannot occupy the threshold or stop traversal.

ACCEPTED-TOP-K(CANDIDATES, QUERY, POLICY, LIMIT, BUDGET)
Input:  bounded CANDIDATES, QUERY, acceptance POLICY, LIMIT, BUDGET
Output: accepted results plus settlement receipt

heap ← EMPTY-MIN-HEAP(LIMIT)
for each candidate in BOUNDED-SCORE-ORDER(CANDIDATES, QUERY, BUDGET)
    if not ELIGIBLE(candidate, POLICY)
        RECORD-INELIGIBLE(candidate.identity)
        continue
    evidence ← SCORE-AND-CALIBRATE(candidate, QUERY, POLICY)
    action ← MINIMUM-LOSS-ACTION(evidence, POLICY.losses)
    if action is INCLUDE
        heap ← KEEP-BEST(heap, candidate, LIMIT)
    else if action is ABSTAIN
        RECORD-ABSTENTION(candidate.identity, evidence)
return SETTLE(heap, BUDGET, SOURCE-STATUS())

Candidate traversal, calibration, and receipts all consume the query-wide budget. An unbounded loop cannot be excused by a strict acceptance threshold.

Eligibility and relevance are different evidence

A private document can be highly relevant and still ineligible for this request. A public document can be eligible and irrelevant. Access checks, document-kind constraints, and source admission are hard predicates; relevance orders the surviving universe.

Do not encode eligibility as a negative score and hope ranking places it low enough. A score can be outweighed, rescaled, or fused. Hard constraints exclude candidates before the objective. The receipt records the policy identity and aggregate exclusion reasons without revealing hidden document identities to an unauthorized consumer.

This also prevents an inference leak. Reporting “three confidential documents matched” may disclose existence even when content is withheld. The access boundary decides which counts, explanations, and source names may cross it.

Citation is part of the result value

A passage offered as grounding evidence carries:

• stable source identity and canonical location;
• the exact admitted span or media region;
• content and artifact revision;
• extraction or transformation provenance;
• relevance and eligibility explanations;
• settlement and access labels; and
• enough surrounding structure to interpret the claim.

A generated sentence with a nearby URL is not automatically supported. The evaluator checks whether the cited span entails or substantiates the claim for the declared task. Citation correctness has at least two axes: citation presence and citation support. Optimizing presence alone produces many beautifully attributed unsupported claims.

NIST's risk-management guidance treats validity, reliability, accountability, transparency, and context as distinct considerations. It is not evidence that a particular retrieval pipeline is safe. It motivates an evaluation checklist whose decisive claims must still be tested on the local artifact.

Precision and latency meet at the decision boundary

A stricter threshold can reduce context size and downstream work, but it may also require deeper candidate traversal to find enough accepted results. A filter can make ranking cheaper if it narrows an indexed universe, or more expensive if eligibility is checked only after costly scoring.

Measure the joint frontier:

• cost-weighted false inclusions and exclusions;
• accepted-set precision and coverage;
• abstention and review rate;
• eligible results found per candidate examined;
• candidate, scoring, and context tokens consumed;
• time to first accepted evidence and terminal settlement;
• citation support and hard-negative inclusion; and
• behavior under unavailable and access-restricted sources.

There is no single universally optimal point. The loss policy names a context of use. A high-stakes action and a casual site-navigation hint should not share the same error ratio merely because they share a search box.

Negative results

Lessons

• A machine reader changes the cost of false-positive retrieval.
• With calibrated relevance probability, asymmetric losses determine the acceptance threshold.
• Retrieval scores need calibration before they can be treated as probabilities.
• Abstention is optimal when its cost is lower than either classification error.
• Post-filtering top-k expects only $ks$ survivors at selectivity $s$.
• Rank inside the eligible universe and keep hard predicates outside scoring.
• Empty, partial, unavailable, refused, and access-withheld settlements differ.
• Attribution preserves source identity; citation support remains an evaluated correctness property.
• Threshold, traversal cost, context budget, and latency form a measured frontier for each context of use.

Practice

1. Derive the acceptance threshold for $C_{FP}=4$ and $C_{FN}=3$.
2. Add a reject cost and calculate all three decision regions.
3. Give a score that orders candidates correctly but is not calibrated.
4. For $k=20$ and $s=0.15$, derive expected survivors and explain why overretrieving $k/s$ is not a guarantee.
5. Construct a ranked list where post-filtering returns zero although five eligible candidates exist below the cutoff.
6. Design a receipt that reports access-filter settlement without leaking the existence of forbidden documents.
7. Write separate tests for citation presence and citation support.

References

1. C. K. Chow. “On Optimum Recognition Error and Reject Tradeoff.” IEEE Transactions on Information Theory 16.1, 1970.
2. Ran El-Yaniv and Yair Wiener. “On the Foundations of Noise-Free Selective Classification.” Journal of Machine Learning Research 11, 2010.
3. Stephen Robertson and Hugo Zaragoza. “The Probabilistic Relevance Framework: BM25 and Beyond.” Foundations and Trends in Information Retrieval 3.4, 2009.
4. National Institute of Standards and Technology. “Artificial Intelligence Risk Management Framework 1.0.” NIST AI 100-1, 2023.