Learn · Time and Senses
budding
What the Index Must Not Reveal
Every derived artifact remains at least as restricted as the media that produced it.
“The central component of the model is a lattice structure derived from the security classes.”
— Dorothy E. Denning, A Lattice Model of Secure Information Flow
This is the thirty-fifth and final chapter in a book about search from first principles. You will distinguish access decisions from information flow; label captions, transcripts, thumbnails, fingerprints, embeddings, intervals, indexes, caches, and counts; prove that compositions of label-monotone stages cannot downgrade confidentiality; construct leaks caused by post-filtering; and design revocation, deletion, cache partitioning, and audit receipts. The chapter returns to the book's opening index: every index is prepaid knowledge, so its contents and observables must be authorized before the future query arrives.
The result row is only the last observable
Suppose principal Alice may read a private video and Bob may not. A captioning model derives “merger blueprint” from the video. The final results page removes the video for Bob, yet four disclosures remain possible:
- autocomplete proposes
merger blueprintonly while the video is indexed; - the result count changes from zero to one before filtering;
- a shared thumbnail cache has a hit instead of a miss; or
- latency changes because a forbidden candidate undergoes expensive reranking.
Authorization applied only to final rows is therefore insufficient. Candidate generation, fusion, snippets, previews, completions, counts, caches, and timing are observable parts of search.
Access control and information flow differ
Attribute-based access control decides whether a subject may perform an operation on an object under policy, attributes, and perhaps environmental conditions. NIST SP 800-162 supplies that general vocabulary. Search also needs to track what information a permitted operation derives and where it flows.
Let confidentiality labels form a partially ordered set ((L,\sqsubseteq)), where
means (b) is at least as restrictive as (a). The join (a\sqcup b) is the least label restrictive enough to protect information from both inputs.
For a transformation (f) consuming labeled inputs (x_1,\ldots,x_n), require
The stage may preserve or strengthen confidentiality; it may not downgrade it. A generated caption from a private video remains private. A fused score using a public title and private transcript is at least as restricted as the private transcript.
An information-flow label records restrictions that must follow data through derivation. Dorothy E. Denning's 1976 lattice model gives the foundational mathematical treatment of security classes and permitted flows.
Monotone stages compose
Consider two label-monotone transformations (f) and (g). For input (x),
and
By transitivity,
Induction extends the proof to every finite pipeline of monotone stages. This is the useful end-to-end law: extraction, analysis, indexing, retrieval, fusion, preview, and caching cannot downgrade confidentiality when each local edge satisfies the same order.
The proof does not establish that a real system assigned labels correctly, that every channel is modeled, or that the training process was authorized. Those are separate evidence obligations. The law makes a class of downgrades structurally testable once the boundaries are complete.
Derived artifacts are still information
The source label follows:
- authored and generated captions;
- OCR tokens and transcript spans;
- thumbnails, crops, and previews;
- content digests and perceptual fingerprints;
- feature vectors and quantized codes;
- temporal segments and keyframes;
- postings, graph nodes, centroids, and fused ranks;
- completions, facets, counts, and explanations; and
- cache keys, values, membership, and timing behavior.
A vector may be hard for a person to read directly, but it was derived to preserve information useful for comparison. Treating it as harmless because it “is only numbers” confuses representation with confidentiality.
Tenant-scoped or policy-scoped indexes can make eligibility structural. Shared indexes require an execution design that admits only eligible candidates without revealing forbidden membership through work or observables. Which is appropriate depends on corpus size, sharing policy, update rate, and the threat model; neither architecture earns safety by name alone.
Revocation and deletion traverse the derivation graph
Revoking Alice's access changes future eligibility but does not automatically erase derived artifacts. Deleting a source requires locating every live descendant: extracts, vectors, postings, graph entries, previews, caches, replicas, queues, logs, and backups under their retention policy.
Maintain a derivation receipt with source identity and revision, artifact kind and generation, transformation identity, input labels, output label, storage locations, and deletion state. The graph supports targeted rebuild and audit. It is not a promise of instantaneous physical erasure from media whose runtime or backup system cannot provide that property.
Revocation races with in-flight queries. A query receipt therefore names the authorization snapshot or generation under which candidates were admitted. Before publication, the system validates that this authority is still current or settles as revoked rather than returning stale evidence.
Algorithm — label-monotone derivation and publication
DERIVE-AUTHORIZED(PRINCIPAL, INPUTS, TRANSFORM, AUTHORITY, LIMIT)
Input: labeled INPUTS, bounded TRANSFORM, current AUTHORITY, positive LIMIT
Output: published artifact or typed refusal
for each input in INPUTS
if not MAY-READ(AUTHORITY, PRINCIPAL, input)
return REFUSED-WITHOUT-MEMBERSHIP
requiredLabel ← JOIN-LABELS(INPUTS)
artifact ← RUN-BOUNDED(TRANSFORM, INPUTS, LIMIT)
if not AT-LEAST-AS-RESTRICTIVE(artifact.label, requiredLabel)
return LABEL-VIOLATION
receipt ← RECORD-DERIVATION(INPUTS, artifact, TRANSFORM, AUTHORITY.generation)
if not AUTHORITY-STILL-CURRENT(PRINCIPAL, INPUTS, AUTHORITY)
return REVOKED
return PUBLISH(artifact, receipt)The refusal does not reveal which input failed. Operational diagnostics may be available to a separately authorized auditor, but the requesting principal does not gain a membership oracle.
Wrong turns
Filter after top-k
Forbidden candidates consume capacity and alter counts, fused ranks, latency, and cache state before removal. Rank within the eligible universe.
Hash private identifiers into public cache keys
Hashing obscures spelling, not membership. A shared observer can probe or correlate deterministic keys. Partition caches by authority and label, and protect cache observables.
Downgrade because a model output is uncertain
Uncertainty concerns truth, not confidentiality. A possibly wrong caption can still reveal its private source.
Promise deletion by removing one result row
The derivation graph contains many descendants and storage lifecycles. Deletion needs a settlement over all declared locations, including honest residuals.
Train first and add access control later
Application filtering cannot undo unauthorized training or reliably remove an item's influence from a trained model. Training authorization, provenance, retention, and removal need their own lifecycle.
The index at greater depth
The opening chapter called a book's index a bargain made before the reader arrives: someone turns documents inside out so a future question can begin with the known coordinate. Thirty-four chapters added analysis, intersection, ranking, skipping, compression, approximation, concurrency, evaluation, machine readers, media, and time.
The final law changes none of that machinery. It completes the bargain. An index is prepaid knowledge, and every observable it prepares must be knowledge the future reader is permitted to receive. Fast search is the art of not looking—but trustworthy search is also the discipline of not revealing what the questioner may not know.
Lessons
- Final-row filtering is not end-to-end search authorization.
- Access decisions and information-flow labels solve related but different problems.
- Every derived artifact remains at least as restricted as all its inputs.
- Label-monotone transformations compose by transitivity.
- Counts, completions, cache state, and timing are observables.
- Eligibility holds before candidate generation and fusion.
- Revocation and deletion traverse every declared descendant and generation.
- Query publication revalidates authority after in-flight work.
- Application filtering cannot repair unauthorized model training.
- An index prepares knowledge before the reader arrives; authorization governs that preparation as well as its final display.
Practice
- Prove composition for three label-monotone stages by induction.
- Compute the label join for public title, team transcript, and private crop.
- Construct a result-count leak despite correct final-row filtering.
- Design cache partition keys that do not cross two tenant authorities.
- List descendants that source deletion must settle for one video.
- Explain why an embedding retains the source label.
- State what the monotonicity proof does not establish about training data.
- Return to any earlier chapter and add authorization to its result receipt.
References
- Dorothy E. Denning. “A Lattice Model of Secure Information Flow.” Communications of the ACM 19.5, 1976.
- Vincent C. Hu et al. “Guide to Attribute Based Access Control Definition and Considerations.” NIST SP 800-162, updated, 2019.
- Butler W. Lampson. “A Note on the Confinement Problem.” Communications of the ACM 16.10, 1973.