You're viewing the readable version of this site. The interactive extras (search, diagrams, read-aloud) need JavaScript and a current browser. Enable JavaScript; if it is already enabled, update your browser.

Learn · Humans Type Badly

budding

A Ruler for Almost

Edit distance turns “nearly the same” into an optimization problem—but the ruler is only as honest as the costs engraved on it.

search, edit-distance, dynamic-programming, approximate-matching, spelling, algorithms, learn

“The string-to-string correction problem is to determine the distance between two strings as measured by the minimum cost sequence of edit operations.”

— Robert A. Wagner and Michael J. Fischer, The String-to-String Correction Problem

This is the fifteenth chapter in a book about search from first principles. You will define an edit policy, derive the Wagner–Fischer recurrence from the last operation, prove its optimal substructure, reduce its storage to two rows, model gap opening with finite state, stop outside a bounded band, and see why unit costs admit a bit-vector representation. You will then separate candidate generation, ranking, and correction decisions. The next chapter replaces distance minimization with a positional subsequence objective.

Equality has no notion of near

An exact dictionary answers whether appol is present. It cannot say that apple is nearby. To ask that question, first state the allowed operations:

  • delete one source symbol;
  • insert one target symbol; or
  • substitute one source symbol for one target symbol.

Give every operation unit cost. The edit distance d(x,y)d(x,y) is the minimum cost of any operation sequence that turns xx into yy. For appol and apple, one substitution and one insertion suffice, so d2d\leq2. No single permitted edit changes both the fourth character and the length, so d=2d=2.

The qualifier “unit cost” is part of the definition, not an implementation detail. A keyboard-neighbor substitution might cost less than an arbitrary one. Opening a long missing span might cost more than extending it. Those policies define different rulers and can rank candidates differently.

Prediction — compute under the declared operations.

Prefixes expose the recurrence

Let x1xnx_1\ldots x_n and y1ymy_1\ldots y_m be two strings. Define

D[i,j]=d(x1xi,y1yj). D[i,j]=d(x_1\ldots x_i,\;y_1\ldots y_j).

The empty-prefix boundaries are forced:

D[i,0]=i,D[0,j]=j. D[i,0]=i,\qquad D[0,j]=j.

For nonempty prefixes, inspect the last operation of an optimal correction:

D[i,j]=min{D[i1,j]+1delete xi,D[i,j1]+1insert yj,D[i1,j1]+[xiyj]match or substitute. D[i,j]=\min\begin{cases} D[i-1,j]+1 & \text{delete }x_i,\\ D[i,j-1]+1 & \text{insert }y_j,\\ D[i-1,j-1]+[x_i\ne y_j] & \text{match or substitute.} \end{cases}

Here [P][P] is 1 when proposition PP is true and 0 otherwise.

Dynamic programming — solve overlapping subproblems once, then compose their answers. Richard Bellman named the method; Wagner and Fischer applied this prefix recurrence to string correction. Read the primary paper.

Why the recurrence is exact

The recurrence is not merely plausible. Take an optimal sequence correcting the first ii source symbols to the first jj target symbols. Its final action must be one of three kinds.

If it deletes xix_i, everything before that final deletion must optimally correct x1..i1x_{1..i-1} to y1..jy_{1..j}. If a cheaper prefix correction existed, substituting it would make the whole sequence cheaper, contradicting optimality. The insertion case is symmetric. If the final action aligns xix_i with yjy_j, the preceding actions optimally correct both shorter prefixes, and the final alignment costs zero for equality or one for substitution.

Thus every optimal sequence appears in one branch and costs at least the minimum. Conversely, append the named final action to an optimal solution for any branch; each branch constructs a valid whole correction. The minimum is therefore both a lower and an upper bound. They meet.

Filling (n+1)(m+1)(n+1)(m+1) cells takes O(nm)O(nm) time. If only the distance is needed, row ii reads only row i1i-1 and the already-computed left cell. Keep the shorter string across the row and storage falls to O(min(n,m))O(\min(n,m)).

Algorithm — unit-cost edit distance with rolling rows

EDIT-DISTANCE(SOURCE, TARGET)
Input:  strings SOURCE and TARGET, with length(TARGET) ≤ length(SOURCE)
Output: unit-cost edit distance between SOURCE and TARGET

previous  [0, 1, ..., length(TARGET)]
for i  1 to length(SOURCE)
    current[0]  i
    for j  1 to length(TARGET)
        change  0
        if SOURCE[i]  TARGET[j]
            change  1
        current[j]  min(previous[j] + 1, current[j - 1] + 1, previous[j - 1] + change)
    SWAP(previous, current)
return previous[length(TARGET)]

Rolling rows discard the path. If an explanation must highlight the edits, retain predecessor choices, recompute a constrained region, or use a divide-and-conquer traceback. Memory savings and witnesses are separate outputs; an API should say which it returns.

A gap has a beginning

Unit edit distance charges a deletion of length rr exactly rr. In many correction policies, one contiguous missing span is likelier than rr independent omissions. An affine gap cost separates opening and extending:

g(r)=go+(r1)ge. g(r)=g_o+(r-1)g_e.

A direct recurrence asks where every possible gap began, adding a scan over prefix positions. Gotoh's insight is that the past relevant to the next cell has only three states:

  • M[i,j]M[i,j]: the alignment ends with xix_i paired to yjy_j;
  • I[i,j]I[i,j]: it ends inside a gap in the source; and
  • E[i,j]E[i,j]: it ends inside a gap in the target.

For minimization, with substitution cost c(i,j)c(i,j):

I[i,j]=min(M[i,j1]+go,I[i,j1]+ge). I[i,j]=\min(M[i,j-1]+g_o,\;I[i,j-1]+g_e).

E[i,j]=min(M[i1,j]+go,E[i1,j]+ge). E[i,j]=\min(M[i-1,j]+g_o,\;E[i-1,j]+g_e).

M[i,j]=min(M[i1,j1],I[i1,j1],E[i1,j1])+c(i,j). M[i,j]=\min(M[i-1,j-1],I[i-1,j-1],E[i-1,j-1])+c(i,j).

The exact initialization depends on whether a length-one gap costs gog_o or go+geg_o+g_e; state that convention and test it. The three-state formulation replaces a search over all gap beginnings with constant work per cell, restoring O(nm)O(nm) time.

The transferable idea is larger than alignment: if the future needs to know a bounded summary of the past, make that summary an explicit state instead of rescanning history.

A threshold changes the reachable region

A spelling corrector often asks whether d(x,y)kd(x,y)\leq k, not for every possible distance. Any path from (0,0)(0,0) to (i,j)(i,j) needs at least |ij||i-j| insertions or deletions. Therefore cells with |ij|>k|i-j|>k cannot participate in a unit-cost answer at most kk.

Computing only the diagonal band of width 2k+12k+1 reduces work to O(kmin(n,m))O(k\min(n,m)) for the bounded-distance decision, plus boundary handling. A row whose in-band values all exceed kk may settle rejection early because nonnegative future costs cannot repair the excess.

This is safe pruning: the rejected cells have a proved lower bound above the threshold. It is not “usually enough” search.

A transposition defines another problem

Fat fingers often reverse adjacent symbols. Damerau's edit model admits transposition. The optimal-string-alignment shortcut adds a local recurrence for adjacent swaps but forbids a substring from being edited more than once. That restriction makes it easier to compute and means it is not the unrestricted Damerau–Levenshtein distance. In particular, optimal-string-alignment distance can violate the triangle inequality.

Metric — a distance that is nonnegative, symmetric, zero only for identical objects, and obeys the triangle inequality. Metric indexes rely on these laws; a useful similarity score need not satisfy them. Read Damerau's error study.

That distinction matters when a later index prunes by metric balls. A function that looks like a distance but violates a required law can make an optimization silently discard valid answers.

A column can become two bit vectors

Under unit insertion, deletion, and substitution costs, vertically adjacent DP cells differ by only 1-1, 00, or +1+1. Instead of storing every numeric cell, represent the positive and negative differences as bit vectors. Equality masks mark all pattern positions matching the current text symbol. Word-sized Boolean operations and addition propagate a whole column's frontier at once.

Myers's algorithm uses this relocatable bit-vector representation to process a pattern in O(m/w)O(\lceil m/w\rceil) machine words per text symbol, where ww is the word width. For a short dictionary word that fits in one word, one iteration advances all pattern positions together.

Boundary — identify what licenses the representation.

Add a position-dependent bonus, an arbitrary substitution matrix, or affine gap state and the tiny step alphabet may disappear. General-score bit-parallel alignment exists, but it carries more machinery. The honest rule is:

representation eligibility = recurrence + cost laws + required output

The problem name alone licenses nothing. A score-only bit-vector path also does not produce edit locations. If the interface promises a highlighted correction path, account for how that witness is recovered.

Distance generates evidence, not decisions

Consider smthin. A distance-only system may prefer a short common word whose edit path is cheap even when a reader intended something. The distance has not malfunctioned; the product asked it to decide a question it does not answer.

A correction pipeline has three distinct stages:

  1. Generate candidates without losing plausible corrections.
  2. Rank them using edit evidence, corpus frequency, prefix or consonant agreement, corpus-local aliases, and surrounding terms.
  3. Decide whether to preserve the literal query, offer alternatives, apply a replacement, or abstain.

Relevance is not confidence. A candidate may rank first by a narrow margin without earning an automatic rewrite. False correction is asymmetric: changing a rare valid identifier can be worse than returning no suggestion.

Keep the literal query as an alternative. Learn abbreviations within the searched corpus rather than importing a timeless universal slang list. Record the edit policy, threshold, candidate source, features, margin, and decision in the query receipt.

Negative results belong beside the winner

Several tempting shortcuts fail for different reasons:

Shortcut Why it looked plausible Counterexample
sort candidates by raw distance one simple number a rare short word beats the intended frequent expansion
divide distance by candidate length compare across lengths changes the objective and can favor long distractors
always take the nearest candidate top one sounds decisive ties and small margins carry little confidence
add transposition to the name only users swap letters recurrence and metric laws change
use the bit-vector path for every policy it is fast on unit costs position bonuses destroy the bounded difference representation
discard the matrix implementation the fast path passed examples a plausible missing candidate has no visible witness

Keep the exhaustive DP as the differential oracle. Generate short strings, cost policies within the supported regime, and thresholds; require the banded or bit-parallel result to equal the oracle. Retain the smallest counterexample when it does not.

Lessons

  • “Almost” is defined by operations and costs, not by an algorithm's name.
  • Wagner–Fischer follows from the exhaustive cases for an optimal final edit.
  • Rolling rows preserve the distance but discard the correction path.
  • Affine gaps become quadratic-time manageable when gap history is summarized by a finite state.
  • A threshold creates a proved diagonal band; it does not justify heuristic truncation elsewhere.
  • Transposition variants have different semantics and laws.
  • Bit-vector speed is licensed by unit-cost differences and score-only output.
  • Candidate generation, ranking, and correction decisions are separate jobs.
  • The simple DP remains the executable specification for every faster path.

Practice

Transfer — select the algorithm from the requested observation.
  1. Fill the complete unit-cost matrix for kitten and sitting, then trace one optimal correction.
  2. Prove that |ij||i-j| is a lower bound on any unit-cost path to cell (i,j)(i,j).
  3. Change substitution cost to two. Which recurrence lines change, and does the rolling-row proof still hold?
  4. Choose go=3g_o=3 and ge=1g_e=1. Compare one gap of length four with four gaps of length one under the stated convention.
  5. Construct a correction policy with a word-boundary bonus and explain why a unit-cost bit-vector result is not an oracle for it.
  6. Design an abstention test using the top two candidate scores. State which empirical calibration it still needs before publication.

References

  1. Robert A. Wagner and Michael J. Fischer. “The String-to-String Correction Problem.” Journal of the ACM 21(1), 1974.
  2. Osamu Gotoh. “An Improved Algorithm for Matching Biological Sequences.” Journal of Molecular Biology 162(3), 1982.
  3. Esko Ukkonen. “Finding Approximate Patterns in Strings.” Journal of Algorithms 6(1), 1985.
  4. Fred J. Damerau. “A Technique for Computer Detection and Correction of Spelling Errors.” Communications of the ACM 7(3), 1964.
  5. Gene Myers. “A Fast Bit-Vector Algorithm for Approximate String Matching Based on Dynamic Programming.” Journal of the ACM 46(3), 1999.