Learn · Learning Bazel
seedling
The Makefile That Lies
Five lines of Make, one undeclared input, and a build that reports success while handing you the wrong answer — the failure every other chapter in this book exists to make impossible.
The problem is not in make at all, but rather in the way make is being used.
— Peter Miller, "Recursive Make Considered Harmful," AUUG'97; AUUGN Journal of AUUG Inc. 19(1), 1998
By the end of this chapter you will have built something wrong on purpose and watched the tool congratulate you. You will have seen the same omission come back as a race that only appears when you build on more than one core, watched three makes read one file two different ways, seen a build turn on a character you cannot see, and looked at the npm script that has no dependency line at all. You will be able to say exactly which piece of information was missing in each case, and why no amount of care with the recipe could have saved you. That missing piece is the subject of this entire book.
Type this
Make a directory and put three files in it. Two are data:
greeting.txt
Hello The first half of the message.
name.txt
world And the second half.
The third is the build:
Makefile
message.txt: greeting.txt
cat greeting.txt name.txt > message.txt
clean:
rm -f message.txt Five lines. The first says: to build message.txt, you need greeting.txt, and here is the command. The command reads both files.
Read the first line again, slowly. To the left of the colon is what gets built. To the right is what it depends on: greeting.txt. The indented line below is the recipe — the shell command that does the work. The recipe reads greeting.txt and name.txt.
One of those two files is on the dependency line. The other is not.
Watch it work
Run it:
$ make
cat greeting.txt name.txt > message.txt
$ cat message.txt
Hello
worldThe first build. Make runs the recipe, both files are read, and the message is correct.
Correct. Both files were read, both halves are there. Run it again and Make does nothing, because nothing changed — which is the entire point of a build system and the reason we put up with them.
Watch it lie
Now change the name:
$ echo everybody > name.txt
$ make
make: `message.txt' is up to date.
$ cat message.txt
Hello
worldThe lie. name.txt changed, the recipe reads name.txt, and Make reports there is nothing to do — while message.txt still holds the old answer.
Stop and look at that for a moment.
You edited a file. The recipe reads that file. The output does not reflect the edit. Make exited zero and told you, in plain English, that there was nothing to do.
There is no error here to find. No warning appeared. No log line hints that something was skipped. If message.txt were a compiled binary instead of a text file, you would now be running last week's code and wondering why your change had no effect — and the harder you looked at your change, the less you would find, because your change is fine.
This is the failure mode that produces the phrase "try a clean build." It works, which is why people keep saying it, and it teaches nobody anything about why the incremental build was wrong.
The state persists. Run make again tomorrow: still up to date, still wrong. Only make clean, or deleting the file by hand, breaks the spell.
What actually went wrong
It is tempting to call this a Make bug, and it is worth resisting that, because the mistake is more interesting than a bug.
Make did exactly what you asked. You told it message.txt depends on greeting.txt. It compared their timestamps, found the target newer than its one declared prerequisite, and correctly concluded there was nothing to do. Every step of that reasoning is sound. The reasoning just ran on a dependency graph missing an edge.
Make knows what you declared. It does not know what your recipe read. When those two disagree, the build's answer is wrong and its report is confident.
That gap — between the declared inputs and the real ones — is where stale builds live. It cannot be closed by being careful, because being careful is exactly what fails: you write the recipe, you get it right, and then six months later someone adds a #include, or an import, or a config file read at startup, and the declaration silently stops matching the reality. Nobody edited the dependency line, because nobody had to. That is why it goes unnoticed.
Watch it race
Everything so far ran one command at a time. Real builds do not, and the moment they stop, the same missing edge stops handing you a stale answer and starts handing you a different answer on every run.
Start a second directory beside the first. greeting.txt is unchanged. The other half of the message now arrives differently:
who.txt
everybody The word itself, in a file of its own — because in a real build the thing you edit and the thing a step reads are rarely the same file.
Makefile
all: name.txt message.txt
name.txt: who.txt
sleep 0.2; cp who.txt name.txt
message.txt: greeting.txt
cat greeting.txt name.txt > message.txt
clean:
rm -f message.txt name.txt name.txt is no longer typed by hand; it is generated from who.txt. That is the only change — one input to the message is now the output of another step, which is what every real build looks like.
message.txt still declares only greeting.txt. Its recipe still reads name.txt. The omission from the first act is exactly where it was.
$ make
sleep 0.2; cp who.txt name.txt
cat greeting.txt name.txt > message.txt
$ cat message.txt
Hello
everybodyOne job at a time. Both steps run, in the right order, and the answer is right.
Correct — and it is worth being precise about why. all lists its two prerequisites in an order, Make walks that list left to right, so name.txt gets built before the step that reads it. Nothing made that happen except the order you happened to type on the first line. You did not declare it, and Make did not work it out. It came out right the way a coin comes up heads.
Now build it the way anybody builds anything that takes longer than a minute.
$ make clean
rm -f message.txt name.txt
$ make -j2
sleep 0.2; cp who.txt name.txt
cat greeting.txt name.txt > message.txt
cat: name.txt: No such file or directory
make: *** [message.txt] Error 1
make: *** Waiting for unfinished jobs....Two jobs. The two recipes start together, and the one that reads name.txt gets there first.
Race condition — a defect whose outcome depends on the relative timing of things happening at once, so the same inputs can produce different results on different runs. More.
Make exits 2. Nothing about the sources changed; the only difference is how many things were allowed to happen at once.
This failure is loud, and loud is the good outcome. Here is the bad one. Look at what the failed build left behind:
$ wc -c < message.txt
6
$ cat message.txt
HelloSix bytes. The shell created message.txt for the redirect and cat wrote the first file into it before dying on the second.
Make does not remove a half-written output when a recipe fails, unless you have asked it to with .DELETE_ON_ERROR, and almost nobody has. So the truncated file stays — carrying a modification time of right now.
You already know what happens next.
$ make
make: Nothing to be done for `all'.
$ cat message.txt
HelloThe tree is now permanently wrong, and every build from here agrees it is fine.
Run it again tomorrow: still nothing to be done. message.txt is newer than greeting.txt, which is the only thing Make was ever told to compare it against. A build failed, and the wreckage it left is now indistinguishable, to the tool, from a finished job.
About that sleep
The sleep 0.2 in the producer is there so this fails the same way on your machine as it did on the machine these transcripts came from. Delete it and the outcome depends on which of the two recipes' shells wins the start-up race: usually fine on your laptop, usually fine on your colleague's, wrong once every few dozen runs on the build machine with more cores and less luck.
That version is strictly worse, and that version is the one people actually have. A defect that fails one run in thirty does not get diagnosed as a defect. It gets a retry button, and a reputation for flakiness, and a note in the onboarding doc telling new hires to just run it again.
The fix is one word
Makefile.fixed
all: message.txt
name.txt: who.txt
sleep 0.2; cp who.txt name.txt
message.txt: greeting.txt name.txt
cat greeting.txt name.txt > message.txt
clean:
rm -f message.txt name.txt The prerequisite line for message.txt now names the file its recipe actually reads. Nothing else differs.
$ make -f Makefile.fixed clean
rm -f message.txt name.txt
$ make -f Makefile.fixed -j8
sleep 0.2; cp who.txt name.txt
cat greeting.txt name.txt > message.txt
$ cat message.txt
Hello
everybodySame two recipes, four times the jobs, and now it is right every time.
The order is no longer something you hoped for. It is something the graph knows, so Make can compute it — and having computed it, can safely run everything the graph does not constrain at the same time.
Carry one thing out of this act: -j did not break the build. The graph was already wrong when you ran it serially. Running one job at a time was never a safety property; it was a schedule that happened to hide the defect, and you had no way of knowing you were relying on it.
Which Make?
Everything above has said "Make" as though that named one program.
Here is a Makefile using one small convenience — a variable whose value comes from running a command:
Makefile
WHO != cat who.txt
message.txt: greeting.txt who.txt
cat greeting.txt > message.txt
echo '$(WHO)' >> message.txt
clean:
rm -f message.txt != assigns the output of a command to a variable. One line, and it is enough to make three implementations disagree.
Three makes on one machine is not unusual: macOS ships one, Homebrew installs another, and a third arrives with anything ported from BSD.
$ make --version | head -1
GNU Make 3.81
$ gmake --version | head -1
GNU Make 4.4.1
$ make
cat greeting.txt > message.txt
echo '' >> message.txt
$ gmake
cat greeting.txt > message.txt
echo 'everybody' >> message.txt
$ bmake
cat greeting.txt > message.txt
echo 'everybody' >> message.txtTwo answers, three programs, and not one error message.
The second line of each echoed recipe is the tell: it became echo '' under one make and echo 'everybody' under the other two. All three exited zero.
macOS ships GNU Make 3.81, released in 2006 — the last version under GPLv2, which is why it has been frozen there for two decades. != arrived in GNU Make 4.0. Before that it is not an operator at all, so the line parses as an ordinary assignment to a variable whose name happens to end in a space and an exclamation mark:
$ make -p -n | grep '^WHO'
WHO ! = cat who.txtAsking Make to dump what it parsed. The variable is real, its name has a space in it, and the command in its value was never run.
$(WHO) is therefore undefined, and an undefined variable in Make expands to nothing at all. Not a warning. Not a line on standard error. The word simply leaves the message, and the file is one blank line longer than it looks.
This is the mild version, because != at least means the same thing in modern GNU Make and in BSD make. Other constructs split along different seams entirely: $(shell …) and $(foreach …) are GNU functions that BSD make has never had, .for loops are BSD's and GNU has never had them, and the two spell conditionals differently. There is no single GNU-versus-BSD line to stay on the right side of. Each feature has its own.
So what is anyone saying when they say the build works? That it works with a particular make, of a particular version, resolved out of a particular PATH — none of which appears anywhere in the repository. The dependency graph was missing an edge. This is missing a node: the tool that runs the build is itself an input to the build, and it is no more declared than name.txt was.
The portable answer is to stop using the language:
Makefile.portable
message.txt: greeting.txt who.txt
cat greeting.txt who.txt > message.txt
clean:
rm -f message.txt No variables, no functions, no shell assignment — a prerequisite list and a command.
That genuinely works everywhere, and people genuinely adopt it, and it is worth naming the price out loud: the way to write a portable Makefile is to write as little Makefile as possible.
The character you cannot see
Three failures so far, and every one of them was about information the Makefile did not carry. This one inverts that. The information is present, correct, and unreadable, because the character that carries it is invisible.
Indent a recipe with spaces where a tab belongs:
Makefile
message.txt: greeting.txt who.txt
cat greeting.txt who.txt > message.txt
clean:
rm -f message.txt Four spaces in front of each recipe line — what most editors insert when you press Tab.
$ make
Makefile:3: *** missing separator. Stop.The diagnostic, in full.
Missing which separator? The message does not say "tab". It does not say "spaces". It names line 3, where the recipe is, rather than the character that is absent from it. If you already know the rule you fix this in two seconds. If you do not, there is nothing on screen to search for that will not also return a thousand unrelated results.
Now indent the same line with eight spaces instead of four:
Makefile.eight
message.txt: greeting.txt who.txt
cat greeting.txt who.txt > message.txt
clean:
rm -f message.txt The identical mistake, at a different width.
$ make -f Makefile.eight
Makefile.eight:3: *** missing separator (did you mean TAB instead of 8 spaces?). Stop.Same mistake, same line, and now Make explains itself.
The hint exists, and it is hard-coded to a width of exactly eight. Four spaces — the default in most editors shipping today — gets the bare message; eight gets the answer. GNU Make 3.81 and 4.4.1 print both of these identically, so this is not a version you can upgrade past. Make explains itself to people configured the way people were configured when it was written.
The direction that does not fail
That was the loud one. Whitespace fails in the other direction too, and that is the one that costs an afternoon.
Makefile.silent
LOUD = 1
PUNCT = .
clean:
rm -f message.txt
ifeq ($(LOUD),1)
PUNCT = !
endif
message.txt: greeting.txt who.txt
cat greeting.txt who.txt > message.txt
printf '%s\n' '$(PUNCT)' >> message.txt A conditional, a variable it should set, and one tab that should not be there.
LOUD is 1, so the ifeq is plainly true and PUNCT should become !. Build it:
$ make -f Makefile.silent message.txt
cat greeting.txt who.txt > message.txt
printf '%s\n' '.' >> message.txtExit zero, no diagnostic, wrong file.
PUNCT is still .. The assignment inside the conditional did not happen, and nothing said so.
Nothing on screen mentions the conditional, so the conditional is what you debug. You check that LOUD really is 1. You add $(info ...) around it. You reread the manual section on ifeq, and everything you read agrees with what you wrote. All of that is time spent studying a line that is behaving perfectly, because the ifeq was never a conditional at all: the tab in front of its body made that line a recipe line, and a recipe line belongs to whichever rule came before it in the file.
Here, that is clean. So the report, when it finally arrives, arrives from a target with no connection to any of this:
$ make -f Makefile.silent clean
rm -f message.txt
PUNCT = !
/bin/sh: PUNCT: command not found
make: *** [clean] Error 127The error lands two rules away from its cause.
Delete one tab and the whole thing is correct.
Warning
This one is GNU-only, and not because BSD make gets it right. ifeq is GNU's spelling; bmake rejects the file rather than misreading it. A bug that does not reproduce under the other implementation is still a bug — it just adds itself to the previous section's list.
Where the tab came from
It is worth knowing, because the reason is smaller and more ordinary than the folklore suggests. The story usually told is that Feldman knew the tab was a mistake but already had users. Asked about it directly, he said that was only partly true:
I used tabs because I was trying to use Lex (still in first version) and had trouble with some other patterns… So I gave up on being smart and just used a fixed pattern (^\t) to indicate rules.
A limitation in the lexer he was building with, promoted into the user-facing syntax of the language. The user base came second, and he is clear-eyed about the result: within a few weeks a dozen friends were using Make, he knew the tab was a bad idea, and he did not want to disrupt them. "So instead I wrought havoc on tens of millions."
Which criticism this is
Be precise, because it is easy to file this under the wrong heading.
This is not the dependency model failing. Targets, prerequisites, and recipes are untouched by any of it; you could fix the notation tomorrow and the stale answer and the race would both survive unchanged. Nor is it a dialect problem, though the implementations do disagree about the fallout.
It is a third thing, and the plainest one in the chapter: a notation in which one of the load-bearing tokens is invisible on screen, cannot be distinguished from a token that means something else, and produces either a diagnostic that declines to name it or no diagnostic at all. Every editor that ever helpfully converted a tab to spaces was making a semantic change while showing you an unchanged file.
The lesson generalizes past Make, and it is the one to carry forward: if a build's meaning depends on something a person cannot see, then reviewing that build is not possible, and no amount of care in the review will change that.
The version with no dependency line at all
Most people reading this do not maintain a Makefile. They maintain this:
{
"scripts": {
"build": "npm run js && npm run copy",
"js": "tsc && vite build",
"copy": "cp -r assets dist/"
}
}
The build step of a mid-sized front-end project, in the form it usually takes.
Ask it the two questions this chapter has been asking. What does build produce? Nothing here says. What does it depend on? Nothing here says either.
&& looks like it is doing the dependency line's job, and it is not. It is an ordering, and a weak one: it means the command before it exited zero. It does not say the bundler consumes what the compiler produced, and nothing anywhere checks that it does.
Make's dependency line was incomplete. Here there is no dependency line to be incomplete.
The usual defense is that this cannot go stale the way Make did, because every run does all three steps unconditionally. That is true, and it is not a defense — it is the same missing information with the incrementality removed. The runner cannot tell what any step read, so its only safe strategy is to redo everything, every time, and charge you for it on every commit.
Except that it does not redo everything, because each of those tools keeps a cache the runner cannot see. The compiler has its own incremental build-info file. The bundler has its own cache directory. cp -r writes over dist/ without removing anything that is no longer generated, so an asset deleted six months ago is still shipping. The freshness decisions did not disappear along with the dependency line. They scattered into three tools that cannot see each other, plus an output directory nobody owns.
None of which makes the scripts field a bad thing. It is a good place to write down the commands a project runs, it never claimed to be more than that, and for a project that rebuilds in four seconds it is exactly the right amount of machinery. The difficulty is that it stands where a build system would stand, and no project announces the day it outgrew it.
Two ideas Make got right
Before the rest of this book takes Make apart, it is worth being precise about how much of it is already correct — because the answer is "almost all of it," and the parts that survive are the parts you will recognize in every chapter that follows.
Stuart Feldman's 1978 Bell Laboratories memo “Make — A Program for Maintaining Computer Programs” — published the next year in Software: Practice and Experience 9(4) — describes the whole model in its first pages: a target, the files it depends on, and a command to regenerate it. From those three things you get a dependency graph, and from the graph you get the two ideas that make build systems worth having.
A target is defined by its inputs. Not by a script someone runs in order, but by a declared relationship: this file comes from those files, by this command. Order falls out of the graph rather than being imposed on it, which is why make -j can run independent branches at the same time without you writing a single line about threads. The idea is sound. You have just watched what it does when the graph it is reading from is missing an edge.
Work is skipped when the inputs have not changed. The build asks what is already valid before it asks what to do. Every fast build in existence, including the caching that shows up in chapter twelve, is this idea taken seriously.
Both ideas are load-bearing. Neither is wrong. What this book changes is not the model but a question the model leaves open: who checks that the declared inputs are the real ones?
The two wrongnesses
Make's answer to that question is "nobody," and it produces two distinct failures. You have already seen the first.
Nothing enforces the declaration. The recipe is a shell command. It can read any file on the machine, and Make will never know. Your dependency line is a promise you made, and it is checked by nobody but you. Break the promise and a serial build gives you yesterday's answer; a parallel build gives you a different answer each run.
Freshness is a timestamp, not the content. Make compares modification times. This is why the mistake above is silent — the timestamp on message.txt is genuinely newer than the timestamp on greeting.txt, so by Make's definition the target is genuinely fresh. It is also why the opposite failure happens: touch a file without changing a byte and Make will faithfully redo work that could not possibly produce a different result.
$ touch greeting.txt
$ make
cat greeting.txt name.txt > message.txtBack in the first directory. Not one byte changed, but the timestamp did — so Make redoes work that cannot possibly produce a different result.
Nothing changed. Make rebuilt anyway. Both failures — doing too little and doing too much — come from the same root: Make is reasoning about when files were written instead of what they contain and what was read.
Two wrongnesses, because the model has two ideas and each is bent a little. The dialects are a third thing, and it is worth keeping separate: that one is not a flaw in the model at all. Feldman's model says nothing about which program reads the file. It simply never occurred to anyone to write that down, and the same missing-declaration habit that leaves an input off a prerequisite line leaves the entire toolchain off the repository.
Imagine a recipe card that says "to make soup, you need broth," and the cook also — every time, without fail — adds salt. Nobody wrote salt on the card. Swap the salt for sugar and the card still says the soup is fine. The cook isn't lying and the card isn't broken. The card is just not a complete description of what the cook does.
Where this goes
The fix is not a better Makefile. You cannot write your way out of this, because the failure is not in what you wrote — it is in the fact that nothing checks what you wrote against what actually happens.
The fix is a build system that runs each step somewhere the undeclared file is not present. Not a warning, not a linter: an environment where reading an undeclared input is impossible, because the input isn't there to read. Declare it and the step works. Forget it and the step fails immediately, loudly, on the machine of whoever forgot — rather than silently, months later, on someone else's.
The same answer settles the race, and for the same reason. A step that cannot read what it did not declare cannot accidentally read a file another step is still writing, because the graph that would have to list the file is the graph the scheduler runs on. Correct parallelism is not a feature added on top; it is what you get once the graph is true.
The tool question needs a second mechanism, and it is the smaller of the two: put the compiler in the graph. If the build knows which version of which program ran, along with everything that program read, then "it works on my machine" stops being a category of statement.
That is the whole idea, and everything else in this book falls out of taking it seriously. The next chapter builds the same two-file message under a system that enforces its own graph, in two languages at once — and pins both compilers in the repository, which is where you should notice the answer to which Make? arriving for free. Chapter four goes looking for a file that isn't there.
Make's model is right: targets defined by inputs, work skipped when inputs are unchanged. What Make lacks is any way to know whether your declaration matches what your recipe really read — and when they disagree, you get a wrong answer with a zero exit code, a race as soon as you pass -j, and no way to tell the two apart. The tool that runs the build is undeclared too, so which program is on the PATH is part of your build's meaning. Every mechanism in this book exists to close that one gap.
Try this in your own repository
Reading about a stale build is not the same as finding one. Two things worth doing before the next chapter, both on code you actually own.
Find an undeclared read. Pick a build step in your current project — a Makefile rule, an npm script, a CI job — and write down every file you believe it reads. Then check: run it and watch, or read the tools it invokes and ask what they open. Compilers read include paths and config files. Bundlers read package.json and tsconfig.json and whatever a plugin decided to look at. Test runners read fixtures.
You are looking for one file that is genuinely read and genuinely undeclared. Most projects have several; the interesting question is how long it takes you to find the first one, because that time is a measure of how much your build is currently trusting to luck.
Break something on purpose. Take that undeclared file and change it. Does your build notice? If it does, work out why — something in the chain is watching a directory rather than a declaration, which is luck rather than design and will stop working when the file moves. If it does not, you have reproduced this chapter on your own code, and you now know one specific way your project can hand you a wrong answer.
There is no answer key for these. The point is not to arrive at a result somebody else already knows — it is to find out what is true of your build, which nobody else can tell you.
What you can now do
Name the exact information a build system needs in order to be trustworthy, and recognize the failure when you hit it in the wild: a build that is confidently, persistently wrong until someone deletes something; a build that only fails on the machine with more cores; a build whose behavior depends on which program answered to the name make; a build whose meaning turns on a character nobody can see in a diff; and a build script that never had a dependency line to get wrong.