You're viewing the readable version of this site. The interactive extras (search, diagrams, read-aloud) need JavaScript and a current browser. Enable JavaScript; if it is already enabled, update your browser.

Learn · Learning Bazel

seedling

Why Did It Rebuild?

The question every build system eventually owes you an answer to — and a demonstration that touching a file does nothing at all.

bazel, debugging, caching, learn

By the end you will be able to ask your build why it did something, read the answer, and explain why a timestamp change is not a change.

The question

Sooner or later a build does something you did not expect. It rebuilds the world after a one-line edit, or it fails to rebuild something you are certain you changed. In most build systems the response is folklore: delete something, try again.

Here you can ask.

Touch a file and nothing happens

Start with a fully built workspace and touch a source file — updating its timestamp without changing a byte:

$ touch api/greeting.schema

$ bazel build //...
INFO: 1 process: 2 action cache hit, 1 internal.

A build with nothing to do. Two steps found their answers already valid; nothing was recompiled.

Compare that with chapter one:

$ touch greeting.txt

$ make
cat greeting.txt name.txt > message.txt

The same gesture, in Make. Not one byte changed, but the timestamp did — so Make redid the work.

Same action, opposite outcomes, and this time the difference favours the newer tool for a reason worth naming.

Make asks when a file was written. This asks what the file contains. A timestamp is a proxy for change, and like most proxies it is wrong in both directions — it fires when nothing changed, and stays quiet when something did.

Chapter one showed the second failure: an edit Make never noticed. This is the first: a change Make invents. Both are gone once the question is about content.

Change something and ask why

Now change the schema for real, and ask the build to explain itself:

$ sed -i '' 's/max_name_length 32/max_name_length 64/' api/greeting.schema

$ bazel build //... --explain=/tmp/explain.txt --verbose_explanations
INFO: 11 processes: 6 action cache hit, 2 internal, 9 darwin-sandbox.

The explain flag writes a log of every action that ran, with the reason it ran.

And read it:

$ grep "Executing action" /tmp/explain.txt
Executing action 'Executing genrule //api:greeting_go':
  action changed since cached execution.
Executing action 'Executing genrule //api:greeting_ts':
  action changed since cached execution.
Executing action 'Transpiling & type-checking TypeScript project //api:api_ts':
  action changed since cached execution.
Executing action 'GoCompilePkg api/api.a':
  action changed since cached execution.
Executing action 'GoCompilePkg server/greeting/greeting.a':
  action changed since cached execution.
Executing action 'GoLink server/server_/server':
  action changed since cached execution.

The explanation. Six actions, each named, each with the reason it could not be served from the cache.

That list is the cascade from chapter thirteen, written down: the schema fed both generators, the generators fed both compilers, and the Go binary relinked. Nothing else ran.

It is also chapter three's graph, explaining itself. The set of actions that reran is exactly the set rdeps would have named — except that this time you did not have to ask in advance.

The rebuild with no diff

The cases so far all had an edit behind them. Here is the one that sends people hunting through git diff for a change that is not there.

Start from a fully built workspace, touch nothing, and add a flag:

$ bazel build //... --compilation_mode=dbg
INFO: 94 processes: 47 action cache hit, 22 internal, 72 darwin-sandbox.

No file changed. Ninety-four processes ran.

Ask why:

$ grep "Executing action" /tmp/explain.txt | sed 's/.*: //' | sort | uniq -c
  89 no entry in the cache (action is new).
   1 unconditional execution is requested.

The reason is not "changed". It is new.

Eighty-nine actions the build had never seen before, from a workspace where nothing was edited.

The flag changed the configuration, and configuration is part of what an action is. Compiling with debug information is not the same command as compiling without it, so it is not the same action, so it has a different key, so nothing in the cache applies. The build is not redoing work — it is doing different work for the first time.

"What changed?" has a larger answer than "which files did I edit." The inputs to an action include the flags it was configured with, so switching a flag can invalidate everything downstream while your version control shows a clean tree.

This is worth recognizing because the reflex it triggers is wrong. Nothing is broken, no cache is misbehaving, and clearing anything will not help — you asked for a different build and got one. Switch back and the original results are still there, waiting, because they were never discarded:

$ bazel build //...
INFO: 3 processes: 138 action cache hit, 2 disk cache hit, 1 internal.

Returning to the default configuration. The earlier work is still cached.

Two configurations, both cached, neither disturbing the other. That is also why a continuous-integration system building with different flags than you do will not warm your cache, however identical the source.

What to reach for, and when

Three flags cover almost every confusing build:

--explain answers "why did this run." Use it when a build does more work than you expected, or when an edit you thought was harmless triggers a rebuild of something distant.

--sandbox_debug answers "what could this step actually see." That is chapter four's flag, and it is what you want when a step fails to find a file that is obviously present.

aquery answers "what will this step actually do." Chapter seven used it to look underneath a rule; it is equally good for checking whether a flag you added reached the command you thought it would.

The common thread: each of these turns a guess into a question with an answer. That is worth insisting on generally — a system you cannot interrogate is a system you end up cargo-culting.

A kitchen that keeps a log saying which dishes it remade and why. When you cannot understand why lunch took so long, you read the log instead of asking around.

Touching a file changes nothing, because freshness is decided by content rather than timestamps — the inverse of chapter one's failure, from the same root cause. --explain names every action that ran and why; --sandbox_debug shows what a step could see; aquery shows what it will do. Between them, "why did my build do that" stops being folklore.

Try this in your own repository

Find a rebuild you cannot explain. Next time your build does more work than you expected, do not clear the cache — find out why first. Whatever tooling you have, spend ten minutes on the question before reaching for the sledgehammer.

Check whether your build can answer at all. If it cannot tell you why it ran a step, that is a property worth knowing about your tools, and it explains why "try a clean build" is the advice your team gives each other.

What you can now do

Ask a build why it did what it did and get a specific answer, and explain to a colleague why touching a file is not a change. One chapter left: what a repository looks like once several people are maintaining it.