Mozilla's build system is a black box to many. This post attempts to shed some light onto how it works.

Configuration File

The first part of building is creating a configuration file. This defines what application to build (Firefox, Firefox OS, Fennec, etc) as well as build options, like to create a release or debug build. This step isn't technically required, but most people do it.

Configuration files currently exist as mozconfig files. By default, most people create a .mozconfig file in the root directory of mozilla-central.

Interaction

All interaction with the build system is currently gated through the client.mk file in the root directory of mozilla-central. Although, I'm trying to land an alternate (and eventual replacement) to client.mk called mach. You can read about it in previous posts on this blog.

When you run make -f client.mk, you are invoking the build system and telling it to do whatever it needs to do build the tree.

Running Configure

The first thing client.mk does to a fresh tree is invoke configure. configure is a shell script in the root directory of the repository. It is generated from the checked-in configure.in file using the GNU autoconf utility. I won't go into detail on how autoconf works because I don't have a beard.

configure accomplishes some important tasks.

First, it validates that the build environment is sane. It performs some sanity testing on the directory tree then looks at the system and build configuration to make sure everything should work.

It identifies the active compiler, locations of common tools and utilities, and ensures everything works as needed. It figures out how to convert desired traits into system-specific options. e.g. the exact argument to pass to the compiler to enable warnings.

Once configure determines the environment is sane, it writes out what it learned.

Currently, configure takes what it has learned and invokes the allmakefiles.sh script in the root directory. This script prints out the set of Makefile's that will be used to build the tree for the current configuration. configure takes the output of filenames and then procedes to generate those files.

Generation of Makefile's is rather simple. In the source tree are a bunch of .in files, typically Makefile.in. These contain special markers. configure takes the set of determined configuration variables and performs substitution of the variable markers in the .in files with them. The .in files with variables substitutes are written out in the object directory. There are also some GYP files in the source tree. configure invokes a tool to convert these into Mozilla-style Makefile's.

configure also invokes configure for other managed projects in mozilla-central, such as the SpiderMonkey source in js/src.

configure finishes by writing out other miscellaneous files in the object directory.

Running Make

The next step of the build is running make. client.mk simply points GNU make (or pymake) at the Makefile in the top-level directory of the object directory and essentially says evaluate.

Build System Tiers

The build system is broken up into different tiers. Each tier represents a major phase or product in the build system. Most builds have the following tiers:

1. base - Builds global dependencies
2. nspr - Builds NSPR
3. js - Builds SpiderMonkey
4. platform - Builds the Gecko platform
5. app - Builds the configured application (e.g. Firefox, Fennec, Firefox OS)

Inside each tier are the distinct sub-tiers:

1. export
2. libs
3. tools

A Makefile generally belongs to 1 main tier. Inside Makefile's or in other included .mk files (make files that are not typically called directly by make) are statements which define which directories belong to which tiers. See toolkit-tiers.mk for an example.

When the top-level Makefile is invoked, it iterates through every tier and every sub-tier within it. It starts at the first tier and evaluates the export target on every Makefile/directory defined in it. It then moves on to the libs target then finally the tools target. When it's done with the tools target, it moves on to the next tier and does the same iteration.

For example, we first start by evaluating the export target of the base tier. Then we evaluate base's libs and tools tiers. We then move on to nspr and do the same. And, we keep going. In other words, the build system makes 3 passes through each tier.

Tiers are composed of directory members. e.g. dom or layout. When make descends into a tier member directory, it looks for specially named variables that tell it what sub-directories are also part of this directory. The DIRS variable is the most common. But, we also use TEST_DIRS, PARALLEL_DIRS, TOOL_DIRS, and a few others. make will invoke make for all defined child directories and for the children of the children, and so on. This is what we mean by recursive make. make essentially recurses into directory trees, evaluating all the directories linearly.

Getting back to the tiers, the sub-tiers export, libs, and tools can be thought of as pre-build, build, and post-build events. Although, this analogy is far from perfect.

export generally prepares the object directory for more comprehensive building. It copies C/C++ header files into a unified object directory, generates header files from IDLs files, etc.

libs does most of the work. It compiles C++ code and performs lots of other main work, such as Jar manifest creation.

tools does a lot of miscellaneous work. If you have tests enabled, this is where tests are typically compiled and/or installed, for example.

Processing a Makefile

For each directory inside a tier, make evaluates the Makefile in that directory for the target/sub-tier specified.

The basic gist of Makefile execution is actually pretty simple.

Mozilla's Makefiles typically look like:

DEPTH := .
topsrcdir := @top_srcdir@
srcdir := @srcdir@
VPATH := @srcdir@

include $(DEPTH)/config/autoconf.mk

IDLSRCS := foo.idl bar.idl
CPPSRCS := hello.cpp world.cpp

include $(topsrcdir)/config/rules.mk

All the magic in Makefile processing happens in rules.mk. This make file simply looks for specially named variables (like IDLSRCS or CPPSRCS) and magically converts them into targets for make to evaluate.

In the above sample Makefile, the IDLSRCS variable will result in an implicit export target which copies IDLs into the object directory and compiles them to .h files. CPPSRCS will result in a libs target that results in each .cpp file being compiled into a .o file.

Of course, there is nothing stopping you from defining targets/rules in Makefile's themselves. This practice is actually quite widespread. Unfortunately, it is a bad practice, so you shouldn't do it. The preferred behavior is to define variables in a Makefile and have rules.mk magically provide the make targets/rules to do stuff with them. Bug 769378 tracks fixing this bad practice.

Conclusion

So, there you have it: a very brief overview of how Mozilla's build system works!

In my next post, I will shed some light onto how much times goes into different parts of the build system.