Firefox Build System: Present and Future

Gregory Szorc
gps@mozilla.com
November 29, 2012

Agenda

• Overview of existing build system
• Performance evaluation of existing build system
• Reinventing build configs
• Better tools and automation through mach
• Other build efforts
• Q&A

What is the Build System

• Build System is overloaded
• Different meanings to different people

The Spectrum Perspective

Wait

Relief

It's over!

What I Consider the Build System

• System bootstrapping
• Build configuration (mozconfigs)
• Actual building
• Running tests
• Packaging

What Happens When You Type Build?

1. mozconfig evaluated

• a mozconfig file is any shell script
• mk_add_options registers make statements to be executed by client.mk
• ac_add_options registers arguments to configure
• Everything else sourced as part of configure

2. Verification and Configuration

• autoconf2.13: configure.in -> configure
• configure -> config.status

3. Build Preparation

On Recursive Make

• ~1160 Makefile for a browser build on OS X
• Generally only 1 Makefile active at a time
• Parallel execution (-j) occurs within a single Makefile
• If Makefile has N < -j tasks, CPU starved
• Can't build dependencies not defined in current Makefile

Benchmarks

*

Build Observations

• Builds use a lot of CPU!
• Mostly CPU heavy except in known areas
• ... but we don't scale with # of cores in use!
• Page cache is important
• ccache adds significant I/O
• GCC uses ~10% more CPU, per-process memory, and ~400MB objdir output
• Hyperthreading: meh

Where Do the Resources Go?

• 63:52 i7-2600K core time to clobber
• ~400MB read I/O (srcdir + toolchain)
• ~5GB write I/O (objdir output)
• ~4.2GB in objdir (Clang), 4.4GB (GCC)
• ~4.6GB in page cache (Clang), 4.8GB (GCC)
• ~3.4GB in page cache just for ccache files!
• ~7.3GB in page cache when ccache enabled
• 1-2GB memory linking libXUL
• 30-300MB per compiler process

Why We Build Slow: Things in Your Control

• Your mozconfig. Don't do debug builds unless you need them.
• Your CPU is slow
• Limited memory causes excessive disk I/O
• Slow disk stalls work
• GCC slower and larger memory consumer than Clang
• ccache adds significant I/O load

Build System Recommendations

• Modern CPU with 4+ physical cores (Core i5 Sandy Bridge or newer)
• 8+GB RAM. 12+ if using ccache or not a headless builder
• A real SSD

Why We Build Slow: Things (Likely) Not in Your Control

• C++ takes a lot of CPU to compile. We have a ton of C++
• Build system not invoking enough commands in parallel
• configure is single process (20-40s)
• NSS -j1 (70s)
• linking not performed in parallel (10-120s)
• sqlite.c not compiled in parallel (20s)
• Excessive rebuilding
• New process overhead on Windows
• pymake slower than GNU make

Thoughts on ccache

• In the ideal build system, ccache isn't needed
• ... if you've compiled already, you don't invoke the compiler!
• Fixing excessive rebuilding should marginalize ccache
• ... unless you build multiple trees (RelEng)
• ... unless you toggle between significantly differing trees (some devs)
• Until then, watch out for I/O overhead of ccache
• ... consider disabling on magnetic hard drives

What Should We Optimize?

• Eliminate the troughs (single core bottlenecks)
• Minimize total CPU requirements
• Precompiled headers
• Smarter C++ template usage?
• Other compiler hacks?
• Prevent excessive rebuilding

Takeaways / Reflection Point

• Overall build system structure and process
• Building Firefox takes a lot of resources
• ... and we don't even use everything available!

More Make Non-Awesomeness

• Hard to maintain
• Simple things are hard. Hard things are really hard
• ... in all fairness, simple dependencies are easy!
• mtime is is only out of date detection
• Lots of foot guns
• Difficult to bulk refactor build config
• Recursive make won't scale into the future

Goals for a Better Build System

• Build faster
• Easier to maintain
• Support multiple build backends (make, Tup, Visual Studio, Eclipse, Ninja, etc)
• Foster new awesomeness
• Curb the foot guns

Python With a Twist

• moz.build files are actually Python files
• But running in a highly specialized execution sandbox
• Can't import Python modules
• Can't do I/O
• Python globals/built-ins must be whitelisted to be exposed

moz.build Example: Locals and Globals

local_variable = 'dir_a'

DIRS = [local_variable]
        

• DIRS is available after execution because it is UPPERCASE
• local_variable is local to the execution environment
• DIRS is defined in a central manifest, with all the other UPPERCASE variables

moz.build Example: Non-Available Built-ins

# Runtime error because import is not available
import os

# Runtime error because OrderedDict is not exposed to the sandbox.
d = OrderedDict()

# Runtime error because open() is not exposed to the sandbox.
fh = open('xpcshell.ini')
        

moz.build Example: Bad Variable Access

# Runtime error because DIRS expects a list, not a string.
DIRS = 'dir_a'
        

# Runtime error because FOO is not a known UPPERCASE variable.
FOO = 'bar'
        

# Runtime error because BAR is not a known UPPERCASE variable.
local_variable = BAR
      

So What Can We Do?

• Not that much (at least if you expect a real Python environment)
• And that's the point

moz.build Traversal

• Traversal is like Makefiles. Start at root file, recurse into children.
• Execute each moz.build file in isolation.
• Output of individual moz.build is dictionary of UPPERCASE variable values:

{
  DIRS: ['dir_a', 'dir_b'],
  CPP_SOURCES: ['nsFoo.cpp', 'nsBar.cpp'],
  XPCOM_MANIFEST_FILES: {
    MyXPCOMService.manifest: {test_only: True}
  },
}
          

Processing the Output

• Build config is a stream of dictionaries. 1 dict for each moz.build file.
• ... massaged into Python classes
• ... class instances define specific aspect of the build config or tree. e.g. XPCOMComponent, DirectoryTraversal, CPPCompilation
• Additional validation performed when converting to classes. Missing file detection, enforcement of higher-level rules and conventions, etc.
• Ultimate output is a stream of class instances

Build Backends

• Just a consumer that facilitates building the tree!
• A make build backend writes out Makefiles.
• A Visual Studio backend writes out Visual Studio .vsproj files
• A Tup backend writes out Tupfiles
• moz.build files allow us to more easily create and experiment with build backends
• New build backends will enable faster builds

Other Consumer Possibilities

• Generate Clang compilation databases
• Generate code autocomplete databases
• One-off build backends (static analysis, address sanitizer)
• Capture Valgrind or GDB macros, settings
• Analyze C/C++ with Clang Python bindings
• Build system dependency analysis

More Radical Ideas for moz.build Files

• Replace xpcshell.ini?
• Replace packaging files like removed-files.in, package-manifest.in?
• Record module owners, peers and/or reviewers for directories?
• Define static analysis and auditing policies?
• Define Bugzilla components?
• Any metadata could be captured by moz.build files!

moz.build Transition Plan

• Identify Makefile functionality/variables to be ported
• Remove variables from Makefile.in, add to moz.build
• Implement consumer that outputs a .mk file
• Recursive make build backend works as it does today. Data just comes from moz.build files instead of Makefile.in
• Repeat until everything defined in moz.build files
• Eventually implement alternate build backends (like non-recursive make or Tup)

Implications of Transition

• For recursive make, not much
• As we roll out new backends, make will no longer build the tree
• ... you'll have to do everything through a more intelligent build driver (mach)

moz.build When?

• Patches written and preliminary r+
• DIRS conversion first
• In holding pattern because B2G backport concerns
• Rinse and repeat with many flag days until Makefile.in eliminated
• Bug 784841 tracks

Concluding with Build Config

• Essentially we're creating an API for the build/tree config
• ... and that API will enable awesomeness

Python Versions Today

• Python is our tooling language
• Currently require Python 2.6 to build the tree
• Some test suites may still run on 2.5 (Talos is even 2.4!)
• Version chaos makes authoring difficult, prevents progress
• Difficult to write Python 3 compatible code
• Lack of handy newer language features
• Numerous bugs in older Pythons

Python Versions Tomorrow

• Days to weeks: Python 2.7 to build, preferably 2.7.3
• Weeks to months: Python 2.7 for ALL OF THE THINGS
• Many months: Python 2.7/3.x dual compatibility (no rush)
• A few years: Require Python 3

Mach

• ... is not a build system!
• is a generic driver to common developer tasks
• All your make targets in one place
• Goal: better user experience
• Goal: enable better, faster, stronger (but not harder) tools and automation
• Current state: young, but improving all the time

Advantages of mach

• Python >> make and shell
• Increased tools cohesion
• Friendlier tools = happier developers = increased community engagement

Even Better Usability

$ mach test test_load_modules.js

• mach is in $PATH and you can run it from anywhere
• mach can deduce file is an xpcshell test and it does the right thing
• Requires some improvements to mach (I'm working on them)
• Requires richer and more consistent metadata (moz.build files!)

Ever Betterer Usability

$ mach test_load_modules.js

• Argument is a file!
• That file is an xpcshell test - let me invoke the xpcshell test runner!
• ...And I'll be sure the tree is built before I try it and will either build it for you or tell you what to do in order to build it

mach Opens Doors

• Much more flexible than make
• Deep integration with test runners
• Unified logging bus
• Forcing tools to be written as libraries, not standalone scripts
• Conveniently brokers all interaction and can make intelligent decisions

mach Ideas

• Configure Mercurial and/or Git with sane defaults
• Easily submit [proper] patches to Bugzilla
• Find reviewers for patch
• Incorporate all those great tools that are floating around
• Organize mach commands into categories
• Shell completion support
• Record and report metrics from building
• ... add stuff to moz.build files, consume with mach

Adding a mach Command

from __future__ import print_function, unicode_literals

from mach.decorators import (
    CommandArgument,
    CommandProvider,
    Command,
)

@CommandProvider
class MyCommandClass(object):
    @Command('doit', help='Run it!')
    @CommandArgument('--debug', '-d', action='store_true',
        help='Do it in debug mode.')
    def doit(self, debug=False):
        print('I did it!')

See MDN for more.

Other In-Progress Improvements

• Custom Python importer so tree isn't littered with .pyc files
• Port driver logic from client.mk to Python
• Eliminate client.mk

Potential Improvements and Ideas

• Rewrite parts of configure in Python and/or parallelize
• Replace mozconfigs with moz.build-like files?
• mach config file?
• Streaming logs from TBPL (possibly with metlog)
• Structured logging from mach eliminates log parsing
• Parallel test runners and/or unify test harness code
• Invent MFBT acronym for moz.build files

Future Crazy Ideas

• What about running a background service with an HTML interface?
• web UI easier to target than terminal
• background building of the tree
• push-button interface
• yo dawg, I hear you like to use Firefox to build Firefox

How Can I Help?

• Learn Python
• squash mach bugs
• Write mach commands - incorporate tools into tree
• Give test runners some API love
• #build, #mach on irc.mozilla.org

It's Over!