Firefox Build System: Present and Future

Gregory Szorc
gps@mozilla.com
November 29, 2012

Agenda

What is the Build System

A subset of the tasks between deciding you want to build Firefox and running it.
Me

The Spectrum Perspective

Realization

  • I want to fix a bug.
  • I want to code a new feature.
  • I want to build it for myself.

Research

Prepare Yourselves

  • Install build dependencies

Grab the Source

  • hg clone https://hg.mozilla.org/mozilla-central
  • git clone git://github.com/mozilla/mozilla-central.git

Configure the Build

  • Create a mozconfig file
  • Configure Mercurial and/or Git

Start a Build

  • ./mach build
  • make -f client.mk

Wait

Relief

It's over!

Testing

  • mochitest
  • reftest
  • xpcshell
  • Marionette
  • ...

Packaging

  • Windows installer
  • OS X dmg
  • Archives (for binaries and even test files)

Development Tasks

  • Create, submit, apply patches
  • Pull new tree revisions
  • Repeat

What I Consider the Build System

Coordinated by a Driver

Dawn of time - 2012: client.mk

2012 and beyond:

mach

What Happens When You Type Build?

1. mozconfig evaluated

2. Verification and Configuration

config.status: Computed Tree Config


      

3. Build Preparation

InputOutput
mozilla-config.h.inmozilla-config.h
autoconf.mk.inautoconf.mk
Makefile.in'sMakefile's
*.gypMakefile's

4. Invoke the Build Backend


      

Example Makefile.in

DEPTH = @DEPTH@
topsrcdir = @top_srcdir@
srcdir = @srcdir@
VPATH = @srcdir@

include $(DEPTH)/config/autoconf.mk

DIRS = tests

CPPSRCS = nsFoo.cpp nsBar.cpp
LOCAL_INCLUDES += -I$(srcdir)/../build

EXTRA_COMPONENTS = toolkitplaces.manifest nsLivemarkService.js

ifdef MOZ_XUL
EXTRA_COMPONENTS += nsPlacesAutoComplete.js nsPlacesAutoComplete.manifest
endif

include $(topsrcdir)/config/rules.mk
        

The Header

DEPTH = @DEPTH@
topsrcdir = @top_srcdir@
srcdir = @srcdir@
VPATH = @srcdir@

include $(DEPTH)/config/autoconf.mk
        

      

The Body

DIRS = tests

CPPSRCS = nsFoo.cpp nsBar.cpp
LOCAL_INCLUDES += -I$(srcdir)/../build

EXTRA_COMPONENTS = toolkitplaces.manifest nsLivemarkService.js

ifdef MOZ_XUL
EXTRA_COMPONENTS += nsPlacesAutoComplete.js nsPlacesAutoComplete.manifest
endif
        

The Footer

include $(topsrcdir)/config/rules.mk
        

      

Tier Traversal

Makefile Map (click me)

Makefile Tree Map

On Recursive Make

Benchmarks

*

Resource Usage - Effects of -j on Clobber Builds

Ubuntu 12.10 x64 on i7-2600K 12GB RAM, 7200 RPM HD, empty page cache
Build Wall Time CPU Time Speedup Overall CPU % CPU Efficiency Read (MB) Write (MB) I/O Wait Read (s?) I/O Wait Write (s?)
-j1 66:10 63:52 1.00x 12.6 96.5% 382 4,898 247 3,321
-j2 38:16 68:44 1.73x 23.2 89.8% 372 4,856 296 2,614
-j3 28:10 70:17 2.35x 32.6 83.2% 370 4,869 317 3,124
-j4 23:39 73:14 2.80x 40.5 77.4% 401 4,861 339 2,920
-j5 23:02 85:38 2.87x 48.6 74.4% 372 4,862 331 3,464
-j6 22:19 97:27 2.96x 56.8 72.8% 372 4,839 342 3,329
-j7 21:49 108:51 3.03x 64.4 71.3% 355 4,828 287 4,000
-j8 21:40 120:35 3.05x 71.8 69.6% 356 4,865 300 4,413
-j9 21:44 121:46 3.04x 71.9 70.0% 355 4,859 306 4,090
-j10 21:25 120:39 3.09x 72.2 70.4% 366 4,832 377 3,781

Resource Usage - Effects of Caches on Clobber Builds

Ubuntu 12.10 x64 on i7-2600K 12GB RAM, 7200 RPM HD. All builds -j8.
Page Cache State ccache State Wall Time Read Bytes (MB) Write Bytes (MB) I/O Wait Read (s) I/O Wait Write (s)
Empty Disabled 21:38 356 4,865 300 4,413
Populated Disabled 20:45 0 4,842 0 4,200
Empty Empty 23:59 361 7,982 425 8,097
Empty Populated 9:34 3,185 4,904 3,401 7,591
Populated Populated 4:04 1 4,317 2 11,921

Resource Usage - Linux Other

Build Wall Time CPU % Read Bytes Write Bytes I/O Wait Read I/O Wait Write
-j8 21:40 71.8 356 4,865 300 4,413
-j12 RAM disk 21:12 75.2 N/A N/A N/A N/A
GCC 4.7 -j8 24:11 72.2 353 5,242 359 3,904
2GB RAM -j12 26:35 59.82 4,681MB 5,252MB 1,495 4,837
-j12 (Flush Page Cache Every 30s) 62:39 51.71 15,455MB 5,392MB 19,375 3,767

Resource Usage - Try Servers

All builds performed using -j12 without ccache enabled
Type Host Wall Time CPU % Read MB Write MB Read Wait Write Wait
Linux64 #1 bld-centos6-hp-032 19:32 82.5 50 4,351 12 6,641
Linux64 #2 bld-centos6-hp-029 19:36 82.5 45 4,452 17 8,722
Linux64 #3 try-linux64-ec2-316 26:17 61.3 111 4,421 43 10,830
Linux64 #4 try-linux64-ec2-607 26:26 61.2 47 4,423 34 16,902
OS X 64 #1 bld-lion-r5-036 21:42 75.9 374 3,669 91 265
OS X 64 #2 bld-lion-r5-026 21:55 75.8 366 3,385 99 321
OS X 64 #3 bld-lion-r5-022 21:30 74.8 368 3,373 129 339
OS X 64 #4 bld-lion-r5-020 21:34 75.5 378 3,380 109 309

Resource Usage - 2011 MacBook Pro

2011 MacBook Pro, 8GB RAM, factory magnetic hard drive
Build Wall Time CPU % Read Bytes Write Bytes I/O Wait Read I/O Wait Write
-j4 26:33 46.48 738MB 3,674MB 223 2,023
-j12 #1 23:19 73.85 1000MB 3,710MB 270 1,998
-j12 #2 22:44 74.03 705MB 3,582MB 271 2,106

Pretty Graphs

Build Observations

Where Do the Resources Go?

Why We Build Slow: Things in Your Control

Build System Recommendations

Why We Build Slow: Things (Likely) Not in Your Control

Dependency Hell


      

Thoughts on ccache

What Should We Optimize?

Takeaways / Reflection Point

More Make Non-Awesomeness

Goals for a Better Build System

All problems in computer science can be solved by another level of indirection.

Goodbye Makefile.in, Hello moz.build

DIRS += [
    'external',
    'component',
    'bug656331_component',
    'component_no_aslr',
]

if CONFIG.OS_ARCH == 'WINNT':
    DIRS += ['windows']

if CONFIG.DEHYDRA_PATH:
    DIRS += ['static-checker']

CPP_SOURCES = ['nsFoo.cpp', 'nsBar.cpp']

add_xpcom_manifest('MyXPCOMService.manifest', test_only=True)
        

Python With a Twist

moz.build Example: Locals and Globals

local_variable = 'dir_a'

DIRS = [local_variable]
        

moz.build Example: Non-Available Built-ins

# Runtime error because import is not available
import os

# Runtime error because OrderedDict is not exposed to the sandbox.
d = OrderedDict()

# Runtime error because open() is not exposed to the sandbox.
fh = open('xpcshell.ini')
        

moz.build Example: Bad Variable Access

# Runtime error because DIRS expects a list, not a string.
DIRS = 'dir_a'
        
# Runtime error because FOO is not a known UPPERCASE variable.
FOO = 'bar'
        
# Runtime error because BAR is not a known UPPERCASE variable.
local_variable = BAR
      

ZOMG Useful Error Messages!

==============================
ERROR PROCESSING MOZBUILD FILE
==============================

The error occurred while processing the following file:

    /home/gps/src/mozilla-central/services/moz.build

The error was triggered on line 6 of that file.

The underlying problem is an attempt to write a reserved UPPERCASE
variable that does not exist.

The variable write causing the error is:

    FOO

Please change the file to not use this variable.

You can view the list of UPPERCASE variables by running:

    ./mach mozbuild-reference
        

So What Can We Do?

moz.build Traversal

{
  DIRS: ['dir_a', 'dir_b'],
  CPP_SOURCES: ['nsFoo.cpp', 'nsBar.cpp'],
  XPCOM_MANIFEST_FILES: {
    MyXPCOMService.manifest: {test_only: True}
  },
}
          

Processing the Output

Enter Consumers

from mozbuild.backend.base import BuildDataConsumer

class MyBuildDataConsumer(BuildDataConsumer):
    """Writes the set of all XPCOM component files to a text file."""

    def __init__(self):
        self.component_files = set()

    def handle_xpcom_component(self, component):
        """Called for every XPCOMComponent instance in the output stream."""
        self.component_files.add(component.path)

    def finished(self):
        """Called when all emitted classes have been consumed."""
        with open('all_xpcom_manifest_files.txt', 'w') as fh:
            for path in self.component_files:
                fh.write(path + '\n')
        

Build Backends

Other Consumer Possibilities

More Radical Ideas for moz.build Files

moz.build Transition Plan

Implications of Transition

moz.build When?

Concluding with Build Config

Any questions?

Python Versions Today

Python Versions Tomorrow

mozboot

$ wget https://hg.mozilla.org/.../bootstrap.py && python bootstrap.py

Or if you have the tree:

$ ./mach bootstrap 
[I] just started using mach today. I approve heartily. Way less voodoo.
mconley

Mach

Advantages of mach

Usability Comparison: Run Single xpcshell Test

What make mach
Command $ SOLO_FILE=test_load_modules.js make -C services/sync/tests check-one $ ./mach xpcshell-test services/common/tests/unit/test_load_modules.js
Required Knowledge
  • services/common/tests/unit/test_load_modules.js is an xpcshell test file
  • make target to run a single xpcshell test is check-one
  • Must define SOLO_FILE variable to run an individual test
  • Must execute make within the directory containing the XPCSHELL_TESTS variable
  • services/common/tests/unit/test_load_modules.js is an xpcshell test file
  • mach command to run an xpcshell test is xpcshell-test
Misc
  • No make help
  • Realistically requires docs on MDN
  • mach help can educate!
  • Shell tab-complete test paths!
  • You can do everything with just the tree

Even Better Usability

$ mach test test_load_modules.js

Ever Betterer Usability

$ mach test_load_modules.js

mach Opens Doors

mach Ideas

Adding a mach Command

from __future__ import print_function, unicode_literals

from mach.decorators import (
    CommandArgument,
    CommandProvider,
    Command,
)

@CommandProvider
class MyCommandClass(object):
    @Command('doit', help='Run it!')
    @CommandArgument('--debug', '-d', action='store_true',
        help='Do it in debug mode.')
    def doit(self, debug=False):
        print('I did it!')

See MDN for more.

Other In-Progress Improvements

Potential Improvements and Ideas

Vision: Bootstrap

Linux, OS X, BSDs, etc

wget https://www.mozilla.org/build_firefox.py && python build_firefox.py

Windows

Download and run https://www.mozilla.org/build_firefox.vbs

Vision: Development Workflow

We activate a shell tied to a source tree and an output directory:

./mach activate obj-firefox
mach build
# wait through faster build, complete with progress indicator

mach run
# Firefox starts!

Future Crazy Ideas

How Can I Help?

It's Over!