As we look ahead to Q4 planning for the Firefox build system, I wanted to take the time to reflect on what was accomplished in Q3 and to simultaneously look forward to Q4 and beyond.

2013 Q3 Build System Improvements

There were notable improvements in the build system during the last quarter.

The issues our customers care most about is speed. Here is a list of accomplishments in that area:

• MOZ_PSEUDO_DERECURSE work to change how make directory traversal works. This enabled the binaries make target, which can do no-op libxul-only builds in just a few seconds. Of all the changes that landed this quarter, this is the most impactful to local build times. This change also enables C++ compilation to scale out to as many cores as you have. Previously, the build system was starved in many parts of the tree when compiling C++. Mike Hommey is responsible for this work. I reviewed most of it.

• WebIDL and IPDL bindings are now compiled in unified mode, reducing compile times and linker memory usage. Nathan Froyd wrote the code. I reviewed the patches.

• XPIDL files are generated much more efficiently. This removed a few minutes of CPU core time from builds. I wrote these patches and Mike Hommey reviewed.

• Increased reliance on install manifests to process file installs. They have drastically reduced the number of processes required to build by performing all actions inside Python processes as system calls and removing the clownshoes of having to delete parts of the object directory at the beginning of builds. When many mochitests were converted to manifests, no-op build times dropped by ~15% on my machine. Many people are responsible for this work. Mike Hommey wrote the original install code for packaging a few months ago. I built in manifest file support, support for symlinks, and made the code a bit more robust and faster. Mike Hommey reviewed these patches.

• Many bugs and issues around dependency files on Windows have been discovered and fixed. These were a common source of clobbers. Mike Hommey found most of these, many during his work to make MOZ_PSEUDO_DERECURSE work.

• The effort to reduce C++ include hell is resulting in significantly shorter incremental builds. While this effort is largely outside the build config module, it is worth mentioning. Ehsan Akhgari is leading this effort. He's been assisted by too many people to mention.

• The build system now has different build modes favoring faster building vs release build options depending on the environment. Mike Hommey wrote most (all?) of the patches.

A number of other non-speed related improvements have been made:

• The build system now monitors resource usage during builds and can graph the results. I wrote the code. Ted Mielczarek, Mike Hommey, and Mike Shal had reviews.

• Support for test manifests has been integrated with the build system. This enabled some build speed wins and is paving the road for better testing UX, such as the automagical mach test command, which will run the appropriate test suite automatically. Multiple people were involved in the work to integrate test manifests with the build system. I wrote the patches. But Ted Mielczarek got primary review. Joel Maher, Jeff Hammel, and Ms2ger provided excellent assistance during the design and implementation phase. The work around mochitest manifests likely wouldn't have happened this quarter if all of us weren't attending an A*Team work week in August.

• There are now in-tree build system docs. They are published automatically. Efforts have been made to purge MDN of cruft. I am responsible for writing the code and most of the docs. Benjamin Smedberg and Mike Shal performed code reviews.

• Improvements have been made to object directory detection in mach. This was commonly a barrier to some users using mach. I am responsible for the code. Nearly every peer has reviewed patches.

• We now require Python 2.7.3 to build, making our future Python 3 compatibility story much easier while eliminating a large class of Python 2.7.2 and below bugs that we constantly found ourselves working around.

• mach bootstrap has grown many new features and should be more robust than ever. There are numerous contributors here, including many community members that have found and fixed bugs and have added support for additional distributions.

• The boilerplate from Makefile.in has disappeared. Mike Hommey is to thank.

• dumbmake integrated with mach. Resulted in friendlier build interface for a nice UX win. Code by Nick Alexander. I reviewed.

• Many variables have been ported from Makefile.in to moz.build. We started Q3 with support for 47 variables and now support 73. We started with 1226 Makefile.in and 1517 moz.build and currently have 941 Makefile.in and 1568 moz.build. Many people contributed to this work. Worth mentioning are Joey Armstrong, Mike Shal, Joshua Cranmer, and Ms2ger.

• Many build actions are moving to Python packages. This enabled pymake inlining (faster builds) and is paving the road towards no .pyc files in the source directory. (pyc files commonly are the source of clobber headaches and make it difficult to efficiently perform builds on read-only filesystems.) I wrote most of the patches and Mike Shal and Mike Hommey reviewed.

• moz.build is now more strict about what it accepts. We check for missing files at config parse time rather than build time, causing errors to surface faster. Many people are responsible for this work. Mike Shal deserves kudos for work around C/C++ file validation.

• mach has been added to the B2G repo. Jonathan Griffin and Andrew Halberstadt drove this.

Current status of the build bystem

Q3 was a very significant quarter for the build system. For the first time in years, we made fundamental changes to how the build system goes about building. The moz.build work to free our build config from the shackles of make files had enabled us to consume that data and do new and novel things with it. This has enabled improvements in build robustness and - most importantly - speed.

This is most evident with the MOZ_PSEUDO_DERECURSE work, which effectively replaces how make traverses directories. The work there has allowed Gecko developers focused on libxul to go from e.g. 50s no-op build times to less than 5s. Combined with optimized building of XPIDL, IPDL, and WebIDL files, processing of file installs via manifests, and C++ header dependency reduction, and a host of other changes, and we are finally turning a corner on build times! Much of this work wouldn't have been possible without moz.build files providing a whole world view of our build config.

The quarter wasn't all roses. Unfortunately, we also broke things. A lot. The total number of required clobbers this quarter grew slightly from 38 in Q2 to 43 in Q3. Many of these clobbers were regressions from supposed improvements to the build system. Too many of these regressions were Windows/pymake only and surely would have been found prior to landing if more build peers were actively building on Windows. There are various reasons we aren't. We should strive to fix them so more build development occurs on Windows and Windows users aren't unfairly punished.

The other class of avoidable clobbers mostly revolves around the theme that the build system is complicated, particularly when it comes to integration with release automation. Build automation has its build logic currently coded in Buildbot config files. This means it's all but impossible for build peers to test and reproduce that build environment and flow without time-intensive, stop-energy abundant excessive try pushes or loading out build slaves. The RelEng effort to extract this code from buildbot to mozharness can't come soon enough. See my overview on how automation works for more.

This quarter, the sheriffs have been filing bugs whenever a clobber is needed. This has surfaced clobber issues to build peers better and I have no doubt their constant pestering caused clobber issues to be resolved sooner. It's a terrific incentive for us to fix the build system.

I have mixed feelings on the personnel/contribution front in Q3. Kyle Huey no longer participates in active build system development or patch review. Ted Mielczarek is also starting to drift away from active coding and review. Although, he does constantly provide knowledge and historical context, so not all is lost. It is disappointing to see fantastic people and contributors no longer actively participating on the coding front. But, I understand the reasons behind it. Mozilla doesn't have a build team with a common manager and decree (a mistake if you ask me). Ted and Kyle are both insanely smart and talented and they work for teams that have other important goals. They've put in their time (and suffering). So I see why they've moved on.

On the plus side, Mike Hommey has been spending a lot more time on build work. He was involved in many of the improvements listed above. Due to review load and Mike's technical brilliance, I don't think many of our accomplishments would have happened without him. If there is one Mozillian who should be commended for build system work in Q3, it should be Mike Hommey.

Q3 also saw the addition of new build peers. Mike Shal is now a full build config module peer. Nick Alexander is now a peer of a submodule covering just the Fennec build system. Aside from his regular patch work, Mike Shal has been developing his review skills and responsibilities. Without him, we would likely be drowning in review requests and bug investigations due to the departures of Kyle and Ted. Nick is already doing what I'd hope he'd do when put in charge of the Fennec build system: looking at a proper build backend for Java (not make) and Eclipse project generation. (I still can't believe many of our Fennec developers code Java in vanilla text editors, not powerful IDEs. If there is one language that would miss IDEs the most, I'd think it would be Java. Anyway.)

There was a steady stream of contributions from people not in the build config module. Joshua Cranmer has been keeping up with moz.build conversions for comm-central. Nathan Froyd and Boris Zbarsky have helped with all kinds of IDL work. Trevor Saunders has helped keep things clean. Ms2ger has been eager to provide assistance through code and reviews. Various community contributors have helped with moz.build conversion patches and improvements to mach and the bootstrapper. Thank you to everyone who contributed last quarter!

Looking to the future

At the beginning of the quarter, I didn't think it would be possible to attain no-op build speeds with make as quickly as make binaries now does. But, Mike Hommey worked some magic and this is now possible. This was a game changer. The code he wrote can be applied to other build actions. And, our other solutions involving moz.build files to autogenerated make files seems to be working pretty well too. This raises some interesting questions with regards to priortization.

Long term, we know we want to move away from make. It is old and clumsy. It's easy to do things wrong. It doesn't scale to handle a single DAG as large as our build system. The latter is particularly important if we are to ever have a build system that doesn't require clobbers periodically.

Up to this point we've prioritized work on moz.build conversion, with the rationale being that it would more soon enable a clean break from make and thus we'd arrive at drastically faster builds sooner. The assumption in that argument was that drastically faster builds weren't attainable with make. Between the directory traversal overhaul and the release of GNU make 4.0 last week (which actually seems to work on Windows, making the pymake slowness a non-issue), the importance of breaking away from make now seems much less pressing.

While we would like to actively move off make, developments in the past few weeks seem to say that we can reassess priorities. I believe that we can drive down no-op builds with make to a time that satisfies many - let's say under 10s to be conservative. Using clever tricks optimizing for common developer workflows, we can probably get that under 5s everywhere, including Windows (people only caring about libxul can get 2.5s on mozilla-central today). This isn't the 250ms we could get with Tup. But it's much better than 45s. If we got there, I don't think many people would be complaining.

So, the big question for goals setting this quarter will be whether we want to focus on a new build backend (likely Tup) or whether we should continue with an emphasis on make. Now, a lot of the work involved applies to both make and any other build backend. But, I have little doubt it would be less overall work to support one build backend (make) than two. On the other hand, we know we want to support multiple build backends eventually. Why wait? In the balance are numerous other projects that have varying impact for developers and release automation. While important in their own right, it is difficult to balance them against build speed. While we could strive towards instantaneous builds, at some point we'll hit good enough and the diminishing returns that accompany them. There is already a small vocal faction advocating for Ninja support, even though it would only decrease no-op libxul build times from ~2.5s to 250ms. While a factor of 10x improvement, I think this is dangerously close to diminishing returns territory and our time investment would be better spent elsehwere. (Of course, once we can support building libxul with Ninja, we could easily get it for Tup. And, I believe Tup wins that tie.). Anyway, I'm sure it will be an interesting discussion!

Whatever the future holds, it was a good quarter for the build system and the future is looking brighter than ever. We have transitioned from a maintain-and-react mode (which I understand has largely been the norm since the dawn of Firefox) to a proactive and future-looking approach that will satisfy the needs of Firefox and its developers for the next ten years. All of this progress is even more impressive when you consider that we still react to an aweful lot of fire drills and unwanted maintenance!

The Firefox build system is improving. I'm as anxioux as you are to see various milestones in terms of build speed and other features. But it's hard work. Wish us luck. Please help out where you can.

When I talk to Mozillians who now work at Facebook, they all say something similar: the tools at Facebook are amazing and put what Mozilla uses to shame. Having worked at places with decent tools, I do know that Mozilla is behind the times. So it was easy for me to dismiss these observations as obvious. But, when you start asking them about specifics, one tool they are quick to name is Phabricator.

For those who don't want to follow the link, Phabricator is your all-in-on software engineering web application and platform. It has issue/bug tracking, code review, repository navigation, wikis, etc. It's the kitchen sink for software lifecycle management. And, it's open source and easy to extend. Sounds promising.

After hearing Facebookers rave about Phabricator and having casually interacted with an instance to review patches to Clang, I was keen to have a closer look.

So, I spent a lazy afternoon and installed Phabricator on an EC2 instance (connection details below). Having interacted with the default setup for a few hours, I can say with high confidence that the Facebookers can back up their claim that Phabricator is pretty damn amazing. My initial impressions are that it's one of those tools that's so good that once you start using it, you'll never tolerate anything subpar again. Again, hearing the Mozillians-turned-Facebookers opine about Bugzilla, code review, etc, I think this initial impression might be right.

In the sections below, I'll talk a bit about Phabricator and why I think it is superior to what Mozilla uses.

Integrated UI

One thing I really love about Phabricator is the integrated UI. Phabricator is essentially a collection of applications tied together by a single web interface. So, everything is readily available under one site/domain/service.

This means everything is easier to find. You search for something and all the different components have their results surfaced. Contrast with Mozilla as of today, where you need separate searches for Bugzilla, MXR/DXR, wiki.mozilla.org, etherpad, MDN, etc.

For IT, it means there is only a single service to run and scale. Less overhead to operate. Less effort to secure.

Arcanist command line tool

Phabricator ships with a command line tool called Arcanist. It is amazing.

Projects put .arcconfig files under version control which contain the settings used to configure Arcanist for that project. They essentially point back towards the Phabricator instance for that project.

You can do a lot with the Arcanism command line interface. Want to create a code review? arc diff. It opens up an editor and allows you to fill in details about the review. It then automatically publishes it for review. No leaving the command line necessary.

You can download and apply patches undergoing code review by specifying their review identifier. e.g. arc patch D1.

You can create tasks for yourself by running arc todo. The tasks are uploaded to Phabricator automatically and show up in your dashboard.

If your repo has configured which tests apply to which paths, arc unit will run all tests impacted by the current patch. (If we ever get around to annotating this, it would significantly reduce load in automation by eliminating unnecessary tests.)

There is an arc land command to land patches on the appropriate landing branch.

Arcanist abstracts the differences between version control systems pretty well. You can use the same commands for Mercurial, Git, or Subversion.

Arcanist is pretty nice. It reminds me a lot of mach and what I originally wanted mach to become with regard to helping to improve developer workflow.

Real Code Review

Phabricator features a real code review tool - Differential.

Things like interdiff actually work. It integrates with the code repository, which means you can expand context right from the review interface. It can detect blocks of code that moved between files. It has a nice file tree for navigating changed files (optional feature). It hides whitespace only changes by default. You can specify a group of people for review and reviews will show up in the dashboard of individuals in that group. It has a flag to say whether a patch underwent linting before upload. (You can configure linting through Arcanist and have your patches linted before upload. If you properly deploy linting, style nits all but go away. As I like to say, every time I write a style nit in code review, my time is wasted and Mozilla is throwing away part of my salary.)

Mozilla is apparently integrating ReviewBoard into Bugzilla. This is good. ReviewBoard's review features are infinitely better than Splinter (Bugzilla's current review interface). However, ReviewBoard is mostly a standalone tool. What makes Phabricator's approach better is that code review integrates tightly with everything else. This enables some pretty awesome scenarios. Keep reading for more.

Auditing

An interesting and potentially very useful feature of Phabricator is auditing. Where Differential handles pre-push code review, the Audit application supports post-push code review. You may say but all changes at Mozilla must go through pre-push code review - there's no use for post-push code review. Yes, but post-push review opens up a whole new realm of possibilities.

It's possible for individual users or projects to set up rules to automatically trigger audits. For example, as build module owner, I'm very interested in ensuring people follow the code review policy of having a build peer review changes to make files. With Phabricator, I was able to easily configure a rule that automatically creates an audit every time a make file has changed but it wasn't signed off by a build peer. Front and center on my dashboard are a list of commits that I should probably look at. Neat!

Audits are potentially useful for reviews that focus less on code. For example, Firefox Health Report and Telemetry are two Firefox features that collect data and send it (anonymously) to Mozilla. Audits could be used to flag a privacy review when probes are changed. We often don't want this privacy review to hold up code landing. But we do want to track that this review occurs before a patch is shipped to our Beta channel users. I think audits would be great for facilitating this.

Of course, there is a web interface to create new audits. Any individual can watch anything they have access to. No process involved. Just log in and create a rule.

Mozilla has audits today, but it isn't user intuitive. They commonly take the form of Bugzilla keywords or whiteboard entries. Phabricator's feature is much more powerful.

Creating rules as a result of actions

Phabricator has an application called Herald that allows users to install rules to do something as a result of an action.

For example, I can create a rule against new review requests that will CC me or automatically add me as a reviewer if a certain file is touched by a patch.

I can have Phabricator email me whenever a specific user makes a commit or when a commit touching a specific file is made.

This feature is self-service. You just open the web interface and program custom rules to your heart's content.

Herald is insanely useful. To my knowledge, Mozilla has nothing quite like it today. I wish we did. If Mozilla doesn't official deploy Phabricator, I may continue to run my own personal instance just so I can have this feature.

Of course, there's an Event Listener API so you can write your own events and rules. The sky is literally the limit here. For example, we could have all uploaded patches automatically submitted to Try so they run the release automation gamut. And, we could have Phabricator's background daemons poll for try results and report them directly in the web interface!

Module ownership defined inside Phabricator

Phabricator has a modules concept built into it called Owners. You create a package that has a single owner and a number of members. This package can be associated with multiple repositories and individual paths inside repositories.

The power of this feature is realized when it is integrated with other components of Phabricator.

For example, there is a checkbox for each package where you can automatically have Phabricator ensure that a package member is involved with code review touching files belonging to a package. It will also automatically report commits not reviewed by members of a package. How insanely useful is that!

Other applications and features.

Phabricator ships with a number of other applications and features.

There's a blogging platform called Phame.

There's a simple wiki called Phriction.

There's a simple polling application called Slowvote.

There's a mechanism to upload and view files.

There's a pastebin-like application called Paste. It integrates with authentication and authorization, so you can limit the audience of pastes.

There's a JSON over HTTP API that allows you to do nearly everything. (I assume this is what Arcanist uses for everything.)

There's a built-in badges mechanism called Tokens.

You can exchange messages and chats with other users (similar to IRC) using Conpherence.

There is a question and answer application called Ponder. People ask questions and other people answer them. You can search questions, of course. This helps build up a searchable knowledge base.

There is a mockups viewing and collaboration tool called Pholio. You can upload mockups as photos and then annotate images through the web interface. e.g. you can create a rectangle and ask a question about what you see in that rectangle.

There is a time tracking application called Phrequent.

Phabricator can receive inbound email to update issues, reviews, etc. You can even file new bugs via email! Having used an email interface to Bugzilla at a previous job, I can say from experience this is extremely useful.

Phabricator allows you to cherry pick which applications you have enabled. Want just code review and repository browsing without issue tracking? You can do that.

Test instance for Mozilla

As I alluded to above, I have installed an instance of Phabricator on EC2. My intention for this instance is for Mozillians to start playing with it and seeing if it's something they'll like.

The test instance is available at http://phabricator.gregoryszorc.com/.

You can log in using OAuth with one of the configured service providers or by creating an account. Please note that I'm only serving on plain HTTP, so your password are sent in the clear.

I have configured a few popular Mozilla repositories with the install. It's easy to configure new repositories. Send me their info and I'll add them!

While the web UI is nice, Arcanist is essential to get the most out of Phabricator. To get arc working with your machine, follow the official instructions to install Arcanist. For your .arcconfig file, use something like:

{
  "project_id": "gecko",
  "conduit_uri": "http://phabricator.gregoryszorc.com/"
}

Then, have Arcanist fetch authentication credentials:

$ arc install-certificate

From there, run arc help and see what you can do!

Of course, you'll need to adjust the project_id when appropriate.

When you use this Phabricator instance, please keep the following in mind:

• I'm hosting this on my own dime and have the most powerful EC2 instance I felt like paying out of pocket. That instance sadly isn't powerful enough. I installed Phabricator on my desktop at home and it flies like a rocket ship. Please don't blame slowness on Phabricator.
• I didn't even attempt to tune the HTTP or MySQL servers yet. Expect more slowness.
• The service and all the data inside could go away at any time. I don't have monitoring of the service (yet). Use at your own risk.
• At the time I wrote this, Phabricator was still indexing some repositories, notably mozilla-central. Anything interacting with source navigation appears to be slow as a result.
• The install is effectively a vanilla, out-of-the-box install. Things like Mozilla styling could presumably be added. There are likely also many sub-par configuration settings. Leave comments and I'll see about changing things.
• The front page insists on a sign-in. However, I believe all of the content should be accessible to the public. Contact me if you know of a way to fix this!
• This could be hooked up with Persona. I couldn't find a Persona auth provider and coding one is beyond my current PHP skills. If someone writes one, let me know and I'll install it!
• I could hook up authentication against Mozilla's LDAP server. However, I don't want your Mozilla credentials going to my personal server nor traveling over the wire in the clear.
• I'm not an expert on Phabricator. I'll try to help where appropriate. But don't count on me being a support desk.

Next steps at Mozilla

How should we use Phabricator at Mozilla? Good question!

I encourage people to play around with my personal instance. Upload some patches. Perform some code reviews. Set up Herald rules to notify you of repository change events. Just keep in mind that I don't guarantee an SLA for my instance. So please don't rely on it.

I anticipate that people will come to the conclusion that Phabricator is awesome and that Mozilla should install an officially supported instance on a mozilla.org domain with proper hardware. I'll likely be reaching out to IT shortly to see how feasible that is. But, I imagine it will get shuffled to the bottom of the priority queue unless people make noise. If you like Phabricator, tell other people about it. Let your management chain know how amazing it is. If enough of or the right people clamor for it, priorities can get changed.

I hope you enjoy Phabricator!

Many people ask/complain about Firefox builds being slow. They are slow. You have every right to complain.

Now in the official build system docs is a page explaining the major reasons why the build system is slow. I encourage everyone to read it. If you have any questions, ping me and I'll update the docs.

Not yet documented there are some new insights we've been getting from Mike Hommey's terrific work to optimize how make traversal works in the current build system.

As part of that work, Mike established a new compile tier in the build system. This tier compiles C/C++ and little else. Before, C/C++ compiling was interleaved with all the other build steps (install JS files, xpt processing, etc), so it was somewhat difficult to isolate the impact of compilation against all other build actions.

Now that we have all compiling happening in one isolated phase, it's pretty obvious that compiling accounts for the majority of CPU and wall time in the build system (at least with non no-op builds). We suspected this before, but we didn't have concrete numbers.

As I've been pulling and rebuilding inbound since this change landed, it's become increasing clear to me the impact of header dependency hell on build times. Even simple changesets touching a few .h files are resulting in hundreds or even thousands of C/C++ files being recompiled. This increases incremental build times from say 1.5 minutes to often over 10 minutes.

With Mike Hommey's work, I feel we finally are on a path to make no-op build times reasonable for many developers. However, until the header dependency graph is significantly reduced, developers will continue to see excessively long build times for incremental builds, especially on pulls. (I notice that my daily inbound pulls take about as long as a clobber build, which means practically every .cpp file is invalidated every day due to .h changes.)

Fortunately, great work has been happening to minimize C/C++ header dependency hell recently. To everyone working on that, everyone who builds Firefox owes you a giant thank you. However, there's much more that can be done. While you've made terrific progress, please don't slow down any time soon.

Do you run Ubuntu and build Firefox? If so, please check out a mozilla-central revision as of a few hours ago and run mach bootstrap. Among other things, the bootstrapper now installs a modern version of Mercurial (2.7 at the time I wrote this). Ubuntu currently ships 2.2.2 in 12.10 and 13.04. It's even worse for older releases. Mercurial has evolved a lot over the last few months and releases and I don't think you'll be disappointed after upgrading.

If you don't run Ubuntu, you should still run mach bootstrap periodically to ensure your system has the optimal packages for developing Firefox.

Many people have written custom web services related to Mozilla projects. Cool! However, many of these services live on random hosts scattered around the internets. These hosts can have frequent downtime, cost Mozilla contributors time and money to operate, and can be hard to discover and trust (I don't want to give my LDAP credentials to a non-Mozilla server).

Did you know Mozilla operates a Platform-as-a-Service (PaaS) infrastructure that allows Mozillians to host web services and daemons on the internets (similar to how EC2 works). Until recently, I didn't either!

If you are a Mozilla employee, learn more about paas.allizom.org. (To non-employees, unfortunately it appears you can't use this yet. And I'm not sure what the plans for that are.)

This service is actually a Mozilla-hosted Stackato instance. You set up your client with your LDAP credentials, provision services such as PostgreSQL, memcache, MySQL, etc and deploy applications - either web applications or standalone daemons. The runtime environment supports nearly everything under the sun - Python, Node.js, Ruby, Perl, PHP, Java, etc. You write an application, drop a few files in the project to describe how to deploy it to Stackato, push it, and it just works. Web applications even get an HTTPS endpoint with a valid root CA chained certificate!

I recently used this service to power my Django-based automation-dashboard project. Aside from the initial hurdles figuring out how Django works (this was the first Django project I've written) and how to hook Django up to Stackato, I'm pretty impressed with the results. Free web hosting with database access!

For me, the selling point of this service is that apparently the transition plan from Stackato-hosted service to IT-managed service is relatively pain free. Since you must perform all the steps to isolate/package your application for Stackato, it's relatively easy for IT to deploy your application.

I encourage Mozillians to look into using this PaaS service for their Mozilla service hosting needs.