Do you run Ubuntu and build Firefox? If so, please check out a mozilla-central revision as of a few hours ago and run mach bootstrap. Among other things, the bootstrapper now installs a modern version of Mercurial (2.7 at the time I wrote this). Ubuntu currently ships 2.2.2 in 12.10 and 13.04. It's even worse for older releases. Mercurial has evolved a lot over the last few months and releases and I don't think you'll be disappointed after upgrading.

If you don't run Ubuntu, you should still run mach bootstrap periodically to ensure your system has the optimal packages for developing Firefox.

Many people have written custom web services related to Mozilla projects. Cool! However, many of these services live on random hosts scattered around the internets. These hosts can have frequent downtime, cost Mozilla contributors time and money to operate, and can be hard to discover and trust (I don't want to give my LDAP credentials to a non-Mozilla server).

Did you know Mozilla operates a Platform-as-a-Service (PaaS) infrastructure that allows Mozillians to host web services and daemons on the internets (similar to how EC2 works). Until recently, I didn't either!

If you are a Mozilla employee, learn more about paas.allizom.org. (To non-employees, unfortunately it appears you can't use this yet. And I'm not sure what the plans for that are.)

This service is actually a Mozilla-hosted Stackato instance. You set up your client with your LDAP credentials, provision services such as PostgreSQL, memcache, MySQL, etc and deploy applications - either web applications or standalone daemons. The runtime environment supports nearly everything under the sun - Python, Node.js, Ruby, Perl, PHP, Java, etc. You write an application, drop a few files in the project to describe how to deploy it to Stackato, push it, and it just works. Web applications even get an HTTPS endpoint with a valid root CA chained certificate!

I recently used this service to power my Django-based automation-dashboard project. Aside from the initial hurdles figuring out how Django works (this was the first Django project I've written) and how to hook Django up to Stackato, I'm pretty impressed with the results. Free web hosting with database access!

For me, the selling point of this service is that apparently the transition plan from Stackato-hosted service to IT-managed service is relatively pain free. Since you must perform all the steps to isolate/package your application for Stackato, it's relatively easy for IT to deploy your application.

I encourage Mozillians to look into using this PaaS service for their Mozilla service hosting needs.

Have you ever wondered what the machines in Mozilla's automation infrastructure are doing all the time? I have. So, I decided to create a visualization of this data. You can find it at http://automation-dashboard.paas.allizom.org/.

When you start looking at the visualizations, you notice something: there's a lot of time when machines aren't doing anything! All that white space between jobs is time machines are not processing jobs. This is capacity Mozilla is failing to utilize.

While some may say Mozilla's automation infrastructure has a load or capacity problem, I say it has an efficiency problem. The average machine in our automation infrastructure is doing work less than 50% of the time during weekdays. Now, some of this might be VMs that are powered off (due to low demand). But considering physical machines are also under-utilized, I'd say it's a global problem.

Oh, and don't get too hung up with a machine-job efficiency metric. While important, it's only part of the problem. When jobs are running, they are typically only using a fraction of the CPU available to them. From data now available in mozharness, we know that many test suites only use 10-15% CPU. If you combine this with sub-50% machine utilization in terms of jobs, I estimate we're only utilizing somewhere between 5-10% of available CPU cycles in our automation infrastructure. We have a magnitude more capacity in the machines we already have. We don't have a capacity problem, we have an efficiency problem. In my opinion we should throw less time and money at new hardware and invest in maximizing the return from what we have.

Edit 2013-09-03

I started a thread about this data in another forum and others have pointed out that the data in this post is incomplete. Notably absent from this data is when on-demand EC2 instances are shut down, when there are or aren't jobs scheduled (if jobs aren't scheduled, poor machine utilization isn't such a big deal), and when some maintainence tasks are performed (e.g. Panda boards are checked for consistency between jobs). While we could probably hook job scheduling data up to the graph and numbers easily, I don't believe data on slave uptime or background tasks is publicly available. Perhaps we should publish this data to facilitate deeper analysis.

A goal of this post was to shed light on how little we utilize some of the machines in our automation infrastructure in order to inspire a conversation and ultimately to address the perceived problem. It was not a goal to point fingers and cast blame. If I inadvertently performed the latter or seemed to jump to conclusions based on incomplete data, I apologize.

I'm a big fan of tools that encourage and/or enforce the following of best practices and that help people become more productive.

One of the tools that Firefox developers interact with nearly daily is Mercurial. As I've observed from coworkers and from community contributors, many don't have Mercurial configured for optimal development. For first-time contributors, this can manifest in patch rejection - an experience that can be embarassing and demotivating. This is frustrating to me because most issues are easily identifiable and correctable. And, when addressed, everyone wins.

Anyway, I'm pleased to announce that there is now a configuration wizard in the Firefox source tree to help with configuring Mercurial. To run it, just type:

./mach mercurial-setup

Currently, it's aimed for first-time contributors. So, it's missing things that more seasoned developers rely on. But you need to start somewhere, right?

Currently, the tool isn't advertised anywhere other than mach help. Please run it and report issues in bug 794580 or file a new report. Once things have baked in, I'd like to add some kind of notification/tips system to mach where it will encourage you to do things like automatically run mach mercurial-setup. Until then, I recommend trying to remember to run mach mercurial-setup every few weeks to ensure your Mercurial environment is up to date and properly configured.

I'd like to thank Nick Alexander for sharing my enthusiasm for helping contributors and for taking the time to review this work.

My Mercurial extension for Firefox development now has an initial implementation of pushlog aggregation and searching.

You first start by synchronizing the pushlog data on Mozilla's servers with the local client:

hg pushlogsync

This takes a while the first time you run it because it has to download all the data. On subsequent runs, it only downloads new data, so it should be much faster.

Then, you can search for the push history of a changeset:

$ hg changesetpushes -a b968708558b9
133967:b968708558b9 Bug 839809:  Make counter-increments and list counting that would go past our internal (int32_t) limit keep the counter at its current value rather than wrapping.  r=dholbert

Per CSS WG resolution regarding counter-styles-3, afternoon of 2013-02-05:
http://krijnhoetmer.nl/irc-logs/css/20130205#l-1590
http://lists.w3.org/Archives/Public/www-style/2013Feb/0392.html

Note that this patch depends on signed integer overflow behavior in C++,
which I believe is portable despite being unspecified.
Tree      Date                Username              Build Info
inbound   2013-02-21T18:12:57 dbaron@mozilla.com    https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=85b91048c1cd
central   2013-02-22T09:43:12 ryanvm@gmail.com      https://tbpl.mozilla.org/?tree=Mozilla-Central&rev=3a7d4085787e
build     2013-02-22T14:31:42 gszorc@mozilla.com    https://tbpl.mozilla.org/?tree=Build-System&rev=3a7d4085787e
fx-team   2013-02-25T01:04:44 ttaubert@mozilla.com  https://tbpl.mozilla.org/?tree=Fx-Team&rev=31466fd86eb7
graphics  2013-02-25T20:02:57 mwoodrow@mozilla.com  https://tbpl.mozilla.org/?tree=Graphics&rev=dcf53b7140cd
ash       2013-02-26T13:50:41 armenzg@mozilla.com   https://tbpl.mozilla.org/?tree=Ash&rev=201b64ad48d8
services  2013-02-28T09:42:45 Ms2ger@gmail.com      https://tbpl.mozilla.org/?tree=Services-Central&rev=31466fd86eb7
aurora    2013-04-01T13:50:56 bbajaj@mozilla.com    https://tbpl.mozilla.org/?tree=Mozilla-Aurora&rev=60a3f369ccf0
beta      2013-05-13T09:59:38 lsblakk@mozilla.com   https://tbpl.mozilla.org/?tree=Mozilla-Beta&rev=60a3f369ccf0
release   2013-06-17T15:53:19 akeybl@mozilla.com    https://tbpl.mozilla.org/?tree=Mozilla-Release&rev=c54e3363712e

(The -a argument prints all trees instead of just the release trees).

I'd like to integrate bug tracking into the mix to facilitate answering questions like when did bug 123456 ride the trains.

I'd also like to integrate release versions and build IDs into the mix. For example, when I look up a changeset, I want to know the first build on the Nightly, Aurora, Beta, and Release channels that change was included in.