Just a few minutes ago, hg.mozilla.org reached an important milestone: deployments are now performed via Ansible from our open source version-control-tools repository instead of via Puppet from Mozilla's private sysadmins repository. This is important for a few reasons.

First, the code behind the operation of hg.mozilla.org is now open source and available for the public to see and change. I strive for my work at Mozilla to be open by default. With hg.mozilla.org's private Puppet repository, people weren't able to see what was going on under the covers. Nor were they empowered to change anything. This may come as a shock, but even I don't have commit privileges to the internal Puppet repository that was previously powering hg.mozilla.org! I did have read access. But any change I wanted to make involved me proxying it through one of two people. It was tedious, made me feel uncomfortable for having to nag people to do my work, and slowed everyone down. We no longer have this problem, thankfully.

Second, having the Ansible code in version-control-tools enables us to use the same operational configuration in production as we do in our Docker test environment. I can now spin up a cluster of Docker containers that behave very similarly to the production servers (which aren't running Docker). This enables us to write end-to-end tests of complex systems running across multiple Docker containers and have relatively high confidence that our production and testing environments behave very similarly. In other words, I can test complex interactions between multiple systems all from my local machine - even from a plane! For example, we can and do test that SSH connections to a simulated production environment running in Docker behave as expected, complete with an OpenSSH server speaking to an OpenLDAP server for SSH public key lookup. While we still have many tests to write, we had no such tests a year ago and every production deployment was a cross-your-fingers type moment. Having comprehensive tests gives us confidence to move fast and not break things.

One year ago, hg.mozilla.org's infrastructure was opaque, didn't have automated tests, and was deployed too seldomly. There was the often correct perception that changing this critical-to-Mozilla service was difficult and slow. Today, things couldn't be more different. The hg.mozilla.org infrastructure is open, we have tests, and we can and do deploy multiple times per day without forward notice and without breaking things. I love this brave new world of open infrastructure and moving fast.