I have very alpha support for Visual Studio project generation for mozilla-central that daring people can dogfood.

I want to emphasize that this is extremely alpha. Normally, I wouldn't release things as fragile as they are. But, I know Windows developers sorely miss Visual Studio, especially IntelliSense. The current Visual Studio projects support IntelliSense, so I want to get this in the hands of Windows developers ASAP.

The current directions for making this all work are a bit hacky. Things will change once things have matured. For now, please excuse the mess.

First, you will need to grab the code. If you use Git, set up a remote to my repository:

git remote add indygreg git://github.com/indygreg/mozilla-central.git
git fetch indygreg

The branch of interest is build-splendid. I periodically rebase this branch on top of master. You have been warned.

You can switch to this branch:

git checkout -b build-splendid indygreg/build-splendid

Alternatively, you can squash it down to a single commit and merge it into your local branch. Once you've done that, you can record the SHA-1 of the commit and cherry-pick that wherever you like!

git merge --squash indygreg/build-splendid
git commit

In the current state, you need to build the tree or the Visual Studio projects will complain about missing files. It doesn't matter if you build the tree before or after Visual Studio projects are generated. But, we might as well get it out of the way. From your MozillaBuild environment, run:

./mach build

That should just work. If it doesn't, you may need to configure mach.ini. See my previous post on how to configure mach.ini. As a reference, my Windows config is:

[build]

configure_extra = --disable-webgl

[compiler]

[paths]
source_directory = c:\dev\src\mozilla-central-git
object_directory = c:\dev\src\mozilla-central-git\objdir

Now, to generate Visual Studio project files:

./mach backendconfig visualstudio

That should take about a minute to finish. When it's done, it should have created objdir/msvc/mozilla.sln. You should be able to load that in Visual Studio!

You will need to regenerate Visual Studio project files when the build config changes. As a rule of thumb, do this every time you pull source. You don't need to perform a full build before you generate Visual Studio files (you do need to perform configure, however). However, if you have not performed a full build, Visual Studio may not be able to find some files, like headers generated from IDLs.

Please close the solution before regenerating the project files. If you don't, Visual Studio puts up a modal dialog for each project file that changed and you have to click through over a hundred of these. It's extremely frustrating. I'm investigating workarounds.

Current State

Currently, it only generates projects for C/C++ compilation (libraries). I still need to add support for IDL, headers, etc. However, each project has proper compiler flags, header search paths, etc. So, IntelliSense is happy and some things do manage to compile!

Many parts are broken and sub-par.

I've only tested on Visual Studio 2008. If you are running Visual Studio \2010, you can try to upgrade the solution. This may work. The backend supports generating solutions for different versions. But, I haven't tested things work on non-2008 and I don't want to expose untested behavior.

Compiling within Visual Studio works for some things. On my system, I get a lot of nullptr not defined errors. I'm not sure why. This will hopefully be worked out soon.

If you do manager to compile within Visual Studio, the output files don't go in the right places. So, if you do a build from the command-line, it will have to re-compile to pick up changes.

Project names are based on the name of the library they produce. I'm not sure if this is the best solution.

Project dependencies are not set up. They will be added later.

Projects for linking libxul or building firefox.exe are not yet provided. Along the same vein, debugging support is not built-in. I'm working on it.

Basically, IntelliSense works. You can now use Visual Studio as a rich editor. Hopefully this is a step in the right direction.

I'm anxious to hear if this works for other people. Please leave comments!

Would you like to build Firefox faster? If so, do the following:

hg qimport http://people.mozilla.org/~gszorc/build-splendid.patch
hg qpush
rm .mozconfig* (you may want to create a backup first)
./mach build

This should just work on OS X, Linux, and other Unix-style systems. Windows support is currently broken, sorry.

mach can do much more than build. Run the following to see what:

./mach --help

Important Info

mach replaces client.mk. mach has its own configuration file. The first time you run mach, it will create the file mach.ini in the same directory as the mach script. This is your new mozconfig file.

The default mach.ini places the object directory into the directory objdir under the top source directory. It also builds an optimized binary without debug info.

Run the following to see which config settings you can add to mach.ini:

./mach settings-create
./mach settings-list

This may fail because I'm still working out the kinks with gettext. If it doesn't work, open python/mozbuild-bs/mozbuild/base.py and search for _register_settings. Open python/mozbuild-bs/mozbuild/locale/en-US/LC_MESSAGES/mozbuild.po for the help messages.

As a point of reference, my mach.ini looks like the following:

[build]
application = browser

configure_extra = --enable-dtrace --enable-tests

[compiler]
cc = /usr/local/llvm/bin/clang
cxx = /usr/local/llvm/bin/clang++

cflags = -fcolor-diagnostics
cxxflags = -fcolor-diagnostics

[paths]
source_directory = /Users/gps/src/mozilla-central-git
object_directory = /Users/gps/src/mozilla-central-git/objdir

I am on OS X and am using a locally-built version of LLVM/Clang, which I have installed to /usr/local/llvm.

You'll notice there are no options to configure make. The patch automatically selects optimal settings for your platform!

Known Issues and Caveats

This is alpha. It works in scenarios in which I have tested it, mainly building the browser application on OS X and Linux. There are many features missing and likely many bugs.

I have been using this as part of my day-to-day development for weeks. However, your mileage may vary.

As stated above, Windows support is lacking. It will appear to work, but things will blow up during building. Don't even try to use it on Windows.

There are likely many bugs. Please don't file Bugzilla bugs, as this isn't part of the build system just yet.

This patch takes over the build system. Do not attempt to use client.mk or run make directly with this patch applied.

If you encounter an issue, your methods of recourse are:

1. Post a comment on this blog post
2. Ping me on irc.mozilla.org. My nick is gps. Try the #buildfaster channel.
3. Send an email to gps@mozilla.com

I am particularly interested in exceptions and build failures.

If you encounter an issue building with this, just reverse the patch and build like you have always done (don't forget to restore your mozconfig file).

If mach.ini does not support everything you were doing in your mozconfig, please send me a copy of your mozconfig so I can implement whatever you need.

Other Info

I will likely write a follow-up post detailing what's going on. If you are curious, the code lives in python/mozbuild-bs. The backend and frontend sub-packages are where the magic is at. Once the backend has been configured, check out hybridmake.mk and all of the splendid.mk files in the object directory.

I am particularly interested in the real-world impact of this patch on people's build times. In this early version of the patch, you likely won't see drastic speed increases. On my MacBook Pro with an SSD, I see end-to-end clobber build times decrease by over a minute. With a little more work, I should be able to shave another minute or two off of that.

I will try to keep the patch up-to-date as I improve the build system. Refresh early and often.

It is hard to believe that my first day as a full-time employee of Mozilla was one year ago today! But, here I am. And, I'd like to think the past year is worth reflecting on.

So, what have I been doing for a year? Good question!

Accomplishments

• First patch and commit to Firefox: bug 673209. Yes, my first patch was to configure.in. It scared me too.
• Number of non-merge commits: 133
• Number of merge commits: 49
• Number of reviews: 31
• Favorite commit message: 88d02a07d390 Fix bug 725478 because we run a buggy Python released almost 5 years ago; r=tigerblood (it was a bustage follow-up).
• Biggest mistake: buildbotcustom change to packaging turned every tree red. All of them. Including release. I think I have philor on record saying it is the worst burnage he's ever seen. Mission accomplished!

Major User-Facing Features:

• Wrote add-on sync for Firefox (with help from many others, especially Dave Townsend and Blair McBride).
• Principle reviewer for Apps in the Cloud (apps sync) in Firefox. (Are reviewers allowed to take credit?)

Honestly, when I see that list, I think that's all? It doesn't feel like a lot, especially since I work on a major user-facing feature (Firefox Sync). I do contribute to more. But, if you ask me what the most significant impact was, I don't think I can call anything else out as major.

There are certainly a lot of minor things:

• Actually managed to understand how Firefox Sync works, including the crypto model. I even documented it! I don't think I knew any of that when I started (I basically knew that Firefox Sync was using strong client-side encryption. What that meant, I had no clue.)
• Filed initial bugs (including some patches) to compile mozilla-central on Windows 8 and MSVC 2011. By the time people started working on Metro development, mozilla-build, configure, and the like already supported building.
• Configured Android Sync Jenkins Builder. I'm told the Android Sync team loves it! You gotta love all the built-in dashboards. I really wish we had this for mozilla-central.
• Made xpcshell test harness write xUnit files. Machine readable output of test results, baby!
• Implement send tab to device API in Firefox Sync. Too bad we don't have UX to use it :(
• Implemented testing-only JavaScript modules. So, instead of the hack that is [head] files in tests, you can Cu.import("resource://testing-common/foo.js");
• Made test package generation quieter. This made build logs about 30% smaller. This means you can get at build results faster.
• Hacked around with JavaScript code coverage using the JS Debugger API. Failed in the process. But, I learned a lot about the JS Debugger API, so I consider it a win.
• Rewrote Firefox Sync's record reconciling algorithm. The old one was full of subtle data loss corner cases. The new algorithm is much more robust (we think - even with 3 reviewers there was still some head scratching).
• Emancipated lots of generic JavaScript code from the Sync tree into the a services-common package. This anticipated Apps in the Cloud and notifications' requirement to use this functionality and allowed those projects to get going quicker. At this point, we're effectively running a mini Toolkit. We really need to port some of this upstream.
• Helped design the next version of the HTTP service to be used by Sync.
• Implemented a standalone JavaScript implementation of the above. The production server used by Mozilla runs on Python (running the Python server in Mozilla's test environment would be too difficult). The server was also implemented using just the spec. This allowed us to clarify many parts of the spec. The Python functional tests can also be executed against the JS server. This gives us confidence that functionality is equivalent and tests hitting the test/JS server will behave the same as if they are hitting the production Python server.
• Implemented a standalone JavaScript client for the above service. Previously, all the logic for this was scattered over about a dozen files and was notoriously difficult to audit and update. I also think it is a decent example of good code. Clean. Highly documented. No hacks.
• Reviewed a skeleton for the notifications service, which will eventually power browser notifications in Firefox.
• Build system patches to better support Clang. Thankfully, Rafael Espíndola has been our Clang champion as of late and is now ensuring the bleeding edge of Clang does not break the tree. Thanks, Rafael! (I actually think he is in the process of switching our OS X builds to use Clang as I type this!)
• Worked with security and crypto people to devise the security model behind the next version of Firefox Sync. (Brian Warner and Ben Adida have been doing most of the crypto work. I'm mostly on the sidelines making sure they design a system that can easily interop with Sync.)
• Helped devise the next version of Sync's server-side storage format. This will make the next version of Sync faster and able to hold more data at lower server cost.
• Gave lots of love to documentation at https://docs.services.mozilla.com/ (especially the Sync docs). It's almost at the point where others can implement Sync clients without having to ask one of the fewer than 10 people on the planet who actually know.
• Contributed many small patches to the build system. Mostly general cleanup so far. Although, I have much bigger plans in the works.
• Many miscellaneous small things. (I get distracted easily.)

Well, that list doesn't seem too shabby. But, a lot of it is smaller ticket items. I don't think there's anything there worth writing home about. Whatever. The future is looking bright for Firefox Sync (Persona integration will make Sync usable by millions more) and new sync backends are coming (including search engine sync). So, I'm fine with not having a longer list of big ticket contributions after only a year.

On Ramping up at Mozilla

I will be the first to admit that I had a difficult time getting into the groove at Mozilla and my first months (dare I say my first half year) were unproductive by my personal standards.

I can't say I wasn't warned. My manager (Mike Connor) told me multiple times that it would happen. I was skeptical, insteading rationalizing that my previous track record of learnly quickly would hold. He was right. I was wrong. I got fat from the humble pie.

There are a few reasons for this. For starters, I was starting a new job. It's almost impossible achieve 100% productivity on your first day. Second, I was working with tools I knew little about, including JavaScript and especially including the flavor of JavaScript used inside Firefox. (Short version: the JavaScript within Firefox is awesome in that it implements bleeding-edge features of the language. Unfortunately, the JavaScript inside Firefox/Gecko is contaminated by this blight called XPCOM. It makes things ugly and not very JavaScript-y. XPCOM also makes the learning curve that much harder because now you have to learn multiple technologies at the same time.) It was daunting.

Not helping matters was the fact that Firefox Sync is complicated. Syncing data is in of itself a difficult problem. Throw in remote servers, an HTTP protocol, a encryption, and interaction with systems inside Firefox that are themselves complicated, and you have a hard problem. My first months were literally spent being a thorn in Philipp von Wieter^H^H^H^H^H^H philikon's side, barraging him with an endless stream of questions. I am forever in beer debt to him because of this. When Philipp left the team to work on Boot 2 Gecko and the rest of the Firefox Sync team was retasked to work on Android Sync shortly thereafter, I was on my own little island to manage Firefox Sync. I kind of felt like Tom Hanks' character in Castaway.

If I have one piece of advice for people starting at Mozilla it's this: be prepared to be humbled by your own ignorance. There is a lot to learn. It's not easy. Don't feel bad when you struggle. The payoff is worth it.

On Life at Mozilla

Despite the hurdles I initially faced ramping up at Mozilla, life at Mozilla is great. This mostly stems from the people, of course.

If you are just looking for technical excellence, I think Mozilla has one of the highest concentrations of any company in the world. Sure, larger companies will have more amazing individuals. But, the number per capita at Mozilla is just staggering. I don't know how many times I've met or talked with someone only to find out later they are considered to be one of the best in his or her respective field. Reading Mozilla's phonebook is like looking at a Who's Who list. For someone like me who loves being a sponge for knowledge, Mozilla is an environment in which I can thrive. Just thinking back at everything I've learned in the past year makes my mind asplode.

On the personal front, the personalities of Mozillians are also top notch. People are generally helpful and supportive. (They need to be for an open source project to thrive.) People recognize good ideas when they hear them and there is comparatively few political battles to be won when enacting change. People themselves are generally interesting and passionate about the world and the work they do. If you are living inside the Mozilla bubble, you may not realize how lucky you have it. I could give specific examples, but I'd be writing all night. Just take my word for it.

If you need something to whet your appetite, just check out the zaniness that is Mozilla Memes. I don't expect you to understand many of the posts unless you are a Mozillian or follower of Reddit and know what internet memes are. But, if you are either, it pretty much sums up a large part of the culture for you. Sometimes I feel like I'm living in one giant, happy meme.

One of the aspects I love most about working at Mozilla is I finally feel that my career interests are aligned with an organization I philosophically agree with. Just read the Mozilla Manifesto. What's not to like?

This is one of the primary factors that convinced me to join Mozilla. After Microsoft acquired the startup where I had my first post-college job (Tellme Networks), I could never hold my head high in Silicon Valley among my friends in the tech sector. Normal people and those outside of Silicon Valley were like, "Microsoft, cool!" But, something just didn't feel right about saying I worked for them. I felt like I was working for the Empire while I really wanted to be working for the Rebel Alliance. I felt like I had to atone for my time at Microsoft. I felt like I needed to cleanse my soul. Mozilla was an obvious answer.

(I don't mean to disparage Microsoft. I actually think the culture has changed since the days when their behavior earned them the reputation that Silicon Valley stills holds them accountable for. Still, I would not work for them in Silicon Valley. Anyway, I'm not here to talk about the past.)

Mozilla is an organization I'm proud to work for. I exercise that pride by frequently wearing my awesome Firefox hoodie. Nearly every time I do, random people come up to me and say how they love Firefox and/or what Mozilla does for the world. Every time they do, it brings a smile to my face. This constantly reinforces what I know to be true: that I'm working for a great organization.

Future at Mozilla

I'm already looking forward to the next year at Mozilla. It is already shaping up to be much, much more productive than my first.

On the roadmap, all of my hacking about with the build system is about to pay dividends. Ever since my first day at Mozilla I have been frustrated with the build system and the developer experience one must go through to contribute to Firefox. After many months of casual (mostly evenings and weekends) experimentation, my work is about to pay off.

I have successfully formulated a plan of attack and helped convince others this is what we need to do. We have since committed to the fundamental components of that plan and are tracking its progress. (I don't mean to take sole or even primary responsibility for this as the credit resides with a number of people. But, I would like to think that the dozens of times I championed aspects of this plan in IRC and in hallway chats before I was the first person to articulate it in a post helped lay the groundwork for the eventual acceptance of this project.) Once we see progress on this project, great things will come from it. I promise.

My work towards making the build system faster had an unintended consequence: the creation of a new tool that serves as a frontend to the build system. One day, I took a step backwards and realized that the potential for such a tool is much greater than simply interacting with the build system. So, I extracted that work from my build system hacking and polished it up a bit. It is now one review away from landing. When it does, thousands of Firefox developers will have a much better experience when developing Firefox. And, my hope is for many more features to follow to make it even more awesome, especially for first-time contributors. I believe this is important to help advance Mozilla's Mission.

Improving the developer experience of Firefox is exciting and it will likely make a lot of people really happy. But, it's neither the most exciting nor most important project I'll contribute to in the upcoming year. The most exciting and important project for me will be refactoring Firefox Sync to make it faster, more robust, sync more data, and, most importantly, usable by more people.

Firefox Sync stands out from similar products in that it keeps your data safe. Really safe. I blogged about this previously. But, I intentionally kept the tone of that post neutral and factual. The truth is that the security model of Firefox Sync makes it look like nearly all other products aren't even trying. I take immense pride in working on a data-sharing feature that makes users' lives better without undermining security. Firefox Sync stands in rare company in this regard.

Unfortunately, in our zeal for the best security possible, we designed a product that isn't usable by the majority of people because it is too complicated to set up and is prone to losing your data. In the end, this doesn't really serve the overall Firefox user base.

We've been hard at work devising the next version of Firefox Sync which will retain the optimum security and privacy settings of the existing product while extending usability at nearly-comparable security and ofer data recovery to the vast majority of our users. This is huge.

Yeah, I'm pretty damn excited about my next year at Mozilla.

The other day I looked at a full Firefox build log from TBPL and noticed that ~84,000 of the ~170,000 lines in the log I looked at was output from archive processes. We were printing thousands of lines showing the files that were being added and extracted from the archives that contain test files!

I thought this was wasteful, so I filed bug 757397 and coded up a patch. Ted agreed that these lines were rather worthless and the patch has landed in mozilla-inbound.

The result of the patch is build logs are about half as big in terms of lines. And, it appears at least 500kb is shaved off the compressed log files as well.

The real world impact is you should be able to load build logs from the server faster because they are smaller.

If you were parsing this data before and are impacted by this, please leave a comment on the aforementioned bug and we'll go from there.

Just landed in mozilla-inbound (Firefox's integration tree) is support for test-only JavaScript modules. That is, JavaScript modules that are utilized by just test code. This is being tracked in bug 748490.

The use case for this feature is sharing common test code, mock types, etc between tests. For example, in the Services code, we have a number of mock types (like a JS implementation of the Sync HTTP server) that need to be utilized across sub-modules. With test-only modules, it is now possible to publish these modules to a common location and import them using the familiar Cu.import() syntax. Previously, you had to perform the equivalent of a #include (possibly by utilizing the [head] section of xpcshell.ini files). The previous method of importing is dirty because you pollute the global object. Furthermore, it is really inconvenient when you wish to utilize shared files from different directories. See this file for an example.

The new method of publishing and consuming test-only JavaScript modules is clean and simple. From your Makefile, define TESTING_JS_MODULES to a list of (JavaScript) files to publish. Optionally, define TESTING_JS_MODULE_DIR to the relative path they should be published to. If the directory variable is not defined, they will be published to the root directory. Here is an example Makefile.in:

DEPTH     = ../..
topsrcdir = @top_srcdir@
srcdir    = @srcdir@

include $(DEPTH)/config/autoconf.mk

TESTING_JS_MODULES = mockserver.js common.js
TESTING_JS_MODULE_DIR = foobar

All test modules are installed to a common directory somewhere in the object directory. Where is not relevant. Just know it is outside the normal distribution directory, so the test modules aren't packaged. This common directory is registered with the resource manager under resource://testing/. So, once a build is performed, you can import these files via Components.utils.import():

Cu.import("resource://testing-common/foobar/mockserver.js");

I hope this feature facilitates better reuse of test code. So, next time you are writing test code, please consider writing writing and publishing it as a module so others can utilize it.

One more thing. Currently, integration with the resource manager is only implemented for xpcshell tests. I'd like to see this supported in all the test runners eventually. I implemented xpcshell support because a) that is the test harness I use almost exclusively and b) it is the only one I'm comfortable modifying. If you want to implement support in another test runner, please have a go at it!