build mozilla

Mozilla versus Eclipse build infrastructure

The Mozilla foundation is similar to the Eclipse foundation in many ways.  However, it has a wholly owned non-profit subsidiary, mozilla.com, which has a sizeable engineering and marketing staff
that work on the Mozilla projects themselves. The Eclipse foundation’s revenue comes from membership dues from member companies. Much of Mozilla’s revenue stream arises from search functionally embedded in Firefox that generates cash flow from companies such as Google, Amazon, Yahoo and eBay.  Of course, Mozilla projects provide more general desktop tools, than the specialized tools we provide at Eclipse, so they have a larger user base.

A while ago, I watched a webinar regarding the Mozilla build process.

http://blip.tv/play/hdlkgojVPAI%2Em4v

It has some interesting numbers. Today, Mozilla has

  • 12 release engineers
  • 17 platforms to support
  • 12 branches to build from
  • 1200 build and test machines
  • 3 colos

Each of their builds consumes 12.40 hours of build time, 54.48 hours of test time and 2.79 days of CPU time. Because of the enormous resources that the build consumes, they run a lot of tests in parallel. With 1200 machines you can do that 🙂

They also run many performance tests with each build. Since the hardware must be identical to compare performance results across machines,  350 of 1200 machines are Mac minis.  It’s the lowest common denominator for hardware that can run on Mac, Windows and Linux operating systems.  In addition, they have a small footprint in a colo, and are relatively inexpensive.

Mozilla’s many Mac minis

At Eclipse we have,

The Eclipse project builds for 15 different platforms but we only run JUnit tests on three.   We don’t have the hardware to run tests on all of them.  Windows, Mac and Linux are represent the vast majority of users and many of the platforms have low download numbers.  The SWT team tests all 15 platforms a few times a milestone and ensure that they start up and work as expected.

The build scheduling engine they use at Mozilla is Buildbot, where we use Hudson on eclipse.org servers.  Their build process used to take ten days to make a release available.  This wasn’t really acceptable.  Mozilla have what they call “zero day” or “chem-spill releases” where they must prepare a release in a single day to address a critical issue.  This is really an admirable feat.  During the Callisto release, I recall having to stay up all night to ensure a build completed to ensure a critical fix was available for the release train the next day.  I’m not sure that with the size of the Eclipse release train today, a zero day release would be possible if a lower level project had to be rebuilt.

Mozilla also run tests on a number mobile devices. To reduce interference between these devices, they are stored in an IKEA shoe closet in the colo.

It’s interesting to see how other open source organizations manage their builds.  How do you see the build infrastructure evolving at Eclipse?

11 comments on “Mozilla versus Eclipse build infrastructure

  1. Nice article Kim. Thanks for the insight.

    Like

  2. Wim, glad you enjoyed it. I have hardware envy 🙂

    Like

  3. What's a colo?

    Like

  4. Love the IKEA rack!

    Like

  5. Villane: In this case, it's a room rented in an ISP to store and run your servers. See http://en.wikipedia.org/wiki/Colocation_centre

    Like

  6. The mobile phones are actually in a Faraday cage room in the Mountain View office AFAIK.

    Like

  7. Ben: Yes, I loved the IKEA rack too.
    Shawn: Thanks for the info!

    Like

  8. Congratulations for this article!!
    It has enlighted our build strategy at the company I work for.

    Like

  9. Thanks for the great post Kim!

    With 15 platforms to support, and only 5 build & test machines, I was wondering how Eclipse manages their build process? Do you run your builds in VMs, or have you partitioned the physical cluster for more specialized / dedicated machines?

    Like

  10. Kim nice post!

    I still have to put a post up to wrap up that presentation and put out the sources.

    I am glad you found the presentation interesting. Feel free to drop me any questions.

    Like

  11. David, we actually have hardware at IBM that is used to compile the C code for the fragments (platform specific libraries) for the 15 platforms we support. These are submitted to the repository in binary form by committers and then consumed by the build process.

    Armen – thanks! I really enjoyed your presentation.

    Sorry for the delay in the comments moderation, my blog settings were misconfigured and wasn't emailing me when there were comments available for moderation 🙂

    Like

Leave a comment