Ben Hearsum (Posts about release-automation)

A Brief History of Code Signing at Mozilla

Ben Hearsum — Thu, 30 Jan 2025 16:18:15 GMT

Shipping large software to end-user devices is a complicated process. Shipping large software securely to end-user devices is even more complicated. Signing the things that ship to end-user devices is one of those complications, and it gets even more complicated when you sign thousands of artifacts per day.

Mozilla has been signing Firefox in some form beginning with Firefox 1.0. This began with detached GPG signatures for builds, and progressed to Authenticode signing for Windows installers in Firefox 1.0.1. Since then it has evolved over time to encompass other platforms, other types of files within our products, and other ways that we ship (such as our own update packages). This post will provide a overview of the what, when, why, and how of code signing at Mozilla over the past ~20 years.

What, when, and why

Early GPG & Authenticode Signing

When we first began signing, it happened on a Windows machine. Late in the release process, after Windows installers had been built, we would download all of the release artifacts to this machine, sign the Windows installers, and generate detached GPG signatures for those before pushing the artifacts elsewhere.

At this time, the private keys and certificates were held on a USB stick that was kept removed from the machine at-rest. A Release Engineer needed to be physically present in Mountain View to perform this step. Once inserted, signing could be done via Remote Desktop rather than at the physical machine (but don't forget to remove the USB stick afterwards!).

GPG signing was done with the standard GPG tools, running in cygwin. Authenticode signing was also done with the standard (at the time) Microsoft 'signcode.exe' tool. An annoying fact about that tool, is that it only accepted the necessary passphrase from a GUI dialog. To work around this, we had an AutoIt script running in the background that injected the passphrase into this dialog whenever it popped up. This interesting way of automating the process meant that mouse movements or keyboard interaction at the wrong time could interfere with the signing process.

This process was partly scripted, but there was still a series of ~15 commands someone had to run by hand (and not mess up) to get everything done. You can see these commands for yourself in our now-ancient Unified Release Process documentation.

Windows internal file signing

Careful readers may have noted that early Authenticode signing only covered the Firefox installer itself, not the EXEs and DLLs inside of it. At some point (I haven't gone to the effort of tracking down exactly where...) we started signing these inner files as well. This process seems to have been lost to the sands of time, but I seem to recall it worked very similarly to the installer signing process, but without the GPG parts.

Improved signing on Windows

The first notable improvement we had to this process was to automate most of the copy/pasting that was done from the wiki. This came in the form of a Makefile that with a few mere inputs, would download, sign, and re-upload the signed builds. The main benefit of this was reduced opportunity for human error.

Not long after that, we had our first real game-changing improvements. Chris AtLee wrote a nice post about it back in 2009, and how his changes took signing time from 8 hours all the way down to sub-15 minutes. This was accomplished through a combination of faster hardware, parallelized signing, in-process compression & decompression, and better caching. His changes also introduced a hefty amount of python into the signing process, which paved the way for the next big improvement...

Automatic signing

Despite the signing process now being very quick once it gets started, it could still sometimes take hours or longer to begin the process. Typically this would happen if our build and repack processes finished at a time when no Release Engineer was around to begin signing. This was solved with what we called "autosign". Rather than require a Release Engineer to be around at the right moment, we adjusted our scripts to allow them to be started ahead of time, and be smart enough to know when all of the files it needs to sign are ready. This work eliminated all wait time between builds being ready and signing running.

Signing Windows builds...on Linux!

In 2011, signing was rearchitected altogether. In short, the idea was to move signing to a highly secured Linux server, and sign builds through an API as part of the build process. This allowed builds to be signed as they were produced, and reduced the number of times builds had to move from one server to another before they shipped. An obvious question here is how we would manage to sign Windows binaries on Linux...as it turns out, the mono project had its own version of signcode that ran natively on Linux that we were able to make use of.

MAR Signing

Shortly after (and perhaps even motivating - I'm not sure at this point) the aforementioned signing server work, we began signing our MAR (Mozilla ARchive) packages that update users from an older version of Firefox to a newer version. Thanks to the earlier work, it was fairly trivial to use the same architecture to sign these files.

macOS .app signing

The idea of signing .app bundles for macOS was filed all the way back in 2007. There was some initial work on this in 2010, but we were unable to land it at that time. Around the same time that MAR signing was happening in 2012, we picked up this work again and managed to drive it home this time.

Unfortunately, there were no tools available at the time to sign macOS builds on anything except a fairly modern macOS machine. For this reason, we had to run additional copies of our signing server on macOS and sign those builds with them. (If you've ever had to run macOS as a server you'll know just how unfortunate this was...)

Taskcluster/signingscript/iscript

In 2018, Mozilla migrated its CI and Release automation from our aging Buildbot systems to Taskcluster. As part of this, signing tasks moved to specialized Taskcluster workers known as "signingscript" and "iscript", used for signing non-macOS and macOS builds respectively. These specialized workers continued to outsource the actual work of signing to the previously discussed signing servers.

An important part of this change is the introduction of Chain of Trust, a significant security enhancement that helps ensure that only authentic artifacts are signed to this day.

Introducing...Autograph!

Autograph is Mozilla's modern code signing service. It was built specifically to provide a signing service that allowed us to keep private key material in Hardware Security Modules (HSMs). Migrating release signing to it was a huge improvement over the existing signing server where Release Engineers had direct access to such things. It was initially used for signing XPIs and APKs, but by the end of 2019 we had migrated all non-macOS signing to it and retired the old Linux signing servers.

In addition to the security enhancements it brought, we saw great performance wins with it as well, largely in thanks to it's support for only requiring a hash of the bytes being signed to be sent over the wire. (This requires that the client has some advanced knowledge of the file being signed, but it saves a tremendous amount of network traffic at our scale.)

Notarization with rcodesign

In 2023 we started making use of rcodesign to notarize and staple our macOS builds. While actual macOS code signing itself continues to happen on macOS machines, this allowed us to move at least some of our operations into the cloud and reduce our reliance on mac hardware.

Tools and tech

I've mentioned a number of tools and technology that we use as part of signing, but I've purposely glossed over some details in the interest of brevity. The following section is a glossary of sorts, and introduces some more under the hood tools that we use as part of signing. If you're interested in the gory details, the links below should be enough to find them for yourself! Or you can stop by #firefox-ci on Matrix to ask questions!

osslsigncode

osslsigncode is a tool that implements parts of Microsoft's signtool.exe. In the past, we used it to directly sign PE files. These days, we use it's support for attaching the signatures that Autograph makes to them.

winsign

Winsign is a python library for signing and manipulating Authenticode signatures. It relies on osslsigncode for writing signatures, and supports signing directly with a private key, or outsourcing the signing process to a passed in function. The latter is what we use, and it's how we inject a call to Autograph into the signing process.

msix-packaging

In 2021 we began shipping Firefox as an MSIX package. As part of this we discovered that osslsigncode does not support signing MSIX packages. Luckily for us, Microsoft's MSIX packaging tools are open source and run on Linux, and we found a fork that contained most of what was needed to support signing. With a few additional modifications, we were able to support signing these packages in our existing systems.

apple-codesign

apple-codesign is a very exciting project from Gregory Szorc which provides 3rd party tools capable of signing, notarizing, and stapling .app bundles and other Apple formats such as .pkg and .dmg. These tools run on Linux, and as noted above, we're already making use of them to notarize and staple our .app bundles. We're extremely excited about this project, and grateful to Gregory Szorc for all the effort he's bit into it. In the future we're looking forward to migrating our actual code signing to these tools which would (finally) allow us to retire our dedicated macOS signing machines.

mardor

mardor is a python tool to manage, and most importantly, sign, MAR files. In the days before Autograph it was used to directly sign MAR files. These days we only use it to inject signatures made by Autograph into the files, similar to our usage of osslsigncode.

signingscript

signingscript is the glue between our CI system (Taskcluster) and Autograph. Through a combination of the tools listed above, custom code in signingscript itself, and communication with Autograph it produces signed builds. It is additionally responsible for notarizing and stapling our macOS builds.

iscript

iscript is essentially a pared down version of signingscript (in fact their code is both derived from our early signing server code), and is responsible for signing our macOS builds. iscript runs on a small cluster of mac minis, which are a huge pain in the butt to manage.

autograph

As noted in an earlier section Autograph is our modern code signing service. It has a simple HTTP API that accepts signing requests and returns signed data or files. In addition to signing various artifacts that we ship it also makes Content Signatures on behalf of addons.mozilla.org, aus5.mozilla.org/Balrog (our update server), and some other backend services that Firefox communicates with, helping to ensure the security and integrity of requests made between Firefox and Mozilla-run services.

Conclusion

What a ride it's been over the last 20 years! We've gone from signing nothing to signing nearly everything in some form. Signing started off as a very manual process, and now happens seamlessly thousands of times per day.

I don't think it would be possible to name everyone that contributed to this, but it took the ideas and efforts of tens, if not hundreds, of people to get to this point: release engineers, build system experts, security folks, and many others were all critical to getting us where we are today.

I've got this post as brief as possible, but if you're interested in more details on any parts here feel free to reach out!

Release Automation Futures: Seamless integration of manual and automated steps

Ben Hearsum — Thu, 19 Mar 2015 14:30:13 GMT

I've written about the history of our Release Automation systems in the past. We've gone from mostly manual releases to almost completely automated since I joined Mozilla. One thing I haven't talked about before is Ship It - our web tool for kicking off releases:

It may be ugly, but having it has meant that we don't have to log on to a single machine to ship a release. A release engineer doesn't even need to be around to start the release process - Release Management has direct access to Ship It to do it themselves. We're only needed to push releases live, and that's something we'd like to fix as well. We're looking at tackling that and other ancillary issues of releases, such as:

Improving and expanding validation of release automation inputs (revisions, branches, locales, etc.)
Scripting the publishing of Fennec to Google Play
Giving release Release Managers more direct control over updates
Updating metadata (ship dates, versions, locales) about releases
Improving security with better authentication (eg, HSMs or other secondary tokens) and authorization (eg, requiring multiple people to push updates)

Rail and I had a brainstorming session about this yesterday and a theme that kept coming up was that most of the things we want to improve are on the edges of release automation: they happen either before the current automation starts, or after the current automation ends. Everything in this list also needs someone to decide that it needs to happen -- our automation can't make the decision about what revision a release should be built with or when to push it to Google Play - it only knows how to do those things after being told that it should. These points where we jump back and forth between humans and automation are a big rough edge for us right now. The way they're implemented currently is very situation-specific, which means that adding new points of human-automation interaction is slow and full of uncertainty. This is something we need to fix in order to continue to ship as fast and effectively as we do.

We think we've come up a new design that will enable us to deal with all of the current human-automation interactions and any that come up in the future. It consists of three key components:

Workflows

A workflow is a DAG that represents an entire release process. It consists of human steps, automation steps, and potentially other types. An important point about workflows is that they aren't necessarily the same for every release. A Firefox Beta's workflow is different than a Fennec Beta or Firefox Release. The workflow for a Firefox Beta today may look very different than for one a few months from now. The details of a workflow are explicitly not baked into the system - they are part of the data that feeds it. Each node in the DAG will have upstreams, downstreams, and perhaps a list of notifications. The tooling around the workflow will respond to changes in state of each node and determine what can happen next. Much of each workflow will end up being the existing graph of Buildbot builders (eg: this graph of Firefox Beta jobs).

We're hoping to use existing software for this part. We've looked at Amazon's Simple Workflow Service already, but it doesn't support any dependencies between nodes, so we're not sure if it's going to fit the bill. We're also looking at Taskcluster which does do dependency management. If anyone knows of anything else that might be useful here please let know!

Ship It

As well as continuing to provide a human interface, Ship It will be the API between the workflow tool and humans/automation. When new nodes become ready it makes that information available to automation, or gives humans the option to enact them (depending on node type). It also receives state changes of nodes from automation (eg, build completion events). Ship It may also be given the responsibility of enforcing user ACLs.

Release Runner

Release Runner is the binding between Ship It and the backend parts of the automation. When Ship It is showing automation events ready to start, it will poke the right systems to make them go. When those jobs complete, it will send that information back to Ship It.

This will likely be getting a better name.

This design still needs some more thought and review, but we're very excited to be moving towards a world where humans and machines can integrate more seamlessly to get you the latest Firefox hotness more quickly and securely.

More on "How far we've come"

Ben Hearsum — Wed, 04 Jun 2014 17:57:16 GMT

After I posted "How far we've come" this morning a few people expressed interest in what our release process looked like before, and what it looks like now. The earliest recorded release process I know of was called the "Unified Release Process". (I presume "unified" comes from unifying the ways different release engineers did things.) As you can see, it's a very lengthy document, with lots of shell commands to tweak/copy/paste. A lot of the things that get run are actually scripts that wrap some parts of the process - so it's not as bad as it could've been. I was around for much of the improvements to this process. Awhile back I wrote a series of blog posts detailing some of them. For those interested, you can find them here:

I haven't gotten around to writing a new one for the most recent version of the release automation, but if you compare our current Checklist to the old Unified Release Process, I'm sure you can get a sense of how much more efficient it is. Basically, we have push-button releases now. Fill in some basic info, push a button, and a release pops out:

How far we've come

Ben Hearsum — Wed, 04 Jun 2014 12:26:51 GMT

When I joined Mozilla's Release Engineering team (Build & Release at the time) back in 2007, the mechanics of shipping a release were a daunting task with zero automation. My earliest memories of doing releases are ones where I get up early, stay late, and spend my entire day on the release. I logged onto at least 8 different machines to run countless commands, sometimes forgetting to start "screen" and losing work due to a dropped network connection. Last night I had a chat with Nick. When we ended the call I realized that the Firefox 30.0 release builds had started mid-call - completely without us. When I checked my e-mail this morning I found that the rest of the release build process had completed without issue or human intervention. It's easy to get bogged down thinking about current problems. Times like this make me realize that sometimes you just need to sit down and recognize how far you've come.

Contribution opportunity: Release Engineering systems

Ben Hearsum — Mon, 02 Dec 2013 14:35:44 GMT

Release Engineering runs a vast array of infrastructure and systems that do much of the continuous integration and releases for Mozilla. Many of our systems are small in their scope but must be able to scale up to support the incredible load that developers put on them. Other systems receive millions of requests every day from live Firefox, Fennec, and Thunderbird installations. Do you want help developer productivity or get releases into users hands more quickly and efficiently? Do you want to gain experience working on systems that must work at scale? If so, Release Engineering is a great place to look. Below are a few interesting bugs that could use some attention. If you're interested in working on any of them I'm interested in mentoring you. You should be familiar with Python, but you don't need to be an expert. Have a look below and contact me directly if anything interests you.

Partial update generation service: Arguably, updates are the most important part of release process. Partial updates in particular help us keep a good user experience by reducing the amount of data a user needs to download, which means they update more quickly. We generate many of these already but creating this service would allow much more flexibility over what and when we generate partial updates. This project would involve writing the service from scratch, most likely in Python.
Update Balrog schema to support multiple partials: Balrog is the code name of our new update server (which I've previously blogged about). It's original design came about before we supported serving partial updates to users on multiple older versions of Firefox. In order to start using Balrog for Betas and Releases we need to add this feature. Balrog is written in Python and this will mostly involve server side changes to it.
Improve update verify output: "Update verify" is a very important test that we run as part of our release automation. It's job is to make sure that all users, regardless of where they're coming from, end up in the same state after updating to the latest release. It's output currently consists of thousands and thousands of lines of text, with test results interspersed. This bug is about finding and implementing a way to make the output easier for a human to make sense of and parse upon failure. The update verify scripts are written in bash, but this could be implemented by modifying them or post-processing the output.
Store history of machine actions requested through API: We recently deployed a new system that helps us manage our thousands of build and test machines. It aims to be a single entry point for information gathering and common operations on them. Currently, the data in it is volatile -- all history of operations is lost when the server is restarted. This bug will involve adding permanent storage (maybe SQL, maybe something else) to that server, which is written in Python.

Smaller & faster updates now accessible to more Firefox users

Ben Hearsum — Tue, 11 Sep 2012 14:17:36 GMT

When a user receives an update to Firefox they get either a partial or complete. A complete is nearly identical to its associated installer, and can update any old Firefox version. A partial is a binary diff of a specific old version against a newer one and only compatible with that specific old version. The size difference a partial and a complete can be huge. For example, the complete MAR for 14.0.1, en-US, win32 was 20MB. The partial from 13.0.1 was 7.4MB (even smaller on other platforms, where PGO doesn't make diffing hard). Until recently, we've only been able to produce partials against a single old version without a lot of extra time consuming and error prone manual work -- meaning that a lot of users who could benefit from a partial weren't receiving them. With bug 773290 (multiple partial MAR support in release automation) resolved, we can now offer partial updates to many versions without the risk and time cost of doing them by hand. When we shipped 15.0 we offered partial updates to users on 14.0.1, 13.0.1, and 12.0 - which collectively represented just over 75% of our installed userbase. For future releases it's possible we'll offer partials to even more previous versions. I'm sure some of you are asking why we can't just do partial updates for ALL old releases. There's a couple of reasons for that. Most importantly, there's big diminishing returns on partial updates. The 13.0.1 -> 15.0 partial was 12MB, the 12.0 -> 15.0 partial was 14MB, and the complete was 20MB. The further you go back the less you gain from getting a partial. Secondly, computing partial updates is not computationally cheap. We ship 88 locales across 4 platforms -- this works out to about 350 partials that need to be calculated. This can be made cheaper through caching and parallelization, but it still ends up adding about 45min to the running time of the release automation for every extra version we want partials for. In turn, this delays QA and other things in the critical path of shipping. I also want to point out that this work does NOT apply to Nightly or Aurora. We have no plans to offer multiple partial updates on those channels at this time. Due to their relatively low userbase and very high frequency of change (almost every 24h), the cost/benefit just doesn't work. However, we will be looking at using this on Beta where the userbase is much larger and the rate of change slower (about once a week). A huge thanks goes out to Rail Aliev and Nick Thomas, who helped work out the design, wrote some parts of it, and provided reviews. We couldn't have had this ready for 15.0 without their help.

release-automation - Part 3: Improvements & Optimizations (2009 to early 2011)

Ben Hearsum — Thu, 23 Feb 2012 21:50:44 GMT

In my last post I talked about the major project of switching the release-automation from Bootstrap driven by Buildbot to being directly implemented in Buildbot, and working out of Mercurial. After a 6 month break from automation work, there were a few spurts of development on the automation over the course of the next two years. Some of these were big new things, like Fennec automation, while others were deliberate attempts to improve the automation. This post will cover the most important changes that happened from late 2009 all the way through early 2011.

Late 2009 to mid 2010

Fennec release-automation

In 2009 Mozilla began working on a version of Firefox for Maemo. Late in that year, we shipped 1.0rc1 with the release-automation. Some people may be thinking "that doesn't sound very hard, it's just another platform right?". Unfortunately, there's a lot of hidden complexity with adding a new platform that doesn't confirm to long-held assumptions, like mobile. While the actual build process is fairly similar there's a lot of pre-build, post-build, and other things that just aren't the same. Fennec was the first product we supported that was built out of multiple source repositories, which not only caused problems for builds (and isn't handled well by Buildbot), but affected how we tag repositories and generate source tarballs. L10n repacks were also completely different for Fennec: not only did we ship individual l10n builds for many locales, but we also shipped builds with multiple locales in them. Doing this meant build process changes as well as a new format to describe locales, revisions, and which types of repacks each one needed. All of this combined ended up being nearly a month of work (and many late nights, Aki tells me) to get up and running! This was the first product we've ever shipped that had automated releases from the start, which is a huge accomplishment for forwarding thinking & planning - something that we simply didn't have time for in the past. It's hard to determine how many hours of end2end time and # of manual touchpoints this saved since it was never manual work to begin with but there's no doubt that we're far better off with it than without.

Major Update

In the latter half of 2009 we started doing a lot of Major Updates. That is, offering 3.0.x users an update to a 3.5.x release. Behind the scenes, each Major Update offer took approximately 4 hours to create and had at least 6 or 7 manual touchpoints in order to do config file bumping, snippet generation, test snippet pushing, and verification of those snippets. Each one had an end2end time of 4 hours or so and had at least 6 or 7 manual touchpoints. If that wasn't bad enough, a single mistake in the configuration file would cause us to have to restart the entire process! Automating this turned out to be one of the easier pieces of new automation because of how similar Major Updates were to the regular updates we already did with every release. When this relatively simple work was done, all of the manual touchpoints were gone completely and because these were now done automatically with a release instead of out of band they moved out of the critical path and therefore had no end2end time impact either! This is always the best kind of new automation =).

Bouncer Entries

In mid-2010 we automated a long standing annoyance: Bouncer entry creation. Like the Major Updates, this was something that was subject to manual error. More importantly, it was _damn_ annoying to do. Bouncer is the piece of software that powers download.mozilla.org, which redirects download requests to our mirror network. Each time we release we need to tell it where to find the files we ship. This translates to one entry for each installer, complete MAR, and partial MAR for each platform. Prior to this being fixed this was done mostly through copy and paste which has a massive margin for error. In the best case scenario this means we'll get some 404s, which are easy to detect and fix. In the worst case we could point at the wrong release entirely, which is an error that may not get caught at all. Fixing this didn't improve our end2end time at all but it did take away the most annoying manual touchpoint, which we were all very happy about. After this change the automation stayed relatively stable for the next 6 months, with only minor bugfixes happening.

Late 2010 to early 2011

At the end of 2010 and start of 2011 we began a huge round of upgrades and optimizations starting with upgrading to a new version of Buildbot. This work wasn't shinyfun, but long overdue after the regular Continuous Integration infrastructure had upgraded many months prior. After that was done some of us spent the next couple of months working hard on some new automation & improvements. This was one of the most exciting and active times for the release-automation. We lowered end2end time by parallelizing some things, we took away many manual touchpoints with new pieces of automation, and we dramatically improved stability through intelligent retrying of failed operations. Also of note is that went back to a model of having standalone scripts doing work and having Buildbot drive those, not unlike the Buildbot+Bootstrap era. This came about after having a lot of challenges implementing some things directly in Buildbot code, which makes it very difficult to make decisions at runtime, and the feeling that we didn't want to tie ourselves to Buildbot forever.

Source Code Tagging

At the time, source repository tagging was one of the rougher parts of the automation. Not only did it often fail to push tags back to a repository due to losing a push race, but load issues caused us to get server side errors. For a period of time it was rare that a release *didn't* have a problem in tagging. Moving the tagging code to an external script made fixing these errors a lot easier. At the same time, we were able to start building up some very useful libraries for working with Mercurial, retrying failed commands, and other things. Since these changes have landed it's been very rare to have issues with tagging, and most of them have been regressions from recently landed things rather than long standing bugs with the tagging scripts.

L10n Repacks

We used to have similar issues with our l10n repacking logic, too. Sometimes the jobs would die while trying to clone a repository or when trying to download or upload a build. Additionally, we used to use a different Buildbot job for each locale, which meant that we would redo steps like "clone/pull from source repository" for every single locale which was quite inefficient. As you may have guessed, we did a similar thing to fix these issues: moved them to a script! Because of the earlier work done with tagging we were able to get retrying of repository cloning for free, and easily add retrying of uploads/downloads. This script also introduced another new technique to the release-automation: chunking (which was shamelessly ripped off of the Mochitest harness). Rather than have 1 Buildbot job for every single locale, the script knows how to compute the overall set of work for all locales and pick a chunk of it to work on.

Automated E-mail

Every release requires a lot of coordination, particularly with Release Drivers and QA. We need to send mail notifications when the release-automation starts, when each platform's en-US build is complete, when each platform's l10n repacks are complete, when updates are ready for testing, and some other events, too. It used to be that the Release Engineer responsible for the release would actively watch the jobs on a Buildbot display and send mail by hand as the jobs completed. Especially as we started doing releases more often, this became extremely tedious and distracting. It also caused artificial delays of up to 8 hours (in the worst case)! By automating these mails we massively reduced manual touchpoints, became more consistent with the messages we sent, allowed Release Engineers to more easily do other work mid-release, and in some extreme cases reduced end2end time of a release by multiple hours. Looking back on it this was one of the most important changes we've ever made, and certainly had the best cost/benefit ratio.

Pushing to Mirrors et. al

When we push a Firefox release out to the mirror network we get past the point of no return. Once it's out there, we have no way to pull it back and no way to guarantee that we overwrite all of the files on all of the mirrors in a timely manner. If we find bugs past that point we have to increment the version number and start again. Because of that we do a full antivirus check and verification of all permissions prior to pushing (in addition to all of the testing that QA already does). These used to be done all by hand - a Release Engineer would log onto a machine at some point between builds being available and prior to pushing, run some commands, and wait. Besides the annoyance of doing it by hand, we would sometimes forget to do this in advance of the release. When that happened these things all of a sudden were in the critical path, and holding up the release. To address both of those issues these checks were automated and done immediately after all release files became available. At the same time we partly automated the mirror push itself. Pushing to mirrors involves running a command like:

rsync -av --exclude=*tests* --exclude=*crashreporter* --exclude=*.log --exclude=*.txt --exclude=*unsigned* --exclude=*update-backup* --exclude=*partner-repacks* --exclude=*.checksums --exclude=logs --exclude=jsshell* --exclude=*/*.asc /pub/mozilla.org/firefox/nightly/10.0.2-candidates/build1/ /pub/mozilla.org/firefox/releases/10.0.2/

With such a non-trivial thing being required every time it's easy to make mistake, so once again, automating is a clear way to reduce manual error.

Autosign

All of the Firefox builds that we distribute are signed in some manner. On Windows, we have Authenticode Signatures; for everything else we have detached GPG signatures. Signing our builds is a crucial part of the release process and right in the middle of the critical path. Because we ship Firefox in so many languages and on multiple platforms it can take awhile to do all of our signing, which means it's important to get it started as soon as possible. In the past, we had to wait for all builds & repacks to complete and then run a long series of manual commands on our signing machine to: download the builds, sign them, verify them, and upload the signed bits. This was OK for awhile, but as we started shipping in more languages on more platforms it became horribly inefficient; downloading the builds alone started to take 30 minutes or more. And again, like many other things, there was lots of opportunity for manual error. Enter: Autosign. This relatively simple improvement adjusted the existing signing logic to be able to detect when it had all of the required bits to start signing. This meant that we could run the commands that would start signing as soon as the release began. The scripts continually download builds in a loop, in parallel with the rest of the automation running, which means we completely remove the "download builds" part of the signing process from the critical path. This also means that the Release Engineer doesn't need to be at work or even awake when all of the builds & repacks complete. In some cases, just like automated e-mail, this can save multiple hours of end2end time.

Summary

The combination of all of the changes above took the automation from a moderately fast system that worked most of the time to a very speedy system that rarely fails. Nearly everyone in Release Engineering had a hand in this, and most of them were done over a two month period! Incredibly, there was still more we found to improve in the following year, which I'll talk about in Part 4!

release-automation - Part 2: Mercurial-based, v1

Ben Hearsum — Wed, 15 Feb 2012 14:19:13 GMT

Around the start of 2008 Mozilla moved Firefox and Gecko development from CVS to Mercurial, with Firefox 3.5 (nee 3.1) as the first release out of the new repository. In addition to that, the underlying build infrastructure had switched from being Tinderbox driven, to being Buildbot driven - which made some of the existing release automation useless. In mid-2008 we started planning to port, rework, and update the release automation for this new environment. The 2008 Firefox Summit conveniently happened right around this time, so we took that opportunity to gather a quorum on the subject and go over all the plans in detail. By the end of the night (and end of the beer, if I recall correctly), we had discussed everything to death a tracking bug. This version of the automation struck a balance between improving the overall design of the system and simply doing straight porting work. The plain porting isn't very interesting, so I'll be mostly focusing on the improvements we made in this post. One of the bigger optimizations we made to to generate files in their final location at build time. In the Bootstrap days we uploaded files to flat directories with long filenames, and then re-arranged them into their final layout later on in the process. With this change made our candidates directories looked a lot more like the the associated release directory. This may not sound like a huge change but it cut our disk space usage per release in half or more, shaved over an hour off the end2end time of the release, and let us put our release file naming logic into the build system, where it more rightly belonged. It also allowed us to make the next optimization: combining the signing processes. In the Bootstrap and pre-Bootstrap worlds we had two separate signing processes: one to sign the internal guts of Firefox win32 builds (firefox.exe, xul.dll, et. al) and one to sign the Firefox installers themselves. Early on, we signed the internal bits and handed them off to QA. Closer to release time, we signed the installers themselves and generated GPG signatures for all files. The only reason I can think of why we would do this is to keep signed installers out of public directories until we're sure we'll be releasing them. This isn't without its drawbacks though. Leaving this until later in the process added unnecessary manual touchpoints, put non-trivial work late in the critical path, and worst of all: It meant QA did not test the exact bits that we shipped to users! (We actually managed to ship unsigned installers once, which isn't possible anymore.) Improving this only required a small rework of our existing signing scripts (and lots of testing, of course!) but it took another 1-2h off of our end2end time and removed another manual touchpoint. It's also worth noting that merely by switching to Mercurial we saved over half an hour in end2end time in tagging. In CVS, we had to create a branch and tag thousands and thousands of files with multiple tags, which takes a very long time. In Mercurial, we have clone a repository, which takes some time, but the tagging itself is near-instant. In addition to the optimizations noted above, tons of work was done porting the existing automation. Many things had to be pulled out of Bootstrap and put into their own scripts to make them usable by both versions of the automation; en-US builds and l10n repacks had to be reimplemented entirely in Buildbot; and some other things that couldn't be pulled out of Bootstrap had to be reimplemented as well. It was a very large undertaking that was primarily worked on by Nick Thomas, Coop, and myself and took months to complete. Firefox 3.1b3 was the first fully automated release with this automation. By the time we worked out most of the kinks we were at end2end time of 8-10h and about 12 manual touchpoints. Next up: Various improvements & optimizations (not as boring as it sounds, I promise!)

release-automation - Part 1: Bootstrap

Ben Hearsum — Tue, 07 Feb 2012 20:14:25 GMT

One of the first tasks I had as a full-time employee of Mozilla was getting the Bootstrap Release framework working with Firefox 3.0 Beta releases. Now, just over 4 years later, our release-automation has changed dramatically in many ways: primary language, supported platforms, scope and extent, reliability, and versatility. I thought it made be interesting to trace the path from there to here, and talk about what's in store for the future, too. Throughout all of this work there's been two overarching goals: 1) Lower the time it takes to go from "go to build" to "updates available for testing" - which we call "end2end time", and 2) Remove the number of machines we have to log into, commands we have to run, and active time we have to spend on a release - known as "manual touchpoints". I'll be referencing these a lot throughout this series. This post will talk about what I know of Bootstrap and my work porting it to Firefox 3.0. In its earliest form Bootstrap was a simple scripted version of much of the previously manual release process. The processes for tagging VCS repositories, creating deliverables (source packages, en-US and localized builds, updates), and some verifications were encapsulated into its scripts. This was a big improvement over the 100% manual, cut+paste-from-a-wiki, process. Instead of logging into many machines and running many commands, the release engineer had to log in to many machines and run a few, very simple commands. The very first release that was Bootstrap-aided was Firefox 1.5.0.9, built on December 6th, 2006. This was before my time, but a former release engineer, Rob Helmer, told me that the end2end time back then could be multiple days, and countless touchpoints. Over time, more parts of the release process were automated with Bootstrap, further reducing the burden on the release engineer. Even with these big improvements some classes of things were still not codified: which machines to run which commands on, when and in what order to run things, who to notify about what. Enter: Buildbot. Integrating Bootstrap into Buildbot was the next logical step in the process. It would handle scheduling and status, while Bootstrap would remain responsible for all of implementation. With this, the release engineer only had to log in to a few machines and run a few, very simple commands. Another big improvement! The first release to benefit from this was Firefox 2.0.0.8, built on October 10th, 2007. This work was largely done by Rob Helmer. Around this time we were gearing up to start shipping the first Firefox 3.0 Beta release and had never tested Bootstrap against that development branch. I was tasked with making whatever changes were necessary to Bootstrap and our Buildbot to make it work. The Buildbot side was largely simple, because of it being at such a high abstraction layer, but back in these days we still had single purpose Buildbot masters, so it involved adding several hundred lines of config code. The Bootstrap side was far more interesting. Until this point, there was a lot of built-in assumptions based on what the 1.8 branch looked like, including:

Releases are done from CVS branches (explicitly _not_ trunk)
Windows build machines run Cygwin
Linux packages are in .gz format
The crash reporting system Talkback is always shipped

By themselves, none of these things are too challenging to deal with, but as a very new hire, the combination took me about a month to find solutions to and fully test, with many rounds of feedback and guidance along the way. With all of that done and landed, we managed to use the new automation to build Firefox 3.0b2 on December 10, 2007. At this point, the end2end time was around 24h and there were about 20 manual touchpoints. Over the next 8 months or so there were a few major improvements of note. Firstly, Nick Thomas fixed bug 409394 (Support for long version names) allowed us to start shipping releases with nicer looking filenames like "Firefox Setup 3.0 Beta 4". Not a crucial thing, but much nicer from the user perspective. bug 422235 (enable fast patcher for release automation) was a massive improvement in update generation, written by schrep. With this work, we went from taking 6-8 hours to generate updates, down to ~1h -- an incredible savings in time. Finally, bug 428063 (Support major releases & quit using rc in overloaded ways) (also fixed by Nick) enabled us to build RCs with Bootstrap. While it may sound simple, there's a lot of things in release automation that depend on filename, and catching them all can be difficult. As well as making it possible to build these, this bug also renamed the internal "rc" notion to "build", to avoid situations where we'd have things like "3.0 RC1 rc1", which was utterly confusing.

So, in the early days there were tons of improvement quickly: Bootstrap itself sped things up and lowered the possibility of error through reducing manual touchpoints. Buildbot + Bootstrap did so again, through the same methods. We also had pure speed-ups through things such as fast patcher. Having these things allowed us to maintain the 2.0.0.x and 3.0.x branches more more easily, and get chemspill releases out quickly and simultaneously. All of this work had to be done incrementally too, because we had to continue shipping releases while the work was happening. It's hard to find good data for releases done with this version of the automation, but I guesstimate that the end2end time was around 12-14 hours and the number of manual touchpoints was still around 20 for a release without major issues. Next up....release-automation on Mercurial, v1.