Ben Hearsum (Posts about releng)

Balrog in 2016 & 2017

Ben Hearsum — Fri, 23 Dec 2016 17:57:55 GMT

This past year has been a big and exciting one for Balrog. We made the transition to Docker, migrated to new infrastructure, participated in Google's Summer of Code for the first time, bootstrapped System Addon updates, and much, much more. In total, 129 tickets were closed, with a significant portion of those being done by volunteers. I'd like to highlight a few things in particular.

Toolchain Improvements

Early on in the year I spent a good deal of time modernizing Balrog's toolchain. I upgraded most of the Python packages, switched to Tox, added Taskcluster for CI, and started using Docker for local development and testing. All of these things combined made it dead-simple to run Balrog on your laptop. This helps with reproducing bugs and testing new work, but the most important aspect ended up being how easy it made it for new developers to get to work. Without this change I wouldn't have felt comfortable proposing Balrog work for Summer of Code, and I highly doubt we would've had many (if any) volunteers working on it.

Looking back on this, I'm now firmly of the belief that if you have some extra time, spending it on things that reduce developer friction is one of the best things you can do. For new projects, make sure you build in time for this at the start.

Volunteers are Awesome

Speaking of volunteers, we've got some awesome ones, and that's pretty much a first for Release Engineering. We've always had a tough time opening up our work to volunteers, primarily because our systems are often difficult to hack on locally. All the work I just talked about to reduce developer friction made this a non-issue for Balrog, and opened the door to start advertising good first bugs, and applying for Summer of Code. I was so amazed and impressed with how many people came knocking when it was made much easier to get started.

But as I look back, I've also realized that there's another part to it: spending the time. When I submitted a Summer of Code proposal, I knew I would have to spend some time mentoring, and it helped put me in the right mindset when volunteers started coming along. While some will just send a PR with a great patch out of the blue, others need a bit more guidance to get started. This doesn't mean that the latter group are less skilled or less valuable, so you shouldn't ignore them. In fact, one of the most active contributors to Balrog is someone that started out in this group. If you can find the time to help these people learn and grow, it can pay dividends down the road.

I want to send out a special thanks to a few of our volunteers in particular. Varun Joshi, our Summer of Code student, who improved the efficiency of our l10n update submissions, and did much of the initial work on System Addons. Njira Perci, who has done the vast majority of the UI work in the past year (I'm pretty sure she's our resident expert on it now!). And Ninad Bhat, who did all of the other work on System Addons, and is now helping out a lot with Multiple Signoff. Without you three I don't know what we would've done!

Transition to Cloud Operations

Early in the year we made the decision to move the production infrastructure of Balrog to CloudOps' platform. While there was a lot of small details to figure out, this went extremely well, and I want to thank the Web Operations, Database Operations, and Cloud Operations teams for making it so smooth. Because it's AWS and Docker based, we now have much more control over the production stack, and are able to scale much more easily as we add more load to Balrog.

System Addon Updates

One of the things I'm most proud of this year is how quickly we were able to spin up updates for System Addons. In the past this may have been a huge headache, but because of Balrog's flexible design we were able to get them working quite quickly, and make some improvements later. Ninad in particular spent considerable time making improvements to that process.

Multifile Updates

Since we began shipping them, Balrog has served updates to Gecko Media Plugins. And since we added a second one, they've been a huge headache for us to manage. After Varun implemented multifile updates, shipping new versions of these plugins became trivial.

Scheduled Changes

Code-wise, this was my biggest accomplishment of the year. The Scheduled Changes work allows us queue changes to be enacted at a later time, and once we have lower latency ADI information, release uptake. Once the latter is available, we'll be able to do much better throttled rollouts rather than the guesswork we currently do. This system has also become the basis for the Multiple Signoff work we started in Q4.

2017

This past year has been a great one for Balrog, but I expect 2017 to be even better. Here's some of the things we're looking at for 2017:

Finishing up the work on Multiple Signoffs, which will make Balrog much more resilient to bad actors or credential theft.
Digging into unifying update requests that Firefox makes. Laura talked a lot about this at Mozloha, and I think we can greatly improve uptake of a lot of different things that Firefox queries for.
Authentication improvements. I'd like to look into switching to Okta or another service that allows for MFA.
Getting more involved with Cloud Operations QA, who want to help us with better load and contract testing in our deployment pieline.
Various improvements to the Rules:
- Supporting new business requirements like CPU/GPU filtering and multiple matching.
- Allowing for Rules to be treated as sets, to make it safer when adjusting intertwined Rules.
- Improvements to the UI to give better insight into the effects of a change to a Rule.
Investigating some architectural changes such as upgrading to Python 3 or adding a service layer.

Here's to another great year of Balrog development!
_{(Photo credit: Donna Oberes)}

Building and Pushing Docker Images with Taskcluster-Github

Ben Hearsum — Thu, 30 Jun 2016 19:29:00 GMT

Earlier this year I spent some time modernizing and improving Balrog's toolchain. One of my goals in doing so was to switch from Travis CI to Taskcluster both to give us more flexibility in our CI configuration, as well as help dogfood Taskcluster-Github. One of the most challenging aspects of this was how to build and push our Docker image, and I'm hoping this post will make it easier for other folks who want to do the same in the future.

The Task Definition

Let's start by breaking down Task definition from Balrog's .taskcluster.yml. Like other Taskcluster-Github jobs, we use the standard taskcluster.docker provisioner and worker.

  - provisionerId: "{{ taskcluster.docker.provisionerId }}"
    workerType: "{{ taskcluster.docker.workerType }}"

Next, we have something a little different. This section grants the Task access to a secret (managed by the Secrets Service). More on this later.

    scopes:
      - secrets:get:repo:github.com/mozilla/balrog:dockerhub

The payload has a few things of note. Because we're going to be building Docker images it makes sense to use Taskcluster's image_builder Docker image as well as enabling the docker-in-docker feature. The taskclusterProxy feature is needed to access the Secrets Service.

    payload:
      maxRunTime: 3600
      image: "taskcluster/image_builder:0.1.3"
      features:
        dind: true
        taskclusterProxy: true
      command:
        - "/bin/bash"
        - "-c"
        - "git clone $GITHUB_HEAD_REPO_URL && cd balrog && git checkout $GITHUB_HEAD_BRANCH && scripts/push-dockerimage.sh"

The extra section has some metadata for Taskcluster-Github. Unlike CI tasks, we limit this to only running on pushes (not pull requests) to the master branch of the repository. Because only a few people can push to this branch, it means that only these can trigger Docker builds.

    extra:
      github:
        env: true
        events:
          - push
        branches:
          - master

Finally, we have the metadata, which is just standard Taskcluster stuff.

    metadata:
      name: Balrog Docker Image Creation
      description: Balrog Docker Image Creation
      owner: "{{ event.head.user.email }}"
      source: "{{ event.head.repo.url }}"

Secrets

I mentioned the "Secrets Service" earlier, and it's the key piece that enables us to securely push Docker images. Putting our Dockerhub password in it means access is limited to those who have the right scopes. We store it in a secret with the key "repo:github.com/mozilla/balrog:dockerhub", which means that anything with the "secrets:get:repo:github.com/mozilla/balrog:dockerhub" scope is granted access to it. My own personal Taskcluster account has it, which lets me set or change the password:

We also have a Role called "repo:github.com/mozilla/balrog:branch:master" which has that scope:

You can see from its name that this Role is associated with the Balrog repository's master branch. Because of this, any Tasks created for as a result of pushes to that branch in that repository and branch may assign the scopes that Role has - like we did above in the "scopes" section of the Task.

Building and Pushing

The last piece of the puzzle here is the actual script that does the building and pushing. Let's look at a few specific parts of it.

To start with, we deal with retrieving the Dockerhub password from the Secrets Service. Because we enabled the taskclusterProxy earlier, "taskcluster" resolves to the hosted Taskcluster services. Had we forgotten to grant the Task the necessary scope, this would return a 403 error.

password_url="taskcluster/secrets/v1/secret/repo:github.com/mozilla/balrog:dockerhub"
dockerhub_password=$(curl ${password_url} | python -c 'import json, sys; a = json.load(sys.stdin); print a["secret"]["dockerhub_password"]')

We build, tag, and push the image, which is very similar to building it locally. If we'd forgotten to enable the dind feature, this would throw errors about not being able to run Docker.

docker build -t mozilla/balrog:${branch_tag} .
docker tag mozilla/balrog:${branch_tag} "mozilla/balrog:${date_tag}"
docker login -e $dockerhub_email -u $dockerhub_username -p $dockerhub_password
docker push mozilla/balrog:${branch_tag}
docker push mozilla/balrog:${date_tag}

Finally, we attach an artifact to our Task containing the sha256 of the Docker images. This allows consumers of the Docker image to verify that they're getting exactly what we built, and not something that may have been tampered on Dockerhub or in transit.

sha256=$(docker images --no-trunc mozilla/balrog | grep "${date_tag}" | awk '/^mozilla/ {print $3}')
put_url=$(curl --retry 5 --retry-delay 5 --data "{\"storageType\": \"s3\", \"contentType\": \"text/plain\", \"expires\": \"${artifact_expiry}\"}" ${artifact_url} | python -c 'import json; import sys; print json.load(sys.stdin)["putUrl"]')
curl --retry 5 --retry-delay 5 -X PUT -H "Content-Type: text/plain" --data "${sha256}" "${put_url}"

The Result

Now that you've seen how it's put together, let's have a look at the end result. This is the most recent Balrog Docker build Task. You can see the sha256 artifact created on it:

And of course, the newly built image has shown up on the Balrog Dockerhub repo:

A Flurry of Balrog Activity

Ben Hearsum — Thu, 24 Mar 2016 16:13:40 GMT

This past quarter I spent some time modernizing Balrog's toolchain to make it more approachable. We've switched from Vagrant to Docker, cleaned up setup.py, started using tox, and updated the sample data included in the repo. At the same time, I started identifying some good first bugs, and put together a proposal for a Summer of Code project.

I feel very lucky that this work has paid off so quickly. There's been great interest in the Summer of Code project, and we've had 5 new volunteers submit patches to Balrog. These people are doing really great work, and I'd like to highlight their contributions today (in no particular order).

Njira Perci

Njira has focused on UI improvements, and has already improved the confirmation dialog for deleting Rules and added the ability to autocomplete Products and Channels in form fields. She continues to hack away and is now working on improving the Releases UI to highlight whether or not a release is active in any way.

Ashish Sareen

Ashish fixed a bug where the Admin server would hit an ISE 500 under certain conditions. With his patch, it now correctly returns a 400 error to the client.

Varun Joshi

Varun has been diving deep into the Admin server. He started off by fixing a small bug where some weirdly formed Releases caused ISE 500s, and has since provided a patch that gives us the ability to mark Releases as "read only". This is something we intend to make use of in our new Release Promotion system to guard accidental (or malicious...) changes to Release metadata.

Aybüke Özdemir

Aybüke enhanced the UI to show rule_ids, which makes it esaier for humans to find them when they need to put them into a script or automation.

Kumar Rishabh

Kumar fixed a very annoying bug where diffs of different versions of Releases would be generated against the wrong base version, making them essentially useless.

You?

If you would like to get involved in the development of Balrog, we'd love to have you. The wiki page can get you bootstrapped, and you can find us all on irc.mozilla.org in #balrog.

Configuring uWSGI to host an app and static files

Ben Hearsum — Thu, 24 Dec 2015 14:44:50 GMT

This week I started using uWSGI for the first time. I'm in the process of switching Balrog from Vagrant to Docker, and I'm moving away from Apache in the process. Because of Balrog's somewhat complicated Apache config this ended up being more difficult than I thought. Although uWSGI's docs are OK, I found it a little difficult to put them into practice without examples, so here's hoping this post will help others in similar situations.

Balrog's Admin app consists of a pretty standard Python WSGI app, and a static Angular app hosted on the same domain. To complicate matters, the version of Angular that we use does not support being hosted anywhere except the root of the domain. It took a bit of futzing, but we came up with an Apache config to host both of these pieces on the same domain pretty quickly:

<VirtualHost *:80>
    ServerName balrog-admin.mozilla.dev
    DocumentRoot /home/vagrant/project/ui/dist/

    # Rewrite virtual paths in the angular app to the index page
    # so that refreshes/linking works, while leaving real files
    # such as the js/css alone.
    <Directory /home/vagrant/project/ui/dist>
        RewriteEngine On
        RewriteCond %{REQUEST_FILENAME} -f [OR]
        RewriteCond %{REQUEST_FILENAME} -d

        RewriteRule ^ - [L]
        RewriteRule ^ index.html [L]
    </Directory>

    # The WSGI app is rooted at /api
    WSGIScriptAlias /api /home/vagrant/project/admin.wsgi
    WSGIDaemonProcess aus4-admin processes=1 threads=1 maximum-requests=50 display-name=aus4-admin
    WSGIProcessGroup aus4-admin
    WSGIPassAuthorization On

    # The WSGI app relies on the web server to do the authentication, and will
    # bail if REMOTE_USER isn't set. To simplify things, we just set this
    # variable instead of prompting for auth.
    SetEnv REMOTE_USER balrogadmin

    LogLevel Debug
    ErrorLog "|/usr/sbin/rotatelogs /var/log/httpd/balrog-admin.mozilla.dev/error_log_%Y-%m-%d 86400 -0"
    CustomLog "|/usr/sbin/rotatelogs /var/log/httpd/balrog-admin.mozilla.dev/access_%Y-%m-%d 86400 -0" combined
</VirtualHost>

Translating this to uWSGI took way longer than expected. Among the problems I ran into were:

Using --env instead of --route's addvar action to set REMOTE_USER (--env turns out to be for passing variables to the overall WSGI app).
Forgetting to escape "$" when passing routes on the command line, which caused my shell to interpret variables intended for uWSGI
Trying to rewrite URLs to a static path, which I only discovered is invalid after stumbling on an old mailing list thread.
Examples from uWSGI's own documentation did not work! I discovered that depending on how it was compiled, you may need to pass "--plugin python,http" to give all of the necessary command line options for what I was doing.

After much struggle, I came up with an invocation that worked exactly the same as the Apache config:


uwsgi --http :8080 --mount /api=admin.wsgi --manage-script-name --check-static /app/ui/dist --static-index index.html --route "^/.*$ addvar:REMOTE_USER=balrogadmin" --route-if "startswith:\${REQUEST_URI};/api continue:" --route-if-not "exists:/app/ui/dist\${PATH_INFO} static:/app/ui/dist/index.html"

There's a lot crammed in there, so let's break it down:

--http :8080 tells uWSGI to listen on port 8080
--mount /api=admin.wsgi roots the "admin.wsgi" app in /api. This means that when you make a request to http://localhost:8080/api/foo, the application sees "/foo" as the path. If there was no Angular app, I would simply use "--wsgi-file admin.wsgi" to place the app at the root of the server.
--manage-script-name causes uWSGI to rewrite PATH_INFO and SCRIPT_NAME according to the mount point. This isn't necessary if you're not using "--mount".
--check-static /app/ui/dist points uWSGI at a directory of static files that it should serve. In my case, I've pointed it at the fully built Angular app. With this, requests such as http://localhost:8080/js/app.js returns the static file from /app/ui/dist/js/app.js.
--static-index index.html tells uWSGI to serve index.html when a request for a directory is made - the default is to 404, because there's no built-in directory indexing.
The --route's chain together, and are evaluated as follows:
If the requested path matches ^/.*$ (all paths will), set the REMOTE_USER variable to balrogadmin.
If the REQUEST_URI starts with /api do not process any more --route's; just satisfy the request. All requests intended for the WSGI app will end up matching here. REQUEST_URI is used instead of PATH_INFO because the latter is written by --manage-script-name
If the requested file does not exist in /app/ui/dist, serve /app/ui/dist/index.html instead. PATH_INFO and REQUEST_URI will still point at the original file, which lets Angular interpret the virtual path and serve the correct thing.

In the end, uWSGI seems to be one of the things that's very scary when you first approach it (I count about 750 command line arguments), but is pretty easy to understand when you get to know it a better. This is almost the opposite of Apache - I find it much more approachable, perhaps because there's such a littany of examples out there, but things like mod_rewrite are very difficult for me to understand after the fact, at least compared to uWSGI's --route's.

Going Faster with Balrog

Ben Hearsum — Fri, 25 Sep 2015 19:15:00 GMT

Go Faster is a broad initiative at Mozilla that is focused on shipping things to users much faster than the current 6 week cycle. One important part of this project is having a mechanism to make Firefox aware of updates they need or may want to download. This is nothing entirely new of course - we've been shipping updates to users since Firefox 1.5 - but with Go Faster we will be updating bits and pieces of Firefox at a time rather than always updating the entire install. In this post I'm going to outline these new types of updates that we've identified, and talk about how things will work in the Glorious Future.

A Primer on Updates

Firefox updates work on a "pull" system, meaning Firefox regularly queries the update server (Balrog) to ask if there's an update available. For example, my Firefox is currently polling for updates by making a GET request to this URL:

https://aus5.mozilla.org/update/3/Firefox/41.0/20150915150946/Linux_x86_64-gcc3/en-GB/beta/default/default/default/update.xml

All of the information in that URL is mapped against a set rules in Balrog, and eventually points to a single release. If that release is newer than the incoming one (based on the incoming version and buildid), Balrog returns the information necessary for the client to update to it:

<updates>
    <update type="minor" displayVersion="41.0" appVersion="41.0" platformVersion="41.0" buildID="20150917150946" detailsURL="https://www.mozilla.org/en-GB/firefox/41.0/releasenotes/">
        <patch type="complete" URL="http://download.mozilla.org/?product=firefox-41.0build3-complete&os=linux64&lang=en-GB" hashFunction="sha512" hashValue="ea0757069363287f67659d8b7d42e0ac6c74a12ce8bd3c7d3e9ad018d03cd6f4640529c270ed2b3f3e75b11320e3a282ad9476bd93b0f501a22d1d9cb8884961" size="48982398"/>
    </update>
</updates>

It's important to note that Balrog only contains metadata about the update. The actual payloads of the updates are hosted on CDN networks.

New Types of Updates

We've identified three different new types of updates that we'll be implementing as part of Go Faster. They are:

System Addons: These are core (aka required) parts of Firefox that happen to be implemented as Addons.
Security Policy: This is a medium sized piece of JSON that instructs NSS about special security policies to enforce for various websites.
Optional Features: These are optional parts of Firefox that may be implemented as Addons or other means.

Each one of these will be implemented as an additional update request to Balrog (we may collapse these into a single request later). Eg, Firefox will look for new System Addons by making a GET request to an URL such as:

https://aus5.mozilla.org/update/3/SystemAddons/41.0/20150915150946/Linux_x86_64-gcc3/en-GB/beta/default/default/default/update.xml

The responses will vary a bit depending on the type of update. More on that below.

System Addons

Seeing as Firefox can't function without them, System Addons may seem like a contradiction at first. The advantages are quite clear though: with them, we can ship updates to self contained pieces of Firefox at a substantionally faster rate. Shipping an update to all of Firefox takes nearly 24 hours (when we're moving as fast as we can); shipping an update to a System Addon could take as little as minutes.

Although they are implemented as Addons, we can't simply ship them through the AMO. Because Firefox cannot function without them we must ship them in the installers and full updates that happen every 6 weeks. This has the nice side effect of minimizing dependency problems -- we won't run into a case where Firefox updates but System Addons don't, which could cause incompatibilities. In between the 6 week cycles Firefox will poll Balrog for updates to System Addons and apply them as they become available. This graph may show this more clearly:

As you can see, Firefox 50.0 can be assumed to have any of Fizz 1.0, 1.1, 1.2, 1.3, or 2.0, while Firefox 51.0 is known to only have Fizz 2.0 (but may receive newer versions later).

When Firefox pings Balrog for System Addon updates, the response will look something like this:

<updates>
    <addons>
        <addon id="fizz@mozilla.org" URL="http://download.cdn.mozilla.net/fizz-1.1.xpi" hashFunction="sha512" hashValue="abcdef123456" version="1.1"/>
        <addon id="pop@mozilla.org" URL="http://download.cdn.mozilla.net/pop-2.5.xpi" hashFunction="sha512" hashValue="abcdef123456" version="2.5"/>
        <addon id="bam@mozilla.org" URL="http://download.cdn.mozilla.net/bam-3.4.xpi" hashFunction="sha512" hashValue="abcdef123456" version="3.4"/>
    </addons>
</updates>

Firefox will compare the list against its currently installed versions and update anything that's out of date. The exact details on where System Addons will live on disk are still being ironed out.

Security Policy

Every version of Firefox ships with the most up-to-date set of security policies that were available when it was built. However, these policies are updated more frequently than we ship, and it's crucial that we keep them up to date to keep our users safe. As with System Addons, Firefox will regularly ping Balrog to check for updated security policies. When one is found, Firefox downloads it from Kinto, which will serve it an incremental update to its security policies. The details of this process have been outlined in much more detail by the Cloud Services team.

The Balrog response for these updates is extremely simple, and simply contains a version that Firefox passes along to Kinto:

<updates>
    <settings>
        <setting id="security" lastModified="129386427328"/>
    </settings>
</updates>

Optional Features

These are parts of Firefox that are not core to the browser, but may be useful to subsets of users. For example: We currently ship a ton of hyphenation dictionaries as part of Firefox for Android. These are locale-specific, so only one ever gets used for each user. We can also distribute opt-in features that not everyone wants or needs, eg: Developer Tools may be a good candidate (there are no plans to do so at this time though).

Optional features may also be implemented in various ways. Hyphenation dictionaries are simple zip files, while something like Developer Tools would likely be an Addon. They will not be included in Firefox installers or update packages. Instead Firefox will regularly query Balrog to see what packages may be available to it. Some things may automatically install based on the user's environment (eg: hyphentation dictionaries for your locale), while other things may require opt-in (eg: optional features).

Balrog responses are not yet set in stone for these, but Kinto is likely to be involved, so the response may end up being similar to the one above for Security Policy updates.

Summary

While System Addons, Security Policy, and Optional Features overlap in some areas, each has its own unique combination of requirements. The chart below summarizes these:

	Required?	Shipped in Installer?	Payload Type	Payload Location
System Addons	Yes	Yes	Addons	CDN
Security Policy	Yes	Yes	JSON	Kinto
Optional Features	No	No	Anything	Kinto

Mozilla Software Release GPG Key Transition

Ben Hearsum — Tue, 21 Jul 2015 18:45:18 GMT

Late last week we discovered the expiration of the GPG key that we use to sign Firefox, Fennec, and Thunderbird nightly builds and releases. We had been aware that this was coming up, but we unfortunately missed our deadline to renew it. This caused failures in many of our automated nightly builds, so it was quickly noticed and acted upon.

Our new GPG key is as follows, and available on keyservers such as gpg.mozilla.org and pgp.mit.edu:

pub   4096R/0x61B7B526D98F0353 2015-07-17
      Key fingerprint = 14F2 6682 D091 6CDD 81E3  7B6D 61B7 B526 D98F 0353
uid                            Mozilla Software Releases 
sub   4096R/0x1C69C4E55E9905DB 2015-07-17 [expires: 2017-07-16]

The new primary key is signed by many Mozillians, the old master key, as well as our OpSec team's GPG key. Nightlies and releases will now be signed with the subkey (0x1C69C4E55E9905DB), and a new one will be generated from the same primary key before this one expires. This means that you can validate Firefox releases with the primary public key in perpetuity.

We are investigating a few options to make sure key renewal happens without delay in the future.

This week in Mozilla RelEng – June 13th, 2014 - double edition

Ben Hearsum — Fri, 13 Jun 2014 15:45:10 GMT

I spaced and forgot to post this last week, so here's a double edition covering everything so far this month. I'll also be away for the next 3 Fridays, and Kim volunteered to take the reigns in my stead. Now, on with it! Major highlights:

We had a busy couple of weeks of important, but relatively boring work. We shipped 5 releases (2 betas, Firefox 30.0 & 24.6.0esr, Thunderbird 24.6.0), did a series of uplifts, and continued moving machines from scl1 to scl3.
Catlee did some investigation into our CDN bandwidth and found that on average, 20% of users who have a partial update available still end up downloading the complete update.
I finished switching B2G updates to aus4.mozilla.org. The updates should now be more reliable, and we'll be able to kill update.boot2gecko.org in the near future. Because we're abiding by standard update channel names now, this also means that the Socorro folks are able to process crashes for most B2G builds.
Mike got his patch that allows RelEng builds to be built with mach ready to land. This is the step along the path making automation steps reproducible by developers, and enabling parallelization of post-build steps.
Ed had us disable coalecing for changes that land within a few minutes of each other. During tree closures for bustage, this should help get the tree open faster because fewer builds will be needed for bisection.

Completed work (resolution is 'FIXED'):

Balrog: Backend

Buildduty

General Automation

Loan Requests

Other

Platform Support

release-automation

Releases

Releases: Custom Builds

Repos and Hooks

Tools

In progress work (unresolved and not assigned to nobody):

Balrog: Backend

Balrog: Frontend

balrog's rules page is excessively wide

Buildduty

General Automation

Loan Requests

Other

[tracking] infrastructure improvements

Platform Support

release-automation

Releases

Releases: Custom Builds

Yandex partner builds for Turkey

Repos and Hooks

[tracker] hg.mozilla.org user repository issues

Tools

More on "How far we've come"

Ben Hearsum — Wed, 04 Jun 2014 17:57:16 GMT

After I posted "How far we've come" this morning a few people expressed interest in what our release process looked like before, and what it looks like now. The earliest recorded release process I know of was called the "Unified Release Process". (I presume "unified" comes from unifying the ways different release engineers did things.) As you can see, it's a very lengthy document, with lots of shell commands to tweak/copy/paste. A lot of the things that get run are actually scripts that wrap some parts of the process - so it's not as bad as it could've been. I was around for much of the improvements to this process. Awhile back I wrote a series of blog posts detailing some of them. For those interested, you can find them here:

I haven't gotten around to writing a new one for the most recent version of the release automation, but if you compare our current Checklist to the old Unified Release Process, I'm sure you can get a sense of how much more efficient it is. Basically, we have push-button releases now. Fill in some basic info, push a button, and a release pops out:

How far we've come

Ben Hearsum — Wed, 04 Jun 2014 12:26:51 GMT

When I joined Mozilla's Release Engineering team (Build & Release at the time) back in 2007, the mechanics of shipping a release were a daunting task with zero automation. My earliest memories of doing releases are ones where I get up early, stay late, and spend my entire day on the release. I logged onto at least 8 different machines to run countless commands, sometimes forgetting to start "screen" and losing work due to a dropped network connection. Last night I had a chat with Nick. When we ended the call I realized that the Firefox 30.0 release builds had started mid-call - completely without us. When I checked my e-mail this morning I found that the rest of the release build process had completed without issue or human intervention. It's easy to get bogged down thinking about current problems. Times like this make me realize that sometimes you just need to sit down and recognize how far you've come.

This week in Mozilla RelEng – May 30th, 2014

Ben Hearsum — Fri, 30 May 2014 19:20:29 GMT

Major highlights:

Nick landed a big patch to Balrog (our new update server) that has unblocked the remaining work to move Beta users over to it.
I moved our Flame builds over to Balrog, which is a huge step towards killing update.boot2gecko.org.
Kim disabled many Android 2.2 tests on trunk branches on Tegras since these now run on 2.3 emulators. This should help reduce the load on our overburdened Tegra pool and is a first step toward deprecating this hardware completely.
Rail brought up our first DNS-less AWS instances. These new instances take jobs more quickly than the older ones, and it's much easier for us to add more to the pool.
Massimo moved our ccache for AWS machines onto SSDs. This should result in a small improvement to build times.
Aki killed the now-unused b2g18 (1.1) branches.