Ben Hearsum (Posts about python)

Collision Detection and History with SQLAlchemy

Ben Hearsum — Fri, 08 Jan 2016 13:58:31 GMT

Balrog is one of the more crucial systems that Release Engineering works on. Many of our automated builds send data to it and all Firefox installations in the wild regularly query it to look for updates. It is an SQLAlchemy based app, but because of its huge importance it became clear in the early stages of development that we had a couple of requirements that went beyond those of most other SQLAlchemy apps, specifically:

Collision Detection: Changes to the database must always be done safely. Balrog must not allow one change to silently override another one.
Full History: Balrog must provide complete auditability and history for all changes. We must be able to associate every change with an account and a timestamp.

Implementing these two requirements ended up being a interesting project and I'd like to share the details of how it all works.

Collision Detection

Anyone who's used Bugzilla for awhile has probably encountered this screen before:

This screenshot shows how Bugzilla detects and warns if you try to make a change to a bug before loading changes someone else has made. While it's annoying when this screen slows you down, it's important that Bugzilla doesn't let you unknowingly overwrite other folks' changes. This is very similar to what we wanted to do in Balrog, except that we needed to enforce it at the API level, not just in the UI. In fact, we decided it was best to enforce it at the lowest level possible to minimize the change of needing to duplicate it in different parts of the app.

To do this, we started by creating a thin wrapper around SQLAlchemy which ensures that each table has a "data_version" column, and requires an "old_data_version" to be passed when doing an UPDATE or DELETE. Here's a slimmed down version of how it works with UPDATE:

class AUSTable(object):
    """Base class for Balrog tables. Subclasses must create self.table as an
    SQLAlchemy Table object prior to calling AUSTable.__init__()."""
    def __init__(self, engine):
        self.engine = engine
        # Ensure that the table has a data_version column on it.
        self.table.append_column(Column("data_version", Integer, nullable=False))

    def update(self, where, what, old_data_version):
        # Enforce the data_version check at the query level to eliminate
        # the possibility of a race condition between the time we can
        # retrieve the current data_version, and when we can update the row.
        where.append(self.table.data_version == old_data_version)

        with self.engine.connect().begin() as transaction:
            row = self.select(where=where, transaction=transaction)
            row["data_version"] += 1
            for col in what:
                row[col] = what[col]

            query = self.table.update(values=what):
            for cond in where:
                query = query.where(cond)
            ret = transaction.execute(query)

            if ret.rowcount != 1:
                raise OutdatedDataError("Failed to update row, old_data_version doesn't match data_version")

And one of our concrete tables:

class Releases(AUSTable):
    def __init__(self, engine, metadata):
        self.table = Table("releases", metadata,
            Column("name", String(100), primary_key=True),
            Column("product", String(15), nullable=False),
            Column("data", Text(), nullable=False),
        )

    def updateRelease(self, name, old_data_version, product, data):
        what = {
            "product": product,
            "data": data,
        }
        self.update(where=[self.table.name == name], what=what, old_data_version=old_data_version)

As you can see, the data_version check is added as a clause to the UPDATE statement - so there's no way we can race with other changes. The usual workflow for callers is to retrieve the current version of the data, modify it, and pass it back along with old data_version (most of the time retrieval and pass back happens through the REST API). It's worth pointing out that a client could pass a different value as old_data_version in an attempt to thwart the system. This is something we explicitly do not try to protect against (and honestly, I don't think we could) -- data_version is a protection against accidental races, not against malicious changes.

Full History

Having history of all changes to Balrog's database is not terribly important on a day-to-day basis, but when we have issues related to updates it's extremely important that we're able to look back in time and see why a particular issue happened, how long it existed for, and who made the change. Like collision detection, this is implemented at a low level of Balrog to make sure it's difficult to bypass when writing new code.

To achieve it we create a History table for each primary data table. For example, we have both "releases" and "releases_history" tables. In addition to all of the Releases columns, the associated History table also has columns for the account name that makes each change and a timestamp of when it was made. Whenever an INSERT, UPDATE, or DELETE is executed, the History table has a new row inserted with the full contents of the new version. These are done is a single transaction to make sure it is an all-or-nothing operation.

Building on the code from above, here's a simplified version of how we implement History:

class AUSTable(object):
    """Base class for Balrog tables. Subclasses must create self.table as an
    SQLAlchemy Table object prior to calling AUSTable.__init__()."""
    def __init__(self, engine, history=True, versioned=True):
        self.engine = engine
        self.versioned = versioned
        # Versioned tables (generally, non-History tables) need a data_version.
        if versioned:
            self.table.append_column(Column("data_version", Integer, nullable=False))

        # Well defined interface to the primary_key columns, needed by History tables.
        self.primary_key = []
        for col in self.table.get_children():
            if col.primary_key:
                self.primary_key.append(col)

        if history:
            self.history = History(self.table.metadata, self)
        else:
            self.history = None

    def update(self, where, what, old_data_version=None, changed_by=None):
        # Because we're a base for History tables as well as normal tables
        # these must be optional parameters, but enforced when the features
        # are enabled.
        if self.history and not changed_by:
            raise ValueError("changed_by must be passed for Tables that have history")
        if self.versioned and not old_data_version:
            raise ValueError("update: old_data_version must be passed for Tables that are versioned")

        # Enforce the data_version check at the query level to eliminate
        # the possibility of a race condition between the time we can
        # retrieve the current data_version, and when we can update the row.
        where.append(self.table.data_version == old_data_version)

        with self.engine.connect().begin() as transaction:
            row = self.select(where=where, transaction=transaction)
            row["data_version"] += 1
            for col in what:
                row[col] = what[col]

            query = self.table.update(values=what):
            for cond in where:
                query = query.where(cond)
            ret = transaction.execute(query)
            if self.history:
                transaction.execute(self.history.forUpdate(row, changed_by))
            if ret.rowcount != 1:
                raise OutdatedDataError("Failed to update row, old_data_version doesn't match data_version")


class History(AUSTable):
    def __init__(self, metadata, baseTable):
        self.baseTable = baseTable
        self.Table("%s_history" % baseTable.table.name, metadata,
            Column("change_id", Integer(), primary_key=True, autoincrement=True),
            Column("changed_by", String(100), nullable=False),
            Column("timestamp", BigInteger(), nullable=False),
        )

        self.base_primary_key = [pk.name for pk in baseTable.primary_key]
        # In addition to the above columns, we need a copy of each Column
        # from our base table.
        for col in baseTable.table.get_children():
            newcol = col.copy()
            # We have our own primary_key Column, and don't want our
            # base table's PK to be part of it.
            if col.primary_key:
                newcol.primary_key = False
            # And while the base table's primary key is always required for
            # history rows, all other columns (including those that are
            # required in the base table) must be nullable.
            else:
                newcol.nullable = True
            self.table.append_column(newcol)

        AUSTable.__init__(self, history=False, versioned=False)

    def forUpdate(self, rowData, changed_by):
        row = {}
        # Copy in the data that's about to be updated in the base table...
        for k in rowData:
            row[k] = rowData[k]
        # ...and add the extra data that we need to track history accurately.
        row["changed_by"] = changed_by
        row["timestamp"] = time.time()
        return self.table.insert(values=rows)

A key thing to notice here is that the History tables are maintained automatically with only a minor tweak to the query interface (addition of "changed_by"). And while not shown here, it's important to note that the History table objects are not queryable directly through any exposed API. Even if an attacker got access to Balrog's REST API with admin permissions, they cannot delete rows from those tables.

If you'd like to see the complete implementation of either of these, you can find it over in the Balrog repository.

Enhancements

These things were implemented a few years ago, and since then we've discovered a couple of rough edges that would be nice to fix.

The biggest complaint is that the History tables are extremely inefficient. Many of our Release objects are a few hundred kilobytes, which means every change to them (thousands per day) significantly grows the releases_history table. We've dealt with this by limiting how long we keep history for certain types of releases, but it's far less than ideal. We'd love to have a more efficient way of storing history. We've discussed storing history as diffs rather than full copies or compressing the data before inserting the rows, but haven't settled on anything yet. If you have any ideas about this we'd love to hear them!

I mentioned earlier how annoying it is when Bugzilla throws you a mid-air collision, and it's no different in Balrog. We get hundreds of them a day when locales l10n repacks all try to update the same Releases. These can be dealt with by retrying but it's very inefficient. We might be able to do a better here if we inspected the details of changes that collide, and only reject them if they try to modify the same parts of an object.

Finally, all of this awesome collision detection and history code is in no way tied to Balrog - the classes that implement it are already very generic. I would love to pull out these features and ship them as their own module, which Balrog (and hopefully others!) can then consume.

Configuring uWSGI to host an app and static files

Ben Hearsum — Thu, 24 Dec 2015 14:44:50 GMT

This week I started using uWSGI for the first time. I'm in the process of switching Balrog from Vagrant to Docker, and I'm moving away from Apache in the process. Because of Balrog's somewhat complicated Apache config this ended up being more difficult than I thought. Although uWSGI's docs are OK, I found it a little difficult to put them into practice without examples, so here's hoping this post will help others in similar situations.

Balrog's Admin app consists of a pretty standard Python WSGI app, and a static Angular app hosted on the same domain. To complicate matters, the version of Angular that we use does not support being hosted anywhere except the root of the domain. It took a bit of futzing, but we came up with an Apache config to host both of these pieces on the same domain pretty quickly:

<VirtualHost *:80>
    ServerName balrog-admin.mozilla.dev
    DocumentRoot /home/vagrant/project/ui/dist/

    # Rewrite virtual paths in the angular app to the index page
    # so that refreshes/linking works, while leaving real files
    # such as the js/css alone.
    <Directory /home/vagrant/project/ui/dist>
        RewriteEngine On
        RewriteCond %{REQUEST_FILENAME} -f [OR]
        RewriteCond %{REQUEST_FILENAME} -d

        RewriteRule ^ - [L]
        RewriteRule ^ index.html [L]
    </Directory>

    # The WSGI app is rooted at /api
    WSGIScriptAlias /api /home/vagrant/project/admin.wsgi
    WSGIDaemonProcess aus4-admin processes=1 threads=1 maximum-requests=50 display-name=aus4-admin
    WSGIProcessGroup aus4-admin
    WSGIPassAuthorization On

    # The WSGI app relies on the web server to do the authentication, and will
    # bail if REMOTE_USER isn't set. To simplify things, we just set this
    # variable instead of prompting for auth.
    SetEnv REMOTE_USER balrogadmin

    LogLevel Debug
    ErrorLog "|/usr/sbin/rotatelogs /var/log/httpd/balrog-admin.mozilla.dev/error_log_%Y-%m-%d 86400 -0"
    CustomLog "|/usr/sbin/rotatelogs /var/log/httpd/balrog-admin.mozilla.dev/access_%Y-%m-%d 86400 -0" combined
</VirtualHost>

Translating this to uWSGI took way longer than expected. Among the problems I ran into were:

Using --env instead of --route's addvar action to set REMOTE_USER (--env turns out to be for passing variables to the overall WSGI app).
Forgetting to escape "$" when passing routes on the command line, which caused my shell to interpret variables intended for uWSGI
Trying to rewrite URLs to a static path, which I only discovered is invalid after stumbling on an old mailing list thread.
Examples from uWSGI's own documentation did not work! I discovered that depending on how it was compiled, you may need to pass "--plugin python,http" to give all of the necessary command line options for what I was doing.

After much struggle, I came up with an invocation that worked exactly the same as the Apache config:


uwsgi --http :8080 --mount /api=admin.wsgi --manage-script-name --check-static /app/ui/dist --static-index index.html --route "^/.*$ addvar:REMOTE_USER=balrogadmin" --route-if "startswith:\${REQUEST_URI};/api continue:" --route-if-not "exists:/app/ui/dist\${PATH_INFO} static:/app/ui/dist/index.html"

There's a lot crammed in there, so let's break it down:

--http :8080 tells uWSGI to listen on port 8080
--mount /api=admin.wsgi roots the "admin.wsgi" app in /api. This means that when you make a request to http://localhost:8080/api/foo, the application sees "/foo" as the path. If there was no Angular app, I would simply use "--wsgi-file admin.wsgi" to place the app at the root of the server.
--manage-script-name causes uWSGI to rewrite PATH_INFO and SCRIPT_NAME according to the mount point. This isn't necessary if you're not using "--mount".
--check-static /app/ui/dist points uWSGI at a directory of static files that it should serve. In my case, I've pointed it at the fully built Angular app. With this, requests such as http://localhost:8080/js/app.js returns the static file from /app/ui/dist/js/app.js.
--static-index index.html tells uWSGI to serve index.html when a request for a directory is made - the default is to 404, because there's no built-in directory indexing.
The --route's chain together, and are evaluated as follows:
If the requested path matches ^/.*$ (all paths will), set the REMOTE_USER variable to balrogadmin.
If the REQUEST_URI starts with /api do not process any more --route's; just satisfy the request. All requests intended for the WSGI app will end up matching here. REQUEST_URI is used instead of PATH_INFO because the latter is written by --manage-script-name
If the requested file does not exist in /app/ui/dist, serve /app/ui/dist/index.html instead. PATH_INFO and REQUEST_URI will still point at the original file, which lets Angular interpret the virtual path and serve the correct thing.

In the end, uWSGI seems to be one of the things that's very scary when you first approach it (I count about 750 command line arguments), but is pretty easy to understand when you get to know it a better. This is almost the opposite of Apache - I find it much more approachable, perhaps because there's such a littany of examples out there, but things like mod_rewrite are very difficult for me to understand after the fact, at least compared to uWSGI's --route's.

Redo 1.3 is released - now with more natural syntax!

Ben Hearsum — Tue, 07 Oct 2014 11:48:02 GMT

We've been using the functions packaged in Redo for a few years now at Mozilla. One of the things we've been striving for with it is the ability to write the most natural code possible. In it's simplest form, retry, a callable that may raise, the exceptions to retry on, and the callable to run to cleanup before another attempt - are all passed in as arguments. As a result, we have a number of code blocks like this, which don't feel very Pythonic:


retry(self.session.request, sleeptime=5, max_sleeptime=15,

      retry_exceptions=(requests.HTTPError, 

                        requests.ConnectionError),

      attempts=self.retries,

      kwargs=dict(method=method, url=url, data=data,

                  config=self.config, timeout=self.timeout,

                  auth=self.auth, params=params)

)

It's particularly unfortunate that you're forced to let retry do your exception handling and cleanup - I find that it makes the code a lot less readable. It's also not possible to do anything in a finally block, unless you wrap the retry in one. Recently, Chris AtLee discovered a new method of doing retries that results in much cleaner and more readable code. With it, the above block can be rewritten as:


for attempt in retrier(attempts=self.retries):

    try:

        self.session.request(method=method, url=url, data=data,

                             config=self.config,

                             timeout=self.timeout, auth=self.auth,

                             params=params)

        break

    except (requests.HTTPError, requests.ConnectionError), e:

        pass

retrier simply handles the the mechanics of tracking attempts and sleeping, leaving your code to do all of its own exception handling and cleanup - just as if you weren't retrying at all. It's important to note that the break at the end of the try block is important, otherwise self.session.request would run even if it succeeded. I released Redo 1.3 with this new functionality this morning - enjoy!

Redo - Utilities to retry Python callables

Ben Hearsum — Wed, 07 May 2014 18:07:24 GMT

We deal with a lot of flaky things in RelEng. The network can drop. Code can have race conditions. Servers can go offline temporarily. Freak errors can happen (more often than you'd think). One of the ways we've learned to cope with this is to add "retry" behaviour to damn near everything that could fail intermittently. We use it so much that we've got a Python library and command line tool that are used all over the place. Last week I finally got around to packaging and publishing ours, and I'm happy to present: Redo - Utilities to retry Python callables. Redo provides a decorator, context manager, plain old function, and even a command line tool to retry all sorts of things that may break. It's very simple to use, here's some examples from the docs: The plain old function:


def maybe_raises(foo, bar=1):

    ...

    return 1



def cleanup():

    os.rmtree("/tmp/dirtydir")



ret = retry(maybe_raises, retry_exceptions=(HTTPError,),

            cleanup=cleanup, args=1, kwargs={"bar": 2})

The decorator:


from redo import retriable



@retriable()

def foo()

    ...



@retriable(attempts=100, sleeptime=10)

def bar():

    ...

The context manager:


def foo(a, b):

    ...



with retrying(foo, retry_exceptions=(HTTPError,)) as retrying_foo:

    r = retrying_foo(1, 3)

You can grab version 1.0 from PyPI, or find it on Github, where you can send issues or pull requests.

How to deal with timezone adjusted "epoch" timestamps in Python

Ben Hearsum — Mon, 28 Oct 2013 17:57:15 GMT

Today I discovered that we have a system that returns "epoch" timestamps, but adjusted for Pacific time. This means that depending on whether daylight savings time is in effect, these timestamps are 7 or 8 hours ahead when interpreted by most tools. These are horribly difficult to deal with as unix timestamps are assumed to be in UTC time. I spent a good deal of time banging my head against Python's datetime and pytz modules (as well as the wall). With some help from John Hopkins I found a solution: In the following example we'll convert the Pacific "epoch" timestamp 1383000394 to a proper epoch timestamp (which is 1382975194). First, we need a tzinfo object for Pacific time: >>> import pytz >>> pacific_time = pytz.timezone("America/Los_Angeles") Next, we need to get the initial timestamp into a datetime object to work with it. Note that it's important to use utcfromtimestamp() here otherwise you'll get a localized datetime object - which will only be useful if the machine you run this on is in Pacific time: >>> from datetime import datetime >>> dt = datetime.utcfromtimestamp(1383000394) >>> dt datetime.datetime(2013, 10, 28, 22, 46, 34) It gets a little weird from here. We need to subtract the Pacific offset from the datetime object in order to get it into an actual UTC time. To do that we can force the datetime object in UTC time and then use its built-in astimezone() method to do the conversion. I think this still leaves a 7 or 8 hour window whenever DST starts or ends where this conversion is an hour off - but it's good enough for my usage: >>> dt = dt.replace(tzinfo=pytz.utc) >>> dt datetime.datetime(2013, 10, 28, 22, 46, 34, tzinfo=) >>> dt = dt.astimezone(pacific_time) >>> dt datetime.datetime(2013, 10, 28, 15, 46, 34, tzinfo=) Now we have a datetime object with the correct time, but claiming to be in Pacific. We can fix that by replacing the tzinfo again: >>> dt = dt.replace(tzinfo=pytz.utc) >>> dt datetime.datetime(2013, 10, 28, 15, 46, 34, tzinfo=) The only thing left to do now is convert to epoch time! >>> import calendar >>> calendar.timegm(dt.utctimetuple()) 1382975194 Voila, the timestamp we were looking for! Much credit to John Hopkins for his code that taught me how to use datetime.replace() and astimezone(). No credit at all goes to Python's datetime module, which is sorely in need of an overhaul.

Loading Python modules from arbitrary files

Ben Hearsum — Tue, 21 May 2013 16:46:36 GMT

tl;dr: Use imp.load_source. I've been hacking on a tool on and off that needs to load Python code from badly named files (eg, "master.cfg"). To my surprise, there wasn't an obvious way to do this. My "go to" method of doing this is with execfile. For example, this will load the contents of master.cfg into "m", with each top level object as a key: m = {} execfile("master.cfg", m) This works well enough for simple cases, but what happens when you try to load a module that loads other modules? It turns out that execfile has a nasty limitation of requiring modules that aren't in sys.path to be in the same directory as the file that calls execfile. You can't even chdir your way around this, you have to copy the files you need to the caller's directory. (We actually have some production code that does this. Someone in #python on Freenode suggested using importlib. That seemed like a fine idea, especially after recently watching Brett Cannon's "How Import Works" talk. Unfortunately, Python 2.7's importlib only has a single method which can only load a module by name. Eventually I came across a Stack Overflow post that pointed me at imp.load_source. This function is similar to execfile in that it loads Python code from a named file. However, it properly handles imports without the need to copy files around. It also has the nice added bonus of returning a module rather than throwing objects into a dict. I ended up with code like this, to load the contents of "foo/bar/master.cfg": >>> import os, sys >>> os.chdir("foo/bar") >>> sys.path.insert(0, "") # Needed to ensure that the current directory is looked at when importing >>> m = imp.load_source("buildbot.master.cfg", "master.cfg") Problem solved!