Ben Hearsum (Posts about debugging)

How to deal with timezone adjusted "epoch" timestamps in Python

Ben Hearsum — Mon, 28 Oct 2013 17:57:15 GMT

Today I discovered that we have a system that returns "epoch" timestamps, but adjusted for Pacific time. This means that depending on whether daylight savings time is in effect, these timestamps are 7 or 8 hours ahead when interpreted by most tools. These are horribly difficult to deal with as unix timestamps are assumed to be in UTC time. I spent a good deal of time banging my head against Python's datetime and pytz modules (as well as the wall). With some help from John Hopkins I found a solution: In the following example we'll convert the Pacific "epoch" timestamp 1383000394 to a proper epoch timestamp (which is 1382975194). First, we need a tzinfo object for Pacific time: >>> import pytz >>> pacific_time = pytz.timezone("America/Los_Angeles") Next, we need to get the initial timestamp into a datetime object to work with it. Note that it's important to use utcfromtimestamp() here otherwise you'll get a localized datetime object - which will only be useful if the machine you run this on is in Pacific time: >>> from datetime import datetime >>> dt = datetime.utcfromtimestamp(1383000394) >>> dt datetime.datetime(2013, 10, 28, 22, 46, 34) It gets a little weird from here. We need to subtract the Pacific offset from the datetime object in order to get it into an actual UTC time. To do that we can force the datetime object in UTC time and then use its built-in astimezone() method to do the conversion. I think this still leaves a 7 or 8 hour window whenever DST starts or ends where this conversion is an hour off - but it's good enough for my usage: >>> dt = dt.replace(tzinfo=pytz.utc) >>> dt datetime.datetime(2013, 10, 28, 22, 46, 34, tzinfo=) >>> dt = dt.astimezone(pacific_time) >>> dt datetime.datetime(2013, 10, 28, 15, 46, 34, tzinfo=) Now we have a datetime object with the correct time, but claiming to be in Pacific. We can fix that by replacing the tzinfo again: >>> dt = dt.replace(tzinfo=pytz.utc) >>> dt datetime.datetime(2013, 10, 28, 15, 46, 34, tzinfo=) The only thing left to do now is convert to epoch time! >>> import calendar >>> calendar.timegm(dt.utctimetuple()) 1382975194 Voila, the timestamp we were looking for! Much credit to John Hopkins for his code that taught me how to use datetime.replace() and astimezone(). No credit at all goes to Python's datetime module, which is sorely in need of an overhaul.

Cleanup is important (for more than just cleanliness)

Ben Hearsum — Wed, 30 Jan 2013 19:13:08 GMT

A few weeks ago we had a very strange problem with a release. We didn't have the time to investigate it, so we added a hack to workaround it and carried on. I've been slowly digging deeper into the issue this week and finally got to the bottom of it. As it turns out the root problem was that our release builds weren't using the same "make" as the non-release builds! This is absolutely unintended, and could've been the source of even more, subtler errors. Eventually, we would've hit a problem that we couldn't have hacked around, and would've had to make scary changes under time pressure to fix it. This is a great example of why it's important to remove hacks from your code and take the time to understand the errors you hit. You can clean your house all you want but if you don't find out where the mess is coming from you'll never keep up with it.

GRUB, the MBR, BIOS bugs?

Ben Hearsum — Tue, 09 Mar 2010 19:22:46 GMT

Recently we ordered a set of server class Linux machines to supplement our pool of VMs. They are lightning fast, especially compared to VMs, but it's been a bit of a bumpy ride getting them ready to go to production. Most notably we've had an mysterious problem where they would occasionally refuse to boot, halting at a "GRUB _" dialogue. It took awhile, but we believe we have this fixed now. This problem first occurred on 2 out of 25 slaves. Catlee quickly discovered that it could be fixed with a simple re-installation of Grub, so that's we did, and moved on. The thought at the time was that the MBR somehow got partially overwritten or otherwise corrupted. A day later, 2 more slaves hit the same issue. Since it was the second time we hit the issue there was some more speculation and digging. We had made a few changes to the machines, including: * Changing the hard disk controller from "IDE" mode to "AHCI". * Changing the kernel to a PAE version. Both of those were pretty quickly dismissed as the causes. It seemed very unlikely that the kernel version could cause an issue with the bootloader, and the problem didn't occur instantly after changing the disk controller mode, so that seemed unlikely too. With other important things happening we again moved on. The next day I did some more googling, this time about GRUB in general, and came across a page detailing the GRUB boot process. In it, it talks about how to dump the contents of the MBR and view it as hex. Seeing that made me *very* eager to compare a working slave vs. a busted one. Unfortunately there was no longer a busted machine to look at. After 5 or so days without issues, and after all other setup and configuration issue was taken care of we decided to move them to production and deal with the GRUB problems if they arose. As luck would have it, 2 machines refused to boot as they were being moved to production. After booting from a rescue disk and dumping the MBR I found that bytes 0x40 through 0x49 differed against a working slave. I also noticed that the MBR of a busted slave was identical to one that had *never* broken, and thus, never had GRUB re-installed. This seemed to rule out MBR corruption. With some more information in my hands I looked for some help or pointers from the GRUB developers, on Freenode. One of them pointed me to this section of the GRUB Manual which documents some key bytes of the MBR. Notably, byte 0x40 is described as "The boot drive. If it is 0xFF, use a drive passed by BIOS.". On a working slave this was set to 0xFF. On a broken one, it was set to 0x80 (which I was told means "first hard drive"). That certainly sounds like something that could affect bootability! After thinking it over a few times I came to the conclusion that *somehow* 0x80 must end up being the wrong device to boot from. I also realized that no slave which had had GRUB re-installed had failed again. With all of that I became confident that re-installing GRUB would fix the problem permanently. I ran all of this by Catlee who told me that GRUB developers had told him that the BIOS could be re-ordering drives semi-randomly. That piece of information seems to fill in the last bit of the puzzle and I'm more confident than ever that GRUB installation will permanently fix the problem. It's still a mystery to me why the BIOS would be re-ordering the drives at random. There's a "BIOSBugs" page on the GRUB wiki which describes a problem where the BIOS sends the *wrong* boot device. Since relying on the BIOS to send the boot device has fixed our problem I don't think it's the same thing. I haven't been able to find any information on this specific issue, or how to find out what boot device the BIOS is sending the Bootloader, which makes it difficult to truly confirm our fix. If anyone has hit this, or knows how to get at this kind of information I'd love to hear from you.