Skip to main content

Leap second crashes major websites

It didn't seem like such a bad idea at first. To correct for minor decelerations in Earth's rotation, timekeepers added an extra second to the clock on 30 June – bringing Earth roughly four-tenths of a second ahead of the official time instead of six-tenths of a second behind.

However, the inclusion of that second – added during the minute prior to 00:00 hours GMT – was not as well-received by systems used to a conventional 60-second minute. And, go figure, one tiny little second managed to bring down a number of the Web's more well-known sites as soon as the clock hit "61" instead of "60".

Sites affected by the second included Reddit, Yelp, Linkedin, all of Gawker media's various Web properties, 4chan, Fark, Stumbleupon and Pirate Bay, to name a few.

"Servers running java apps such as Hadoop and ElasticSearch and java doesn't appear to be working. We believe this is related to the leap second happening tonight becuase [sic] it happened at midnight GMT," wrote Eric Ziegenhorn in a Mozilla bug update entitled, "Java is choking on leap second."

And, of course, there were the natural issues that arise when Linux servers haven't been correctly configured to allow for the extra second to take place – a common issue when systems are configured under the assumption that every minute is always going to be 60 seconds.

"So bad that, after exactly forty years from the first leap second, systems and applications still rely on these assumptions and can crash badly when, during a leap second insertion, they find themselves in a situation they didn't expect," writes Marco Marongiu, a senior system administrator at Opera, in a fairly lengthy post designed to show Linux sysadmins how they can reconfigure their systems to fix the issue.

Google's sites remained unaffected by the addition of the extra second, but the inclusion of leap seconds and other time adjustments has been something that Google's already prepared itself for – since 2008, in fact, when the last leap second was thrown into the mix.

"The solution we came up with came to be known as the 'leap smear.' We modified our internal NTP servers to gradually add a couple of milliseconds to every update, varying over a time window before the moment when the leap second actually happens. This meant that when it became time to add an extra second at midnight, our clocks had already taken this into account, by skewing the time over the course of the day," wrote Google site reliability engineer Christopher Pascoe in a blog post last year.

"All of our servers were then able to continue as normal with the new year, blissfully unaware that a leap second had just occurred."