BBC issues explanation for iPlayer and website outage

As promised, the BBC has come forward with an explanation of what happened last weekend to cause major issues with iPlayer and the Beeb's website (many found they could not watch programmes, and the latter was reduced to a "simple" mode while the problem persisted).

The issues, which were announced as fixed on Tuesday, were actually caused by two unfortunately timed incidents.

In a blog post, Richard Cooper, Controller Digital Distribution at the BBC, explained: "We have a system comprising 58 application servers and 10 database servers that provides programme and clip metadata. This data powers various BBC iPlayer applications for the devices that we support (which is over 1200 and counting) as well as modules of programme information and clips on many sites across BBC Online."

"At 9.30 on Saturday morning (19 July 2014) the load on the database went through the roof, meaning that many requests for metadata to the application servers started to fail."

"The immediate impact of this depended on how each product uses that data. In many cases the metadata is cached at the product level, and can continue to serve content while attempting to revalidate. In some cases (mostly older applications), the metadata is used directly, and so those products started to fail."

This issue was compounded by a second problem, whereby a caching layer pool that manages the iPlayer and Beeb homepage failed, and the "the failure made all of those products inaccessible".

Check out the blog post for the full and detailed lengthy explanation.

What isn't clear is what caused the spike of the massive load on the database, and whether it was a malicious action or not (such as a DDoS attack). The BBC said that it is now undertaking forensics to trace the root causes, and it will "put in place the measures necessary to minimise the chances of such interruptions in the future".

Meanwhile, folks are still complaining about some issues with iPlayer right now, namely iPlayer Radio categories not displaying recent programmes. However, the Beeb says this is a separate issue, and an "oversight" when they implemented a tweak to make radio programmes visible for the last 30 days. A fix is apparently incoming.