Skip to main content

Amazon EC2 cloud crash brings sites to standstill

UPDATE: Amazon Web Services issued the following explanation for the EC2 and Elastic Block Storage outages:

8:54 AM PDT "We’d like to provide additional color on what were working on right now (please note that we always know more and understand issues better after we fully recover and dive deep into the post mortem). A networking event early this morning triggered a large amount of re-mirroring of EBS volumes in US-EAST-1. This re-mirroring created a shortage of capacity in one of the US-EAST-1 Availability Zones, which impacted new EBS volume creation as well as the pace with which we could re-mirror and recover affected EBS volumes. Additionally, one of our internal control planes for EBS has become inundated such that it’s difficult to create new EBS volumes and EBS backed instances. We are working as quickly as possible to add capacity to that one Availability Zone to speed up the re-mirroring, and working to restore the control plane issue. We’re starting to see progress on these efforts, but are not there yet. We will continue to provide updates when we have them."

Amazon's Elastic Cloud Compute (EC2) service, used by a host of well-known companies and services, has crashed - taking down half the internet with it.

Smart-arse networking site Quora, geo-location service service for show-offs, FourSquare, and Twitterholic app Hootsuite are among the victims who've been royally rogered after choosing Amazon to service their backend.

The problems are believed to have begun after 02:18 PDT (09:18 GMT), when Amazon admitted it was having issues with its EC2 cloud, as well as its own Relational Database Service - both are which are based at the same data centre in North Virginia.

"We can confirm connectivity errors impacting EC2 instances and increased latencies impacting EBS volumes in multiple availability zones in the US-EAST-1 region. Increased error rates are affecting EBS CreateVolume API calls. We continue to work towards resolution," the company announced in a somewhat garbled statement.

The company's cloud hosting status page is still currently reporting problems at the North Virginia centre.

As long as Amazon's elastic service remains snapped, services that rely on EC2 are at a standstill. Question-and-answer site Quora is entirely hosted on the service, and continues to be unreachable at the time of writing.

Amazon has so far not said how long it will take to fix the problem.