Opponents of the NSA's secretive data-gathering campaign PRISM will have had something to cheer about in recent months, as massive electrical failures have crippled the organisation's newest datacentre in Bluffdale, Utah. However, the agency has claimed that "the failures that occurred during testing have been mitigated."
The Wall Street Journal reported on Monday that there have been a total of 10 "meltdowns" over the past 13 months at the new Utah datacentre, which have dramatically set back its operational capacity. The latest failure was on 25 September.
It's thought that each failure has cost upwards of $100,000 (£62,644), which, added to the at least $1 million (£62.64 million) a month cost of keeping the centre flowing with electricity, makes for some eye-watering losses.
The enormous new data-processing centre, which is set to be the agency's largest yet, covers 247 acres, includes 1.2 million square feet of enclosed space, four separate 'data halls', and is tabled to cost $1.4 billion (£876.88 million).
That's without even considering the cost of the new Cray XC30 supercomputers set to power the centre's processing system. The NSA promises that the Utah centre will "host the power, space, cooling and communications needed to support specialized computing." The accumulated processors will offer speeds of 100 petaflops, or one thousand trillion (1,000,000,000,000,000, or 10^15) calculations per second.
They will be "the most sophisticated supercomputers and largest reserves of data storage on the planet", and at any one time will require an average of 65 megawatts of electricity to run.
However, multiple failures have crippled the project. Most of these problems have been in the form of 'arc failures', which one official described as "a flash of lightning inside a 2-foot box." The failures can have dramatic consequences: creating fiery explosions; melting metal casings and components, as well as causing widespread circuit failure.
Rigorous testing has taken place, with 160 checks over 50,000 man-hours.
"During the testing and commissioning of the Utah Data Center, problems were discovered with certain parts of the electrical system," the Army Corps of Engineers said. The spokesperson went on to defend the project, saying "issues such as these can arise in any project and are the reason the Corps tests and reviews every aspect of any project prior to releasing it to the customer."
"The customer", in this case, being the NSA. However, some people aren't so sure. A source familiar with the construction of the database told Forbes magazine that "the problem, and we all know it, is that they put the appliances too close together."
The source went on to say "they used wiring that's not adequate to the task."
Despite the problems, the NSA claims that it's on top of things. "The failures that occurred during testing have been mitigated," a spokesperson said, before adding somewhat defensively that "a project of this magnitude requires stringent management, oversight and testing."
Most analysts have pointed out that "mitigated" doesn't mean the same as 'solved'. The NSA has offered no comment on when the centre will be operational.
Meanwhile, a protest group has obtained the contract to clean up the highway just outside the datacentre. It has promised to carry anti-NSA placards while it carries out the task.
Image: Business Insider