Businesses despair as script error at 123-Reg deletes chunk of the internet [Updated]

An error at hosting and domain provider 123-Reg over the weekend has ended up deleting part of the internet, taking hundreds of websites offline with some unable to recover any of their data.

Businesses saw their websites disappear on Saturday after a script error affected a number of customers using a virtual private server (VPS) to store their data. Those that have their own backups will be able to restore their websites, but the unlucky businesses that backed up their data with 123-Reg will be unable to do so.

A statement from 123-Reg on Saturday said: "Some customers on our VPS platform may currently experience connectivity issues. Our engineers are aware of this issue and are currently in the process of resolving this."

An email sent out later by the company's brand director, as seen by MailOnline, said: "This script is run to show us the number of machines active against the master database. An error on the script showed 'zero-records' response from the database for some live VPS. For those customers, this created a 'failure' scenario...effectively deleting what was on the host. As a result of our team's investigations, we can conclude that the issues faced having resulted in some data loss for some customers."

"We have recovery running on the VPS servers and some are restoring to new disks. We have also begun copying recovered VPS images to new hosts and we expect some VPS to be back up and running throughout the night and in to tomorrow."

Customers are understandably extremely unhappy about the loss of their websites, with several companies - such as e-commerce businesses relying solely on their online presence to generate revenues.

Nigel Tozer, Solutions Marketing Director, EMEA at Commvault commented: “If there was ever a good argument for offsite backup, this is it; a backup on-site is just another copy, it’s not really a backup. Many organisations are turning to cloud providers such as Azure and AWS as the modern way getting data safely offsite, even if they run a service themselves, as in this case.

"Businesses should choose software that can securely send data into the cloud and also enables you to recover your systems there, converting to the right hypervisor at the same time. This means you have off-site protection and disaster recovery without the cost usually associated with duplicating infrastructure and premises.”

UPDATE: 123-Reg has issued the following statement: "We suffered a technical fault, which has affected a minority of 123 Reg customers. The fault was limited to 67 servers out of 115,000 (across Europe) and it is important to note that only a selection of customers on the 67 servers affected has been impacted. We are investigating the restoration of each VPS on a case-by-case basis and are working individually with customers to keep them informed of the website recovery process. We are working as quickly as we can to restore service to normal.

"Our VPS product is an unmanaged service and we always recommend that customers implement backups to safeguard against unexpected issues. Customers who had purchased 123-Reg backups can be online now as can those who are using another solution for website backup. If customers restore from their own backups, this will not overwrite our efforts. Additionally, customers who have restored from their own backups are now hosted on new servers. By using new servers, we will ensure that we do not overwrite the previous servers and impact the data recovery process. We are working with Kroll, the leading data recovery specialist to manage the process of restoration.

"We would like to extend our apologies to affected customers and assure them that we are doing everything we can to restore their data as quickly as possible. We are keeping customers informed as we restore their websites."

Photo Credit: Mr.Exen/Shutterstock