To file archive or not to file archive?

New independent research, the second annual BridgeHead Software Information Lifecycle Management (ILM) Audit, confirms that significant IT data growth continues unmitigated.

In both the US and UK there is a continuing trend towards storing more data on primary storage. In 2005 only 16% of UK respondents had more than 5 terabytes (TB) of online data, in 2006 this has grown to 22%, an increase of over a third. Similarly, those storing in excess of 10TB also jumped by 30%.

The degree to which respondents believe primary storage is consumed by redundant data also continues to increase. Over 18% more respondents than last year believe they could recoup over 50% of their most expensive primary disk space by removing unwanted data. While continued data growth is not surprising in general, the year-to-year comparison captured a clear discrepancy between how different types of data are driving this growth.

Both in the US and UK, the growth of online data is dominated by unstructured data - typically end-user files created outside of an IT-managed application. There has been an 81% increase in respondents who calculated that over half of their data was unstructured files. While on the one hand, this dramatic increase may be a leader of greater corporate productivity, it is a potential nightmare for IT managers.

User files tend to be created and stored on network file shares without the benefit of IT's most structured management and control systems, making them hard to find, manage and correctly store for compliance and data protection.

In contrast to unstructured data, data held in database applications currently makes up to 30% of server data for nearly 75% of respondents. For more than two thirds of UK respondents, email data has grown in tandem with overall data growth and still makes up to 20% of server-based data.

The ILM Audit reveals that businesses have recognised the need to tackle unstructured files. To the question: "What are your key storage-related areas of interest in the next 12 months?", file archiving has made the biggest leap forward compared to the responses in 2005. Twenty eight percent increase in UK respondents listed file archiving as a key area of interest, whereas email archiving registered a lower increase of 16% and disaster recovery only increased by 4%.

Despite the growing awareness of file archiving, 6% UK respondents said that they would likely not be able to find and restore a three month old file and 27% said they didn't know how long retrieval of a 3-month old file would take. The continued lack of confidence in long-term retention as the apparent adoption of archiving rises leads one to wonder whether the archiving strategies being adopted are effective or if there isn't a more acute awareness that better tools are needed.

Tony Cotterill, CEO of BridgeHead Software, said: "While the figures are interesting by themselves, they clearly illustrate that companies simply cannot let up in their efforts to archive. Even with almost 85% of companies claiming to archive in some way, the fact that data on primary storage is still growing at 25% minimum indicates that the most effective archiving tools and technologies are not yet as prevalent as they should be.

"The big question for those who continue to put off archiving is 'why?'. Information Lifecycle Management is now considered a mature concept and thinking is moving on to concepts such as Protected Data Lifecycle Management (PDLM). So there is a danger of a gap developing between the storage management 'haves' and the 'have nots', with the have nots accumulating not only data, but also excessive storage costs and painful manual procedures as well," he added.

Protected DLM integrates data protection, business continuance, and disaster recovery strategies into the long-term retention and management of data as its lifecycle requirements cause it to be copied into and subsequently repositioned entirely to a secondary storage archive. It does this by allowing archives to be defined as multiple copies on multiple media types and it uses a distributed architecture to allow these copies to be written and managed at different network locations.

Protected DLM represents the full integration of archiving with other vital storage management processes into a single enterprise-wide facility for ensuring that data is available for both operational and disaster recovery, that it is protected and compliantly retained for suitable periods, and that the most cost effective storage technology can be leveraged to minimize storage and storage management costs.