Skip to main content

Navigating the budgetary and infrastructure challenges of capturing, sharing and preserving data

(Image credit: Image source: Shutterstock/alexskopje)

“Action without vision is only passing time; vision without action is merely daydreaming; but vision with action can change the world.” - Nelson Mandela

The recently published fifth annual 2020 Data Storage Outlook report from Spectra Logic explores the ways in which the world manages, accesses, uses and preserves its ever-growing data repositories. The report provides an annual overview of trends and predictions in storage technology development, availability and strategy in order to support data storage and media manufacturers, application developers, and all sizable users of data as they plan for their requirements and strategies for the coming years. As Nelson Mandela wisely says in the quote above, action alone and vision alone have no significant effect, but vision coupled with action can change the world. This combination will be the key to meeting data storage and data management challenges now and will be vital in driving the technological leaps that will be required by the generations to come.

With that said, the year 2020 started as no other in modern history. Today, more than ever, as many sectors of the economy remain unclear due to the coronavirus outbreak, the vision and steps taken to advance technology and fortify supply chains today will further preserve the world’s treasury of information for tomorrow.

Technology advancements meet growing requirements

For the foreseeable future, customers’ storage growth requirements will be fulfilled by technology providers who will continue to innovate with faster performance and higher capacities to meet increasing demand. As noted in the report, every storage category is exhibiting technology improvements.  Memory hosted Xpoint technology is becoming the latest high-performance standard for database storage. At the flash layer, 3D fabrication technology is allowing for the creation of higher density parts while lowering the cost per gigabyte. Moreover, disk manufacturers are closing in on delivery of HAMR and MAMR technologies that will allow them to initially provide disk drives of 20TB while also enabling a technology roadmap that could achieve 50TB or greater over the next ten years. And, tape has enough technology headroom that it will achieve storage capacities of 100TB or higher on a single cartridge in the next decade. In fact, as smaller tape systems make way for cloud-based solutions, many organizations, including cloud providers, will continue to adopt tape for long-term archive tiers.  Not to be left out, cloud providers have a unique opportunity to adopt new storage technologies, based on the sheer size of their storage needs and small number of localities, ahead of volume commercialization of these technologies.

The data storage dilemma

Customers continue to face the dilemma of what data should be stored on which medium at what time. Many organizations today face unnecessary storage costs due to the fact that up to 80 percent of digital assets are stored on the wrong storage tier.

A new generation of software tools is emerging that allows end users to identify the usage patterns of their data and then provide for the movement of infrequently accessed data to less expensive storage targets, greatly improving data storage efficiencies while mitigating storage costs.

Previously, Spectra Logic proposed a logical two-tier architecture comprised of a Primary Tier (for data ingest and work in progress) and a Perpetual Tier (for data sharing, distributing and archiving). The 2020 report expands upon this, envisioning that the first logical tier’s storage requirements will be satisfied entirely through solid-state storage technologies while the second-tier requirements will be satisfied by magnetic disk and tape deployed through object storage either on-premise or in the cloud.

Designing with the cloud in mind

Over the last few years, storage administrators have had to consider what data should be placed in the cloud, what data should be located on-premise, and what data should be stored in both locations. Each location offers benefits and cost trade-offs. The demand by storage customers to use cloud-based storage prompted many legacy storage providers to ‘shoehorn’ basic cloud capability into their existing products. Primarily this has consisted of providing customers with the ability to send disaster recovery copies of their on-premise data to the cloud. This shoehorn pattern that has been seen before such as in the adoption of flash technology into disk arrays. The first generation of storage systems to utilize flash were existing products that were designed before flash storage was available. Customers gained some benefit, but not the full scope of the technology. Second- and third-generation solutions were designed with flash in mind and provided tremendous capability to the customer and over the last few years these solutions have become a growing segment in the storage system business.

Cloud integration by on-premise storage systems is in this first phase. Next-phase products will be designed from the ground up with the cloud in mind. These products will allow seamless integration of applications into the storage infrastructure, regardless of storage location -- whether in the cloud and/or in multiple on-premise locations. Complex customer workflows will be supported, through policies set by the customer, that allow data to be automatically moved to the right location(s), to the right storage tiers, at the right time. With this capability, customers will have the freedom to decide which processes they want to run locally and which ones in the cloud – all without having to think about the underlying storage system.

Future storage technologies

In addition, there are many interesting storage ideas being pursued in laboratory settings at different levels of commercialization: storing data in DNA, 3D RAM, (5-dimension optical) hologram storage – plus many that are not yet known. Historically many storage technologies have shown promise in the prototype phase, but have been unable to make the leap to production products that meet the cost, ruggedness, performance, and most importantly, reliability of the current technologies in the marketplace. However, given the advent of cloud providers, the avenue to market for some of these technologies might become easier.

Planning for the Future

Spectra’s projections in the 2020 Data Storage Outlook Report do not anticipate shortages or rising media costs. But, due to the coronavirus pandemic there could be short-term supply-side shortages. it is unclear at this point whether reduced demand will result in a balanced or unbalanced market.  Nevertheless, whether still battling the pandemic or fully on the other side of it, global corporations, government entities, cloud providers, research institutions, and curators must continue to plan today for the impending growth in data that will impact their organizations, agencies and customers well into the future.

The full Spectra Logic 2020 Digital Data Storage Outlook report, including detailed analysis of the different storage media, strategies and architectures, and the size of the digital universe, can be accessed here

David Trachy, Senior Director of Advanced Technology Development, Spectra Logic