How object storage manages big data in the IoT

There’s no question: The era of big data has been a paradigm shift for businesses. Now, creating an easily searchable 'data lake' — either on-site or cloud-based — is a key determinant of a company’s success. Conceivably, trillions of files of any size must be managed and protected without degradation in performance.

Dependence on intelligent and cost-effective data storage will only grow in the next decade. Last year, Gartner forecast that 6.4 billion connected things would be in use worldwide this year and predicted that number would reach 20.8 billion by 2020. With 5.5 million new things connected every day, data storage and access systems are critical to the foundation of today’s business technology.

With the Internet of Things on the rise, data flows into servers not only from keyboards and mice, but also from a rich ecosystem of sensors and surveillance equipment worldwide, massively increasing an intake of data that often must be stored and rapidly accessible for years.

But this wealth of data becomes vulnerable when companies rely on legacy storage. Such systems are locked to specific hardware and were never designed to respond to today’s unpredictable and rapidly exploding data landscape.

Instead, a software-defined approach such as object storage allows the necessary flexibility — in terms of hardware upgrades or changes to workload and use cases — to compete in a data-intensive world.

Data + metadata = the need for object storage

It’s not just that there is a massive amount of data in the world; it’s that the rate of data production grows every day. In 2013, ScienceDaily reported that 90 per cent of all the data created in human history had been produced in the previous two years. Organisations often devote parallel computing clusters (such as Google’s 1,000 computers per search query) and other techniques just to manage that data.

Such improvements to accessibility are critical to the bottom line. For Fortune 1000 companies, a 10 per cent boost in data management can return $65 (£45) million in revenue. But not all low-hanging fruit has been plucked, either. It has been estimated that we are analysing less than 1 per cent of all data.

In fact, it’s not only data itself that raises the bar for companies. It’s metadata, too. Metadata is the additional information stored along with primary content — data about the data. It’s crucial for indexing and organising almost any additional information, such as descriptions of an event, user sentiment, tags, dates, geospatial information, or other characteristics.

Metadata permits a more sophisticated approach to analytics. For data management, it greatly enhances data reuse, database 'housecleaning', accountability, and security. Properly tagged content is searchable and, therefore, monetisable. For product development teams, mining the granular data available from devices allows teams to better understand how devices are used and how to make them more efficient and useful.

Taken together, the growing torrent of content and its associated metadata means that companies that effectively leverage their 'long-tail' and unstructured data will be positioned to reap benefits from analysing their data that far exceed what they could do with traditional data storage and management technology.

Object storage is designed to meet this specific opportunity. But what exactly is it, and how does it work?

Object storage and the IoT

High-tech sensors are only as good as the data they acquire, and that data is only as good as the mechanisms that ensure its accuracy, security, and accessibility. For this reason, antiquated approaches to storage like network-attached storage are woefully inadequate for taking advantage of data in IoT. Here are four ways object storage can benefit any company working in the IoT industry:

Added flexibility

With increased automation, software-defined object storage transforms commodity hardware for the enterprise into resilient and infinitely scalable data lakes accessible via HTTP. Instead of using files or blocks, it treats data as modular objects so that each is like a cell in a beehive, containing an ID number, metadata and attributes, and the content itself. As a result, pooled resources are managed elastically, depending on the shifting needs of performance, accessibility, value, and other factors.

Reduced costs

Keeping costs down is a primary concern as IoT missions scale out, so the efficiencies of object storage running on commodity hardware are attractive. But this does not mean reverting to the 'silo' approach to data storage.

Object storage gives you online access to petabytes of data using nodes. A cluster of such nodes can serve as a scalable public or private cloud-based file repository. Because you can use standard servers and internal solid-state disk or hard drive disk storage, it provides much lower total cost of ownership, both in the hardware itself and the human power required to manage it.

Object storage can also reduce data centre costs because some systems can even power down storage and nodes. As a result, it saves on power and cooling, making it as economical as offline media storage like tape.

Increased data protection

Of course, data protection needs to be in place for the long haul — you may even go decades between accessing certain pieces of data. Object storage manages this concern transparently by periodically checking the integrity of data and dynamically correcting any issues found.

Enhanced searchability and analysis

How do you actually find that data when you need it? Best-of-breed object storage architectures attach metadata to the actual object, so you can easily search as needed and retrieve data. This metadata also gives you the ability to sort and analyse data for actionable insights.

Exploring object storage for your IoT business

Understanding the contextual nature of each piece of data in the IoT (such as the type or state of a sensor) is integral to making the most of that data. So object storage’s effective management of metadata is a game changer and should be implemented into any company working in the IoT. To set up an object storage system, it’s wise to consider these four steps:

Explore your company’s needs

As a business working in IoT, how much data does your company see on a given day? How much data do you anticipate needing to store? Are you transitioning your data storage from a traditional storage system? If so, how soon do you want everything to be transferred over?

Seek out an expert

Often an experienced sales engineer or value-added reseller can save you time and money by identifying the right storage solution for your needs based on years of helping people with similar use cases.

Test the waters

Put together a proof of concept so your company can test an object storage system to prove it’s compatible with a given software environment. Beware of vendors that will not allow you to test their software prior to purchase: It may be indicative of problems with their product or a sign that they do not offer the needed technical support to ensure proper integration.

Map your destination

A plan for future growth is necessary to ensure that as data acquisition and analysis expands, the system can scale to accommodate it.

This last step is vital. Ignoring the long-range needs of an organisation can ultimately limit opportunities down the road, whether in predictive analytics, big data, monetisation of data, or even legal protections (such as medical or financial contexts).

Given the complexities and unpredictability of the new world of big data and the IoT, it’s clear that data management is one of the most important aspects of business practice in the 21st century. Fortunately, object storage approaches are allowing for resilient, highly available, and web-accessible metadata-driven interactions with content — and with a lower total cost of ownership than traditional storage systems.

Jonathan Ring is co-founder and CEO of Caringo