How search will save us from the Big Data flood

Information technology is gaining increasing influence in all aspects of everyday life. Big Data is at the vanguard, and the world as we know it is about to undergo a radical change.

While the primary goal in the early days of technology was to process large volumes of data and to conduct searches within it, the current development is definitely trending toward intelligent assistance systems.

1. Big Data, a Treasure Chest

Companies are often referred to as living organisms, a metaphor which works neatly in the minds of many a wishful-thinking managing director, but fails to correspond to the nitty-gritty corporate reality. The fact of the matter is that what really makes the operative difference for a company is not the joined forces within, working together toward a common goal, but rather the trenches running through that company, that severely limit its ability to move forward - trenches between departments that are in competition, trenches between employees who want nothing to do with each other, trenches between top management and the rest of the staff.

Trenches, too, between the individual applications used by a company, which do not communicate well with each other. The effects of these internal shortcomings: the company's existing data, information and knowledge are widely dispersed and only rarely manage to cross the divide.

A company’s internal data can be a veritable buried treasure. Those who manage to unearth it and use it effectively for their business are the companies who will take the lead. Now is the time in which Big Data technologies will be increasingly used to gather the data, information and subsequently the knowledge that is scattered throughout a company, to consolidate it across all the entrenchments and to use it as a key advantage in everyday business - a golden asset that benefits every employee in the company, regardless of how deep the divides may be.

2. Making the Connection

A major strength of Big Data is its ability to recognise patterns and correlations where people can only see data chaos. There are already hundreds of examples from all walks of life - business, science, medicine and sports.

Analyses can predict the course of an influenza epidemic in minute detail or analyse typical behaviour patterns to anticipate when a customer is toying with the idea of terminating a contract. The ability to make these connections connotes comparisons to how the human mind works.

Even though IT technology may never come close to human intelligence, systems are already in use today that help people accomplish things more intelligently. The development and use of these intelligent assistance systems will be intensified in the coming months and years to help people and companies navigate in a steadily more complex world and to create the necessary competitive edge in business.

3. Customised information processing

Everyone has his or her own particular way of acquiring and processing knowledge effectively, and relating it in a way that is profitable for all involved. A system that supports an individual approach to information must have, first and foremost, comprehensive knowledge about that person: his preferences, his background and experience, his behavioural tendencies in certain situations.

More and more, enterprise search solutions are being employed to personally customise a company website for users, visitors and employees. From the design of the user interface to the structuring of its contents and countless things in between – the possibilities for tailoring to the user’s needs and preferences are endless.

The Big Data application automatically collects and analyses any and all information necessary for this customised access on the basis of user behaviour. The more often a person accesses the system, the more accurately the information presented to him will be consistent with his needs. This means, for instance, that while all employees of a company will have access to the same knowledge base, they will do so in completely different ways, with significantly increased productivity and greater job satisfaction as a result.

4. Self-learning systems

Big Data systems such as enterprise search solutions can be instrumental in helping companies economically handle a flood of correspondence. Insurance companies, for instance, need to cope daily with countless incoming data, both structured and unstructured, in the form of letters or e-mails.

To make things easier for the agent, enterprise search can extract all relevant information from the incoming mail, things like the names of people or places, license plate numbers and type of damage. Based on this information, the damage reports can be classified automatically, increasing processing speed.

Should errors in classification occur, they can be corrected manually, after which the system recognises the new classification and applies it from that point forward. Viewed in this light, enterprise search solutions - under human supervision – are self-learning applications. The longer they are in use, the more accurate the results, a trend that’s gaining speed in 2015.

5. Closing the data flood gates

The "Internet of Things" is a buzzword that pervades today’s media and refers to the network of everyday consumer objects that have built-in interactive elements such as sensors which are integrated into the global network of the Internet.

The repercussions of this all-encompassing connectivity are fairly predictable: the amount of data will skyrocket. A single airplane collects around 10 terabytes of data in 30 minutes. Considering there are 25,000 flights per day worldwide, this amounts to a total of 250 petabytes. Add to that everything from smart refrigerators to smart snow cannons, some of the countless objects rapidly joining the Internet By the year 2020, an estimated 25 billion things will be connected.

It is one of the basic tasks of Big Data – as the name implies - to deal with big amounts of data. This means that incoming streams of information are continuously analysed and filtered by user-defined criteria. At the end of this process you are left with usable information which ideally can help you form concrete answers out of a mass of data. The job of analysing that data is moving progressively to the end appliance, a trend that will continue in 2015, so that the snow guns of the future will supply filtered data.

6. Data Scientists & Big Questions

When following the rapid development of Big Data, one is tempted to believe that this revolutionary technology could soon replace humans in many areas. And indeed this is exactly what will happen – specifically in those areas that can be described as "monkey business" - monotonous clerical work which doesn’t add any value. Big Data ensures that people can concentrate on more important things to improve productivity.

Moreover, even the best Big Data system is worthless if it is not planned, implemented and used by specialists. These experts, the so-called "data scientists" are a pleasantly exotic mixture of mathematicians, statisticians and computer scientists. In addition, they need to have a deep understanding of business. These prerequisites could get pretty much any Big Data engine to start if the right questions are asked. Because Big Data needs big questions.

The only fly in the ointment is that data scientists are still a very rare breed, especially in Europe, though less so in the US. It is to be hoped that more and more young professions decide to specialise in this groundbreaking sector.

Daniel Fallmann, founder and managing director of Mindbreeze

Image source: Shuttterstock/Bruce Rolff