Big data and security analytics collide

This article was originally published on Technology.Info.
As part of our continuing strategy for growth, ITProPortal has joined forces with Technology.Info to help us bring you the very best coverage we possibly can.

Big data - an easily digestible name for the emergence of commodity software designed to allow synchronous n-dimensional analytics - quite the mouthful to anyone without a background specialising in the data sciences. Data has always been 'big', an intrinsic side-effect of Moore's law can be expressed as "Utilisation will always expand to fill capacity." No, the real nature of 'big data' is 'big queries' - the ability to ask questions of our data that have previously been computationally unfeasible.

We've had big data for years - terabytes of logs to sift through to find that single log entry that delivers the smoking gun to us - and we'll regale you with stories of waiting hours, days even, for that search to return results. If big data was nothing more than a leap beyond isometric increases in the speed of querying our vast repositories of data in accordance to their volume, the average security analyst would be quite happy with that.

And yet, big data has become the next big thing - a critical re-evaluation and re-tooling of our analytical abilities. This is not about being able to query more data but being able to query all data. Beyond being able to 'grep' through log data faster is the ability to distil everything we have ever recorded from our information systems, into information pictures that no single human mind could perceive from the uninstalled source material.

The convergence of data science to security analytics was not an overnight event, more so because it was not a creation of the information security world to begin with. The path of convergence first came with an overlapping field - fraud detection and investigation - where data analytics has been a key driver for many years now in identifying what constitutes normal and abnormal patterns of activity. For anyone who has ever found their debit card locked out after a transaction they consider normal, well there's the data analytics in action, running into an edge case. These algorithms are refined over time, iteration by iteration, and their designers learn to ask ever more elegant questions about their datasets.

Big data can achieve nothing by itself. It is merely an engine to enable the asking of better questions - questions that arise only through experience with real world data. To express those questions programmatically from big data systems requires a certain set of technical skills that are only hastily covered in the current educational tracks for Infosec. If 'security big data' is going to do more than keep buzzword-pace with the rest of the technology world, it will inevitably draw upon prior expertise from other fields. True, they will have to acquire some of the experience and domain knowledge of the security field - a task I suspect will be far less challenging to people with a background in data science than for our current crop of security graduates to replicate in reverse.

The

hubris

of the Infosec field, to believe it deals with entirely unique and unsolvable problems may finally see new light as other domains of expertise come to accept that security is everybody's

problem. Information security has matured - after two decades of relevance we should expect nothing less - but are we following suit with it? Big data was not our creation, and there exists far more talent for asking the right questions from data, outside of our field. If this is our new normal, the core technology that drives all workflow and action, how are we going to address that in education, training and certification?

Information security expertise requires experience and competence across a wide variety of information technology domains, yet how will we address the incursion of a skill so few of us are qualified with beyond cursory familiarity, only to find ourselves exclaiming "Help, a data scientist took my security job!"?

Topics