Skip to main content

Why big data is making security exciting again

As security attacks become both more persistent and complex, the rules-based approach to network and data protection – where security information and event management (SIEM) systems are designed to track and mitigate 'known threats' – is becoming increasingly ineffective; heralding in a new age problem where 'unknown threats' pose the new danger.

So what happens when a new and previously unidentified piece of malware comes calling? Once it's inside the network, an attack can ensure that evidence of its presence is hidden in the massive amounts of 'normal' data that enterprise systems generate – and trying to locate it is extremely difficult.

Furthermore, given that SIEM's tend to overlook a lot of data or it simply gets lost, this can significantly impact organisations ability to spot new threats - especially when data being dismissed could potentially provide valuable clues to spotting new advanced threats.

That's why the new frontier of enterprise security is statistical analysis and pattern recognition in big data – specifically, machine data.

In this new world of security, CSOs and IT teams have to unlearn their over-reliance on traditional data protection technologies such as anti-virus software, firewalls, and security information and event management systems. The non-stop barrage of attacks that the enterprise faces has turned security into a reactive, administrative role.

Security should be an exciting industry to work in, but too often, both seasoned professionals and new entrants are just responding to systems alerts rather than applying their knowledge and thinking more laterally about threats.

To address this, security professionals need to have much greater oversight of everything that's happening inside the enterprise. How? By being able to quickly analyse and sift through the machine data generated by interactions with IT systems in order to identify unusual patterns and abnormal behaviours which could indicate that an attack is taking place.

While big data analysis technologies exist that can help to identify possible anomalies, it still requires human insight and intelligence to interpret what they might mean.

For example, the presence of URL strings that are 4-5 times longer than normal could indicate the possible presence of command and control instructions attempting to launch a web protocol attack. Another 'tell' could be a network access password being entered 10 times faster than it's possible for a human to type. Or an excessive amount of outbound DNS traffic or DNS requests could indicate that an employee's machine has become part of a botnet.

Interrogating machine data is also an excellent way of spotting when a security threat is being created internally – not by a clever piece of malware, but by a malicious insider who may feel entitled to intellectual property and wants to take it with them to their new job. Questions you could be asking you data are, why is a user repeatedly trying to access a file they don't have permission to view, why is there a significant change in the mix of categories of websites they surf to, or why has their ID card been used to enter the office when they're meant to be on holiday in the Bahamas?

Achieving this level of operational intelligence not only opens up new possibilities for how companies defend themselves against the myriad security threats that they face, but also re-engages the interest and creativity of the IT teams entrusted with the task of overall risk mitigation.

While there may be no 'silver bullet' for advanced threat detection, big data represents a compelling way to change the tide of online warfare back in favour of the good guys.

Register for IP EXPO 2013 now.

If you register with, you'll receive:

- Fast-track access to the seminar programme

- Entry into a prize draw for an exclusive gourmet dining experience at IP EXPO ONE Place Dining.

- PLUS: As a loyal reader of ITProPortal, you'll also be able to kick back in the exclusive ITProPortal lounge, enjoying complimentary beverages and the chance to chat to our expert team of technology writers.