Skip to main content

Pointing the automation big guns at log data

(Image credit: Future)

Log data is a valued ally in the war against cybercrime. Security companies large and small alike gather logs and security sensor data from almost all software applications and systems. This data is useful for monitoring, understanding and managing the network – and for securing it. Organisations use this data to help them detect an attack in progress and respond quickly and appropriately. Logs are also used after the fact to piece together what took place.  

However, not all logging is equally useful or accurate. Much of the time, the data is difficult to understand, inconsistent and focused solely on system break/fix. It doesn’t contain much information to help reveal if something malicious was or is happening in the network. If logging doesn’t have the information needed for security monitoring, it will adversely affect all your other security measures. So how does your team pull out the big guns on the problem? Automation is the key.

Challenges with the status quo

Although standard logging formats and methods exist, no standard agreement exists about how the data should be used for specific purposes. Consequently, most logs are extremely inconsistent and difficult to leverage effectively for any purpose. In addition, data sources are self-focused, referring only to themselves and containing irrelevant redundancies. There is no universally accepted set of informational fields across data categories, vendors and applications. That is unfortunate, since a standard like this would enable organisations to get much more value from their reams of log data.

Adding to the dilemma are incomplete rules or playbooks with which to analyse the data. Log management has traditionally had a hard time with custom log formats. This makes life for the security analytics teams all the more difficult. The majority of security teams are using almost none of the log data for security monitoring and detection. Imagine having 100 data sources but having rule logic applied to just 5-10 of them (and perhaps just one or two rules for most); how truly effective is your monitoring at this point?

Logs can’t indicate in and of themselves if they contain malicious or benign activity. In other words, they have plenty of noise but no signal. It would be supremely helpful for IT to know whether malicious activity is occurring, but that is not the case. Because logs don’t contain this helpful information, cloud security remains ineffective.

IT security and IT Ops are often at loggerheads because they typically share the same infrastructure but have quite different business requirements. IT security manages the SIEM and therefore wants to collect as much information as possible. Usually, IT Ops doesn’t have enough authority to make engineering home in on what they need for their detection and response duties. The two teams forget that they are on the same side.

There’s a pervasive attitude of “it’s not broke, don’t fix it.” That mentality has been the status quo all along – even though, in reality, the way logging is done actually is broken. It won’t change until an outside force creates movement. Though vendors lack motivation to fix logging deficits, security teams are motivated toward change. But how much leverage does a five-person security team have on Microsoft to produce better logs for Active Directory or Office 365?

Logs can’t indicate in and of themselves if they contain malicious or benign activity. In other words, they have plenty of noise but no signal. It would be supremely helpful for IT to know whether malicious activity is occurring, but that is not the case. Because logs don’t contain this helpful information, logs become, frankly, a slog.

The industry work-around won’t work long-term

A truly strong detection program requires a significant budget outlay and a large number of skilled security analysts – a tall order for many cash-strapped organisations. Many organisations opt instead, then, for a managed security service provider – even though the majority of MSSP clients are not satisfied with their services. If organisations opt to go it alone, they hire more people or buy more solutions to deal with their bad data.

But this is one challenge that you can’t hire your way out of these days. The cybersecurity skills gap remains a real and growing problem. Worldwide, there are almost three million unfilled cybersecurity positions – a reality that organisations need to acknowledge so that they can look for another solution besides throwing more people at the problem.

This talent shortage affects the security analysts you already have, too. Long shifts of monitoring oceans of alerts looking for signals of malicious activity creates mental fatigue. ESG and the Information Systems Security Association International found in a research study that the talent gap makes for a bigger workload. Highly trained security professionals spend most of their time doing laborious grunt work instead of the high-value work that is more engaging and satisfying. This is a scenario ripe for burnout.

Yet even with all this hard work, analysts are able to review fewer than one in a million of the logs collected in their SIEMs or data lakes. It’s not the number of logs you collect that matters, but how many of them you can review.

What organisations need to do to be part of the change

With these realities in mind, here are some best practices and questions to get teams beyond the log slog and into a truly effective cybersecurity posture.

  • Crank up your security sensors: Give your security team the most comprehensive visibility possible – and require your vendors to give better security visibility their products, as well.
  • Pay attention to the logs with a signal – whether they are malicious or not.
  • Log only from sources that matter – and gather the information that can effectively support detection and remediation.

Log only from sources that matter – and gather the information that can effectively support detection and remediation.

  • What’s the most effective way to monitor your logging for malicious activity?
  • How many malicious scenarios do your logs identify?
  • How difficult is it to parse them so they are normalised with other, similar technologies?
  • What standard log formats do you use for output?
  • How difficult it is it to introduce new logs based on new attack techniques? What’s your update frequency?
  • Do you have a centre of excellence for security monitoring of your technology?

Automation is a linchpin to making this happen effectively. Autonomous analysts currently exist that can operate at machine speed and put all relevant logs and contextual information within the bounds of a complete analysis. Such analysts automate and autonomously perform the tasks of a skilled Level 1 security operations analyst. They are able to observe every log and reason about it using more than 50 dimensions. That would significantly increase the successful identification of attackers in your environment and the success of your security strategy.

Looking ahead

Logs have historically been over-collected and under-used. There is too much noisy garbage mixed in with the important signals your IT security team needs to find. In addition, there just aren’t enough skilled workers to keep moving through your SOC as the burnout cycle continues. It is possible to overcome the status quo, however, by implementing automation, knowing what questions to ask and demanding better logging capabilities. These new practices will create happier security analysts and stronger network security.

Chris Calvert, vice president, product strategy, co-founder, Respond Software