Manual threat intelligence management: Doing it the hard way - Part 1

This is the first post of a series on manual threat intelligence management. 

Threat intelligence is a popular topic in security circles these days. Many organisations are now utilising a threat feed that comes bundled with some other security product, such as McAfee’s GTI or IBM’s X-Force feeds. 

Lots of products, notably SIEMs, have added support for some sort of integration with specific threat intelligence feeds or more generic imports via STIX/TAXII. With many now hoping to take advantage of the large number of open source and free intelligence feeds available. Some are even investing in commercial intelligence feeds. 

However, as many organisations quickly discover, without effective management of the threat intelligence lifecycle, making effective use of this valuable information is nearly impossible. Today, an organisation has two choices for managing threat intelligence, these are to deploy a threat intelligence management platform, or a manual in-house management program. 

In this blog series, the steps required to set up a manual threat intelligence lifecycle program will be outlined for those who prefer this approach. Effective threat intelligence management consists of six main functions or processes:

  • Threat intelligence source selection
  • Threat intelligence capture
  • Threat intelligence processing
  • Actioning threat intelligence
  • Threat intelligence analysis
  • Threat intelligence maintenance

Each of these requires consideration of multiple challenges and requires particular skillsets be present or contracted. We will explore each of them in detail over the course of a series of blog posts. In this post we will look at Threat Intelligence Source selection and Threat Intelligence Capture. 

Source Selection is actually not the first step in setting up a manual threat intelligence program. Before any threat intelligence can be made useful, you must first have something against which to compare it. This will usually be a log management system or SIEM technology, collecting logs or other key information from security devices in your environment. 

Without this critical foundation, there is no way to correlate what is happening in your environment against the intelligence you are collecting, and therefore no way to know when you are communicating with any of the malicious indicators you have identified. Choose carefully, as the limitations of the chosen solution may reduce your options when it comes time to integrate your threat intelligence.  

Source selection

Assuming you have an adequate solution in place, you are ready to select the intelligence sources from which you wish to collect. You can choose from free/open source feeds or you may purchase a feed from one of the several dozen vendors in the market today. There are well over a hundred free or open source intelligence feeds available. 

Many of these feeds get their indicators from the same sources and report on the same indicators, creating large areas of overlap and duplication of data. This is an important consideration, as too much overlap can negatively impact the later stages of the threat intelligence management process. There are dozens of paid feeds available as well. Each has their own areas of focus, and costs vary widely. Although the quality of paid feeds is high, the cost of subscribing to multiple feeds can add up quickly. 

Careful attention should be paid to contract negotiations with feed vendors so that you are absolutely clear about which of their feeds you will have access to and which you will not. Another important consideration should be the methods supported for ingesting those feeds. A flexible API (Application Program Interface) would be an advantage in this instance, since you will be integrating each of these sources in-house.

Capturing threat intelligence

Once you have settled on the sources you wish to collect, a method of collection must be established. If you have lots of sources identified, you are likely to be forced to support several different methods of collection. In some cases, delivery will be automated, such as TAXII over email, or received by email, but in a format that must be converted such as a CSV, PDF, XML, or even free text. Some websites will publish threat intelligence in HTML or XML formats, from which users may either capture it manually or script an automated method to scrape the site at a predetermined interval. STIX and TAXII are widely supported standards for formatting and delivery, but support is by no means universal. An API may be available for some feeds. This is certainly the case for most commercial feeds, but may or may not be the case with open source or free intelligence feeds. 

The API’s themselves will generally require reviewing reference documentation to understand how to access them, how to request and/or retrieve data, as well as limitations on use such as rate limits. Leveraging API’s to ingest feeds can be fairly straightforward but does require scripting or some other mechanism to actually pull the data and do something with it. Additional care and feeding may be required over time as API’s do change as features are deprecated or added, and tweaks are made for improved efficiency. Major overhauls of APIs are not unheard of and may break a lot of automation if previous APIs are deprecated. Monitoring API sources for updates is an important part of keeping feed collection running smoothly. 

You should automate as much of the threat intelligence collection process as possible. This can be done mostly via scripting but may require some additional efforts around collecting via email or web scraping. Putting in this effort pays off over time as manual collection consumes time few teams have to spare. It also frequently takes analysts away from their primary duties while they focus on the mechanics of manual collection. Source selection itself may end up being limited due to the inability to regularly capture the available data without it being manually collected. 

Up next in the series: Processing threat intelligence and actioning threat intelligence

Chris Black, Sr. Sales Engineer, Anomali
Image Credit: ESB Professional / Shutterstock