Operationalizing Indicators

Are you inundated with reports from vendors, notes hurriedly documented in a text file, or a CSV file filled with indicators from the last few years? You're not alone.  Our industry is rife with reports and other communications that are poorly designed for easy extraction. Today we will show you how you can automate operationalizing indicators from a variety of documents. We will cover how to automatically extract indicators from a vendor report using Bro's Intel Framework to detect the indicators on the wire.  The end result is time saved.   Here's how it works.

Getting Started

For this exercise we are going to work from the recent report on the Lazarus Group, dubbed Operation Blockbuster. Go ahead and download the  C2 Indicators resource from the website so you can walk through this as we go along.

We will assume that you have recently installed the latest version of stoQ using the supplied installation script. Otherwise, you will need to ensure that the following plugins are installed:

• reader/iocregex
• reader/tika
• reader/pdftext
• worker/iocextract
• connector/bro-intel

Not sure which plugins you have installed? No problem. Simply run stoQ with the list argument, as below.




The bro-intel plugin was just released, so if you have a previously installed version of stoQ, you will want to upgrade stoQ to the latest version (v0.9.11, as of this writing) and also install the latest public plugins.

In order to extract text from documents, stoQ leverages Apache Tika. Tika is a toolkit written in Java that allows text and metadata extraction from thousands of document formats. You will need to ensure that you have a Tika Server up and running to fully leverage IOC extraction. If you don't have it set up, you can quickly install a Tika Server using the installation script for stoQ, or you can follow that installation directions located on the Apache Tika page.

Extracting Indicators of Compromise

Now, let's get started extracting the IOCs from the Operation Blockbuster Indicator resource. In this scenario, we have decompressed the report and placed it in the ~/stoq/ingest directory.

Next, we are simply going to run the iocextract worker plugin against this file.

Once we run the command, we will have our results printed to our console. In the below screen shot, we have truncated the results for brevity. When you run this command, you will also see hashes to include md5, sha1, and sha256. In addition to extracting IPv4 addresses, md5, sha1, and sh256 hashes, the iocextract plugin also supports the extraction of domains, MAC, e-mail and IPv6 addresses, as well as URLs. These are automatically extracted, if they exist in the document. There is no need for any additional configurations.

The extraction process will ensure that the results are de-duped, so you will not have to worry about removing any duplicates.

In addition, if there are unwanted indicators that are extracted, you can easily add them to the whitelist.  Simply add the indicator to plugins/reader/iocregex/whitelist.txt, and you'll never have to worry about it again.

Output into the Bro Intel Framework

Now, let's get to the meat of this post, which is operationalizing the extracted IOC's. If you don't already use Bro's Intel Framework, you should give it a try. For those not familiar with Bro, it is a network analysis framework that allows a network defender to do a lot of pretty amazing things with network traffic. For example, you can automatically extract files from your pcap or live network traffic; log all HTTP requests; Generate a hash of every file downloaded and uploaded; log FTP connections; extract e-mail headers and content; write Bro scripts to do custom and even more amazing things with your network traffic.

As previously mentioned, Bro also has a very useful Intel Framework built in. What this means for you is Bro can automatically alert you if an IOC is detected within your network traffic. Want to monitor for an IP address, no problem. Simply add it to your list of indicators. What about a hash? Just add it. Domain? Yep, you guessed it, just add it.

What if you want to use  stoQ to automatically extract indicators from reports, and add them to Bro's Intel Framework? No problem, there is a plugin for that. Simply ensure the connector/bro-intel plugin is installed and define it as your connector plugin, like so:

We now have a nicely formatted Bro Intel Framework file that is ready for operational use on your network.

Conclusion

As you can see, stoQ is extremely flexible and highly modular. It doesn't just have to be used for file analytics with MongoDB or Elasticsearch. You can leverage it's agility to complete a multitude of tasks that were previously difficult and time consuming. In this example, we've taken a CSV file containing hashes and IP addresses and quickly converted them into a Bro Intel Framework file with minimal effort. Do you have a PDF file, Word or Excel document with indicators in them? No problem, give it a shot and see how easy it is to operationalize your indicators.