Thursday, May 19, 2016

Know thy e-mail with stoQ

Have you ever needed to check if your users received a malcious file with a specific filename? Perhaps you know of a malicious e-mail address from a threat intel service, but have no way to know if they ever e-mailed your users. Would you like to be able to scan every attachment in every e-mail within in your enterprise with yara or any other tool? Today, we're releasing our newest plugin for stoQ, the SMTP Parser that allows you to do just that. You can now parse every e-mail coming into your enterprise, insert it into any data store you wish to use, and scan every attachment with your application of choice. Let's get started and show you how you can quickly deploy this in your environment today.

Getting Started

Before we get to the fun stuff, let's make sure our environment is setup properly so you can play along. We will assume that you have a basic stoQ environment already setup and ready to go with the latest version (v0.9.18) and plugins. If you don't, you can quickly get one setup using Docker, Vagrant, or using the installation script. Need more help? Feel free to take a look at the documentation or send us a note.

This post will also assume that you are doing everything from an Ubuntu box.


Ok, now that we have stoQ all setup, let's setup a basic postfix server. First, let's install a barebones postfix server.
vagrant@vagrant$ sudo apt-get install postfix

For this use case, we will select "Internet Site" from the menu and set the domain name to your e-mail domain. In this case, we will just use 'yourdomain'.

Next, we need to make sure that Postfix will use Maildir as it's mailbox format.

vagrant@vagrant$ sudo postconf -e 'home_mailbox = Maildir/'

Let's go ahead and restart postfix now.
vagrant@vagrant$ sudo service postfix restart

Handling E-mails

Now, we get to the fun part. There are several ways we can ingest e-mails. For now, we will keep it simple. We will handle any e-mail that is sent to stoq@yourdomain.

First, make sure you are logged in as the stoQ user.
vagrant@vagrant$ sudo su - stoq

Because we've not received any e-mail yet, let's create the Maildir directory structure and set permissions so we can tell stoQ to monitor it.
stoq@vagrant$ mkdir ~/Maildir/new
stoq@vagrant$ chmod -R go-wrx ~/Maildir

You should now be the stoQ user and be in a python virtual environment. Let's run the SMTP worker plugin and monitor our Maildir for new e-mails.
stoq@vagrant$ ./ smtp -I dirmon -F ~/Maildir/new

     .d8888b.  888             .d88888b.
    d88P  Y88b 888            d88P" "Y88b
    Y88b.      888            888     888
     "Y888b.   888888 .d88b.  888     888
        "Y88b. 888   d88""88b 888     888
          "888 888   888  888 888 Y8b 888
    Y88b  d88P Y88b. Y88..88P Y88b.Y8b88P
     "Y8888P"   "Y888 "Y88P"   "Y888888"
            Analysis. Simplified.

[*] root: Monitoring /usr/local/stoq/Maildir/new for new files...

What we've done here is started the smtp worker plugin, told stoQ to use the directory monitor plugin as the source, and that the ~/Maildir/new directory should be monitored for new files. Now, let's open up a new terminal and send a quick test e-mail to the stoQ user so we can see what happens.

stoq@vagrant$ echo "This is a test e-mail" | mail -s "Test" stoq@localhost

Looking back at the window we have stoQ running in, you should see something similar to this:

    "results": [
            "uuid": "776b4232-c8cc-4d65-9f5e-43d4167e9f54",
            "path": "/usr/local/stoq/Maildir/new/1462987239.V801I637afM519163.vagrant-ubuntu-trusty-64",
            "payload_id": 0,
            "archive": "file",
            "md5": "fa11ac74cdb54adfe8c2e30969e01839",
            "sha256": "17c13ab194086fe602a957bbbf949525d3e15a89efc6ad92b9562039870e5399",
            "plugin": "smtp",
            "sha512": "3466bbd6f35449a16f69cf2a4b516dc2ffdbe7818e4208a927c2c0edfbb7652cbba7a629216496dcf6f9ab5b16769f92ff3d7012a09b00a6a641e3b5656411b4",
            "scan": {
                "uuid": "03150da1-6331-4c73-b8b4-43ad3244e860",
                "date": "Wed, 11 May 2016 17:20:39 +0000 (UTC)",
                "delivered-to": "stoq@localhost",
                "x-mailer": "mail (GNU Mailutils 2.99.98)",
                "body": "This is a test e-mail\n\r\n--69b1cbbb-0f6f-4a26-ac4a-08faa1f144a6--\r\n",
                "message-id": "<20160511172039 .7d2b7637b0="" vagrant-ubuntu-trusty-64="">",
                "subject": "Test",
                "x-original-to": "stoq@localhost",
                "from": "stoq@vagrant-ubuntu-trusty-64 (stoQ)",
                "return-path": "",
                "from_isnew": true,
                "received": "by vagrant-ubuntu-trusty-64 (Postfix, from userid 999)\n\tid 7D2B7637B0; Wed, 11 May 2016 17:20:39 +0000 (UTC)",
                "to": "",
                "puuid": "776b4232-c8cc-4d65-9f5e-43d4167e9f54",
                "body_html": "",
                "att": []
            "source_meta": {},
            "sha1": "4fe7324d8ad495e0c1a42d819c29948caadcb8a6",
            "filename": "1462987239.V801I637afM519163.vagrant-ubuntu-trusty-64",
            "size": 472
    "date": "2016-05-11T17:20:40.022749",
    "payloads": 1

What just happened here? You've successfully analyzed an e-mail with stoQ from the stoQ users inbox. Looking at the results, you can see all of the details about the e-mail itself within the scan key. It tells you who the e-mail went to, the subject, every other header in the raw session, along with the body of the e-mail itself. We've also used the iocextract plugins to extract all IP addresses and URLs in the e-mail and headers, making them easily available for analysis. In short, the entire email session was extracted and made parseable by stoQ.

Now, let's get to some even cooler things. What if we want to extract every attachment and scan it using yara, exif, trid, and peinfo workers? It's already configured to do just that. Just send an e-mail to your stoQ user with an attachment and each attachment will be extracted and scanned with each of the previously mentioned worker plugins. Done and done.

What if we want to scan each attachment with another worker plugin, such as clamav. Each plugin has it's own configuration file that ends with .stoq To get the smtp worker scanning attachment with clamav, all we need to do is edit it's stoq configuration file located at plugins/worker/smtp/smtp.stoq.

hashpayload = True
saveresults = True

# Should a queuing plugin be used? True or False
# publisher worker plugin is required to be installed
use_queue = False

# Default workers that all attachments will be scanned with
workers_list = yara, exif, trid, peinfo, clamav

# What is the default attachment connector plugin to use?
attachment_connector = file

# Should bloom filters be used to track specified SMTP headers?
# If True, the SMTP plugin may only be run once at a time
use_bloom = True

# Where is the bloom filters file located?
bloom_directory = plugins/worker/smtp/email_filters

# Expected maximum values in bloom filter
bloom_size = 50000000

# Desired False Positive rate
bloom_fp_rate = 0.001

# How often should the bloomfilter be written to disk?
bloom_update = 60

# SMTP header to extract a value from and insert into bloom filter
bloomfield_list = from

We've added clamav to the workers_list option. Now, when we receive an e-mail each attachment will also be scanned with clamav.

Enterprise Deployment

Let's get everything setup so we can use this in an enterprise environment. For this use case, we will use RabbitMQ as our queueing system of choice and saving all results and attachments to disk. We will also use supervisor to run stoQ as a daemon in the background.

If you installed stoQ using Vagrant or the installation script, then you have RabbitMQ installed and operational on your localhost. If not, you'll need to install and setup a RabbitMQ server and configure the stoQ RabbitMQ plugin to use that server by editing plugins/source/rabbitmq/rabbitmq.stoq.

Let's install supervisor so we can daemonize stoQ.

vagrant@vagrant$ sudo apt-get install supervisor

Now, let's add our stoQ workers so supervisor can start and monitor the processes.

vagrant@vagrant$ sudo vi /etc/supervisor/conf.d/stoq.conf

And then add the below to the newly created file.

command=/usr/local/stoq/.stoq-pyenv/bin/python %(program_name)s -I rabbitmq -C file

command=/usr/local/stoq/.stoq-pyenv/bin/python %(program_name)s -I rabbitmq -C file

command=/usr/local/stoq/.stoq-pyenv/bin/python %(program_name)s -I rabbitmq -C file

command=/usr/local/stoq/.stoq-pyenv/bin/python %(program_name)s -I rabbitmq -C file

command=/usr/local/stoq/.stoq-pyenv/bin/python %(program_name)s -I rabbitmq -C file

command=/usr/local/stoq/.stoq-pyenv/bin/python %(program_name)s -I rabbitmq -C file

command=/usr/local/stoq/.stoq-pyenv/bin/python %(program_name)s -I dirmon -F /usr/local/stoq/Maildir/new -C file -w smtp

Ok, that's a lot. Let's go over what we just did. We've told supervisor to start multiple stoQ workers (exif, yara, peinfo, trid, clamav, and smtp). These stoQ workers were all executed with the source plugin of rabbitmq and the connector plugin of file. What this means is these workers will monitor RabbitMQ for new messages, process each of those messages, then save the results to a file on disk. We could just as easily change the connector plugin to use mongodb, elasticsearch, or any of the other available plugins use for saving results.

The final program being run in the supervisor configuration file is the stoQ publisher worker. This worker has a very simple function, which is take events (such as a new file being created) and then send that event on to a queueing system, such as RabbitMQ. In this case, we are using the publisher worker to detect that a new e-mail has been received, and it will then send a message to the queue for the smtp worker.

Now, we need to make sure the smtp worker knows to use RabbitMQ to publish messages to all of the other workers when it extracts an attachment. Let's update the smtp worker configuration file (plugins/worker/smtp/smtp.stoq) update it use_queue to True.

# Should a queuing plugin be used? True or False
# publisher worker plugin is required to be installed
use_queue = True

We are pretty much all set now. All we need to do is load the new configuration file and start the stoQ services.

vagrant@vagrant$ sudo supervisorctl reload

Let's check the status to make sure everything started up ok.

vagrant@vagrant$ sudo supervisorctl status
clamav:clamav_00                 RUNNING    pid 6534, uptime 0:00:01
exif:exif_00                     RUNNING    pid 6533, uptime 0:00:01
peinfo:peinfo_00                 RUNNING    pid 6536, uptime 0:00:01
publisher:publisher_00           RUNNING    pid 6538, uptime 0:00:01
smtp:smtp_00                     RUNNING    pid 6535, uptime 0:00:01
trid:trid_00                     RUNNING    pid 6539, uptime 0:00:01
yara:yara_00                     RUNNING    pid 6532, uptime 0:00:01

You should see everything running. If not, check out logs/stoq.log for any errors that may have occurred.

Now, go ahead and try sending an email to your stoQ user and include an attachment with that e-mail. Once each plugin is done scanning, it will write the results to disk in their respective folder in results/. For instance, the yara worker will write its results to results/yara/ and the smtp worker will write to results/smtp/.

Once you're ready, you can now forward any e-mail you would like scanned to your stoQ user account and review the results at your leisure.


We've shown you how to build a basic postfix server for use with stoQ, and what the parsed data looks like when sent through the smtp worker. You have also learned how to scan extracted attachments with other stoQ plugins and how to configure those plugins. We then went over how to bring it all together so you can scan all of your e-mail and their attachments in your enterprise environment.

Why would you want to do this? Imagine having complete visibility into all of your e-mail and their attachments. No longer will you have to shrug your shoulders when asked by leadership if you've ever received an e-mail from a malicious actor. You will never have to wonder if a malicious file has been sent to your users. You no longer have to pay tens of thousands of dollars annually to big name vendors just to have visibility into your own traffic. You can automate more tasks allowing your analysts to focus on what matters, not on trying to get information that should be simple to obtain.

Want to see how to use this with your enterprise Splunk deployment? We will be presenting stoQ’ing your Splunk this June at DFIR 2016 in Austin, TX. Come by, learn more about stoQ and how you can leverage it with the power of Splunk to simplify your security operations.

Wednesday, April 13, 2016

Operationalizing Indicators

Are you inundated with reports from vendors, notes hurriedly documented in a text file, or a CSV file filled with indicators from the last few years? You're not alone.  Our industry is rife with reports and other communications that are poorly designed for easy extraction. Today we will show you how you can automate operationalizing indicators from a variety of documents. We will cover how to automatically extract indicators from a vendor report using Bro's Intel Framework to detect the indicators on the wire.  The end result is time saved.   Here's how it works.

Getting Started

For this exercise we are going to work from the recent report on the Lazarus Group, dubbed Operation Blockbuster. Go ahead and download the  C2 Indicators resource from the website so you can walk through this as we go along.

We will assume that you have recently installed the latest version of stoQ using the supplied installation script. Otherwise, you will need to ensure that the following plugins are installed:

  • reader/iocregex
  • reader/tika
  • reader/pdftext
  • worker/iocextract
  • connector/bro-intel
Not sure which plugins you have installed? No problem. Simply run stoQ with the list argument, as below.

The bro-intel plugin was just released, so if you have a previously installed version of stoQ, you will want to upgrade stoQ to the latest version (v0.9.11, as of this writing) and also install the latest public plugins.

In order to extract text from documents, stoQ leverages Apache Tika. Tika is a toolkit written in Java that allows text and metadata extraction from thousands of document formats. You will need to ensure that you have a Tika Server up and running to fully leverage IOC extraction. If you don't have it set up, you can quickly install a Tika Server using the installation script for stoQ, or you can follow that installation directions located on the Apache Tika page.

Extracting Indicators of Compromise

Now, let's get started extracting the IOCs from the Operation Blockbuster Indicator resource. In this scenario, we have decompressed the report and placed it in the ~/stoq/ingest directory.

Next, we are simply going to run the iocextract worker plugin against this file.

Once we run the command, we will have our results printed to our console. In the below screen shot, we have truncated the results for brevity. When you run this command, you will also see hashes to include md5, sha1, and sha256. In addition to extracting IPv4 addresses, md5, sha1, and sh256 hashes, the iocextract plugin also supports the extraction of domains, MAC, e-mail and IPv6 addresses, as well as URLs. These are automatically extracted, if they exist in the document. There is no need for any additional configurations.

The extraction process will ensure that the results are de-duped, so you will not have to worry about removing any duplicates.

In addition, if there are unwanted indicators that are extracted, you can easily add them to the whitelist.  Simply add the indicator to plugins/reader/iocregex/whitelist.txt, and you'll never have to worry about it again.

Output into the Bro Intel Framework

Now, let's get to the meat of this post, which is operationalizing the extracted IOC's. If you don't already use Bro's Intel Framework, you should give it a try. For those not familiar with Bro, it is a network analysis framework that allows a network defender to do a lot of pretty amazing things with network traffic. For example, you can automatically extract files from your pcap or live network traffic; log all HTTP requests; Generate a hash of every file downloaded and uploaded; log FTP connections; extract e-mail headers and content; write Bro scripts to do custom and even more amazing things with your network traffic.

As previously mentioned, Bro also has a very useful Intel Framework built in. What this means for you is Bro can automatically alert you if an IOC is detected within your network traffic. Want to monitor for an IP address, no problem. Simply add it to your list of indicators. What about a hash? Just add it. Domain? Yep, you guessed it, just add it.

What if you want to use  stoQ to automatically extract indicators from reports, and add them to Bro's Intel Framework? No problem, there is a plugin for that. Simply ensure the connector/bro-intel plugin is installed and define it as your connector plugin, like so:

We now have a nicely formatted Bro Intel Framework file that is ready for operational use on your network.


As you can see, stoQ is extremely flexible and highly modular. It doesn't just have to be used for file analytics with MongoDB or Elasticsearch. You can leverage it's agility to complete a multitude of tasks that were previously difficult and time consuming. In this example, we've taken a CSV file containing hashes and IP addresses and quickly converted them into a Bro Intel Framework file with minimal effort. Do you have a PDF file, Word or Excel document with indicators in them? No problem, give it a shot and see how easy it is to operationalize your indicators. 

Thursday, March 31, 2016

The stoQ Workflow

We've previously discussed how to setup and operate your instance of stoQ.  But to take a break from the technical, today we want to talk about the stoQ workflow and give some examples and hopefully some inspiration about how you can use it to your advantage.

Fundamentally, stoQ is structured to be in the middle of your analyst workflow.  It works best when it can leverage other security tools to ingest objects and a database to store the output. 
Let’s take a look at how stoQ is structured:

For sources, stoQ can ingest data from individual files, monitor a directory for new files, pull from a database, or from an API.  This is where stoQ at scale can be extremely powerful.  Extracting files from threat vectors like HTTP or e-mail can be sent to stoQ for automatic enrichment and processing.  This is when file extraction from Suricata or Bro can be used to ingest files for stoQ processing.  Sending all executables, PDFs, or Office Documents would allow us to automatically analyze our higher risk file types as they hit our network. 

Reader, Decoder, Extractor and Carver plugins that can be used to run the gamut of common activities against ingested files.  Use reader plugins to extract text and look for keywords.  Use decoder plugins to automatically handle XOR-encoded content or decode base64 strings.  Extractors can automate the tasks like decompressing archives and deflating streams in PDF documents.  Carver plugins are used to extract hidden payloads, like shellcode or flash files embedded within Word Documents.  These enriched objects are then passed back to the stoQ framework for additional processing.   

The Worker class plugins are what interact with external systems to allow objects to be processed elsewhere and returns the information back to stoQ to be associated with the object.  This allows stoQ to interact with scripts (exiftool, TRiD, Yara, etc.) and APIs (FireEye, VirusTotal, ThreatCrowd, etc.) and get even more data about our objects.  Remember, we are able to automate all of this and can quickly scale to requirements.

Once stoQ has ingested, enriched, and processed an object, the results are sent to a Connector plugin for storage.  This can be as simple as a regular text file or a database, or as complex as multiple databases spread across multiple data centers.

Leveraging that data with something like ElasticSearch or Splunk can give us a very rich resource of metadata for the objects that have passed through stoQ.  This large and detailed dataset can be used to find larger trends and anomalies in your environment.   stoQ enables you to craft queries and alerts for all of this metadata. 

You can now search across all yara hits in your enterprise, or list all processed file names.  We will discuss how to leverage this data in a future post – today’s goal was just to explore a sample workflow using stoQ. 

Tuesday, February 9, 2016

Using stoQ with Suricata's File Extraction Capability

We previously discussed why stoQ matters, how to install it, and what it can do for you. This blog takes a slightly more advanced look at how stoQ scales to provide an enterprise view of files downloaded from the internet.

stoQ was originally designed to support the enterprise level. When used in conjunction with a tool that provides file extraction capabilities, such as Suricata or Bro, one quickly realizes the agility and power of stoQ.

For example, assume you would like to use Suricata to extract every exe, swf, and jar file downloaded from the internet into your environment and then use stoQ to analyze each of them. In this example, you are interested in analyzing each file with Exiftoolyara, and ClamAV. You will then store the results from each of these tools into a preferred datastore. In this case, ElasticSearch will be leveraged; however, stoQ is very flexible, so, you can just as easily switch out Suricata for Bro, or ElasticSearch for MongoDB.

For the sake of simplicity, let's outline some assumptions:
Depending on the size of your network, Suricata may extract files faster than you can scan them. To keep up with the throughput, let's leverage the queuing platform RabbitMQ. If you installed stoQ using the installation script, both the RabbitMQ plugin and the server are already installed and configured.

Make sure you are in the stoQ virtualenv before starting any of your processes:

The first thing that needs to happen to get stoQ processing these files is a directory monitor needs to be setup so stoQ knows when Suricata has extracted a file off of the network. All you have to do is run:

This will start the publisher worker plugin, which is used to send messages to our queuing platform of choice, RabbitMQ. Then, load the Suricata directory monitor plugin to monitor /var/log/suricata/files, and finally publish a message to the RabbitMQ queues for exif, yara, and clamav. 

In short, once Suricata extracts a file off of the network, it will be saved to /var/log/suricata/files, at which point stoQ will detect it as a new file, then publish a message to each worker queue so it processes the file.

Now that we have files be extracted and sent to stoQ, you will need to start a worker (exif, yara, and clamav) to process those payloads:

This tells stoQ to load the exif worker plugin, then load the RabbitMQ plugin and monitor its queue, and when a message is found, process it and send any results to ElasticSearch. Now, open another terminal and do the same for yara:

And now, start the ClamAV plugin in yet another terminal:

Normally these processes would be running in the background, but for this example, you will want to see the output from the plugins should there be any.

At this point, all results from stoQ should be inserted into your ElasticSearch instance.  By default, each worker plugin will insert into an index named after itself. For example, yara will insert into the yara index, exif will insert into the exif index, and so on. You can retrieve the results from each of the worker indexes from elasticsearch via curl:

Another more user-friendly option is to simply use Kibana for searching and interacting with the results. As an example, we created a very simple dashboard using output from stoQ's worker plugins yara, TRiD, and peinfo.

You now have the ability to scan every file extracted off your network with yara, exif, and clamav. The results from each have also been inserted into ElasticSearch for easy searching. This can very easily be expanded to scan each file with OPSWAT, VirusTotal, or anything else for which a plugin exists. If one isn't written, with a little python experience, you can easily write one yourself.

Thursday, January 14, 2016

Introduction to stoQ

Simply put: we were tired of wasting time.  As network defenders, we have seen the market explode with countless security tools promising better awareness, functionality, and speed (we'll make you taller AND better looking!!!).  Unfortunately, most of these solutions have one thing in common -- they don't talk to each other, and in many cases, results are lost the moment our screen scrolls away or our browser window closes. As analysts, we have been frustrated by this inefficiency for some time now.  It doesn't have to be this difficult. We just want our data, and we want to be able to easily search it in our datastore of choice.  So, we decided to do something about it.

As we developed stoq, our mantra was "make no assumptions."  We wanted to develop a tool for analysts that would allow them to organize and automate the data they received on a daily basis in an easy-to-use fashion using whatever tools were important to them.  Analysts are really particular with their pet tools, so we didn't assume stoQ had to be wedded to any one set. Consequently, we created more than 40 plugins for everything from input to output, to include any intermediary analytics such as carving and extracting. If you want to scan a file against a new API or tool, support a different database, extract content from a file, or just about anything else, stoQ can do it.  You'll notice popular workers for FireEye, yara, and OPSWAT, all of which have plug-and-play functionality within stoQ.  We're expanding these workers each week. Want one in particular?  Send us an email.

Ready to go?  Perhaps the easiest way to get started is to download stoQ here. Once you have it ready, we recommend scanning a single file using a tool like, exiftool, and displaying the results to the console.  Stay tuned over the coming weeks as we post tutorials to help you in standing up and optimizing stoQ.

Here's a more advanced example that uses Bro and/or Suricata to extract files from your network and analyze them with stoQ. Once ingested by stoQ it will ensure compressed, XOR'd, OLE streams, and base64 encoded content is decoded or extracted. stoQ will then scan those independent streams against exiftool, Fireeye, Yara, Virustotal, and #totalhash. Once stoQ receives the results from exiftool, Fireeye, and yara stoQ will insert them into ElasticSearch. For the results from Virustotal and #totalhash, stoQ will insert them into Mongodb instead. No matter how you want your data to be ingested, analyzed, and stored, stoQ can simplify and automate that process.

stoQ is an extremely agile and robust framework that helps to automate the majority of redundant tasks we as analysts do on a regular basis. Carving executables or SWF's out of a file should not be a time consuming and cumbersome task. Scanning files with yara, exif, or clamav should not be difficult, nor should those results be lost as you work between tools. You and your network should be able to focus on what matters, not on jumping from tool to tool, and then manually connecting the dots. 

You can browse and download the source code for stoQ here, and also check out more than 40 publicly available plugins here. Additionally, the most up-to-date documentation for stoQ can be found here.

In future posts, we will be focusing on blog posts that will outline the use cases for stoQ and how to quickly enhance your visibility of threats.