The stoQ Workflow

We've previously discussed how to setup and operate your instance of stoQ.  But to take a break from the technical, today we want to talk about the stoQ workflow and give some examples and hopefully some inspiration about how you can use it to your advantage.

Fundamentally, stoQ is structured to be in the middle of your analyst workflow.  It works best when it can leverage other security tools to ingest objects and a database to store the output. 

Let’s take a look at how stoQ is structured:

Ingestion
For sources, stoQ can ingest data from individual files, monitor a directory for new files, pull from a database, or from an API.  This is where stoQ at scale can be extremely powerful.  Extracting files from threat vectors like HTTP or e-mail can be sent to stoQ for automatic enrichment and processing.  This is when file extraction from Suricata or Bro can be used to ingest files for stoQ processing.  Sending all executables, PDFs, or Office Documents would allow us to automatically analyze our higher risk file types as they hit our network. 

Enrichment

Reader, Decoder, Extractor and Carver plugins that can be used to run the gamut of common activities against ingested files.  Use reader plugins to extract text and look for keywords.  Use decoder plugins to automatically handle XOR-encoded content or decode base64 strings.  Extractors can automate the tasks like decompressing archives and deflating streams in PDF documents.  Carver plugins are used to extract hidden payloads, like shellcode or flash files embedded within Word Documents.  These enriched objects are then passed back to the stoQ framework for additional processing.   

Processing

The Worker class plugins are what interact with external systems to allow objects to be processed elsewhere and returns the information back to stoQ to be associated with the object.  This allows stoQ to interact with scripts (exiftool, TRiD, Yara, etc.) and APIs (FireEye, VirusTotal, ThreatCrowd, etc.) and get even more data about our objects.  Remember, we are able to automate all of this and can quickly scale to requirements.

Export

Once stoQ has ingested, enriched, and processed an object, the results are sent to a Connector plugin for storage.  This can be as simple as a regular text file or a database, or as complex as multiple databases spread across multiple data centers.

Leveraging that data with something like ElasticSearch or Splunk can give us a very rich resource of metadata for the objects that have passed through stoQ.  This large and detailed dataset can be used to find larger trends and anomalies in your environment.   stoQ enables you to craft queries and alerts for all of this metadata. 

You can now search across all yara hits in your enterprise, or list all processed file names.  We will discuss how to leverage this data in a future post – today’s goal was just to explore a sample workflow using stoQ. 

Using stoQ with Suricata's File Extraction Capability

We previously discussed why stoQ matters, how to install it, and what it can do for you. This blog takes a slightly more advanced look at how stoQ scales to provide an enterprise view of files downloaded from the internet.

stoQ was originally designed to support the enterprise level. When used in conjunction with a tool that provides file extraction capabilities, such as Suricata or Bro, one quickly realizes the agility and power of stoQ.

For example, assume you would like to use Suricata to extract every exe, swf, and jar file downloaded from the internet into your environment and then use stoQ to analyze each of them. In this example, you are interested in analyzing each file with Exiftool, yara, and ClamAV. You will then store the results from each of these tools into a preferred datastore. In this case, ElasticSearch will be leveraged; however, stoQ is very flexible, so, you can just as easily switch out Suricata for Bro, or ElasticSearch for MongoDB.

For the sake of simplicity, let's outline some assumptions:

• You've installed stoQ using the installation script
• Suricata is installed, running, and configured for file extraction
• All files are extracted into /var/log/suricata/files
• An ElasticSearch server is operational
• Everything is running on the same server, localhost (just for the sake of simplicity)

Depending on the size of your network, Suricata may extract files faster than you can scan them. To keep up with the throughput, let's leverage the queuing platform RabbitMQ. If you installed stoQ using the installation script, both the RabbitMQ plugin and the server are already installed and configured.

Make sure you are in the stoQ virtualenv before starting any of your processes:

The first thing that needs to happen to get stoQ processing these files is a directory monitor needs to be setup so stoQ knows when Suricata has extracted a file off of the network. All you have to do is run:

This will start the publisher worker plugin, which is used to send messages to our queuing platform of choice, RabbitMQ. Then, load the Suricata directory monitor plugin to monitor /var/log/suricata/files, and finally publish a message to the RabbitMQ queues for exif, yara, and clamav. 

In short, once Suricata extracts a file off of the network, it will be saved to /var/log/suricata/files, at which point stoQ will detect it as a new file, then publish a message to each worker queue so it processes the file.

Now that we have files be extracted and sent to stoQ, you will need to start a worker (exif, yara, and clamav) to process those payloads:

This tells stoQ to load the exif worker plugin, then load the RabbitMQ plugin and monitor its queue, and when a message is found, process it and send any results to ElasticSearch. Now, open another terminal and do the same for yara:

And now, start the ClamAV plugin in yet another terminal:

Normally these processes would be running in the background, but for this example, you will want to see the output from the plugins should there be any.

At this point, all results from stoQ should be inserted into your ElasticSearch instance.  By default, each worker plugin will insert into an index named after itself. For example, yara will insert into the yara index, exif will insert into the exif index, and so on. You can retrieve the results from each of the worker indexes from elasticsearch via curl:

Another more user-friendly option is to simply use Kibana for searching and interacting with the results. As an example, we created a very simple dashboard using output from stoQ's worker plugins yara, TRiD, and peinfo.

You now have the ability to scan every file extracted off your network with yara, exif, and clamav. The results from each have also been inserted into ElasticSearch for easy searching. This can very easily be expanded to scan each file with OPSWAT, VirusTotal, or anything else for which a plugin exists. If one isn't written, with a little python experience, you can easily write one yourself.