ELK Stack and Blue Coat Logs

Have been experimenting lately with using ELK stack to process Blue Coat web surfing logs. ELK is Elasticsearch + Logstash + Kibana. Logstash receives, transforms and submits logs to Elasticsearch, which is used as a search engine and data repository. Kibana connects to Elasticsearch and provides visualization and reporting capabilities. More: https://www.elastic.co/webinars/introduction-elk-stack

Installing ELK stack is fairly easy, pick a Linux distro, install Java (required by Elasticsearch and Logstash), then install three additional packages:

Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html

Logstash: https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html

Kibana: https://www.elastic.co/guide/en/kibana/current/setup.html

Or use this all-in-one guide (Ubuntu 14.04): http://christophe.vandeplas.com/2014/06/setting-up-single-node-elk-in-20-minutes.html

Next, Logstash needs to be configured to ingest Blue Coat logs. Logstash configuration consists of three section — define input (i.e. where are the logs coming from), define filters (parse and transform logs), define output (i.e. send the to Elasticsearch). Here’s what needs to be done at a high level in the filter section:

  • parse fields from the log entry
  • identify and parse the field(s) for the timestamp info; I have combined date and time into one field and defined the format for the combined field
  • define which fields should be treated as numbers (i.e. cs-bytes, sc-bytes, time-taken, etc.) so that they can be used appropriately for analysis in Kibana
  • do geo-IP lookup for relevant server IP address; my logs didn’t have s-supplicant-ip field, so I had to do DNS lookup of the server host name first
  • parse user agent string to have information about OS, device type, browser type etc.
  • split up web site categories into an array (in case a web site matches several categories) to it’s more useful for analysis

Logstash configuration details to follow…

ELK Stack and Blue Coat Logs, Part 2

ELK Stack and Blue Coat Logs, Part3

Random rumblings about #InfoSec. The opinions expressed here are my own and not necessarily those of my employer.

Random rumblings about #InfoSec. The opinions expressed here are my own and not necessarily those of my employer.