ELK Stack and Blue Coat Logs, Part 2

Part 1 here.

Now that we have ELK stack instance up and running, time to configure Logstash for log ingestion and parsing. We’ll start with the easy part — input and output.

Input and output sections of LogStash configuration define where the data is coming from and where it is going to, which usually is Elasticsearch cluster.

There are several input plugins you might want to consider, including stdin, file, tcp, syslog. Stdin is really great for debugging, so you can run Logstash and manually feed the logs files (and, potentially, uncompressing them on the fly). There’s no required parameters, so your input section can be as simple as:

input {
stdin { }
}

File input plugin allows reading information from files and supports file rotation. There’s one required parameter path to specify file(s) location — you can specify a wildcard to monitor multiple files. For example:

input {
file {
path => "/path/to/logs/*.log"
}
}

Use tcp to read logs over the network (TCP connection). Port is a required parameter. TLS/SSL encryption is supported (see ssl_* parameters in the Logstash documentation — link below). For example:

input {
tcp {
port => 10514
}
}

Syslog input plugin allows receiving syslog logs. When using it, make sure there is no port conflict with system syslog daemon. No required parameters.

input {
syslog { }
}

More information on Logstash input plugins: https://www.elastic.co/guide/en/logstash/current/input-plugins.html

For output, two common options are elasticsearch and stdout.

Stdout is used mostly for debugging to check whether your Logstash filters (data parsing) works as expected. There’s no required parameters, but setting codec to rubydebug makes output much more readable. For example:

output {
stdout {
codec => rubydebug
}
}

Elasticsearch allows data output to Elasticsearch for indexing and subsequent access through Kibana. There’s no required parameters, but you should at least specify hosts and point it to your Elasticsearch cluster nodes:

elasticsearch {
hosts => ["localhost"]
}

More information on Logstash output plugins: https://www.elastic.co/guide/en/logstash/current/output-plugins.html

In the next part we’ll configure the log data parsing and transformation.

Random rumblings about #InfoSec. The opinions expressed here are my own and not necessarily those of my employer.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store