promoting Filebeat as preferred way to forward logs

added sample Filebeat configuration, working with server-side Beats plugin 2.0.3 (see spujadas#12)

02-beats-input.conf

@@ -0,0 +1,8 @@
+input {
+  beats {
+    port => 5044
+    ssl => true
+    ssl_certificate => "/etc/pki/tls/certs/logstash-beats.crt"
+    ssl_key => "/etc/pki/tls/private/logstash-beats.key"
+  }
+}

Dockerfile

@@ -78,16 +78,19 @@ ADD ./elasticsearch.yml /etc/elasticsearch/elasticsearch.yml
 
 ### configure Logstash
 
-# cert/key
+# certs/keys for Beats and Lumberjack input
 RUN mkdir -p /etc/pki/tls/certs && mkdir /etc/pki/tls/private
 ADD ./logstash-forwarder.crt /etc/pki/tls/certs/logstash-forwarder.crt
 ADD ./logstash-forwarder.key /etc/pki/tls/private/logstash-forwarder.key
+ADD ./logstash-beats.crt /etc/pki/tls/certs/logstash-beats.crt
+ADD ./logstash-beats.key /etc/pki/tls/private/logstash-beats.key
 
 # filters
 ADD ./01-lumberjack-input.conf /etc/logstash/conf.d/01-lumberjack-input.conf
+ADD ./02-beats-input.conf /etc/logstash/conf.d/02-beats-input.conf
 ADD ./10-syslog.conf /etc/logstash/conf.d/10-syslog.conf
 ADD ./11-nginx.conf /etc/logstash/conf.d/11-nginx.conf
-ADD ./30-lumberjack-output.conf /etc/logstash/conf.d/30-lumberjack-output.conf
+ADD ./30-output.conf /etc/logstash/conf.d/30-output.conf
 
 # patterns
 ADD ./nginx.pattern ${LOGSTASH_HOME}/patterns/nginx
@@ -101,7 +104,7 @@ RUN chown -R logstash:logstash ${LOGSTASH_HOME}/patterns
 ADD ./start.sh /usr/local/bin/start.sh
 RUN chmod +x /usr/local/bin/start.sh
 
-EXPOSE 5601 9200 9300 5000
+EXPOSE 5601 9200 9300 5000 5044
 VOLUME /var/lib/elasticsearch
 
 CMD [ "/usr/local/bin/start.sh" ]

docker-compose.yml

@@ -12,3 +12,4 @@ elk:
     - "5601:5601"
     - "9200:9200"
     - "5000:5000"
+    - "5044:5044"

docs/index.md

@@ -37,19 +37,20 @@ To pull this image from the [Docker registry](https://hub.docker.com/r/sebp/elk/
 
 Run the container from the image with the following command:
 
-	$ sudo docker run -p 5601:5601 -p 9200:9200 -p 5000:5000 -it --name elk sebp/elk
+	$ sudo docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -p 5000:5000 -it --name elk sebp/elk
 
 This command publishes the following ports, which are needed for proper operation of the ELK stack:
 
 - 5601 (Kibana web interface).
 - 9200 (Elasticsearch JSON interface).
-- 5000 (Logstash server, receives logs from Logstash forwarders – see the *[Forwarding logs](#forwarding-logs)* section below).
+- 5044 (Logstash Beats interface, receives logs from Beats such as Filebeat – see the *[Forwarding logs](#forwarding-logs)* section below).
+- 5000 (Logstash Lumberjack interface, receives logs from Logstash forwarders – see the *[Forwarding logs](#forwarding-logs)* section below).
 
 **Note** – The image also exposes Elasticsearch's transport interface on port 9300. Use the `-p 9300:9300` option with the `docker` command above to publish it.
 
 The figure below shows how the pieces fit together.
 
-![](http://i.imgur.com/RVW12Md.png)
+![](http://i.imgur.com/wDertsM.png)
 
 Access Kibana's web interface by browsing to `http://<your-host>:5601`, where `<your-host>` is the hostname or IP address of the host Docker is running on (see note), e.g. `localhost` if running a local native version of Docker, or the IP address of the virtual machine if running a VM-hosted version of Docker (see note).
 
@@ -68,6 +69,7 @@ If you're using [Docker Compose](https://docs.docker.com/compose/) to manage you
 	  ports:
 	    - "5601:5601"
 	    - "9200:9200"
+	    - "5044:5044"
 	    - "5000:5000"
 
 You can then start the ELK container like this:
@@ -102,7 +104,7 @@ Wait for Logstash to start (as indicated by the message `Logstash startup comple
 
 	this is a dummy entry
 
-**Note** - You can create as many entries as you want. Use `^C` to go back to the bash prompt.
+**Note** – You can create as many entries as you want. Use `^C` to go back to the bash prompt.
 
 If you browse to `http://<your-host>:9200/_search?pretty` (e.g. [http://localhost:9200/_search?pretty](http://localhost:9200/_search?pretty) for a local native instance of Docker) you'll see that Elasticsearch has indexed the entry:
 
@@ -127,60 +129,24 @@ Make sure that the drop-down "Time-field name" field is pre-populated with the v
 
 Forwarding logs from a host relies on a forwarding agent that collects logs (e.g. from log files, from the syslog daemon) and sends them to our instance of Logstash.
 
-The forwarding agent that was originally used with Logstash was Logstash forwarder, but with the introduction of the [Beats platform](https://www.elastic.co/products/beats) it may eventually be phased out in favour of Filebeat. The two approaches are therefore presented below.
-
-### Forwarding logs with Logstash forwarder <a name="forwarding-logs-logstash-forwarder"></a>
-
-Install [Logstash forwarder](https://github.com/elasticsearch/logstash-forwarder) on the host you want to collect and forward logs from (see the *[References](#references)* section below for links to detailed instructions).
-
-Here is a sample configuration file for Logstash forwarder, that forwards syslog and authentication logs, as well as [nginx](http://nginx.org/) logs.
-
-	{
-	  "network": {
-	    "servers": [ "elk:5000" ],
-	    "timeout": 15,
-	    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
-	  },
-	  "files": [
-	    {
-	      "paths": [
-	        "/var/log/syslog",
-	        "/var/log/auth.log"
-	       ],
-	      "fields": { "type": "syslog" }
-	    },
-	    {
-	      "paths": [
-	        "/var/log/nginx/access.log"
-	       ],
-	      "fields": { "type": "nginx-access" }
-	    }
-	   ]
-	}
-
-By default (see `/etc/init.d/logstash-forwarder` if you need to tweak anything):
-
-- The Logstash forwarder configuration file must be located in `/etc/logstash-forwarder`.
-- The Logstash forwarder needs a syslog daemon (e.g. rsyslogd, syslog-ng) to be running.
-
-In the sample configuration file, make sure that you:
-
-- Replace `elk` in `elk:5000` with the hostname or IP address of the ELK-serving host.
-- Copy the `logstash-forwarder.crt` file (which contains the Logstash server's certificate) from the ELK image to `/etc/pki/tls/certs/logstash-forwarder.crt`.
-
-**Note** – The ELK image includes configuration items (`/etc/logstash/conf.d/11-nginx.conf` and `/opt/logstash/patterns/nginx`) to parse nginx access logs, as forwarded by the Logstash forwarder instance above.
+The forwarding agent that was originally used with Logstash was Logstash forwarder, but with the introduction of the [Beats platform](https://www.elastic.co/products/beats) it will be phased out in favour of Filebeat, which should now be the preferred option. The two approaches are described below.
 
 ### Forwarding logs with Filebeat <a name="forwarding-logs-filebeat"></a>
 
 Install [Filebeat](https://www.elastic.co/products/beats/filebeat) on the host you want to collect and forward logs from (see the *[References](#references)* section below for links to detailed instructions).
 
+**Note** – The `nginx-filebeat` subdirectory of the [source Git repository on GitHub](https://github.com/spujadas/elk-docker) contains a sample `Dockerfile` which enables you to create a Docker image that implements the steps below.
+
 Here is a sample `/etc/filebeat/filebeat.yml` configuration file for Filebeat, that forwards syslog and authentication logs, as well as [nginx](http://nginx.org/) logs.
 
 	output:
 	  logstash:
 	    enabled: true
 	    hosts:
 	      - elk:5044
+	    tls:
+		  certificate_authorities:
+      	    - /etc/pki/tls/certs/logstash-beats.crt
 	    timeout: 15
 
 	filebeat:
@@ -197,32 +163,59 @@ Here is a sample `/etc/filebeat/filebeat.yml` configuration file for Filebeat, t
 
 In the sample configuration file, make sure that you replace `elk` in `elk:5044` with the hostname or IP address of the ELK-serving host.
 
+You'll also need to copy the `logstash-beats.crt` file (which contains the CA certificate – or server certificate as the certificate is self-signed – for Logstash's Beats input plugin) from the ELK image to `/etc/pki/tls/certs/logstash-beats.crt`.
+
 **Note** – The ELK image includes configuration items (`/etc/logstash/conf.d/11-nginx.conf` and `/opt/logstash/patterns/nginx`) to parse nginx access logs, as forwarded by the Filebeat instance above.
 
-Additionally, the ELK image needs to extended (see [Extending the image](#extending-image) below for more information) for Logstash to accept Beat events:
+Before starting Filebeat for the first time, run this command (replace `elk` with the appropriate hostname) to load the default index template in Elasticsearch:
 
-- An input configuration file such as the following (which you can name `/etc/logstash/conf.d/02-beats-input.conf`) must be added to the image.
+		curl -XPUT 'http://elk:9200/_template/filebeat?pretty' -d@/etc/filebeat/filebeat.template.json
 
-		input {
-		  beats {
-		    port => 5044
-		  }
-		}
+Start Filebeat:
+
+		sudo /etc/init.d/filebeat start
 
-- Port 5044 must be exposed in order to be publishable (and therefore reachable) from Filebeat.
+### Forwarding logs with Logstash forwarder <a name="forwarding-logs-logstash-forwarder"></a>
 
-**Note** – This Beats-enabled configuration of Logstash may be natively included in a future version of the ELK image.
+**Note** – This approach is deprecated: [using Filebeat](#forwarding-logs-filebeat) is now the preferred way to forward logs.
 
-Once the ELK image has been extended and started (with port 5044 published), on the log-emitting host where Filebeat has been installed:
+Install [Logstash forwarder](https://github.com/elasticsearch/logstash-forwarder) on the host you want to collect and forward logs from (see the *[References](#references)* section below for links to detailed instructions).
 
-- Before starting Filebeat for the first time, run this command (replace `elk` with the appropriate hostname) to load the default index template in Elasticsearch:
+Here is a sample configuration file for Logstash forwarder, that forwards syslog and authentication logs, as well as [nginx](http://nginx.org/) logs.
 
-		curl -XPUT 'http://elk:9200/_template/filebeat?pretty' -d@/etc/filebeat/filebeat.template.json
+	{
+	  "network": {
+	    "servers": [ "elk:5000" ],
+	    "timeout": 15,
+	    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
+	  },
+	  "files": [
+	    {
+	      "paths": [
+	        "/var/log/syslog",
+	        "/var/log/auth.log"
+	       ],
+	      "fields": { "type": "syslog" }
+	    },
+	    {
+	      "paths": [
+	        "/var/log/nginx/access.log"
+	       ],
+	      "fields": { "type": "nginx-access" }
+	    }
+	   ]
+	}
 
-- Start Filebeat:
+By default (see `/etc/init.d/logstash-forwarder` if you need to tweak anything):
 
-		sudo /etc/init.d/filebeat start
+- The Logstash forwarder configuration file must be located in `/etc/logstash-forwarder`.
+- The Logstash forwarder needs a syslog daemon (e.g. rsyslogd, syslog-ng) to be running.
+
+In the sample configuration file, make sure that you replace `elk` in `elk:5000` with the hostname or IP address of the ELK-serving host.
 
+You'll also need to copy the `logstash-forwarder.crt` file (which contains the CA certificate – or server certificate as the certificate is self-signed – for Logstash's Lumberjack input plugin) from the ELK image to `/etc/pki/tls/certs/logstash-forwarder.crt`.
+
+**Note** – The ELK image includes configuration items (`/etc/logstash/conf.d/11-nginx.conf` and `/opt/logstash/patterns/nginx`) to parse nginx access logs, as forwarded by the Logstash forwarder instance above.
 
 ### Linking a Docker container to the ELK container <a name="linking-containers"></a>
 
@@ -232,7 +225,7 @@ If you want to forward logs from a Docker container to the ELK container, then y
 
 First of all, give the ELK container a name (e.g. `elk`) using the `--name` option:
 
-	$ sudo docker run -p 5601:5601 -p 9200:9200 -p 5000:5000 -it --name elk sebp/elk
+	$ sudo docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -p 5000:5000 -it --name elk sebp/elk
 
 Then start the log-emitting container with the `--link` option (replacing `your/image` with the name of the Logstash-forwarder-enabled image you're forwarding logs from):
 
@@ -254,6 +247,7 @@ With Compose here's what example entries for a (locally built log-generating) co
 	  ports:
 	    - "5601:5601"
 	    - "9200:9200"
+	    - "5044:5044"
 	    - "5000:5000"
 
 ## Building the image <a name="building-image"></a>
@@ -325,13 +319,13 @@ See Docker's page on [Managing Data in Containers](https://docs.docker.com/userg
 
 ## Security considerations <a name="security-considerations"></a>
 
-As it stands this image is meant for local test use, and as such hasn't been secured: access to the ELK services is not restricted, and a default authentication server certificate (`logstash-forwarder.crt`) and private key (`logstash-forwarder.key`) are bundled with the image.
+As it stands this image is meant for local test use, and as such hasn't been secured: access to the ELK services is not restricted, and default authentication server certificates (`logstash-*.crt`) and private keys (`logstash-*.key`) for the Logstash input plugins are bundled with the image.
 
 To harden this image, at the very least you would want to:
 
 - Restrict the access to the ELK services to authorised hosts/networks only, as described in e.g. [Elasticsearch Scripting and Security](http://www.elasticsearch.org/blog/scripting-security/) and [Elastic Security: Deploying Logstash, ElasticSearch, Kibana "securely" on the Internet](http://blog.eslimasec.com/2014/05/elastic-security-deploying-logstash.html).
 - Password-protect the access to Kibana and Elasticsearch (see [SSL And Password Protection for Kibana](http://technosophos.com/2014/03/19/ssl-password-protection-for-kibana.html)).
-- Generate a new self-signed authentication certificate for the Logstash server (e.g. `cd /etc/pki/tls; sudo openssl req -x509 -batch -nodes -subj "/CN=elk/" -days 3650 -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt` for a 10-year certificate issued to a server with hostname `elk`) or (better) get a proper certificate from a commercial provider (known as a certificate authority), and keep the private key private.
+- Generate a new self-signed authentication certificate for the Logstash input plugins (e.g. `cd /etc/pki/tls; sudo openssl req -x509 -batch -nodes -subj "/CN=elk/" -days 3650 -newkey rsa:2048 -keyout private/logstash-beats.key -out certs/logstash-beats.crt` for a 10-year certificate issued to a server with hostname `elk` for the Beats input plugin) or (better) get a proper certificate from a commercial provider (known as a certificate authority), and keep the private key private.
 
 ## References <a name="references"></a>
 

logstash-beats.crt

@@ -0,0 +1,18 @@
+-----BEGIN CERTIFICATE-----
+MIIC6zCCAdOgAwIBAgIJANPZwuf+5wTLMA0GCSqGSIb3DQEBCwUAMAwxCjAIBgNV
+BAMMASowHhcNMTUxMjI4MTA0NTMyWhcNMjUxMjI1MTA0NTMyWjAMMQowCAYDVQQD
+DAEqMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAp+jHFvhyYKiPXc7k
+0c33f2QV+1hHNyW/uwcJbp5jG82cuQ41v70Z1+b2veBW4sUlDY3yAIEOPSUD8ASt
+9m72CAo4xlwYKDvm/Sa3KJtDk0NrQiz6PPyBUFsY+Bj3xn6Nz1RW5YaP+Q1Hjnks
+PEyQu4vLgfTSGYBHLD4gvs8wDWY7aaKf8DfuP7Ov74Qlj2GOxnmiDEF4tirlko0r
+qQcvBgujCqA7rNoG+QDmkn3VrxtX8mKF72bxQ7USCyoxD4cWV2mU2HD2Maed3KHj
+KAvDAzSyBMjI+qi9IlPN5MR7rVqUV0VlSKXBVPct6NG7x4WRwnoKjTXnr3CRADD0
+4uvbQQIDAQABo1AwTjAdBgNVHQ4EFgQUVFurgDwdcgnCYxszc0dWMWhB3DswHwYD
+VR0jBBgwFoAUVFurgDwdcgnCYxszc0dWMWhB3DswDAYDVR0TBAUwAwEB/zANBgkq
+hkiG9w0BAQsFAAOCAQEAaLSytepMb5LXzOPr9OiuZjTk21a2C84k96f4uqGqKV/s
+okTTKD0NdeY/IUIINMq4/ERiqn6YDgPgHIYvQheWqnJ8ir69ODcYCpsMXIPau1ow
+T8c108BEHqBMEjkOQ5LrEjyvLa/29qJ5JsSSiULHvS917nVgY6xhcnRZ0AhuJkiI
+ARKXwpO5tqJi6BtgzX/3VDSOgVZbvX1uX51Fe9gWwPDgipnYaE/t9TGzJEhKwSah
+kNr+7RM+Glsv9rx1KcWcx4xxY3basG3/KwvsGAFPvk5tXbZ780VuNFTTZw7q3p8O
+Gk1zQUBOie0naS0afype5qFMPp586SF/2xAeb68gLg==
+-----END CERTIFICATE-----

logstash-beats.key

@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQCn6McW+HJgqI9d
+zuTRzfd/ZBX7WEc3Jb+7BwlunmMbzZy5DjW/vRnX5va94FbixSUNjfIAgQ49JQPw
+BK32bvYICjjGXBgoO+b9Jrcom0OTQ2tCLPo8/IFQWxj4GPfGfo3PVFblho/5DUeO
+eSw8TJC7i8uB9NIZgEcsPiC+zzANZjtpop/wN+4/s6/vhCWPYY7GeaIMQXi2KuWS
+jSupBy8GC6MKoDus2gb5AOaSfdWvG1fyYoXvZvFDtRILKjEPhxZXaZTYcPYxp53c
+oeMoC8MDNLIEyMj6qL0iU83kxHutWpRXRWVIpcFU9y3o0bvHhZHCegqNNeevcJEA
+MPTi69tBAgMBAAECggEAGh1xQZhYqcHtsmhoXF1NfinB5XrAcMpVPLCGfgbyYTOk
+iX+1SmIN7++DNtr6iICjF62ZEwz/evET4LPJnsd5SpzUYb2XIELY1Uy9NfqYEwJs
+XzmBnhSjxCy3AHdZqiyqv7FdZot8Pv8avwUHpUU/SXwfpdG/D6pM54uuKh8tWRfp
+6Fun0x3tFLhr1iY/jwxXx+V5zZ1A7AyHnelSv1u7gnxd2WPNJsyoYp7iWkSbdmjr
+fThE8CsTSgL/ndKOmxnPLs7l7ZUipyBgwjmUoZxCZk1I4w9njXY9mti67+6/SAj7
+i26/c7p6H31C+FAqksdKOHWh+zCg8zf1kMW3x3P3MQKBgQDUncHjIZ2rDuoWrOIc
+ng6IbuyuSHjUDCs59Z2HQbAVe/0IgDKNspdfddgC5XSN8q5GIWvNQtJjQzuCBRJC
+SxKkncOcTH6J3eQ4e4+sqDbHIagHQwuSBQRYkjd/KN63HEEJoh5UB/r/77CRdOhT
+m6dBm7QvrlXUpcm+z2V3TwvahQKBgQDKK7Qg0mBKm8QVneZUqGRbzaLcOsDefzdP
+IKRhWphAia70z6eoPHgR5MaqCFTPAajFaX6xBzIkPT8g8YUMPHhnw5eJRh/58hbU
+84KBw9jNGjE+H+OTT8+qLicP9EoMOeSVknYIX/zPj7xww0w2mF6tYroKKsiQHZhv
+eB16YjqAjQKBgD1BCffm2mbK0DQiMK5v9t3lnziC1pS4wMdc9Lpf+VvnMbn+PRJH
+roapC8eh1ZeDoCPCQy2Kn9RLLVzDG0SQHlngvddMznPnwnVnW7gxaj6qep9E+JNj
+8KGX1ndDDg8RC8e7tiMdfXm401TEqp5TzLcBJcNK5Z1y+hGH7MKXumGFAoGATmkI
+4bn2Url7IY8uKCNvWRO2WIgJCcJ5Zx0X5BJI/q7nxldLhTp+ryH10ziL/AV+uaIi
+2vIZhmiitVo26foCEOyRN1KVUFGOfWU8dqvIyDOiaZ/gmd/YgP6Jc+yhU4CYoVI+
+qRzhZncu9OUqB/qsrb6evRa+1vZDiughNrgmTHkCgYBggzulPfHEBqhlcg6PCsUj
+W6YcxEEPojkPPBsG/aMkGOmCr75I2w81lcjyUv54AiVXpSKqgvs+Zg3nF1WRxMVG
+vevXsCc4wdJPn669J68uh34eHvMBJQ8I00P7tBcW1RjXpvaH/HLUMO51vnfMwBtR
+OrW1ssSF9AvUG1VUmnMSgA==
+-----END PRIVATE KEY-----

nginx-filebeat/Dockerfile

@@ -0,0 +1,65 @@
+# Dockerfile to illustrate how Filebeat can be used with nginx
+# Filebeat 1.0.1
+
+# Build with:
+# docker build -t <repo-user>/filebeat-nginx-example .
+
+# Run with:
+# docker run -p 80:80 -it --link <elk-container-name>:elk \
+#     --name filebeat-nginx-example <repo-user>/filebeat-nginx-example
+
+FROM nginx
+MAINTAINER Sebastien Pujadas http://pujadas.net
+ENV REFRESHED_AT 2015-11-28
+
+
+###############################################################################
+#                                INSTALLATION
+###############################################################################
+
+### install Filebeat
+
+RUN apt-get update -qq \
+ && apt-get install -qqy curl \
+ && apt-get clean
+
+RUN curl -L -O https://download.elastic.co/beats/filebeat/filebeat_1.0.1_amd64.deb \
+ && dpkg -i filebeat_1.0.1_amd64.deb \
+ && rm filebeat_1.0.1_amd64.deb
+
+
+###############################################################################
+#                                CONFIGURATION
+###############################################################################
+
+### tweak nginx image set-up
+
+# remove log symlinks
+RUN rm /var/log/nginx/access.log /var/log/nginx/error.log
+
+
+### configure Filebeat
+
+# config file
+ADD filebeat.yml /etc/filebeat/filebeat.yml
+
+# CA cert
+RUN mkdir -p /etc/pki/tls/certs
+ADD logstash-beats.crt /etc/pki/tls/certs/logstash-beats.crt
+
+###############################################################################
+#                                    DATA
+###############################################################################
+
+### add dummy HTML file
+
+COPY html /usr/share/nginx/html
+
+
+###############################################################################
+#                                    START
+###############################################################################
+
+ADD ./start.sh /usr/local/bin/start.sh
+RUN chmod +x /usr/local/bin/start.sh
+CMD [ "/usr/local/bin/start.sh" ]

nginx-filebeat/filebeat.yml

@@ -0,0 +1,21 @@
+output:
+  logstash:
+    enabled: true
+    hosts:
+      - elk:5044
+    timeout: 15
+    tls:
+      certificate_authorities:
+      - /etc/pki/tls/certs/logstash-beats.crt
+
+filebeat:
+  prospectors:
+    -
+      paths:
+        - /var/log/syslog
+        - /var/log/auth.log
+      document_type: syslog
+    -
+      paths:
+        - "/var/log/nginx/*.log"
+      document_type: nginx-access