Update to v8.0.0

[antoineco]

Sending this PR early so people can experiment with the new v8.x series.
Marking as draft until v8.0.0 gets officially released (2022-02-08 according to the Elastic product end of life dates page).

---

Closes #543

List of changes impacting docker-elk:

Logstash

• The output to Elasticsearch is handled as a data stream.

  Starting with v8.0.0, the elasticsearch output for Logstash sends log data to a data stream instead of logstash-* indices by default. The name of the default data stream is logs-generic-default. docker-elk remains unopinionated and simply uses Elastic's defaults like it always has, so users who prefer to retain the old behaviour need to explicitly opt-out of data streams in their Logstash pipelines.

  Refs:

  • https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html
  • https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-data-streams

• The (legacy) monitoring data collection is now disabled.

  This feature was deprecated since v7.9.0, and removed in v8.0.0.

  Ref: https://www.elastic.co/guide/en/logstash/current/monitoring-internal-collection-legacy.html

Elasticsearch

• The command line tool elasticsearch-setup-passwords was deprecated in favour of a new elasticsearch-reset-password tool.

  Passwords for built-in users must now be generated one by one.

  Ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-passwords.html

Kibana

• An index pattern for logs-* indices is automatically created.

  It used to be required to manually create an index pattern for indices managed by Logstash, even when using the default Logstash indices. This is no longer the case since the output data is now being handled as a data stream, and Kibana automatically creates index patterns for these.

Enterprise Search

• Kibana is now the new management interface, and the only one available moving forward.

  The old standalone Enterprise Search interface was removed in v8.0.0.

  Ref: https://www.elastic.co/guide/en/enterprise-search/current/user-interfaces.html

[antoineco]

Tests are currently failing with a pesky error, which seems to be caused by the Logstash TCP input in particular (possibly related to logstash-plugins/logstash-input-tcp#176):

object mapping for [host] tried to parse field [host] as object, but found a concrete value

It feels to me like all this is related to the ECS schema Elastic is pushing to their products.

Possibly relevant:

• https://discuss.elastic.co/t/problem-with-transfer-filebeat-6-1-3-logstash-6-1-3-elasticsearch-6-1-3/136264
• https://www.elastic.co/guide/en/logstash/current/plugins-inputs-tcp.html#plugins-inputs-tcp-ecs_metadata
• https://www.elastic.co/guide/en/ecs/current/ecs-host.html

---

edit: Indeed, the tcp output automatically injects a host and a port fields at the top level of the generated documents (see the issue I linked above):

{
      "@version" => "1",
          "host" => "gateway",
          "port" => 45732,
       "message" => "dockerelk",
    "@timestamp" => 2021-08-11T17:06:55.679Z
}

Where the mapping for that host field is defined is a mystery for me though. I couldn't find a reference to such field in the index template that gets automatically created by the elasticsearch output.

I started a forum discussion to try clarifying it.

---

edit 2: I originally fixed this using a Logstash filter, but it turns out this is no longer required starting with 8.0.0-rc1. Logstash doesn't "pollute" the top scope anymore.