Skip to content

Commit

Permalink
Add troubleshooting section for the date filter
Browse files Browse the repository at this point in the history
Relates: elastic#16530
  • Loading branch information
robbavey committed Oct 11, 2024
1 parent 6064587 commit 2952c7b
Show file tree
Hide file tree
Showing 2 changed files with 59 additions and 0 deletions.
58 changes: 58 additions & 0 deletions docs/static/troubleshoot/ts-date.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
[[ts-date]]
==== Date Filter issues and solutions

[discrete]
[[ts-date-locale]]
===== Date Parse Failures when using certain Locales

*Symptoms*

The {ls} Date Filter is unexpectedly unable to parse dates after upgrading to {ls} `8.15+`:

A date filter configured to parse dates as follows:

```
date { locale => "en_gb" match => [ "[date]" , "yyyy MMM dd HH:mm:ss" ] target => "[parsed_date]"
```

when parsing the date `2024 Sep 30 11:11:11` would, prior to `8.15.0`, have created the following entry in the event:

```
"parsed_date" => 2024-09-30T11:11:11.000Z,
```

After `8.15.0`, this would have failed to parse with the following added instead:

```
"tags" => [
[0] "_dateparsefailure"
]
```

as it now expects the date to be `2024 Sept 30 11:11:11` for that specific locale.


*Background*

Between JDK17 and JDK21, a change was made in the locale provider provided by the JVM internal.
JDK17 and JDK21 uses the https://cldr.unicode.org/[CLDR] locale provider by default, and between the release of JDK17 and JDK21, the version
of CLDR used by the default changed from version 37 to version 42.

This change included an update that changed the abbreviated form of September from `Sep` to `Sept` in certain locales, including `en_GB` and `en_AU`.
This has caused some parse failures when there is a mismatch between the abbreviated form of date between Logstash and where the date was created.

For example, if the systems running Logstash are running with either an `en_AU` or `en_GB locale`, they will
expect September dates to use the `Sept` abbreviated form when the format states `MMM`.
If the systems generating these dates are creating dates with a Sep abbreviation, parsing will fail.

{ls} `8.15.0` upgraded the bundled JDK version to JDK21, which included the updated version of the `CLDR`, changing the
date parsing behavior for certain locales.

** Mitigations**

* Change the locale in the date filter to a locale that still uses `Sep` as the abbreviated form for September, such as `en_US`.
* Switch out the locale provider from CLDR to compat by setting `-Djava.locale.providers=COMPAT,SPI` in `jvm.options`.
Note that this will fail for anyone using JDK23 and above, which drops support for the `compat` locale provider
* Add filters to substitute `Sep` with `Sept` for all fields where the date is being parsed by the date filter.
Make sure these filters are *before* the date filter in the pipeline configuration.

1 change: 1 addition & 0 deletions docs/static/troubleshoot/ts-plugins.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@

include::ts-kafka.asciidoc[]
include::ts-azure.asciidoc[]
include::ts-date.asciidoc[]

0 comments on commit 2952c7b

Please sign in to comment.