Elasticsearch blocks Kibana upgrading from 8.18 to 9.0

[TinaHeiligers]

Elasticsearch Version

9.0

Installed Plugins

No response

Java Version

bundled

OS Version

Whatever version CI for Kibana runs on

Problem Description

Kibana has an integration test that expects upgrading from 8.18 to 9.0 to pass.
This test failed during es promotion checks and was skipped. This test is vital to ensure that kibana does upgrade successfully for the accepted upgrade path from 8.18 to 9.0

Error:

java.lang.IllegalStateException: cannot upgrade a node from version [8.18.0] directly to version [9.1.0], upgrade to version [8.19.0] first.
	at org.elasticsearch.env.NodeMetadata.verifyUpgradeToCurrentVersion(NodeMetadata.java:122) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.env.NodeMetadata.upgradeToCurrentVersion(NodeMetadata.java:135) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:635) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:333) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.node.NodeConstruction.validateSettings(NodeConstruction.java:542) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:286) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.node.Node.<init>(Node.java:185) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:296) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:296) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:90) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]

Steps to Reproduce

Run kibana from source on the main branch.
Unskip the test
Push a PR to check against the es promotion checks.
The test fails with error containing
[8.18.0] directly to version [9.1.0], upgrade to version [8.19.0] first.

Logs (if relevant)

No response

[rjernst]

I suspect this is because 9.0 and 9.1 currently share a transport version; we haven't yet created a unique transport version for 9.0. But just to clarify @TinaHeiligers , Kibana was upgrading from a snapshot of the ES 8.18.0 to a snapshot of 9.0.0? This definitely wasn't trying to upgrade to a 9.1.0 snapshot (ie main branch)?

[TinaHeiligers]

Kibana was upgrading from a snapshot of the ES 8.18.0 to a snapshot of 9.0.0?

Correct, the failure was in testing 9.0.0 upgrade path. The test was skipped on both the 9.0 and main and it is the 9.0 promotion that we need to have resolved.

[rjernst]

Can you lay out the exact steps the test takes? The error message indicates 9.1.0 is running, and the version being upgraded from is 8.18.0, which is not a valid upgrade path.

[TinaHeiligers]

The error message indicates 9.1.0 is running, and the version being upgraded from is 8.18.0, which is not a valid upgrade path.

@rjernst you're correct and these issues are a result from us having an "inbetween" minor that's made the supported upgrade paths awkward.

@jloleysens might have more info into the conclusion that it's an es issue.

For now, I can't think of an elegant solution until we're over the upgrade hump and have fully tested that this won't have user impact.

[jloleysens]

Apologies, I copy pasted the wrong message in previous discussions. You can see the following error message toward the bottom of these buildkite logs:

 cannot upgrade a node from version [8.18.0] directly to version [9.0.0], upgrade to version [8.19.0] first

This upgrade path should be OK. We are running Kibana on branch 9.0 and starting Kibana with this data archive:

https://github.com/elastic/kibana/blob/db87cd970c1ba0b114e6cfe58c0c8ddd961757e5/x-pack/test/upgrade_assistant_integration/fixtures/data_archives/upgrade_assistant.zip

(Did not test locally, but I'm pretty sure that is the data archive leading to this error at ES startup)

[rjernst]

When was this last tested? When 9.0 was first cut it (incorrectly) knew about 8.19, thinking that was 8.last (to it), but that was fixed in 803b7c17

[TinaHeiligers]

When was this last tested?

The failure was on Thursday, 30th Jan at 18.09 UTC,
testing Kibana on against an es snapshot build, so before the fix.