Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing test: Jest Integration Tests.src/core/server/integration_tests/saved_objects/migrations/group3 - incompatible_cluster_routing_allocation retries the INIT action with a descriptive message when cluster settings are incompatible #158318

Open
kibanamachine opened this issue May 23, 2023 · 74 comments · Fixed by #165319, #197429, #198794 or #204561 · May be fixed by #209828
Open

Failing test: Jest Integration Tests.src/core/server/integration_tests/saved_objects/migrations/group3 - incompatible_cluster_routing_allocation retries the INIT action with a descriptive message when cluster settings are incompatible #158318

kibanamachine opened this issue May 23, 2023 · 74 comments · Fixed by #165319, #197429, #198794 or #204561 · May be fixed by #209828
Assignees
@afharo
Labels
failed-es-promotion failed-test A test failure on a tracked branch, potentially flaky-test Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented May 23, 2023 •
edited
Loading

A test failed on a tracked branch

Error: Missing version for public endpoint GET /api/observability_onboarding/custom_logs/step/{name}
    at parseEndpoint (/var/lib/buildkite-agent/builds/kb-n2-4-spot-05ef58b5d43908c8/elastic/kibana-on-merge/kibana/packages/kbn-server-route-repository/src/parse_endpoint.ts:23:11)
    at /var/lib/buildkite-agent/builds/kb-n2-4-spot-05ef58b5d43908c8/elastic/kibana-on-merge/kibana/x-pack/plugins/observability_onboarding/server/routes/register_routes.ts:39:47
    at Array.forEach (<anonymous>)
    at forEach (/var/lib/buildkite-agent/builds/kb-n2-4-spot-05ef58b5d43908c8/elastic/kibana-on-merge/kibana/x-pack/plugins/observability_onboarding/server/routes/register_routes.ts:37:10)
    at ObservabilityOnboardingPlugin.setup (/var/lib/buildkite-agent/builds/kb-n2-4-spot-05ef58b5d43908c8/elastic/kibana-on-merge/kibana/x-pack/plugins/observability_onboarding/server/plugin.ts:66:19)
    at PluginWrapper.setup (/var/lib/buildkite-agent/builds/kb-n2-4-spot-05ef58b5d43908c8/elastic/kibana-on-merge/kibana/packages/core/plugins/core-plugins-server-internal/src/plugin.ts:105:26)
    at PluginsSystem.setup [as setupPlugins] (/var/lib/buildkite-agent/builds/kb-n2-4-spot-05ef58b5d43908c8/elastic/kibana-on-merge/kibana/packages/core/plugins/core-plugins-server-internal/src/plugins_system.ts:131:40)
    at PluginsService.setupPlugins [as setup] (/var/lib/buildkite-agent/builds/kb-n2-4-spot-05ef58b5d43908c8/elastic/kibana-on-merge/kibana/packages/core/plugins/core-plugins-server-internal/src/plugins_service.ts:166:52)
    at Server.setup (/var/lib/buildkite-agent/builds/kb-n2-4-spot-05ef58b5d43908c8/elastic/kibana-on-merge/kibana/packages/core/root/core-root-server-internal/src/server.ts:348:26)
    at Root.setup (/var/lib/buildkite-agent/builds/kb-n2-4-spot-05ef58b5d43908c8/elastic/kibana-on-merge/kibana/packages/core/root/core-root-server-internal/src/root/index.ts:66:14)
    at Object.<anonymous> (/var/lib/buildkite-agent/builds/kb-n2-4-spot-05ef58b5d43908c8/elastic/kibana-on-merge/kibana/src/core/server/integration_tests/saved_objects/migrations/group3/incompatible_cluster_routing_allocation.test.ts:126:5)

First failure: CI Build - main
@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label May 23, 2023
@botelastic botelastic bot added the needs-team Issues missing a team label label May 23, 2023
@kibanamachine kibanamachine added the Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc label May 23, 2023
@botelastic botelastic bot removed the needs-team Issues missing a team label label May 23, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-core (Team:Core)

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@dokmic
Copy link
Contributor

dokmic commented Jul 28, 2023

This was caused by #157802, which is unrelated to the failed test. The issue can be safely closed.

@dokmic dokmic closed this as completed Jul 28, 2023
@kibanamachine kibanamachine reopened this Aug 7, 2023
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@afharo
Copy link
Member

afharo commented Aug 24, 2023

This time, caused by the same reason as #163273. Closing.

@afharo afharo closed this as completed Aug 24, 2023
andrew-goldstein added a commit to andrew-goldstein/kibana that referenced this issue Aug 28, 2023
@andrew-goldstein
- adds langsmith to transformIgnorePatterns in `packages/kbn-test…
7df734f 
…/jest_integration_node/jest-preset.js` to resolve the following error:

```
Jest Integration Tests
src/core/server/integration_tests/saved_objects/migrations/group3
incompatible_cluster_routing_allocation retries the INIT action with a descriptive message when cluster settings are incompatible

Failures in tracked branches: 4
elastic#158318

Buildkite Job
https://buildkite.com/elastic/kibana-pull-request/builds/153873#018a3421-e8f1-48e5-afdf-43496e4d1b0d

Jest encountered an unexpected token

Jest failed to parse a file. This happens e.g. when your code or its dependencies use non-standard JavaScript syntax, or when Jest is not configured to support such syntax.

Out of the box Jest supports Babel, which will be used to transform your files into valid JS based on your Babel configuration.

By default "node_modules" folder is ignored by transformers.

Here's what you can do:
 • If you are trying to use ECMAScript Modules, see https://jestjs.io/docs/ecmascript-modules for how to enable it.
 • If you are trying to use TypeScript, see https://jestjs.io/docs/getting-started#using-typescript
 • To have some of your "node_modules" files transformed, you can specify a custom "transformIgnorePatterns" in your config.
 • If you need a custom transformation specify a "transform" option in your config.
 • If you simply want to mock your non-JS modules (e.g. binary assets) you can stub them out with the "moduleNameMapper" config option.

You'll find more details and examples of these config options in the docs:
https://jestjs.io/docs/configuration
For information about custom transformations, see:
https://jestjs.io/docs/code-transformation

Details:

/var/lib/buildkite-agent/builds/kb-n2-4-spot-f55f234db461f53e/elastic/kibana-pull-request/kibana/node_modules/langsmith/dist/index.js:1
({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,jest){export { Client } from "./client.js";
```
@kibanamachine kibanamachine reopened this Aug 28, 2023
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@afharo
Copy link
Member

afharo commented Aug 28, 2023

The error is

Error: Plugin "translations" can't be started since it isn't set up.
    at PluginWrapper.start (/var/lib/buildkite-agent/builds/kb-n2-4-spot-2d4ab435ffee02e8/elastic/kibana-on-merge/kibana/packages/core/plugins/core-plugins-server-internal/src/plugin.ts:117:13)
    at PluginsSystem.start [as startPlugins] (/var/lib/buildkite-agent/builds/kb-n2-4-spot-2d4ab435ffee02e8/elastic/kibana-on-merge/kibana/packages/core/plugins/core-plugins-server-internal/src/plugins_system.ts:188:40)
    at PluginsService.startPlugins [as start] (/var/lib/buildkite-agent/builds/kb-n2-4-spot-2d4ab435ffee02e8/elastic/kibana-on-merge/kibana/packages/core/plugins/core-plugins-server-internal/src/plugins_service.ts:194:56)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Server.start (/var/lib/buildkite-agent/builds/kb-n2-4-spot-2d4ab435ffee02e8/elastic/kibana-on-merge/kibana/packages/core/root/core-root-server-internal/src/server.ts:419:5)
    at Root.start (/var/lib/buildkite-agent/builds/kb-n2-4-spot-2d4ab435ffee02e8/elastic/kibana-on-merge/kibana/packages/core/root/core-root-server-internal/src/root/index.ts:77:14)

It seems unrelated again.

@afharo afharo closed this as completed Aug 28, 2023
@kibanamachine kibanamachine reopened this Aug 29, 2023
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

afharo added a commit to afharo/kibana that referenced this issue Aug 31, 2023
@afharo
[Flaky elastic#158318] Catch start-thrown errors as they are irrele…
f525ae4 
…vant to this test
@afharo afharo mentioned this issue Aug 31, 2023
Merged
1 task
@afharo afharo self-assigned this Aug 31, 2023
afharo added a commit that referenced this issue Sep 1, 2023
@afharo
[Flaky #158318] Catch start-thrown errors as they are irrelevant to…
bc37326 
… this test (#165319)
@kibanamachine kibanamachine reopened this Sep 29, 2023
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

JoseLuisGJ pushed a commit to JoseLuisGJ/kibana that referenced this issue Dec 19, 2024
@afharo @JoseLuisGJ
Update data archives for SO integration tests (elastic#204561)
899e397 
## Summary

Bumping the data archives to 8.last (8.18 at the moment) so that ES 9.0
finds it compatible.

Resolves elastic#158318.
Resolves elastic#163254.
Resolves elastic#163255.
Resolves elastic#166190.
Resolves elastic#166199.
Resolves elastic#167676.
benakansara pushed a commit to benakansara/kibana that referenced this issue Jan 2, 2025
@afharo @benakansara
Update data archives for SO integration tests (elastic#204561)
76a7e3e 
## Summary

Bumping the data archives to 8.last (8.18 at the moment) so that ES 9.0
finds it compatible.

Resolves elastic#158318.
Resolves elastic#163254.
Resolves elastic#163255.
Resolves elastic#166190.
Resolves elastic#166199.
Resolves elastic#167676.
CAWilson94 pushed a commit to CAWilson94/kibana that referenced this issue Jan 13, 2025
@afharo @CAWilson94
Update data archives for SO integration tests (elastic#204561)
1d45b20 
## Summary

Bumping the data archives to 8.last (8.18 at the moment) so that ES 9.0
finds it compatible.

Resolves elastic#158318.
Resolves elastic#163254.
Resolves elastic#163255.
Resolves elastic#166190.
Resolves elastic#166199.
Resolves elastic#167676.
viduni94 pushed a commit to viduni94/kibana that referenced this issue Jan 23, 2025
@afharo @viduni94
Update data archives for SO integration tests (elastic#204561)
71c7ceb 
## Summary

Bumping the data archives to 8.last (8.18 at the moment) so that ES 9.0
finds it compatible.

Resolves elastic#158318.
Resolves elastic#163254.
Resolves elastic#163255.
Resolves elastic#166190.
Resolves elastic#166199.
Resolves elastic#167676.
@kibanamachine kibanamachine reopened this Jan 27, 2025
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-elasticsearch-snapshot-verify - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-elasticsearch-snapshot-verify - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-elasticsearch-snapshot-verify - 9.0

@kibanamachine
Copy link
Contributor Author

New failure: kibana-elasticsearch-snapshot-verify - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-elasticsearch-snapshot-verify - 9.0

@kibanamachine
Copy link
Contributor Author

New failure: kibana-elasticsearch-snapshot-verify - main

mistic added a commit that referenced this issue Jan 30, 2025
@mistic
skip flaky es promotion suites (#158318)
777e1fe
mistic added a commit that referenced this issue Jan 30, 2025
@mistic
skip flaky es promotion suites (#158318)
9b1d55f
@mistic
Copy link
Member

mistic commented Jan 30, 2025

Skipped.

main: 777e1fe
9.0: 9b1d55f

@TinaHeiligers
Copy link
Contributor

Failure from logs:

[node-01][.ds-.logs-deprecation.elasticsearch-default-2024.09.19-000001] Error while parsing document for index [.ds-.logs-deprecation.elasticsearch-default-2024.09.19-000001]: [1:87] failed to parse field [data_stream.dataset] of type [constant_keyword] in document with id 'IGF5qJQBMulxMvzvf5Tr'. Preview of field's value: 'elasticsearch.deprecation' org.elasticsearch.index.mapper.DocumentParsingException: [1:87] failed to parse field [data_stream.dataset] of type [constant_keyword] in document with id 'IGF5qJQBMulxMvzvf5Tr'. Preview of field's value: 'elasticsearch.deprecation'

Maybe this has to do with the version of ES these tests are running against, rather than a kibana issue? This test has been troublesome since 2023, regularly failing shortly after each fix.

We need to come up with a different testing strategy that's more robust.

@TinaHeiligers TinaHeiligers mentioned this issue Jan 31, 2025
Merged
1 task
TinaHeiligers added a commit that referenced this issue Feb 1, 2025
@TinaHeiligers
test fix core migrations cluster routing allocation v9 (#209176)
4c043d2 
part of #158318 (v9)

Updates data archive for cluster routing allocation integration test for
v9


- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 9.0

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 9.0

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 9.0

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 9.0

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 9.0

@mistic
Copy link
Member

mistic commented Feb 4, 2025

@TinaHeiligers looks like this got flaky now 😃

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 9.0

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 9.0

@TinaHeiligers
Copy link
Contributor

@mistic go ahead and skip the test if it's causing problems. We'll look into the flakiness.

@TinaHeiligers
Copy link
Contributor

TinaHeiligers commented Feb 5, 2025 •
edited
Loading

The failure for backporting from main to 9.0.0 is the same as seen recently for other backported commits:

fatal exception while booting Elasticsearch java.lang.IllegalStateException: cannot upgrade a node from version [8.18.0] directly to version [9.1.0], upgrade to version [8.19.0] first.

There's an open issue in ES to fix this: elastic/elasticsearch#121693.
The data archive against which the test is run is from 8.18.0 and upgrading from 8.18.0 to 9.1.0 isn't supported.

Perhaps, we should be excluding these tests during backports from main to 8.18?

I'll try again, pushing a new archive from 8.19.0 only to main.

@TinaHeiligers TinaHeiligers linked a pull request Feb 5, 2025 that will close this issue
Draft
@TinaHeiligers
Copy link
Contributor

@mistic we don't technically need these tests on 9.1, only upgrading to 9.1
Is there a way we can keep them skipped on main and not skipped in 9.0, like a tag or something similar?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@afharo afharo

Labels
failed-es-promotion failed-test A test failure on a tracked branch, potentially flaky-test Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc
Projects
None yet
Milestone
No milestone
9 participants
@pgayvallet @jbudz @mistic @afharo @TinaHeiligers @dokmic @elasticmachine @gsoldevila @kibanamachine