Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New OpenSearch client cannot handle some node roles #18585

Closed
mpfz0r opened this issue Mar 12, 2024 · 5 comments
Closed

New OpenSearch client cannot handle some node roles #18585

mpfz0r opened this issue Mar 12, 2024 · 5 comments
Labels
blocker If not finished by release date, the release will be postponed. bug

Comments

@mpfz0r
Copy link
Contributor

mpfz0r commented Mar 12, 2024 •
edited
Loading

Seeing this exception in my logs since I merged in current master.
It doesn't seem to break anything obvious, but..

Update: This actually breaks archiving for me. Updating this to blocker

2024-03-12 15:20:26,464 ERROR: org.graylog2.shared.rest.exceptionmappers.AnyExceptionClassMapper - Unhandled exception in REST resource
org.graylog.shaded.opensearch2.org.opensearch.OpenSearchException: Could not read nodes list from indexer!
	at org.graylog.storage.opensearch2.OpenSearchClient.exceptionFrom(OpenSearchClient.java:254) ~[classes/:?]
	at org.graylog.storage.opensearch2.OpenSearchClient.execute(OpenSearchClient.java:177) ~[classes/:?]
	at org.graylog.storage.opensearch2.ClusterAdapterOS2.nodesInfo(ClusterAdapterOS2.java:299) ~[classes/:?]
	at org.graylog2.telemetry.rest.TelemetryService.getSearchClusterInfo(TelemetryService.java:209) ~[classes/:?]
	at org.graylog2.telemetry.rest.TelemetryService.getTelemetryResponse(TelemetryService.java:114) ~[classes/:?]
	at org.graylog2.telemetry.rest.TelemetryResource.get(TelemetryResource.java:69) ~[classes/:?]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[jersey-server-3.1.5.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146) ~[jersey-server-3.1.5.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189) ~[jersey-server-3.1.5.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[jersey-server-3.1.5.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93) ~[jersey-server-3.1.5.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) ~[jersey-server-3.1.5.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) ~[jersey-server-3.1.5.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) ~[jersey-server-3.1.5.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:261) [jersey-server-3.1.5.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [jersey-common-3.1.5.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [jersey-common-3.1.5.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [jersey-common-3.1.5.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [jersey-common-3.1.5.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [jersey-common-3.1.5.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [jersey-common-3.1.5.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:240) [jersey-server-3.1.5.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:697) [jersey-server-3.1.5.jar:?]
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:367) [jersey-container-grizzly2-http-3.1.5.jar:?]
	at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:190) [grizzly-http-server-4.0.1.jar:4.0.1]
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:212) [metrics-core-4.2.25.jar:4.2.25]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: jakarta.json.stream.JsonParsingException: Invalid enum 'cluster_manager-eligible'
	at org.opensearch.client.json.JsonEnum$Deserializer.deserialize(JsonEnum.java:116) ~[opensearch-java-2.9.0.jar:?]
	at org.opensearch.client.json.JsonEnum$Deserializer.deserialize(JsonEnum.java:102) ~[opensearch-java-2.9.0.jar:?]
	at org.opensearch.client.json.JsonEnum$Deserializer.deserialize(JsonEnum.java:61) ~[opensearch-java-2.9.0.jar:?]
	at org.opensearch.client.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:343) ~[opensearch-java-2.9.0.jar:?]
	at org.opensearch.client.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:308) ~[opensearch-java-2.9.0.jar:?]
	at org.opensearch.client.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:87) ~[opensearch-java-2.9.0.jar:?]
	at org.opensearch.client.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:81) ~[opensearch-java-2.9.0.jar:?]
	at org.opensearch.client.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:185) ~[opensearch-java-2.9.0.jar:?]
	at org.opensearch.client.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:146) ~[opensearch-java-2.9.0.jar:?]
	at org.opensearch.client.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:87) ~[opensearch-java-2.9.0.jar:?]
	at org.opensearch.client.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:91) ~[opensearch-java-2.9.0.jar:?]
	at org.opensearch.client.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:55) ~[opensearch-java-2.9.0.jar:?]
	at org.opensearch.client.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:369) ~[opensearch-java-2.9.0.jar:?]
	at org.opensearch.client.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:355) ~[opensearch-java-2.9.0.jar:?]
	at org.opensearch.client.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:87) ~[opensearch-java-2.9.0.jar:?]
	at org.opensearch.client.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:81) ~[opensearch-java-2.9.0.jar:?]
	at org.opensearch.client.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:185) ~[opensearch-java-2.9.0.jar:?]
	at org.opensearch.client.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:146) ~[opensearch-java-2.9.0.jar:?]
	at org.opensearch.client.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:87) ~[opensearch-java-2.9.0.jar:?]
	at org.opensearch.client.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:91) ~[opensearch-java-2.9.0.jar:?]
	at org.opensearch.client.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:55) ~[opensearch-java-2.9.0.jar:?]
	at org.opensearch.client.transport.rest_client.RestClientTransport.decodeResponse(RestClientTransport.java:317) ~[opensearch-java-2.9.0.jar:?]
	at org.opensearch.client.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:283) ~[opensearch-java-2.9.0.jar:?]
	at org.opensearch.client.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:143) ~[opensearch-java-2.9.0.jar:?]
	at org.opensearch.client.opensearch.nodes.OpenSearchNodesClient.info(OpenSearchNodesClient.java:145) ~[opensearch-java-2.9.0.jar:?]
	at org.graylog.storage.opensearch2.ClusterAdapterOS2.lambda$nodesInfo$19(ClusterAdapterOS2.java:299) ~[classes/:?]
	at org.graylog.storage.opensearch2.OpenSearchClient.execute(OpenSearchClient.java:175) ~[classes/:?]
	... 31 more
  • Graylog Version: 6.0.0-SNAPSHOT
  • OpenSearch Version: 2.11 (two nodes cluster with data tiering config)
$ http :9200/_cat/nodes
HTTP/1.1 200 OK

172.23.0.2 23 98 2 0.50 1.48 1.95 dimrs cluster_manager,cluster_manager-eligible,data,ingest,remote_cluster_client,search * opensearch-gl-on-prem-1
172.23.0.3 20 98 2 0.50 1.48 1.95 dimrs cluster_manager,cluster_manager-eligible,data,ingest,remote_cluster_client,search - opensearch-gl-on-prem-2
@mpfz0r mpfz0r added bug blocker If not finished by release date, the release will be postponed. labels Mar 12, 2024
@mpfz0r
Copy link
Contributor Author

mpfz0r commented Mar 14, 2024

Turns out that cluster_manager-eligible isn't an actual role.
Not sure how that ended up in my config 🙄
OpenSearch doesn't seem to care to verify role settings.

Case closed. Thanks @dennisoelkers for taking a look 🙏

@mpfz0r mpfz0r closed this as completed Mar 14, 2024
@dennisoelkers
Copy link
Member

Nevertheless I opened an issue for the client: opensearch-project/opensearch-java#893

@dennisoelkers dennisoelkers changed the title Exception thrown from TelemetryResource Exception thrown from TelemetryResource when any OpenSearch node has custom role Mar 14, 2024
@dennisoelkers dennisoelkers reopened this Mar 14, 2024
@dennisoelkers dennisoelkers removed the blocker If not finished by release date, the release will be postponed. label Mar 14, 2024
@dennisoelkers
Copy link
Member

Reopening this, because it might be an issue for users too.

@mpfz0r
Copy link
Contributor Author

mpfz0r commented Mar 14, 2024

Also this happens for the role search

Caused by: jakarta.json.stream.JsonParsingException: Invalid enum 'search'

But search is documented and needed for searchable snapshots:
https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/snapshots/searchable_snapshot/#configuring-a-node-to-use-searchable-snapshots

@mpfz0r mpfz0r changed the title Exception thrown from TelemetryResource when any OpenSearch node has custom role New OpenSearch client cannot handle some node roles Mar 15, 2024
@bernd bernd added the blocker If not finished by release date, the release will be postponed. label Mar 18, 2024
@janheise
Copy link
Contributor

fixed by #18648

@mpfz0r mpfz0r closed this as completed Mar 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
blocker If not finished by release date, the release will be postponed. bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@bernd @dennisoelkers @janheise @mpfz0r