[CI] RelocationIT testIndexSearchAndRelocateConcurrently failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #4621 / rhel-7_platform-support-unix
• elasticsearch-periodic-platform-support #4596 / windows-2022_checkpart1_platform-support-windows

Reproduction Line:

./gradlew ":server:internalClusterTest" --tests "org.elasticsearch.recovery.RelocationIT.testIndexSearchAndRelocateConcurrently" -Dtests.seed=1831477A5FD3EB9A -Dtests.locale=wae-Latn-CH -Dtests.timezone=Iran -Druntime.java=23

Applicable branches:
8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.Exception: Test abandoned because suite timeout was reached.

Issue Reasons:

• [8.x] 2 failures in test testIndexSearchAndRelocateConcurrently (0.9% fail rate in 234 executions)
• [8.x] 2 failures in pipeline elasticsearch-periodic-platform-support (33.3% fail rate in 6 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[nicktindall]

Flagging this as medium risk, the test logs some resource leaks before it fails, I think there's a good chance this is a legitimate bug

1> [2024-10-31T00:07:52,271][ERROR][o.e.t.LeakTracker        ] [[Cleaner-0]] LEAK: resource was not cleaned up before it was garbage-collected. |  
-- | --
  | 1> Recent access records: |  
  | 1> Created at: |  
  | 1> 	in [Thread-193][testIndexSearchAndRelocateConcurrently] |  
  | 1> 	org.elasticsearch.action.search.ArraySearchPhaseResults.<init>(ArraySearchPhaseResults.java:28) |  
  | 1> 	org.elasticsearch.action.search.QueryPhaseResultConsumer.<init>(QueryPhaseResultConsumer.java:86) |  
  | 1> 	org.elasticsearch.action.search.SearchPhaseController.newSearchPhaseResults(SearchPhaseController.java:841) |  
  | 1> 	org.elasticsearch.action.search.TransportSearchAction$AsyncSearchActionProvider.newSearchPhase(TransportSearchAction.java:1500) |  
  | 1> 	org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1305) |  
  | 1> 	org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:1038) |  
  | 1> 	org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$4(TransportSearchAction.java:365) |  
  | 1> 	org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:247) |  
  | 1> 	org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:109) |  
  | 1> 	org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77) |  
  | 1> 	org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:531) |  
  | 1> 	org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:316) |  
  | 1> 	org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:121)

[ywangd]

I think this is likely the same or a similar issue as #115056

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[ywangd]

I rerouted to the search team to take a look. Feel free to bounce it back if it does not feel right.