Refactor read context streams to async streams

[kotwanikunal]

Description

• Refactor ReadContext streams to async streams to prevent joining on all streams causing a blocking ReadContext
• Original commit: andrross@d237c6f#diff-4eb8a623572c23131b1bc7bb5e75b1ba7d99a3efef0563cf607d35e43f83ac4c

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Compatibility status:

Checks if related components are compatible with change 22110a4

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/26502/
• CommitID: 90b8d72
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: null ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/26503/
• CommitID: dcd4dfe
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/26504/
• CommitID: dcd4dfe
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: ❌
• URL:
• CommitID: dcd4dfe
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/26526/
• CommitID: dcd4dfe
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/26594/
• CommitID: 22110a4
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[kotwanikunal]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/26594/
• CommitID: 22110a4
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

org.opensearch.repositories.blobstore.BlobStoreRepositoryRemoteIndexTests.testRetrieveShallowCopySnapshotCase1


REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.repositories.blobstore.BlobStoreRepositoryRemoteIndexTests.testRetrieveShallowCopySnapshotCase1" -Dtests.seed=3F918EAE17312403 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ja-JP -Dtests.timezone=Europe/Sarajevo -Druntime.java=20

Ran the test locally with the seed, seems to pass:

OpenSearch % ./gradlew ':server:test' --tests "org.opensearch.repositories.blobstore.BlobStoreRepositoryRemoteIndexTests.testRetrieveShallowCopySnapshotCase1" -Dtests.seed=3F918EAE17312403 -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ja-JP -Dtests.timezone=Europe/Sarajevo

> Configure project :
========================= WARNING =========================
         Backwards compatibility tests are disabled!
See https://github.com/opensearch-project/OpenSearch/issues/4173
===========================================================
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.3
  OS Info               : Mac OS X 13.5.2 (aarch64)
  JDK Version           : 11 (Amazon Corretto JDK)
  JAVA_HOME             : /Users/kkotwani/.sdkman/candidates/java/11.0.18-amzn
  Random Testing Seed   : 3F918EAE17312403
  In FIPS 140 mode      : false
=======================================
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.BootstrapForTesting (file:/Users/kkotwani/workspace/OpenSearch/test/framework/build/distributions/framework-3.0.0-SNAPSHOT.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.BootstrapForTesting
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.gradle.api.internal.tasks.testing.worker.TestWorker (file:/Users/kkotwani/.gradle/wrapper/dists/gradle-8.3-all/6en3ugtfdg5xnpx44z4qbwgas/gradle-8.3/lib/plugins/gradle-testing-base-8.3.jar)
WARNING: Please consider reporting this to the maintainers of org.gradle.api.internal.tasks.testing.worker.TestWorker
WARNING: System::setSecurityManager will be removed in a future release

BUILD SUCCESSFUL in 28s
52 actionable tasks: 1 executed, 51 up-to-date

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/26601/
• CommitID: 22110a4
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[kotwanikunal]

#8762

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/26601/
• CommitID: 22110a4
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.test.rest.ClientYamlTestSuiteIT.test {p0=scroll/11_clear/Body params with array param override query string}
      1 org.opensearch.test.rest.ClientYamlTestSuiteIT.classMethod
      1 org.opensearch.indices.replication.SegmentReplicationAllocationIT.testAllocationWithDisruption

• URL: https://build.ci.opensearch.org/job/gradle-check/26610/
• CommitID: 22110a4
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #10284 (22110a4) into main (e156582) will increase coverage by 0.00%.
Report is 2 commits behind head on main.
The diff coverage is 78.78%.

@@            Coverage Diff            @@
##               main   #10284   +/-   ##
=========================================
  Coverage     71.18%   71.19%           
+ Complexity    58241    58239    -2     
=========================================
  Files          4830     4830           
  Lines        274592   274585    -7     
  Branches      40020    40022    +2     
=========================================
+ Hits         195469   195489   +20     
+ Misses        62701    62685   -16     
+ Partials      16422    16411   -11     

Files	Coverage Δ
...rg/opensearch/repositories/s3/S3BlobContainer.java	80.63% <100.00%> (+1.48%)	⬆️
...bstore/AsyncMultiStreamEncryptedBlobContainer.java	59.18% <100.00%> (+1.73%)	⬆️
...arch/common/blobstore/stream/read/ReadContext.java	100.00% <100.00%> (ø)
...tore/stream/read/listener/ReadContextListener.java	100.00% <100.00%> (ø)
...earch/index/store/RemoteSegmentStoreDirectory.java	90.74% <100.00%> (+1.06%)	⬆️
...ices/replication/RemoteStoreReplicationSource.java	90.47% <100.00%> (ø)
...ommon/blobstore/AsyncMultiStreamBlobContainer.java	0.00% <0.00%> (ø)
...in/java/org/opensearch/index/shard/IndexShard.java	68.74% <85.71%> (-0.78%)	⬇️
...blobstore/stream/read/listener/FilePartWriter.java	78.78% <0.00%> (-14.32%)	⬇️

... and 460 files with indirect coverage changes

[kotwanikunal]

Closing in favor of #10349