Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[3.0] Update neural-search for OpenSearch 3.0 compatibility #1141

Merged
merged 8 commits into from
Feb 17, 2025
Merged

[3.0] Update neural-search for OpenSearch 3.0 compatibility #1141

merged 8 commits into from
Feb 17, 2025

Conversation

martin-gaievski
Copy link
Member

@martin-gaievski martin-gaievski commented Jan 24, 2025 •
edited
Loading

Description

Prepare neural search code base to upcoming 3.x release.

Changes in this PR:

consume API of 3.0-alpha and related dependencies:

  • lucene 10.1
  • latest core OS code
  • consume latest format of lower level query explain response

changes in logic:

  • accept new format of wrapped query for index alias (core rewrites query into bool, now they do additional rewrite that query and structure of bool clauses can change)

infra:

  • version bump for BWC to 2.19 and 2.20-snapshot
  • fixed flaky BWC tests
  • remove some redundant CI actions: precommit and bwc for windows, extra aggs tests for jdk 21

Related Issues

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@martin-gaievski martin-gaievski added skip-changelog Maintenance Add support for new versions of OpenSearch/Dashboards from upstream labels Jan 24, 2025
@yuye-aws
Copy link
Member

@martin-gaievski Thanks for working on this PR. Feel free to ping me for code review once it's ready.

This was referenced Jan 30, 2025
Open
Merged
@martin-gaievski martin-gaievski mentioned this pull request Feb 5, 2025
Closed
@martin-gaievski martin-gaievski force-pushed the point_main_to_alpha1 branch 4 times, most recently from 615830a to 02b1249 Compare February 6, 2025 20:39
@peterzhuamazon
Copy link
Member

knn has updated to 3.0.0.0-alpha1 now.

Thanks.

@martin-gaievski
Copy link
Member Author

we also need ml-commons to be 3.0 alpha ready, neural consumes both knn and ml-commons

@martin-gaievski martin-gaievski changed the base branch from main to 3.0.0-alpha1 February 11, 2025 00:32
@martin-gaievski martin-gaievski changed the base branch from 3.0.0-alpha1 to main February 11, 2025 00:35
@martin-gaievski martin-gaievski changed the base branch from main to 3.0.0-alpha1 February 11, 2025 16:38
@junqiu-lei
Copy link
Member

Waiting ml-commons PR merged to unblock this PR. opensearch-project/ml-commons#3530

@junqiu-lei junqiu-lei mentioned this pull request Feb 12, 2025
Merged
2 tasks
@peterzhuamazon
Copy link
Member

Waiting ml-commons PR merged to unblock this PR. opensearch-project/ml-commons#3530

Pending opensearch-project/ml-commons#3426 as the other one is a duplication.

@martin-gaievski martin-gaievski force-pushed the point_main_to_alpha1 branch 2 times, most recently from a2a538b to 47303e1 Compare February 13, 2025 01:47
@dbwiddis dbwiddis mentioned this pull request Feb 13, 2025
Merged
8 tasks
@peterzhuamazon
Copy link
Member

Seems like conflicts with main now.

@martin-gaievski martin-gaievski force-pushed the point_main_to_alpha1 branch 3 times, most recently from dca84f5 to ec3e60c Compare February 14, 2025 02:57
@peterzhuamazon
Copy link
Member

Well @reta @martin-gaievski after @cwperks rerun the snapshots publishing, I think it is passing for now 😄

Thanks.

@martin-gaievski martin-gaievski changed the title Fix failures after core upgrade to lucene 10 [3.0] Update neural-search for OpenSearch 3.0 compatibility Feb 14, 2025
@martin-gaievski martin-gaievski force-pushed the point_main_to_alpha1 branch 3 times, most recently from 1ffa7d6 to 2b2ef5d Compare February 15, 2025 01:03
@martin-gaievski martin-gaievski changed the base branch from 3.0.0-alpha1 to main February 15, 2025 01:53
@martin-gaievski martin-gaievski marked this pull request as ready for review February 15, 2025 01:55
zhichao-aws
zhichao-aws approved these changes Feb 17, 2025
View reviewed changes
Copy link
Member

@zhichao-aws zhichao-aws left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for creating the PR.

.github/workflows/CI.yml
@@ -117,28 +117,6 @@ jobs:
name: coverage-report-${{ matrix.os }}-${{ matrix.java }}
path: ./jacocoTestReport.xml

Precommit-neural-search-windows:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any background to remove the workflows on windows?

Copy link
Member Author

@martin-gaievski martin-gaievski Feb 17, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mainly it's to offload some work from CI to make it lighter => less flaky and faster. in other plugins (e.g. k-NN https://github.com/opensearch-project/k-NN/actions/runs/13340226034) they use just a single platform, and I cannot recall a case when we caught any error specific to win or linux platform.

@zhichao-aws zhichao-aws mentioned this pull request Feb 17, 2025
Merged
5 tasks
vibrantvarun
vibrantvarun reviewed Feb 17, 2025
View reviewed changes
Copy link
Member

@vibrantvarun vibrantvarun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One minor comment. Overall looks good.

@vibrantvarun vibrantvarun merged commit c36ca15 into opensearch-project:main Feb 17, 2025
74 checks passed
@vibrantvarun
Copy link
Member

Approved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vibrantvarun vibrantvarun vibrantvarun left review comments

@heemin32 heemin32 heemin32 approved these changes

@zhichao-aws zhichao-aws zhichao-aws approved these changes

@navneet1v navneet1v Awaiting requested review from navneet1v navneet1v is a code owner

@VijayanB VijayanB Awaiting requested review from VijayanB VijayanB is a code owner

@vamshin vamshin Awaiting requested review from vamshin vamshin is a code owner

@jmazanec15 jmazanec15 Awaiting requested review from jmazanec15 jmazanec15 is a code owner

@naveentatikonda naveentatikonda Awaiting requested review from naveentatikonda naveentatikonda is a code owner

@junqiu-lei junqiu-lei Awaiting requested review from junqiu-lei junqiu-lei is a code owner

@sean-zheng-amazon sean-zheng-amazon Awaiting requested review from sean-zheng-amazon sean-zheng-amazon is a code owner

@model-collapse model-collapse Awaiting requested review from model-collapse model-collapse is a code owner

@zane-neo zane-neo Awaiting requested review from zane-neo zane-neo is a code owner

@yuye-aws yuye-aws Awaiting requested review from yuye-aws yuye-aws is a code owner

@minalsha minalsha Awaiting requested review from minalsha minalsha is a code owner

Assignees
No one assigned
Labels
Infrastructure Maintenance Add support for new versions of OpenSearch/Dashboards from upstream
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@martin-gaievski @yuye-aws @peterzhuamazon @junqiu-lei @reta @vibrantvarun @heemin32 @zhichao-aws