Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenSearch Sink add support for sending pipeline parameter in BulkRequest #4609

Merged
merged 4 commits into from
Jun 14, 2024

Conversation

sb2k16
Copy link
Member

@sb2k16 sb2k16 commented Jun 6, 2024

Description

In order for OpenSearch API source to support URL parameter pipeline for the Document API Bulk, the OpenSearch sink needs to be able to read the pipeline parameter received in the event and then send it to OpenSearch on Bulk API request. This pull request is intended to add support in OpenSearch sink to do the following:

  • Read pipeline config from the pipeline sink configuration
  • Set the value in the BulkRequest if received in the data prepper event
    • Support added for create and index actions

Example pipeline configuration to support this:

version: "2"
opensearch-api-pipeline:
  source:
    ...
  sink:
    - opensearch:
        ...
        pipeline: "${getMetadata(\"my-pipeline-metadata\")}"
        ...

Issues Resolved

Contributes to #248

Check List

  • [X ] New functionality includes testing.
  • New functionality has a documentation issue. Please link to it in this PR.
    • New functionality has javadoc added
  • [ X] Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…x and create action operation for bulk api requests

Signed-off-by: Souvik Bose <[email protected]>
Signed-off-by: Souvik Bose <[email protected]>
dlvenable
dlvenable previously approved these changes Jun 7, 2024
Copy link
Member

@dlvenable dlvenable left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @sb2k16 ! This will be a helpful feature for users!

kkondaka
kkondaka previously approved these changes Jun 13, 2024
@kkondaka kkondaka dismissed their stale review June 13, 2024 16:16

Looks like some integration tests are failing

@kkondaka
Copy link
Collaborator

@sb2k16 DCO check is failing

@kkondaka kkondaka merged commit f9c5399 into opensearch-project:main Jun 14, 2024
63 of 68 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants