-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add partial support of advanced input transformations (ingest) #220
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… unnecessary data from ml processor ui config; Signed-off-by: Tyler Ohlsen <[email protected]>
Signed-off-by: Tyler Ohlsen <[email protected]>
Signed-off-by: Tyler Ohlsen <[email protected]>
Signed-off-by: Tyler Ohlsen <[email protected]>
Signed-off-by: Tyler Ohlsen <[email protected]>
…lation; Signed-off-by: Tyler Ohlsen <[email protected]>
Signed-off-by: Tyler Ohlsen <[email protected]>
ohltyler
requested review from
dbwiddis,
owaiskazi19,
joshpalis,
amitgalitz and
jackiehanyang
as code owners
July 17, 2024 20:06
ohltyler
changed the title
Add partial support of advanced input transformations
[WIP] Add partial support of advanced input transformations
Jul 17, 2024
ohltyler
changed the title
[WIP] Add partial support of advanced input transformations
[WIP] Add partial support of advanced input transformations (ingest)
Jul 17, 2024
Signed-off-by: Tyler Ohlsen <[email protected]>
Signed-off-by: Tyler Ohlsen <[email protected]>
Signed-off-by: Tyler Ohlsen <[email protected]>
ohltyler
changed the title
[WIP] Add partial support of advanced input transformations (ingest)
Add partial support of advanced input transformations (ingest)
Jul 17, 2024
mend failure can be ignored - it is occasionally failing to run and marking as failed. No new deps are added in this PR. |
dbwiddis
approved these changes
Jul 18, 2024
minalsha
approved these changes
Jul 18, 2024
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
Jul 18, 2024
Signed-off-by: Tyler Ohlsen <[email protected]> (cherry picked from commit 62d18e4)
ohltyler
added a commit
that referenced
this pull request
Jul 18, 2024
…#221) Signed-off-by: Tyler Ohlsen <[email protected]> (cherry picked from commit 62d18e4) Co-authored-by: Tyler Ohlsen <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR begins the process of supporting the ML inference processor form inputs to support an advanced transformation flow. The scope of this PR is to set up the base form changes to the input map / output map inputs, the base transformation modals for configuring advanced input / output flows, and partial support of input transformations in the context of ingest. More specifically:
MLProcessorInputs
componentMLProcessorInputs
component instead of the genericConfigFieldList
. This is required, since this particular processor we need to maintain more specific and complex state unique to this processor type. We still keep a parent/generic component with aswitch
to select the appropriate processor UX component based on the processor type. The main idea is switching from the components being flexible on a per-field basis ---> per-processor-type basis. This allows easy onboarding of new processor types we will eventually add.InputTransformModal
with partial implementation. Specifically, the ability to dynamically fetch the input schema. Future PRs will integrate with JSONPath and the ML interfaces to determine the output schema.bulk
API and updating the default document ingestion to be an array instead of an individual doc. This expands the capability and flexibility for users when trying to test/ingest with a set of documents_simulate
ingest pipeline API for executing proposed ingest pipeline configurations when users are configuring advanced inputs. For example, given some pipeline that looks likeProcessor A -> Processor B -> Processor C
, suppose a user wants to configureProcessor D
at the end. They go into the advanced view to configure an input transformation. To fetch the transformed document data up toProcessor D
, this PR adds logic to 1/ create a temporary pipeline configuration containingProcessor A -> Processor B -> Processor C
, collect the currently-configured documents in the form, run _simulate, and display the transformed doc results. See demo for a visual explanationDemo video, showing the advanced flow. Expected input to each processor is shown. For the first/initial processor, the expected input is simply the list of documents. For the second processor, the expected input is a transformed version of the document, containing the embeddings values. Internally, this creates an ingest pipeline containing all of the preceding processors and executes
_simulate
against the provided documents. Note a single input_map transform is made to convert thehello
field to the model's expectedinput
field.screen-capture.4.webm
Additional note: there is no defined UX on this, subject to change. But core functionality will remain the same.
Issues Resolved
Makes progress on #23
Check List
--signoff
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.