Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(core) Support for multi source dependency #90

Open
awspbade opened this issue Apr 12, 2022 · 3 comments
Open

(core) Support for multi source dependency #90

awspbade opened this issue Apr 12, 2022 · 3 comments

Comments

@awspbade
Copy link

Some common scenarios needs to wait for more than one file to go for the next step in the pipeline, common examples are joining different tables to enrich the data

I think there're two ways to make this.

  1. Make WaitingStage X time to let all files are ready. Simpler method I guess.
  2. I read some forums that propose: a manifest file as an input of the process that needs N files, also to control a full cycle of execution all files needs a prefix per cycle. Every time you receive a new file you check the prefix and look for all files needed using the same prefix, if all are uploaded you run the next step.

Both alternatives could be used in different scenarios.

@malachi-constant
Copy link
Contributor

Agreed that this is a valuable feature add. Let's groom and add to the roadmap.

@kukushking kukushking changed the title Support for multi source dependency (core) Support for multi source dependency Jun 13, 2022
@awspbade
Copy link
Author

awspbade commented Feb 7, 2023

is this feature going to be developed?

@malachi-constant
Copy link
Contributor

@awspbade I'm hoping to look into this soon. We are working on an upcoming major version beta and this will be addressed after all essential features and enhancements are in.

It will be an interesting ask though given that we handle all stage orchestration with event rules, maybe it warrants a statemachine or some orchestrator to handle logical conditions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants