You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Some common scenarios needs to wait for more than one file to go for the next step in the pipeline, common examples are joining different tables to enrich the data
I think there're two ways to make this.
Make WaitingStage X time to let all files are ready. Simpler method I guess.
I read some forums that propose: a manifest file as an input of the process that needs N files, also to control a full cycle of execution all files needs a prefix per cycle. Every time you receive a new file you check the prefix and look for all files needed using the same prefix, if all are uploaded you run the next step.
Both alternatives could be used in different scenarios.
The text was updated successfully, but these errors were encountered:
@awspbade I'm hoping to look into this soon. We are working on an upcoming major version beta and this will be addressed after all essential features and enhancements are in.
It will be an interesting ask though given that we handle all stage orchestration with event rules, maybe it warrants a statemachine or some orchestrator to handle logical conditions.
Some common scenarios needs to wait for more than one file to go for the next step in the pipeline, common examples are joining different tables to enrich the data
I think there're two ways to make this.
Both alternatives could be used in different scenarios.
The text was updated successfully, but these errors were encountered: