Help exploring conditional execution in Dagster Graphs

[s000ne]

Overview

Hi there, I'm new to Dagster and would greatly appreciate any advice you can provide.

I'm currently exploring the implementation of a graph structure similar to the one depicted below, utilizing a single asset.

Here's what I'm aiming to accomplish:

• In cases where the input data from Upstream contains invalid entries, I'd like to have the option to execute a filtering operation.
• Due to the computational intensity of this filtering task, I'm interested in parallelizing it, assuming the use of Dynamic Graph.
• Regardless of whether the filtering operation is executed, I aim to perform a data type transformation and generate an output.

Do you have any insights or suggestions for achieving these objectives within the Dagster framework?

Notes

• I'm considering leveraging DynamicOutput to expedite the processing of some_heavy_process.
• While attempting to incorporate conditional branching, similar to With conditional branching, I'm uncertain about how to connect it to a single operation.

[alangenfeld]

Conditional branching is covered here:

https://docs.dagster.io/concepts/ops-jobs-graphs/graphs#with-conditional-branching

You can use a "fan-in" dependency to prevent a final op from skipping when one of its input branches skips:

https://docs.dagster.io/concepts/ops-jobs-graphs/graphs#with-a-fixed-fan-in

[s000ne]

@alangenfeld
Thank you for your response.
I was able to successfully resolve the issue using the method you suggested.

[shaygenhealth]

@alangenfeld is the documentation incorrect?

Docs for fan-in say: "The downstream op executes only if all of the outputs were successfully created by the upstream op:"
But you mentioned that a "fan-in" dependency should still run when one of its input branches skips.