Conditional branching and fan-in for Ops whose upstream gives out dynamic outputs

[spandan-sharma]

This current bug in dagster is discussed in this thread: #7060 (comment).

Of-course it would be awesome if that was fixed as it's very limiting in what we can do with graph composability just now, but in the meantime I'm all ears to alternate solutions should anyone have any.

Briefly: Op op0 gives out dynamic output. The downstream op1 does conditional branching using Out and Output with say 2 branches yes and no (this part is pretty standard). The yes branch feeds into downstream op2_yes and the no branch to op2_no. Then op3 does a fan-in of op2_yes and op2_no. Then op4 is the final downstream which depends on op3 and that's it.

Since op0 gives out dynamic output, AFAIK i can only use the map and collect functions, but how do I introduce the conditional branching along the way? Because I couldn't figure that out, I separated op1, op2_yes, op2_no and op3 into another graph g1 and kept op0 and op4 in the main graph g0. Then simply hook it up like this:

# in g0:
dyn_op0 = op0()
op4(dyn_op0.map(g1).collect())

Now g1 can do whatever conditional branchings and fan-ins it wants internally, except it errors out due to the mentioned bug.

# in g1:
yes, no = op1()
# The following is the fan-in that errors out since input to g1 is dynamic.
# Without fan-in everything works.
return op3([op2_yes(yes), op2_no(no)])

Is there anyway to not introduce g1 (until the bug is fixed) and do everything in g0 assuming it helps avoid that bug? Is there any other alternative?