Diagram section builder

[mxgrey]

Currently diagrams support node and various builtin operations as building blocks to define a diagram.

Nodes are currently the only "reusable" unit for diagrams. A node builder can be defined once and then used across any number of diagrams. Nodes are quite versatile since they can contain entire workflows internally, but ultimately they can only represent a functional unit that takes one input and produces a set of streams and one final output.

In practical development, it is likely that users will want to be able to define an entire section of a diagram as a reusable unit. A section may be interfaced with via some number of inputs, some number of outputs, and some number of buffers. Internally a section can connect these inputs, outputs, and buffers according to whatever the section builder defines.

Ops schema

"ops": {
    "my_section": {
        "type": "section",
        "builder": "my_section_builder",
        "config": { ... },
        "connect": {
            "output1": "my_node",
            "output2": { "builtin": "terminate" }
        }
    },
    "my_node": {
        "type": "node",
        "builder": "my_node_builder",
        "next": { "section": "my_section", "input": "input1" }
    }
}

Section Builder

The Rust API for registering a section "builder" would strongly resemble the node builder API:

pub fn register_section_builder<Config, SectionImpl>(
    &mut self,
    options: SectionBuilderOptions,
    builder: impl FnMut(&mut Builder, Config) -> SectionImpl + 'static,
)
where
    Config: JsomSchema + DeserializeOwned,
    SectionImpl: Section,
{
    ...
}

We would define a trait Section which can be derived for structs whose fields are all section elements, e.g.:

#[derive(Section)]
struct MySection {
    /// This is the main thing to input data to
    main_input: InputSlot<Dog>,
    other_input: InputSlot<Cat>,
    some_stream_output: Output<Bark>,
    final_output: Output<Fur>,
    some_buffer: Buffer<Treats>,
}

The Section derive macro would iterate through the fields and collect the inputs, outputs, and buffers accordingly. This would be able to generate a schema for the struct based on its fields. The field name in the struct will be used as the key name for the section. I think Vec<_> fields should not be supported, at least for now, because otherwise we'd need to come up with a way for the diagram schema to select specific elements of a collection.

Section Template

We should also be able to define reusable section templates inside the serialized diagram format:

{
    "start": { "section": "my_section", "input": "input1" },
    "ops": {
        "my_section": {
            "type": "section",
            "template": "my_section_template",
            "connections": {
                "output1": "foo",
                "output2": { "builin": "terminate" }
            }
        }
    },
    "templates": {
        "my_section_template": {
            "inputs": [
                "input1",
                "input2"
            ],
            "buffers": [
                "buffer1",
                "buffer2"
            ],
            "outputs": [
                "output1",
                "output2"
            ],
            "ops": {
                "input1": {
                    "type": "node",
                    "builder": "my_node_builder",
                    "next": "my_fork"
                },
                "my_fork": {
                    "type": "fork_result",
                    "ok": "input2",
                    "err": "output1"
                },
                "input2": {
                    "type": "node",
                    "builder": "other_node_builder",
                    "next": "output2"
                },
                "buffer1": { "type": "buffer" },
                "buffer2": { "type": "buffer" }
            }
        }
    }
}

In the templates dictionary of the diagram file, users can define named section templates which they can then refer to in the diagram's ops section. The template definition contains:

• inputs - which named operations within the template can a user directly input a value into
  • Every name in inputs must have a corresponding named operation in the template's ops or else we produce a diagram error
• outputs - which outputs from the template can a user receive a message from
  • Every name in outputs must be the target of some operation in the template's ops or else we produce a diagram error
• buffers - which buffers inside the template can a user interact with
  • Every name in buffers must match a buffer inside the template's ops

It should also be possible to use sections (either from builders or from other templates) within templates:

"templates": {
    "my_outer_template": {
        "inputs": [
            {
                "name": "main_input",
                "section": "inner",
                "input": "inner_input"
            }
        ],
        "outputs": [
            "main_output"
        ],
        "ops": {
            "inner": {
                "type": "section",
                "builder": "my_section_builder",
                "connections": {
                    "inner_output": "main_output"
                }
            }
        }
    }
}

Section Builder vs Template

The key difference between section builder vs template is how the underlying section is defined.

• A section builder can be any function that produces a valid section interface (some set of inputs, outputs, and buffers)
  • Section builders can take in a json config object whose schema is decided by the builder
  • The implementation details of the section generated by a builder is opaque to the diagram using it, so the final workflow could look different depending on runtime conditions or depending on the config used to build the section
• A section template is embedded inside the diagram file
  • The definition of any template used by a diagram travels with the overall diagram file, so its implementation is transparent.
  • The workflow produced by a template is fixed within the diagram data. It cannot be modified by a configuration or altered during runtime except by changing the template contents itself.

The key similarity between section builders and templates is that both are used the same way within the ops of a diagram.