Use Elyra for kubeflow notebook pipelines

[delgadom]

This is ridiculous: https://medium.com/codait/creating-notebook-pipelines-using-elyra-and-kubeflow-pipelines-f9606449cc53

https://github.com/elyra-ai/elyra

[brews]

Could we consider kubeflow-pipelines? Does that get us the features we want or do we want all of elyra?

Edit: clarifying

Edit2: I don't think we get the notebook pipeline UI extension with kubeflow-pipeline without installing all of kubeflow. So, elyra is needed if we want to shortcut this.

[bolliger32]

whoa this looks like a pretty cool project. @brews I don't know much about kubeflow-pipelines, but I would imagine one of the key features of this elyra thing is the fact that it uses notebooks natively, so that would mesh with (I think) a lot of different teams' workflows. I can certainly imagine this being a useful tool for organizing our coastal pipelines. Might load it onto the coastal image and start playing around with it to check it out. It wasn't totally obvious to me whether it would be easy to just plug in our GCS bucket or not. At the very least, it would be helpful for us to have a visual flowchart of notebooks even if we never actually executed them through this tool (tho obviously there are easier ways to make a flowchart lol)