integrate with dask-ctl

[keewis]

There are at least a few related packages that have significant overlap:

• NCAR-jobqueue
• dask-ctl

Since in particular dask-ctl seems to be a bit more mature, it would be good to see if we can rewrite this package to use dask-ctl underneath, and to upstream any features that are missing.

This would then become a collection of hpc configuration files, and maybe a thin wrapper that would translate names to paths and feed that to dask_ctl.lifecycle.create_cluster.

[keewis]

looking at dask/dask-jobqueue#544 it seems we took the easy way out: we print the scheduler address to stdout or into a file, and we don't really care if the scheduler is on the current machine (because in the most common workflow we're usually on a compute node anyways)

cc @jacobtomlinson for reference, but I will open a new issue on dask-ctl to see what the best way forward would be.

[jacobtomlinson]

Sounds great, let me know how I can help! For reference dask-jobqueue currently does not fully support dask-ctl for the reasons you mention about the scheduler being a subprocess of the Python code that created the cluster. But for cluster creation from a spec file you should be all good!