API for working with labeled multidimensional data?

[aiqc]

Hi there,

Extremely excited about the potential of dagger and that the contributor base is growing.

In the long term, is there any plan for an API for working with labeled multi-dimensional data:
http://xarray.pydata.org/en/stable/getting-started-guide/why-xarray.html#core-data-structures

• User-defined names for columns and channels (higher dimensions).
• Attributes like shape, index, and dtype at each dimension.
• group_by across across dimensions.

It looks like these projects started but stopped:

• https://meggart.github.io/YAXArrays.jl/dev/#Package-Features
• https://discourse.julialang.org/t/implementing-an-equivalent-of-pythons-xarray-labeled-nd-arrays-in-julia/1741

[jpsamaroo]

Hey! I think the approach I'd like to take with Dagger is to have two different options for arrays:

1. Use a Dagger.DArray to wrap other arrays
2. Let packages implement their array functionality on top of Dagger or the DArray

For xarrays (which https://github.com/meggart/YAXArrays.jl seems to be the active Julia implemention; check the commit log), I think the latter option would probably make the most sense, since it seems like the xarray has some extra semantics and operations that regular AbstractArray subtypes don't implement. Option 1 is better for "simple" arrays like GPU arrays or various LinearAlgebra array types. There was discussion at JuliaIO/DiskArrays.jl#34 about integrating DiskArrays with Dagger, which I'd like to help support. Since DiskArrays is a dependency of YAXArrays, adding Dagger there should also provide features to YAXArrays.

[rafaqz]

DimensionalData.jl/GeoData.jl are equally active and more widely used than YAXarrays.jl.

But integrating DiskArrays.jl is the best option as its depended on by both GeoData.jl and YAXArrays.jl.

Mostly these wrapper arrays need to be the outside wrapper, as you say.