Feedback on NamedArray

[dcherian]

A place to collect community feedback on the NamedArray design doc

We propose the creation of a standalone Python package, "named-array". This package is envisioned to be a version of the xarray.Variable data structure, cleanly separated from the heavier dependencies of Xarray. named-array will provide a lightweight, user-friendly array-like data structure with named dimensions, facilitating convenient indexing and broadcasting. The package will use existing scientific Python community standards such as established array protocols and the new Python array API standard, allowing users to wrap multiple duck-array objects, including, but not limited to, NumPy, Dask, Sparse, Pint, CuPy, and Pytorch.

[NowanIlfideme]

Amazing - I'm very happy that this is starting to happen. It always felt a bit "off" how xarray was implementing different duck array backends. Having this be a usable middle-layer will be quite amazing, especially for use in probabilistic programming + pytorch.

Regarding attrs metadata, will this be a str -> str mapping or str -> pyobject? If the second, I assume it won't provide any guarantees, besides possibly application-specific things.

[dcherian]

Glad to see your excitement @NowanIlfideme . attrs will be a MutableMapping as in DataArray and Variable at present

[mdsumner]

is there consideration of non-explicit coordinate "labelling"? i.e just an extent of a dimension (min, max edge to edge) and not degenerate rectilinear coord arrays and their various entropy-issues. It could be a transform per dimension (offset,scale,[shear relative to other dim/s) which generalizes to nd, or just a special case extent for always regular grids. 🙏

[benbovy]

@mdsumner IIUC this is rather something that we could currently address at the level of Xarray flexible indexes with "functional indexes" (e.g., #8004 and examples in #7041). The Xarray data model only requires that every index in a Dataset / DataArray must relate to at least one coordinate. For the rest things are now very flexible, i.e., an index may relate to a set of scalar or lazy ("virtual") coordinates, with arbitrary dimension(s) and with parameters like scale, offset... as coordinate metadata.

[mdsumner]

thanks so much I will explore, appreciate the explanation I'm a bit lost 👌

[mathause]

Are there considerations to allow more than one dimension with the same name? E.g. for a variance covariance matrix?

[benbovy]

Isn't it already supported?

var = xr.Variable(("x", "x"), [[0, 1], [2, 3]])

var.dims
# ('x', 'x')

ds = xr.Dataset({"foo": var})
# <xarray.Dataset>
# Dimensions:  (x: 2)
# Dimensions without coordinates: x
# Data variables:
#     foo      (x, x) int64 0 1 2 3

ds.foo
# <xarray.DataArray 'foo' (x: 2)>
# array([[0, 1],
#        [2, 3]])
# Dimensions without coordinates: x, x

[benbovy]

The dimension part in the ds.foo DataArray repr is a bit confusing, though (it reads as a 1-dimensional array).

[mathause]

Thanks, I thought this was not working (turns out it's just not practical to use, I had #1378 in mind), but in this case you can ignore my suggestion.

[benbovy]

Ah I see thanks for pointing to #1378.

[SimonHeybrock]

Have there been any discussions about (not) including attrs? After several discussions we (Scipp) have concluded that attrs are kind of harmful on a low level and we decided to not support it any more. Since NamedArray is intended as a low-level building block for other libraries I wonder if this should be considered here?

See our summary: https://scipp.github.io/reference/developer/adr/0016-do-not-support-attrs.html

I think this was obviously a controversial decision since it impacts our users, and we are not at the point yet where I could claim that we have completed a successful transition.

[dcherian]

attrs will be included. At the moment, we are going with always preserving attrs without any configuration allowed.

[benbovy]

A recent message on the Xarray mailing list about using Xarray as a low level library makes me wonder: would it be worth exposing somewhere a simpler version of open_dataset() that would basically return a (plain) dictionary of NamedArray objects? I.e., being able to load and wrap data from a given file or store without all the extra coordinate decoding and index stuff... I haven't seen such proposal in the design docs but maybe I've overlooked it.

I don't think it would be easy moving all I/O backend logic along with NamedArray if eventually it gets factored out in a separate package. Would it be easier to refactor that into another namedarray-io package? I have no idea :-).

[vnmabus]

I already mentioned it in data-apis/array-api#698 (comment), but what I think that would be nice to have is a NamedArray (or more generally a MetadataArray?) that wraps arrays following the array API standard and its fully-compliant with the array API standard, adding named dimensions/coordinates (and maybe additional metadata) on top. Apparently, this will not be the case for the proposal here, because it will broadcast by dimension name (which I think is a mistake for many use cases).