Xarray-Beam is a Python library for building Apache Beam pipelines with Xarray datasets.
The project aims to facilitate data transformations and analysis on large-scale multi-dimensional labeled arrays, such as:
- Ad-hoc computation on Xarray data, by dividing a
xarray.Datasetinto many smaller pieces ("chunks"). - Adjusting array chunks, using the Rechunker algorithm.
- Ingesting large, multi-dimensional array datasets into an analysis-ready, cloud-optimized format, namely Zarr (see also Pangeo Forge).
- Calculating statistics (e.g., "climatology") across distributed datasets with arbitrary groups.
For more about our approach and how to get started, read the documentation!
🚨 Warning: Xarray-Beam is new and unpolished 🚨
Expect sharp edges 🔪 and performance cliffs 🧗, particularly related to the management of lazy data with Dask and reading/writing data with Zarr. We have used it to efficiently process ~25 TB datasets. We expect it to scale to PB size datasets, but that's easier said than done. We welcome feedback and contributions from early adopters, and hope to have it ready for wider audience soon.
Xarray-Beam requires recent versions of immutabledict, xarray, dask, rechunker and zarr, and the latest release of Apache Beam (2.31.0 or later). For best performance when writing Zarr files, use Xarray 0.19.0 or later.
Xarray-Beam is an experiment that we are sharing with the outside world in the hope that it will be useful. It is not a supported Google product. We welcome feedback, bug reports and code contributions, but cannot guarantee they will be addressed.
See the "Contribution guidelines" for more.
Contributors:
- Stephan Hoyer
- Jason Hickey
- Cenk Gazen
- Alex Merose