0.13.0.dev12

Changes in 0.13.0.dev12

Fixes

• Intermediate: Fixed Viewer.show() and Viewer.info().

Full Changelog: v0.13.0.dev11...v0.13.0.dev12

0.13.0.dev11

Changes in 0.13.0.dev11

Enhancements

• Added Notebook
  xcube-viewer-in-jl.ipynb
  that explains how xcube Viewer can now be utilised in JupyterLab
  using the new (still experimental) xcube JupyterLab extension
  xcube-jl-ext.

Fixes

• Intermediate: Ensure Viewer() creates a server with reverse prefix set.

• Intermediate: Ensure Viewer.add_dataset() provides a dataset title.

• Intermediate: Fixed xcube.webapi.viewer.Viewer
  so it can find ~/.xcube/jupyterlab/lab-info.json.

Other

• Intermediate: Added important TODO, should be made a GH issue after 0.13.

• Removed deprecated example examples/tile.

Full Changelog: v0.13.0.dev10...v0.13.0.dev11

0.13.0.dev10

What's Changed

• Replaced numpy.bool by Python bool by @forman in #796

Full Changelog: v0.13.0.dev9...v0.13.0.dev10

0.13.0.dev9

Changes in 0.13.0.dev9

Enhancements

• xcube Server has been rewritten almost from scratch.

  • Introduced a new endpoint ${server_url}/s3 that emulates
    and AWS S3 object storage for the published datasets. (#717)
    The bucket name can be either:

    • s3://datasets - publishes all datasets in Zarr format.
    • s3://pyramids - publishes all datasets in a multi-level levels
      format (multi-resolution N-D images)
      that comprises level datasets in Zarr format.

    Datasets published through the S3 API are slightly
    renamed for clarity. For bucket s3://pyramids:

    • if a dataset identifier has suffix .levels, the identifier remains;
    • if a dataset identifier has suffix .zarr, it will be replaced by
      .levels only if such a dataset doesn't exist;
    • otherwise, the suffix .levels is appended to the identifier.
      For bucket s3://datasets the opposite is true:
    • if a dataset identifier has suffix .zarr, the identifier remains;
    • if a dataset identifier has suffix .levels, it will be replaced by
      .zarr only if such a dataset doesn't exist;
    • otherwise, the suffix .zarr is appended to the identifier.

    With the new S3 endpoints in place, xcube Server instances can be used
    as xcube data stores as follows:

    store = new_data_store(
        "s3", 
        root="datasets",   # bucket "datasets", use also "pyramids"
        max_depth=2,       # optional, but we may have nested datasets
        storage_options=dict(
            anon=True,
            client_kwargs=dict(
                endpoint_url='http://localhost:8080/s3' 
            )
        )
    )

  • The limited s3bucket endpoints are no longer available and are
    replaced by s3 endpoints.

  • Added new endpoint /viewer that serves a self-contained,
    packaged build of
    xcube Viewer.
    The packaged viewer can be overridden by environment variable
    XCUBE_VIEWER_PATH that must point to a directory with a
    build of a compatible viewer.

  • The --show option of xcube serve
    has been renamed to --open-viewer.
    It now uses the self-contained, packaged build of
    xcube Viewer. (#750)

  • The --show option of xcube serve
    now outputs various aspects of the server configuration.

  • Added experimental endpoint /volumes.
    It is used by xcube Viewer to render 3-D volumes.

• xcube Server is now more tolerant with respect to datasets it can not
  open without errors. Implementation detail: It no longer fails if
  opening datasets raises any exception other than DatasetIsNotACubeError.
  (#789)

• xcube Server's colormap management has been improved in several ways:

  • Colormaps are no longer managed globally. E.g., on server configuration
    change, new custom colormaps are reloaded from files.
  • Colormaps are loaded dynamically from underlying
    matplotlib and cmocean registries, and custom SNAP color palette files.
    That means, latest matplotlib colormaps are now always available. (#687)
  • Colormaps can now be reversed (name suffix "_r"),
    can have alpha blending (name suffix "_alpha"),
    or both (name suffix "_r_alpha").
  • Loading of custom colormaps from SNAP *.cpd has been rewritten.
    Now also the isLogScaled property of the colormap is recognized. (#661)
  • The module xcube.util.cmaps has been redesigned and now offers
    three new classes for colormap management:
    • Colormap - a colormap
    • ColormapCategory - represents a colormap category
    • ColormapRegistry - manages colormaps and their categories

• The xcube filesystem data stores such as "file", "s3", "memory"
  can now filter the data identifiers reported by get_data_ids(). (#585)
  For this purpose, the data stores now accept two new optional keywords
  which both can take the form of a wildcard pattern or a sequence
  of wildcard patterns:

  1. excludes: if given and if any pattern matches the identifier,
     the identifier is not reported.
  2. includes: if not given or if any pattern matches the identifier,
     the identifier is reported.

• Added convenience method DataStore.list_data_ids() that works
  like get_data_ids(), but returns a list instead of an iterator. (#776)

Fixes

• xcube CLI tools no longer emit warnings when trying to import
  installed packages named xcube_* as xcube plugins.

• The xcube.util.timeindex module can now handle 0-dimensional
  ndarrays as indexers. This effectively avoids the warning
  Can't determine indexer timezone; leaving it unmodified.
  which was emitted in such cases.

• xcube serve will now also accept datasets with coordinate names
  longitude and latitude, even if the attribute long_name isn't set.
  (#763)

• Function xcube.core.resampling.affine.affine_transform_dataset()
  now assumes that geographic coordinate systems are equal by default and
  hence a resampling based on an affine transformation can be performed.

• Fixed a problem with xcube server's WMTS implementation.
  For multi-level resolution datasets with very coarse low resolution levels,
  the tile matrix sets WorldCRS84Quad and WorldWebMercatorQuad have
  reported a negative minimum z-level.

• Implementation of function xcube.core.geom.rasterize_features()
  has been changed to account for consistent use of a target variable's
  fill_value and dtype for a given feature.
  In-memory (decoded) variables now always use dtype float64 and use
  np.nan to represent missing values. Persisted (encoded) variable data
  will make use of the target fill_value and dtype. (#778)

• Relative local filesystem paths to datasets are now correctly resolved
  against the base directory of the xcube Server's configuration, i.e.
  configuration parameter base_dir. (#758)

• Fixed problem with xcube gen raising FileNotFoundError
  with Zarr >= 2.13.

• Provided backward compatibility with Python 3.8. (#760)

Other

• The CLI tool xcube edit has been deprecated in favour of the
  xcube patch. (#748)

• Deprecated CLI xcube tile has been removed.

• Deprecated modules, classes, methods, and functions
  have finally been removed:

  • xcube.core.geom.get_geometry_mask()
  • xcube.core.mldataset.FileStorageMultiLevelDataset
  • xcube.core.mldataset.open_ml_dataset()
  • xcube.core.mldataset.open_ml_dataset_from_local_fs()
  • xcube.core.mldataset.open_ml_dataset_from_object_storage()
  • xcube.core.subsampling.get_dataset_subsampling_slices()
  • xcube.core.tiledimage
  • xcube.core.tilegrid

• The following classes, methods, and functions have been deprecated:

  • xcube.core.xarray.DatasetAccessor.levels()
  • xcube.util.cmaps.get_cmap()
  • xcube.util.cmaps.get_cmaps()

• A new function compute_tiles() has been
  refactored out from function xcube.core.tile.compute_rgba_tile().

• Added method get_level_for_resolution(xy_res) to
  abstract base class xcube.core.mldataset.MultiLevelDataset.

• Removed outdated example resources from examples/serve/demo.

• Account for different spatial resolutions in x and y in
  xcube.core.geom.get_dataset_bounds().

• Make code robust against 0-size coordinates in
  xcube.core.update._update_dataset_attrs().

• xcube Server has been enhanced to load multi-module Python code
  for dynamic cubes both from both directories and zip archives.
  For example, the following dataset definition computes a dynamic
  cube from dataset "local" using function "compute_dataset" in
  Python module "resample_in_time.py":

    Path: resample_in_time.py
    Function: compute_dataset
    InputDatasets: ["local"]

  Users can now pack "resample_in_time.py" among any other modules and
  packages into a zip archive. Note that the original module name
  is now a prefix to the function name:

    Path: modules.zip
    Function: resample_in_time:compute_dataset
    InputDatasets: ["local"]

  Implementation note: this has been achieved by using
  xcube.core.byoa.CodeConfig in
  xcube.core.mldataset.ComputedMultiLevelDataset.

• Instead of the Function keyword it is now
  possible to use the Class keyword.
  While Function references a function that receives one or
  more datasets (type xarray.Dataset) and returns a new one,
  Class references a callable that receives one or
  more multi-level datasets and returns a new one.
  The callable is either a class derived from
  or a function that returns an instance of
  xcube.core.mldataset.MultiLevelDataset.

• Module xcube.core.mldataset has been refactored into
  a sub-package for clarity and maintainability.

Full Changelog: v0.13.0.dev8...v0.13.0.dev9

0.13.0.dev8

Changes in 0.13.0.dev8

Other

• Added convenience method DataStore.list_data_ids() that works
  like get_data_ids(), but returns a list instead of an iterator. (#776)

Fixes

• Implementation of function xcube.core.geom.rasterize_features()
  has been changed to account for consistent use of a target variable's
  fill_value and dtype for a given feature.
  In-memory (decoded) variables now always use dtype float64 and use
  np.nan to represent missing values. Persisted (encoded) variable data
  will make use of the target fill_value and dtype. (#778)

Full Changelog: v0.13.0.dev7...v0.13.0.dev8

0.13.0.dev7

Changes in 0.13.0.dev7

Other

• xcube Server has been enhanced to load multi-module Python code
  for dynamic cubes both from both directories and zip archives.
  For example, the following dataset definition computes a dynamic
  cube from dataset "local" using function "compute_dataset" in
  Python module "resample_in_time.py":

    Path: resample_in_time.py
    Function: compute_dataset
    InputDatasets: ["local"]

  Users can now pack "resample_in_time.py" among any other modules and
  packages into a zip archive. Note that the original module name
  is now a prefix to the function name:

    Path: modules.zip
    Function: resample_in_time:compute_dataset
    InputDatasets: ["local"]

  Implementation note: this has been achieved by using
  xcube.core.byoa.CodeConfig in
  xcube.core.mldataset.ComputedMultiLevelDataset.

• Instead of the Function keyword it is now
  possible to use the Class keyword.
  While Function references a function that receives one or
  more datasets (type xarray.Dataset) and returns a new one,
  Class references a callable that receives one or
  more multi-level datasets and returns a new one.
  The callable is either a class derived from
  or a function that returns an instance of
  xcube.core.mldataset.MultiLevelDataset.

• Module xcube.core.mldataset has been refactored into
  a sub-package for clarity and maintainability.

• Provided backward compatibility with Python 3.8. (#760)

0.13.0.dev6

Changes in 0.13.0.dev6

Changes in 0.13.0.dev5

Other

• xcube server Python scripts can now import modules from
  the script's directory.
• Loading of dynamic cubes is now logged.

0.13.0.dev5

Changes in 0.13.0.dev5

Intermediate changes

• xcube server Python scripts can now import modules from
  the script's directory.
• Loading of dynamic cubes is now logged.

Changes in 0.13.0.dev4

• xcube serve correctly resolves relative paths to datasets (#758)

Changes in 0.13.0.dev3

• A new function compute_tiles() has been
  refactored out from function xcube.core.tile.compute_rgba_tile().
• Added method get_level_for_resolution(xy_res) to
  abstract base class xcube.core.mldataset.MultiLevelDataset.
• Removed outdated example resources from examples/serve/demo.
• Account for different spatial resolutions in x and y in
  xcube.core.geom.get_dataset_bounds().
• Make code robust against 0-size coordinates in
  xcube.core.update._update_dataset_attrs().

Changes in 0.13.0.dev2

Intermediate changes

• Fixed unit test w.r.t. change in 0.13.0.dev1

• xcube now tries to prevent indexing timezone-naive variables with
  timezone-aware indexers, or vice versa.

Changes in 0.13.0.dev1

Intermediate changes

• Include package data xcube/webapi/meta/res/openapi.html.

Changes in 0.13.0.dev0

Enhancements

• xcube Server has been rewritten almost from scratch.

  • Introduced a new endpoint ${server_url}/s3 that emulates
    and AWS S3 object storage for the published datasets. (#717)
    The bucket name can be either:

    • s3://datasets - publishes all datasets in Zarr format.
    • s3://pyramids - publishes all datasets in a multi-level levels
      format (multi-resolution N-D images)
      that comprises level datasets in Zarr format.

    Datasets published through the S3 API are slightly
    renamed for clarity. For bucket s3://pyramids:

    • if a dataset identifier has suffix .levels, the identifier remains;
    • if a dataset identifier has suffix .zarr, it will be replaced by
      .levels only if such a dataset doesn't exist;
    • otherwise, the suffix .levels is appended to the identifier.
      For bucket s3://datasets the opposite is true:
    • if a dataset identifier has suffix .zarr, the identifier remains;
    • if a dataset identifier has suffix .levels, it will be replaced by
      .zarr only if such a dataset doesn't exist;
    • otherwise, the suffix .zarr is appended to the identifier.

    With the new S3 endpoints in place, xcube Server instances can be used
    as xcube data stores as follows:

    store = new_data_store(
        "s3", 
        root="datasets",   # bucket "datasets", use also "pyramids"
        max_depth=2,       # optional, but we may have nested datasets
        storage_options=dict(
            anon=True,
            client_kwargs=dict(
                endpoint_url='http://localhost:8080/s3' 
            )
        )
    )

  • The limited s3bucket endpoints are no longer available and are
    replaced by s3 endpoints.

  • The --show option of xcube serve is no longer available. (#750)
    We may reintroduce it, but then with a packaged build of
    xcube Viewer that matches the current xcube Server version.

• xcube Server's colormap management has been improved in several ways:

  • Colormaps are no longer managed globally. E.g., on server configuration
    change, new custom colormaps are reloaded from files.
  • Colormaps are loaded dynamically from underlying
    matplotlib and cmocean registries, and custom SNAP color palette files.
    That means, latest matplotlib colormaps are now always available. (#687)
  • Colormaps can now be reversed (name suffix "_r"),
    can have alpha blending (name suffix "_alpha"),
    or both (name suffix "_r_alpha").
  • Loading of custom colormaps from SNAP *.cpd has been rewritten.
    Now also the isLogScaled property of the colormap is recognized. (#661)
  • The module xcube.util.cmaps has been redesigned and now offers
    three new classes for colormap management:
    • Colormap - a colormap
    • ColormapCategory - represents a colormap category
    • ColormapRegistry - manages colormaps and their categories

Other

• Deprecated CLI xcube tile has been removed.

• Deprecated modules, classes, methods, and functions
  have finally been removed:

  • xcube.core.geom.get_geometry_mask()
  • xcube.core.mldataset.FileStorageMultiLevelDataset
  • xcube.core.mldataset.open_ml_dataset()
  • xcube.core.mldataset.open_ml_dataset_from_local_fs()
  • xcube.core.mldataset.open_ml_dataset_from_object_storage()
  • xcube.core.subsampling.get_dataset_subsampling_slices()
  • xcube.core.tiledimage
  • xcube.core.tilegrid

• The following classes, methods, and functions have been deprecated:

  • xcube.core.xarray.DatasetAccessor.levels()
  • xcube.util.cmaps.get_cmap()
  • xcube.util.cmaps.get_cmaps()

• Fixed problem with xcube gen raising FileNotFoundError
  with Zarr >= 2.13.

Full Changelog: v0.13.0.dev4...v0.13.0.dev5

0.13.0.dev4

Changes in 0.13.0.dev4

Fixes

• xcube server Python scripts can now import modules from
  the script's directory.
• xcube serve correctly resolves relative paths to datasets (#758)

0.13.0.dev3

Changes in 0.13.0.dev3

Other

• A new function compute_tiles() has been
  refactored out from function xcube.core.tile.compute_rgba_tile().
• Added method get_level_for_resolution(xy_res) to
  abstract base class xcube.core.mldataset.MultiLevelDataset.
• Removed outdated example resources from examples/serve/demo.
• Account for different spatial resolutions in x and y in
  xcube.core.geom.get_dataset_bounds().
• Make code robust against 0-size coordinates in
  xcube.core.update._update_dataset_attrs().