0.11.0.dev2

Changes in 0.11.0 (in development)

Enhancements

• xcube serve can now serve datasets with arbitrary spatial
  coordinate reference systems. Before xcube 0.11, datasets where forced
  to have a geographical CRS such as EPSG:4326 or CRS84.

• xcube serve can now provide image tiles for two popular tile grids:

  1. global geographic grid, with 2 x 1 tiles at level zero (the default);
  2. global web mercator grid, with 1 x 1 tiles at level
     zero ("Google projection", OSM tile grid).

  The general form of the xcube tile URL is currently

  /datasets/{ds_id}/vars/{var_name}/tile2/{z}/{y}/{x}
  

  The following query parameters can be used

  • crs: set to CRS84 to use the geographical grid (the default),
    or EPSG:3857 to use the web mercator grid.
  • cbar: color bar name such as viridis or plasma,
    see color bar names of matplotlib. Defaults to bone.
  • vmin: minimum value to be used for color mapping. Defaults to 0.
  • vmax: maximum value to be used for color mapping. Defaults to 1.
  • retina: if set to 1, tile size will be 512 instead of 256.

• The WMTS provided by xcube serve has been reimplemented from scratch.
  It now provides two common tile matrix sets:

  1. WorldCRS84Quad global geographic grid, with 2 x 1 tiles at level zero;
  2. WorldWebMercatorQuad global web mercator grid, with 1 x 1 tiles
     at level zero.

  New RESTful endpoints have been added to reflect this:

  /wmts/1.0.0/{TileMatrixSet}/WMTSCapabilities.xml
  /wmts/1.0.0/tile/{Dataset}/{Variable}/{TileMatrixSet}/{TileMatrix}/{TileRow}/{TileCol}.png
  

  The existing RESTful endpoints now use tile matrix set WorldCRS84Quad by default:

  /wmts/1.0.0/WMTSCapabilities.xml
  /wmts/1.0.0/tile/{Dataset}/{Variable}/{TileMatrix}/{TileRow}/{TileCol}.png
  

  The key-value pair (KVP) endpoint /wmts/kvp now recognises the
  TileMatrixSet key for the two values described above.

• Support for multi-level datasets aka ND image pyramids has been
  further improved (#655):

  • Introduced new parameter agg_methods for writing multi-level datasets
    with the "file", "s3", and "memory" data stores.
    The value of agg_methods is either a string "first",
    "min", "max", "mean", "median" or a dictionary that maps
    a variable name to an aggregation method. Variable names can be patterns
    that may contain wildcard characters '*' and '?'. The special aggregation
    method "auto" can be used to select "first" for integer variables
    and "mean" for floating point variables.
  • The xcube level CLI tool now has a new option --agg-methods (or -A)
    for the same purpose.

• The xcube package now consistently makes use of logging.
  We distinguish general logging and specific xcube logging.
  General logging refers to the log messages emitted by any Python module
  while xcube logging only refers to log messages emitted by xcube modules.

  • The output of general logging from xcube CLI tools can now be
    configured with two new CLI options:

    • --loglevel LEVEL: Can be one of CRITICAL, ERROR,
      WARNING, INFO, DETAIL, DEBUG, TRACE, or OFF (the default).
    • --logfile PATH: Effective only if log level is not OFF.
      If given, log messages will be written into the file
      given by PATH. If omitted, log messages will be redirected
      to standard error (sys.stderr).

    The output of general logging from xcube CLI is disabled by default.
    If enabled, the log message format includes the level, date-time,
    logger name, and message.

  • All xcube modules use the logger named xcube
    (i.e. LOG = logging.getLogger("xcube")) to emit
    messages regarding progress, debugging, errors. Packages that extend
    the xcube package should use a dot suffix for their logger names, e.g.
    xcube.cci for the xcube plugin package xcube-cci.

  • All xcube CLI tools will output log messages, if any,
    on standard error (sys.stderr).
    Only the actual result, if any,
    is written to standard out (sys.stdout).

  • Some xcube CLI tools have a --quiet/-q option to disable output
    of log messages on the console and a --verbose/-v option to enable
    it and control the log level. For this purpose the option -v
    can be given multiple times and even be combined: -v = INFO,
    -vv = DETAIL, -vvv = DEBUG, -vvvv = TRACE.
    The quiet and verbose settings only affect the logger named xcube
    and its children.
    If enabled, a simple message format will be used, unless the general
    logging is redirected to stdout.

Fixes

• Fixed a problem where the DataStores configuration of xcube serve
  did not recognize multi-level datasets. (#653)

• Opening of multi-level datasets with filesystem data stores now
  recognizes the cache_size open parameter.

Other changes

• The xcube tile CLI tool has been deprecated. A new tool is planned that can work
  concurrently on dask clusters and also supports common tile grids such as
  global geographic and web mercator.

• The xcube.util.tiledimage module has been deprecated and is no longer
  used in xcube. It has no replacement.

• The xcube.util.tilegrid module has been deprecated and is no longer
  used in xcube.
  A new implementation is provided by xcube.core.tilingscheme
  which is used instead.

• All existing functions of the xcube.core.tile module have been
  deprecated and are no longer used in xcube. A newly exported function
  is xcube.core.tile.compute_rgba_tile() which is used in place of
  other tile generating functions.

0.11.0.dev1

Changes in 0.11.0 (in development)

Enhancements

• xcube serve can now serve datasets with arbitrary spatial
  coordinate reference systems. Before xcube 0.11, datasets where forced
  to have a geographical CRS such as EPSG:4326 or CRS84.

• xcube serve can now provide image tiles for two popular tile grids:

  1. global geographic grid, with 2 x 1 tiles at level zero (the default);
  2. global web mercator grid , with 1 x 1 tiles at level
     zero ("Google projection", OSM tile grid).

  The general form of the xcube tile URL is

  /datasets/{ds_id}/vars/{var_name}/tile2/{z}/{y}/{x}
  

  The following query parameters can be used

  • crs: set to CRS84 to use the geographical grid (the default),
    or EPSG:3857 to use the web mercator grid.
  • cbar: color bar name such as viridis or plasma,
    see color bar names of matplotlib. Defaults to bone.
  • vmin: minimum value to be used for color mapping. Defaults to 0.
  • vmax: maximum value to be used for color mapping. Defaults to 1.
  • retina: if set to 1, tile size will be 512 instead of 256.

• The WMTS provided by xcube serve has been reimplemented from scratch.
  It now provides two common tile matrix sets:

  1. WorldCRS84Quad global geographic grid, with 2 x 1 tiles at level zero;
  2. WorldWebMercatorQuad global web mercator grid, with 1 x 1 tiles
     at level zero.

  New RESTful endpoints have been added to reflect this:

  /wmts/1.0.0/{TileMatrixSet}/WMTSCapabilities.xml
  /wmts/1.0.0/tile/{Dataset}/{Variable}/{TileMatrixSet}/{TileMatrix}/{TileRow}/{TileCol}.png
  

  The existing RESTful endpoints now use tile matrix set WorldCRS84Quad by default:

  /wmts/1.0.0/WMTSCapabilities.xml
  /wmts/1.0.0/tile/{Dataset}/{Variable}/{TileMatrix}/{TileRow}/{TileCol}.png
  

  The key-value pair (KVP) endpoint /wmts/kvp now recognises the
  TileMatrixSet key for the two values described above.

Other changes

• All components of the xcube.util.tiledimage module have been
  deprecated and are no longer used in xcube.

• TODO: All components of the xcube.util.tilegrid module have been
  deprecated and are no longer used in xcube. Entirely new implementations
  are provided in xcube.core.tilegrid which are used instead.

• TODO: All components of the xcube.core.tile module have been
  deprecated and are no longer used in xcube. Entirely new implementations
  are provided in xcube.core.tile2 which are used instead.
  In version xcube 0.12 these will renamed to xcube.core.tile.

0.11.0.dev0

Changes in 0.11.0 (in development)

Enhancements

• xcube serve can now serve datasets with arbitrary spatial
  coordinate reference systems. Before xcube 0.11, datasets where forced
  to have a geographical CRS such as EPSG:4326 or CRS84.

• xcube serve can now provide image tiles for two popular tile grids:

  1. global geographic grid, with 2 x 1 tiles at level zero (the default);
  2. global web mercator grid , with 1 x 1 tiles at level
     zero ("Google projection", OSM tile grid).

  The general form of the xcube tile URL is

  /datasets/{ds_id}/vars/{var_name}/tile2/{z}/{y}/{x}
  

  The following query parameters can be used

  • crs: set to CRS84 to use the geographical grid (the default),
    or EPSG:3857 to use the web mercator grid.
  • cbar: color bar name such as viridis or plasma,
    see color bar names of matplotlib. Defaults to bone.
  • vmin: minimum value to be used for color mapping. Defaults to 0.
  • vmax: maximum value to be used for color mapping. Defaults to 1.
  • retina: if set to 1, tile size will be 512 instead of 256.

• The WMTS provided by xcube serve has been reimplemented from scratch.
  It now provides two common tile matrix sets:

  1. WorldCRS84Quad global geographic grid, with 2 x 1 tiles at level zero;
  2. WorldWebMercatorQuad global web mercator grid, with 1 x 1 tiles
     at level zero.

  New RESTful endpoints have been added to reflect this:

  /wmts/1.0.0/{TileMatrixSet}/WMTSCapabilities.xml
  /wmts/1.0.0/tile/{Dataset}/{Variable}/{TileMatrixSet}/{TileMatrix}/{TileRow}/{TileCol}.png
  

  The existing RESTful endpoints now use tile matrix set WorldCRS84Quad by default:

  /wmts/1.0.0/WMTSCapabilities.xml
  /wmts/1.0.0/tile/{Dataset}/{Variable}/{TileMatrix}/{TileRow}/{TileCol}.png
  

  The key-value pair (KVP) endpoint /wmts/kvp now recognises the
  TileMatrixSet key for the two values described above.

Other changes

• All components of the xcube.util.tiledimage module have been
  deprecated and are no longer used in xcube.

• TODO: All components of the xcube.util.tilegrid module have been
  deprecated and are no longer used in xcube. Entirely new implementations
  are provided in xcube.core.tilegrid which are used instead.

• TODO: All components of the xcube.core.tile module have been
  deprecated and are no longer used in xcube. Entirely new implementations
  are provided in xcube.core.tile2 which are used instead.
  In version xcube 0.12 these will renamed to xcube.core.tile.

0.10.2

Changes in 0.10.2

Enhancements

• Added new module xcube.core.subsampling for function
  subsample_dataset(dataset, step) that is now used by default
  to generate the datasets level of multi-level datasets.

• Added new setting Authentication.IsRequired to the xcube serve
  configuration. If set to true, xcube Server will reject unauthorized
  dataset requests by returning HTTP code 401.

• For authorized clients, the xcube Web API provided by xcube serve
  now allows granted scopes to contain wildcard characters *, **,
  and ?. This is useful to give access to groups of datasets, e.g.
  the scope read:dataset:*/S2-*.zarr permits access to any Zarr
  dataset in a subdirectory of the configured data stores and
  whose name starts with "S2-". (#632)

• xcube serve used to shut down with an error message
  if it encountered datasets it could not open. New behaviour
  is to emit a warning and ignore such datasets. (#630)

• Introduced helper function add_spatial_ref()
  of package xcube.core.gridmapping.cfconv that allows
  adding a spatial coordinate reference system to an existing
  Zarr dataset. (#629)

• Support for multi-level datasets has been improved:

  • Introduced new parameters for writing multi-level datasets with the
    "file", "s3", and "memory" data stores (#617). They are
    • base_dataset_id: If given, the base dataset will be linked only
      with the value of base_dataset_id, instead of being copied as-is.
      This can save large amounts of storage space.
    • tile_size: If given, it forces the spatial dimensions to be
      chunked accordingly. tile_size can be a positive integer
      or a pair of positive integers.
    • num_levels: If given, restricts the number of resolution levels
      to the given value. Must be a positive integer to be effective.
  • Added a new example notebook
    5_multi_level_datasets.ipynb
    that demonstrates writing and opening multi-level datasets with the
    xcube filesystem data stores.
  • Specified xcube Multi-Resolution Datasets
    definition and format.

• xcube gen2 returns more expressive error messages.

Fixes

• Fixed problem where the dataset levels of multi-level datasets were
  written without spatial coordinate reference system. In fact,
  only spatial variables were written. (#646)

• Fixed problem where xcube Server instances that required
  user authentication published datasets and variables for
  unauthorised users.

• Fixed FsDataAccessor.write_data() implementations,
  which now always return the passed in data_id. (#623)

• Fixes an issue where some datasets seemed to be shifted in the
  y-(latitude-) direction and were misplaced on maps whose tiles
  are served by xcube serve. Images with ascending y-values are
  now tiled correctly. (#626)

Other

• The xcube level CLI tool has been rewritten from scratch to make use
  of xcube filesystem data stores. (#617)

• Deprecated numerous classes and functions around multi-level datasets.
  The non-deprecated functions and classes of xcube.core.mldataset should
  be used instead along with the xcube filesystem data stores for
  multi-level dataset i/o. (#516)

  • Deprecated all functions of the xcube.core.level module
    • compute_levels()
    • read_levels()
    • write_levels()
  • Deprecated numerous classes and functions of the xcube.core.mldataset
    module
    • FileStorageMultiLevelDataset
    • ObjectStorageMultiLevelDataset
    • open_ml_dataset()
    • open_ml_dataset_from_object_storage()
    • open_ml_dataset_from_local_fs()
    • write_levels()

• Added packages python-blosc and lz4 to the xcube Python environment
  for better support of Dask distributed and the Dask service
  Coiled.

• Replace the dependency on the rfc3339-validator PyPI package with a
  dependency on its recently created conda-forge package.

• Remove unneeded dependency on the no longer used strict-rfc3339 package.

Full Changelog: v0.10.1...v0.10.2

0.10.2.dev0

Changes in 0.10.2 (in development)

Enhancements

• xcube serve used to shut down with an error message
  if it encountered datasets it could not open. New behaviour
  is to emit a warning and ignore such datasets. (#630)

• Introduced parameter base_dataset_id for writing multi-level
  datasets with the "file", "s3", and "memory" data stores.
  If given, the base dataset will be linked only with the
  value of base_dataset_id, instead of being copied as-is.
  This can save large amounts of storage space. (#617)

Fixes

• Fixed FsDataAccessor.write_data() implementations,
  which now always return the passed in data_id. (#623)

• Fixes an issue where some datasets seemed to be shifted in the
  y-(latitude-) direction and were misplaced on maps whose tiles
  are served by xcube serve. Images with ascending y-values are
  now tiled correctly. (#626)

Other

• Replace the dependency on the rfc3339-validator PyPI package with a
  dependency on its recently created conda-forge package.

• Remove unneeded dependency on the no longer used strict-rfc3339 package.

0.10.1

Changes in 0.10.1

Fixes

• Deprecated argument xy_var_names in function GridMapping.from_dataset,
  thereby preventing a NotImplementedError. (#551)

Other Changes

• For compatibility, now also xcube.__version__ contains the xcube
  version number.

0.10.0

Changes in 0.10.0

Incompatible Changes

• The configuration DataStores for xcube serve changed in an
  incompatible way with xcube 0.9.x: The value of former Identifier
  must now be assigned to Path, which is a mandatory parameter.
  Path may contain wildcard characters **, *, ?.
  Identifier is now optional, the default is
  "${store_id}~${data_id}. If given, it should only be used to
  uniquely identify single datasets within a data store
  pointed to by Path. (#516)

Enhancements

• It is now possible to use environment variables in most
  xcube configuration files. Unix bash syntax is used, i.e.
  ${ENV_VAR_NAME} or $ENV_VAR_NAME. (#580)

  Supported tools include

  • xcube gen --config CONFIG
  • xcube gen2 --stores STORES_CONFIG --service SERVICE_CONFIG
  • xcube serve -c CONFIG

• Changed the xcube gen tool to extract metadata for pre-sorting inputs
  from other than NetCDF inputs, e.g. GeoTIFF.

• Optimized function xcube.core.geom.rasterize_features().
  It is now twice as fast while its memory usage dropped to the half. (#593)

Fixes

• xcube serve now also serves datasets that are located in
  subdirectories of filesystem-based data stores such as
  "file", "s3", "memory". (#579)

• xcube serve now accepts datasets whose spatial
  resolutions differ up to 1%. (#590)
  It also no longer rejects datasets with large dimension
  sizes. (Formerly, an integer-overflow occurred in size
  computation.)

• DatasetChunkCacheSize is now optional in xcube serve
  configuration. (Formerly, when omitted, the server crashed.)

• Fixed bug that would cause that requesting data ids on some s3 stores would
  fail with a confusing ValueError.

• Fixed that only last dataset of a directory listing was published via
  xcube serve when using the DataStores configuration with
  filesystem-based datastores such as "s3" or "file". (#576)

Other

• Pinned Python version to < 3.10 to avoid import errors caused by a
  third-party library.

• Values obs and local for the FileSystem parameter in xcube
  configuration files have been replaced by s3 and file, but are kept
  temporarily for the sake of backwards compatibility.

All Issues addressed

• Martin's gen support for geotiff inputs by @martin-boettcher in #568
• naming of secrets GH PAT in the GH actions workflow by @dzelge in #575
• Pinning openssl to fix failing unittest on mac by @AliceBalfanz in #578
• Use env vars in configs by @forman in #582
• Restrict python version by @TonioF in #584
• Retrieval of Data IDs fails on some s3 stores by @TonioF in #587
• Only last configured Identifier is published via xcube serve by @TonioF in #589
• now also serve nested datasets by @forman in #588
• Revise spatial x,y resolutions check by @forman in #591
• extract CLI: ensure CSV times are timezone-naive by @pont-us in #596
• Daskify geometry operations, Part 1 by @forman in #594
• Fixing tests for accessing data from xcube server via python api by @AliceBalfanz in #592
• Toniof 598 gridmapping test fail by @TonioF in #599
• Unify Dataset IO by @TonioF in #535
• Tile server: ensure timezone-naive time selector by @pont-us in #602
• Replaced file system values obs and local with s3 and file by @TonioF in #600
• Corrected retrieval of store params by @TonioF in #606
• adapt to zarr 2.11 by @TonioF in #607

New Contributors

• @martin-boettcher made their first contribution in #568

Full Changelog: v0.9.2...v0.10.0

0.9.2

Changes in 0.9.2

Fixes

• A xcube.core.store.fs.impl.FSDataStore no longer raises exceptions when
  root directories in data store configurations do not exist. Instead, they
  are created when data is written.

0.9.2.dev1

Changes in 0.9.2 (in development)

• Use xcube cci v0.9.2.dev0 in dockerfile

0.9.2.dev0

Changes in 0.9.2 (in development)

• Be more robust with respect to non-existing data store root directories.