Skip to content

Commit

Permalink
Merge remote-tracking branch 'pytroll/main' into main
Browse files Browse the repository at this point in the history
  • Loading branch information
Dario Stelitano committed Jul 24, 2023
2 parents 9541b70 + f099cd3 commit 3369c93
Show file tree
Hide file tree
Showing 80 changed files with 7,865 additions and 1,951 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ jobs:
# may break the conda-forge libraries trying to use newer glibc versions
run: |
python -m pip install \
--index-url https://pypi.anaconda.org/scipy-wheels-nightly/simple/ \
--index-url https://pypi.anaconda.org/scientific-python-nightly-wheels/simple/ \
--trusted-host pypi.anaconda.org \
--no-deps --pre --upgrade \
matplotlib \
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/deploy-sdist.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ jobs:

- name: Publish package to PyPI
if: github.event.action == 'published'
uses: pypa/[email protected].6
uses: pypa/[email protected].8
with:
user: __token__
password: ${{ secrets.pypi_password }}
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ repos:
- id: bandit
args: [--ini, .bandit]
- repo: https://github.com/pre-commit/mirrors-mypy
rev: 'v1.2.0' # Use the sha / tag you want to point at
rev: 'v1.4.1' # Use the sha / tag you want to point at
hooks:
- id: mypy
additional_dependencies:
Expand Down
3 changes: 3 additions & 0 deletions AUTHORS.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ The following people have made contributions to this project:
- [Ulrik Egede (egede)](https://github.com/egede)
- [Joleen Feltz (joleenf)](https://github.com/joleenf)
- [Stephan Finkensieper (sfinkens)](https://github.com/sfinkens) - Deutscher Wetterdienst
- [Gionata Ghiggi (ghiggi)](https://github.com/ghiggi)
- [Andrea Grillini (AppLEaDaY)](https://github.com/AppLEaDaY)
- [Blanka Gvozdikova (gvozdikb)](https://github.com/gvozdikb)
- [Nina Håkansson (ninahakansson)](https://github.com/ninahakansson)
Expand All @@ -35,6 +36,7 @@ The following people have made contributions to this project:
- [Gerrit Holl (gerritholl)](https://github.com/gerritholl) - Deutscher Wetterdienst
- [David Hoese (djhoese)](https://github.com/djhoese)
- [Marc Honnorat (honnorat)](https://github.com/honnorat)
- [Lloyd Hughes (system123)](https://github.com/system123)
- [Mikhail Itkin (mitkin)](https://github.com/mitkin)
- [Tommy Jasmin (tommyjasmin)](https://github.com/tommyjasmin)
- [Jactry Zeng](https://github.com/jactry)
Expand Down Expand Up @@ -85,3 +87,4 @@ The following people have made contributions to this project:
- [praerien (praerien)](https://github.com/praerien)
- [Xin Zhang (zxdawn)](https://github.com/zxdawn)
- [Yufei Zhu (yufeizhu600)](https://github.com/yufeizhu600)
- [Youva Aoun (YouvaEUMex)](https://github.com/YouvaEUMex)
64 changes: 64 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,67 @@
## Version 0.43.0 (2023/07/03)

### Issues Closed

* [Issue 2519](https://github.com/pytroll/satpy/issues/2519) - MSG Dust RGB adding coastilnes and grid to the image
* [Issue 2506](https://github.com/pytroll/satpy/issues/2506) - Add xarray_kwargs capability to the geocat reader ([PR 2507](https://github.com/pytroll/satpy/pull/2507) by [@joleenf](https://github.com/joleenf))
* [Issue 2502](https://github.com/pytroll/satpy/issues/2502) - Cropping S3 image not working
* [Issue 2494](https://github.com/pytroll/satpy/issues/2494) - avhrr_l1b_gaclac fails to read most files from NOAA CLASS ([PR 2501](https://github.com/pytroll/satpy/pull/2501) by [@sfinkens](https://github.com/sfinkens))
* [Issue 2490](https://github.com/pytroll/satpy/issues/2490) - ninjogeotiff writer adds offset/scale factor when this is not meaningful ([PR 2491](https://github.com/pytroll/satpy/pull/2491) by [@gerritholl](https://github.com/gerritholl))
* [Issue 2483](https://github.com/pytroll/satpy/issues/2483) - Cacheing doesn't work with `scn.crop` ([PR 2485](https://github.com/pytroll/satpy/pull/2485) by [@djhoese](https://github.com/djhoese))
* [Issue 2465](https://github.com/pytroll/satpy/issues/2465) - Possibility of dual licensing: GPL-3.0 & MIT
* [Issue 2464](https://github.com/pytroll/satpy/issues/2464) - MITIFF writer using pillow: turn off compression due to rowsperstrip issues
* [Issue 2463](https://github.com/pytroll/satpy/issues/2463) - seviri_l1b_native reader issue with reading remote files (azure)
* [Issue 2409](https://github.com/pytroll/satpy/issues/2409) - Inconsistent behavior of time attributes in EUM L1 GEO readers ([PR 2420](https://github.com/pytroll/satpy/pull/2420) by [@YouvaEUMex](https://github.com/YouvaEUMex))
* [Issue 1749](https://github.com/pytroll/satpy/issues/1749) - Load from blended scene
* [Issue 859](https://github.com/pytroll/satpy/issues/859) - Doesn't recognize MODIS L2 file

In this release 12 issues were closed.

### Pull Requests Merged

#### Bugs fixed

* [PR 2522](https://github.com/pytroll/satpy/pull/2522) - Fix CF tests due to new xarray release
* [PR 2516](https://github.com/pytroll/satpy/pull/2516) - Fix SEVIRI native reader failing when missing main header
* [PR 2510](https://github.com/pytroll/satpy/pull/2510) - Fix warnings from NWCSAF reader
* [PR 2507](https://github.com/pytroll/satpy/pull/2507) - Fix HDF4 support in geocat reader with hardcoded engine ([2506](https://github.com/pytroll/satpy/issues/2506))
* [PR 2492](https://github.com/pytroll/satpy/pull/2492) - Fix xarray version for cf tests
* [PR 2491](https://github.com/pytroll/satpy/pull/2491) - Change logic for ninjogeotiff gradient/axisintercept tags ([2490](https://github.com/pytroll/satpy/issues/2490))
* [PR 2485](https://github.com/pytroll/satpy/pull/2485) - Fix angle caching not handling a specific type of irregular chunking ([2483](https://github.com/pytroll/satpy/issues/2483))
* [PR 2481](https://github.com/pytroll/satpy/pull/2481) - Fix NWCSAF reading for NOAA-21

#### Features added

* [PR 2521](https://github.com/pytroll/satpy/pull/2521) - Add a median filter modifier
* [PR 2508](https://github.com/pytroll/satpy/pull/2508) - Add support for OLCI L2 files which are missing Frame_IDs
* [PR 2504](https://github.com/pytroll/satpy/pull/2504) - Improve flexibility of olci level2 reader
* [PR 2501](https://github.com/pytroll/satpy/pull/2501) - Add Pygac reference to avhrr_l1b_gaclac documentation ([2494](https://github.com/pytroll/satpy/issues/2494))
* [PR 2499](https://github.com/pytroll/satpy/pull/2499) - Add option to clip negative ABI radiances
* [PR 2497](https://github.com/pytroll/satpy/pull/2497) - Enable to pass a custom function to Scene.aggregate
* [PR 2489](https://github.com/pytroll/satpy/pull/2489) - Add "neutral_resolution_band" kwarg to RatioSharpenedRGB/SelfSharpenedRGB
* [PR 2480](https://github.com/pytroll/satpy/pull/2480) - Add helper-function for reading SEVIRI L1.5 Native header.
* [PR 2449](https://github.com/pytroll/satpy/pull/2449) - Generalise the `true_color_reproduction` composite and enhancement
* [PR 2420](https://github.com/pytroll/satpy/pull/2420) - Fix inconsistent behavior of time attributes in EUM L1 GEO readers ([2409](https://github.com/pytroll/satpy/issues/2409))
* [PR 2259](https://github.com/pytroll/satpy/pull/2259) - Refactor `CFWriter.save_datasets` and enable retrieval of equivalent xr.Dataset with `scn.to_xarray()`
* [PR 2117](https://github.com/pytroll/satpy/pull/2117) - Add reader for GMS-5 VISSR data

#### Documentation changes

* [PR 2514](https://github.com/pytroll/satpy/pull/2514) - Fix argument name in DayNightComposite example document
* [PR 2501](https://github.com/pytroll/satpy/pull/2501) - Add Pygac reference to avhrr_l1b_gaclac documentation ([2494](https://github.com/pytroll/satpy/issues/2494))
* [PR 2478](https://github.com/pytroll/satpy/pull/2478) - Fix eccodes package names in setup.py, update documentation for setting up development environment.
* [PR 2474](https://github.com/pytroll/satpy/pull/2474) - Reorganize seviri_l2_grib.yaml file and add more documentation to seviri_l1b_native.py

#### Clean ups

* [PR 2523](https://github.com/pytroll/satpy/pull/2523) - Convert CF Writer tests to pytest
* [PR 2486](https://github.com/pytroll/satpy/pull/2486) - Fix leftover deprecated nosetest teardown methods
* [PR 2478](https://github.com/pytroll/satpy/pull/2478) - Fix eccodes package names in setup.py, update documentation for setting up development environment.
* [PR 2474](https://github.com/pytroll/satpy/pull/2474) - Reorganize seviri_l2_grib.yaml file and add more documentation to seviri_l1b_native.py

In this release 28 pull requests were closed.


## Version 0.42.2 (2023/05/10)

### Issues Closed
Expand Down
3 changes: 3 additions & 0 deletions continuous_integration/environment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,12 @@ dependencies:
- xarray!=2022.9.0
- dask
- distributed
- dask-image
- donfig
- appdirs
- toolz
- Cython
- numba
- sphinx
- cartopy
- panel>=0.12.7
Expand Down Expand Up @@ -54,6 +56,7 @@ dependencies:
- xarray-datatree
- pint-xarray
- ephem
- bokeh<3
- pip:
- trollsift
- trollimage>=1.20
Expand Down
2 changes: 2 additions & 0 deletions doc/rtd_environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,13 @@ dependencies:
- pip
- appdirs
- dask
- dask-image
- defusedxml
- donfig
# 2.19.1 seems to cause library linking issues
- eccodes>=2.20
- graphviz
- numba
- numpy
- pillow
- pooch
Expand Down
2 changes: 1 addition & 1 deletion doc/source/composites.rst
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,7 @@ In the case below, the image shows its day portion and day/night
transition with night portion blacked-out instead of transparent::

>>> from satpy.composites import DayNightCompositor
>>> compositor = DayNightCompositor("dnc", lim_low=85., lim_high=88., day_night="day_only", need_alpha=False)
>>> compositor = DayNightCompositor("dnc", lim_low=85., lim_high=88., day_night="day_only", include_alpha=False)
>>> composite = compositor([local_scene['true_color'])

RealisticColors
Expand Down
17 changes: 17 additions & 0 deletions doc/source/config.rst
Original file line number Diff line number Diff line change
Expand Up @@ -258,6 +258,23 @@ as part of the :func:`~satpy.modifiers.angles.get_angles` and
used by multiple modifiers and composites including the default rayleigh
correction.

Clipping Negative Infrared Radiances
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

* **Environment variable**: ``SATPY_READERS__CLIP_NEGATIVE_RADIANCES``
* **YAML/Config Key**: ``readers.clip_negative_radiances``
* **Default**: False

Whether to clip negative infrared radiances to the minimum allowable value before
computing the brightness temperature.
If ``clip_negative_radiances=False``, pixels with negative radiances will have
``np.nan`` brightness temperatures.

Clipping of negative radiances is currently implemented for the following readers:

* ``abi_l1b``


Temporary Directory
^^^^^^^^^^^^^^^^^^^

Expand Down
17 changes: 14 additions & 3 deletions doc/source/dev_guide/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ can do this using conda_::

.. _conda: https://conda.io/

This will create a new environment called "satpy-dev" with Python 3.8
This will create a new environment called "satpy-dev" with Python 3.11
installed. The second command will activate the environment so any future
conda, python, or pip commands will use this new environment.

Expand All @@ -68,6 +68,12 @@ should be run from the root of the cloned Satpy repository (where the
You can now edit the python files in your cloned repository and have them
immediately reflected in your conda environment.

All the required dependencies for a full development environment, i.e. running the
tests and building the documentation, can be installed with::

conda install eccodes
pip install -e ".[all]"

Running tests
=============

Expand All @@ -80,6 +86,7 @@ libraries. If you want to run all Satpy tests you will need to install
additional dependencies that aren't needed for regular Satpy usage. To install
them run::

conda install eccodes
pip install -e ".[tests]"

Satpy tests can be executed by running::
Expand Down Expand Up @@ -115,8 +122,12 @@ Documentation
=============

Satpy's documentation is built using Sphinx. All documentation lives in the
``doc/`` directory of the project repository. After editing the source files
there the documentation can be generated locally::
``doc/`` directory of the project repository. For building the documentation,
additional packages are needed. These can be installed with ::

pip install -e ".[all]"

After editing the source files there the documentation can be generated locally::

cd doc
make html
Expand Down
3 changes: 3 additions & 0 deletions satpy/_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,9 @@ def impr_files(module_name: str) -> Path:
'demo_data_dir': '.',
'download_aux': True,
'sensor_angles_position_preference': 'actual',
'readers': {
'clip_negative_radiances': False,
},
}

# Satpy main configuration object
Expand Down
124 changes: 124 additions & 0 deletions satpy/_scene_converters.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
# Copyright (c) 2023 Satpy developers
#
# This file is part of satpy.
#
# satpy is free software: you can redistribute it and/or modify it under the
# terms of the GNU General Public License as published by the Free Software
# Foundation, either version 3 of the License, or (at your option) any later
# version.
#
# satpy is distributed in the hope that it will be useful, but WITHOUT ANY
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
# A PARTICULAR PURPOSE. See the GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along with
# satpy. If not, see <http://www.gnu.org/licenses/>.
"""Helper functions for converting the Scene object to some other object."""

import xarray as xr

from satpy.dataset import DataID


def _get_dataarrays_from_identifiers(scn, identifiers):
"""Return a list of DataArray based on a single or list of identifiers.
An identifier can be a DataID or a string with name of a valid DataID.
"""
if isinstance(identifiers, (str, DataID)):
identifiers = [identifiers]

if identifiers is not None:
dataarrays = [scn[ds] for ds in identifiers]
else:
dataarrays = [scn._datasets.get(ds) for ds in scn._wishlist]
dataarrays = [dataarray for dataarray in dataarrays if dataarray is not None]
return dataarrays


def to_xarray(scn,
datasets=None, # DataID
header_attrs=None,
exclude_attrs=None,
flatten_attrs=False,
pretty=True,
include_lonlats=True,
epoch=None,
include_orig_name=True,
numeric_name_prefix='CHANNEL_'):
"""Merge all xr.DataArray(s) of a satpy.Scene to a CF-compliant xarray object.
If all Scene DataArrays are on the same area, it returns an xr.Dataset.
If Scene DataArrays are on different areas, currently it fails, although
in future we might return a DataTree object, grouped by area.
Parameters
----------
scn: satpy.Scene
Satpy Scene.
datasets (iterable):
List of Satpy Scene datasets to include in the output xr.Dataset.
Elements can be string name, a wavelength as a number, a DataID,
or DataQuery object.
If None (the default), it include all loaded Scene datasets.
header_attrs:
Global attributes of the output xr.Dataset.
epoch (str):
Reference time for encoding the time coordinates (if available).
Example format: "seconds since 1970-01-01 00:00:00".
If None, the default reference time is retrieved using "from satpy.cf_writer import EPOCH"
flatten_attrs (bool):
If True, flatten dict-type attributes.
exclude_attrs (list):
List of xr.DataArray attribute names to be excluded.
include_lonlats (bool):
If True, it includes 'latitude' and 'longitude' coordinates.
If the 'area' attribute is a SwathDefinition, it always includes
latitude and longitude coordinates.
pretty (bool):
Don't modify coordinate names, if possible. Makes the file prettier,
but possibly less consistent.
include_orig_name (bool).
Include the original dataset name as a variable attribute in the xr.Dataset.
numeric_name_prefix (str):
Prefix to add the each variable with name starting with a digit.
Use '' or None to leave this out.
Returns
-------
ds, xr.Dataset
A CF-compliant xr.Dataset
"""
from satpy.writers.cf_writer import EPOCH, collect_cf_datasets

if epoch is None:
epoch = EPOCH

# Get list of DataArrays
if datasets is None:
datasets = list(scn.keys()) # list all loaded DataIDs
list_dataarrays = _get_dataarrays_from_identifiers(scn, datasets)

# Check that some DataArray could be returned
if len(list_dataarrays) == 0:
return xr.Dataset()

# Collect xr.Dataset for each group
grouped_datasets, header_attrs = collect_cf_datasets(list_dataarrays=list_dataarrays,
header_attrs=header_attrs,
exclude_attrs=exclude_attrs,
flatten_attrs=flatten_attrs,
pretty=pretty,
include_lonlats=include_lonlats,
epoch=epoch,
include_orig_name=include_orig_name,
numeric_name_prefix=numeric_name_prefix,
groups=None)
if len(grouped_datasets) == 1:
ds = grouped_datasets[None]
return ds
else:
msg = """The Scene object contains datasets with different areas.
Resample the Scene to have matching dimensions using i.e. scn.resample(resampler="native") """
raise NotImplementedError(msg)
Loading

0 comments on commit 3369c93

Please sign in to comment.