Preload (FCI) filehandlers for eager processing

doc/source/advanced/index.rst

@@ -0,0 +1,94 @@
+Advanced topics
+===============
+
+This section of the documentation contains advanced topics that are not relevant
+for most users.
+
+.. _preload:
+
+Preloading segments for improved timeliness
+-------------------------------------------
+
+Normally, data need to exist before they can be read.  This requirement
+impacts data processing timeliness.  For data arriving in segments,
+Satpy can process each segment immediately as it comes in.
+This feature currently only works with the :mod:`~satpy.readers.fci_l1c_nc` reader.
+This is experimental and likely to be instable and might change.
+
+Consider a near real time data reception situation where FCI segments are
+delivered one by one.  Classically, to produce full disc imagery, users
+would wait for all needed segments to arrive, before they start processing
+any data by passing all segments to the :class:`~satpy.scene.Scene`.
+For a more timely imagery production, users can create the Scene, load the
+data, resample, and even call :meth:`~satpy.scene.Scene.save_datasets`
+before the data are complete (:meth:`~satpy.scene.Scene.save_datasets` will wait until the data
+are available, unless ``compute=False``).
+Upon computation, much of the overhead in Satpy
+internals has already been completed, and Satpy will process each segment
+as it comes in.
+
+To do so, Satpy caches a selection of data and metadata between segments
+and between repeat cycles.  Caching between segments happens in-memory
+and needs no preparation from the user, but where data are cached
+between repeat cycles, the user needs to create this cache first from
+a repeat cycle that is available completely::
+
+  >>> from satpy.readers import create_preloadable_cache
+  >>> create_preloadable_cache("fci_l1c_nc", fci_files)
+
+This needs to be done only once as long as data or metadata cached
+between repeat cycles does not change (for example, the rows at which
+each repeat cycle starts and ends).  To make use of eager processing, set
+the configuration variable ``readers.preload_segments``.  When creating
+the scene, pass only the path to the first segment::
+
+  >>> satpy.config.set({"readers.preload_segments": True})
+  >>> sc = Scene(
+  ...   filenames=[path_to_first_segment],
+  ...   reader="fci_l1c_nc")
+
+Satpy will figure out the names of the remaining segments and find them as
+they come in.  If the data are already available, processing is similar to
+the regular case.  If the data are not yet available, Satpy will wait during
+the computation of the dask graphs until data become available.
+
+For additional configuration parameters, see the :ref:`configuration documentation <preload_settings>`.
+
+Known limitations as of Satpy 0.51:
+
+- Mixing different file types for the same reader is not yet supported.
+  For FCI, that means it is not yet possible to mix FDHSI and HRFI data.
+- When segments are missing, processing times out and no image will be produced.
+  There is currently no way to produce an incomplete image with the missing
+  segment left out.
+- Dask may not order the processing of the chunks optimally.  That means some
+  dask workers may be waiting for chunks 33–40 as chunks 1–32 are coming in
+  and are not being processed.  Possible workarounds:
+  - Use as many workers are there are chunks (for example, 40).
+  - Use the dask distributed scheduler using
+    ``from dask.distributed import Client; Client()``.  This has only
+    limited support in Satpy and is highly experimental.   It should be possible
+    to read FCI L1C data, resample it
+    using the gradient search resampler, and write the resulting data using the
+    ``simple_image`` writer.  The nearest neighbour resampler or the GeoTIFF
+    writer do not currently work (see https://github.com/pytroll/satpy/issues/1762).
+    If you use this scheduler, set the configuration variable
+    ``readers.preload_dask_distributed`` to True.
+    This is not currently recommended in a production environment.  Any feedback
+    is highly welcome.
+- Currently, Satpy merely checks the existence of a file and not whether it
+  has been completely written.  This may lead to incomplete files being read,
+  which might lead to failures.
+
+For more technical background reading including hints
+on how this could be extended to other readers, see the API documentations for
+:class:`~satpy.readers.netcdf_utils.PreloadableSegments` and
+:class:`~satpy.readers.yaml_reader.GEOSegmentYAMLReader`.
+
+.. versionadded:: 0.52
+
+.. toctree::
+    :hidden:
+    :maxdepth: 1
+
+    preloaded_reading

doc/source/config.rst

@@ -274,6 +274,47 @@ Clipping of negative radiances is currently implemented for the following reader
 
 * ``abi_l1b``, ``ami_l1b``
 
+.. _preload_settings:
+
+Preloading segments
+^^^^^^^^^^^^^^^^^^^
+
+.. note::
+
+   Preloading is an advanced topic and experimental.  See :ref:`preload`
+   for details.
+
+* **Environment variable**: ``SATPY_READERS__PRELOAD__ENABLE``
+* **YAML-Config Key**: ``readers.preload.enable``
+* **Default**: False
+
+Preload segments for those readers where it is supported.
+
+* **Environment variable**: ``SATPY_READERS__PRELOAD__STEP``
+* **YAML-Config Key**: ``readers.preload.step``
+* **Default**: 2
+
+When preloading, internal time in seconds to check if a file has become
+available.
+
+* **Environment variable**: ``SATPY_READERS__PRELOAD__ATTEMPTS``.
+* **YAML-Config Key**: ``readers.preload.attempts``.
+* **Default**: 300
+
+When preloading, how many times to check if a file has become available
+before giving up.
+
+* **Environment variable**: ``SATPY_READERS__PRELOAD__ASSUME_DISTRIBUTED``
+* **YAML-Config Key**: ``readers.preload.assume_distributed``.
+* **Default**: False
+
+When preloading, assume we are working in a dask distributed environment.
+When active, Satpy workers will secede and rejoin while waiting for files.
+This might partially avoid the problem that tasks waiting for later files
+are blocking workers, while tasks working on earlier files are needlessly
+waiting in the queue.  However, Satpy has limited compatibility with
+dask distributed.  Please refer to the note at :ref:`preload` before
+considering this option.
 
 Temporary Directory
 ^^^^^^^^^^^^^^^^^^^

doc/source/index.rst

@@ -70,6 +70,7 @@ Documentation
     enhancements
     writing
     multiscene
+    advanced/index
     dev_guide/index
 
 .. toctree::

satpy/_config.py

@@ -53,6 +53,12 @@
     "sensor_angles_position_preference": "actual",
     "readers": {
         "clip_negative_radiances": False,
+        "preload": {
+            "enable": False,
+            "step": 2,
+            "attempts": 300,
+            "assume_distributed": False,
+        },
     },
 }
 

satpy/etc/readers/fci_l1c_nc.yaml

@@ -21,33 +21,67 @@ file_types:
       - "{pflag}_{location_indicator},{data_designator},MTI{spacecraft_id:1d}+{data_source}-1C-RRAD-FDHSI-{coverage}-{subsetting}-{component1}-BODY-{component3}-{purpose}-{format}_{oflag}_{originator}_{processing_time:%Y%m%d%H%M%S}_{facility_or_tool}_{environment}_{start_time:%Y%m%d%H%M%S}_{end_time:%Y%m%d%H%M%S}_{processing_mode}_{special_compression}_{disposition_mode}_{repeat_cycle_in_day:>04d}_{count_in_repeat_cycle:>04d}.nc"
     expected_segments: 40
     required_netcdf_variables: &required-variables
-      - attr/platform
-      - data/{channel_name}/measured/start_position_row
-      - data/{channel_name}/measured/end_position_row
-      - data/{channel_name}/measured/radiance_to_bt_conversion_coefficient_wavenumber
-      - data/{channel_name}/measured/radiance_to_bt_conversion_coefficient_a
-      - data/{channel_name}/measured/radiance_to_bt_conversion_coefficient_b
-      - data/{channel_name}/measured/radiance_to_bt_conversion_constant_c1
-      - data/{channel_name}/measured/radiance_to_bt_conversion_constant_c2
-      - data/{channel_name}/measured/radiance_unit_conversion_coefficient
-      - data/{channel_name}/measured/channel_effective_solar_irradiance
-      - data/{channel_name}/measured/effective_radiance
-      - data/{channel_name}/measured/x
-      - data/{channel_name}/measured/y
-      - data/{channel_name}/measured/pixel_quality
-      - data/{channel_name}/measured/index_map
-      - data/mtg_geos_projection
-      - data/swath_direction
-      - data/swath_number
-      - index
-      - state/celestial/earth_sun_distance
-      - state/celestial/subsolar_latitude
-      - state/celestial/subsolar_longitude
-      - state/celestial/sun_satellite_distance
-      - state/platform/platform_altitude
-      - state/platform/subsatellite_latitude
-      - state/platform/subsatellite_longitude
-      - time
+      # key/value; keys are names, value is a list of string on how this may be
+      # cached between segments or between repeat cycles or neither
+      attr/platform:
+        - segment
+        - rc
+      data/{channel_name}/measured/start_position_row:
+        - rc
+      data/{channel_name}/measured/end_position_row:
+        - rc
+      data/{channel_name}/measured/radiance_to_bt_conversion_coefficient_wavenumber:
+        - segment
+        - rc
+      data/{channel_name}/measured/radiance_to_bt_conversion_coefficient_a:
+        - segment
+        - rc
+      data/{channel_name}/measured/radiance_to_bt_conversion_coefficient_b:
+        - segment
+        - rc
+      data/{channel_name}/measured/radiance_to_bt_conversion_constant_c1:
+        - segment
+        - rc
+      data/{channel_name}/measured/radiance_to_bt_conversion_constant_c2:
+        - segment
+        - rc
+      data/{channel_name}/measured/radiance_unit_conversion_coefficient:
+        - segment
+        - rc
+      data/{channel_name}/measured/channel_effective_solar_irradiance:
+        - segment
+        - rc
+      data/{channel_name}/measured/effective_radiance: []
+      data/{channel_name}/measured/x:
+        - segment
+        - rc
+      data/{channel_name}/measured/y:
+        - rc
+      data/{channel_name}/measured/pixel_quality: []
+      data/{channel_name}/measured/index_map: []
+      data/mtg_geos_projection:
+        - segment
+        - rc
+      data/swath_direction:
+        - segment
+      data/swath_number:
+        - rc
+      index: []
+      state/celestial/earth_sun_distance:
+        - segment
+      state/celestial/subsolar_latitude:
+        - segment
+      state/celestial/subsolar_longitude:
+        - segment
+      state/celestial/sun_satellite_distance:
+        - segment
+      state/platform/platform_altitude:
+        - segment
+      state/platform/subsatellite_latitude:
+        - segment
+      state/platform/subsatellite_longitude:
+        - segment
+      time: []
     variable_name_replacements:
       channel_name:
         - vis_04
@@ -66,12 +100,24 @@ file_types:
         - ir_105
         - ir_123
         - ir_133
+    # information needed for preloading
+    segment_tag: count_in_repeat_cycle
+    time_tags: &tt
+      - processing_time
+      - start_time
+      - end_time
+    variable_tags:  # other tags where RC caching doesn't care about variation
+      - repeat_cycle_in_day
   fci_l1c_hrfi:
     file_reader: !!python/name:satpy.readers.fci_l1c_nc.FCIL1cNCFileHandler
     file_patterns:
       - "{pflag}_{location_indicator},{data_designator},MTI{spacecraft_id:1d}+{data_source}-1C-RRAD-HRFI-{coverage}-{subsetting}-{component1}-BODY-{component3}-{purpose}-{format}_{oflag}_{originator}_{processing_time:%Y%m%d%H%M%S}_{facility_or_tool}_{environment}_{start_time:%Y%m%d%H%M%S}_{end_time:%Y%m%d%H%M%S}_{processing_mode}_{special_compression}_{disposition_mode}_{repeat_cycle_in_day:>04d}_{count_in_repeat_cycle:>04d}.nc"
     expected_segments: 40
     required_netcdf_variables: *required-variables
+    segment_tag: count_in_repeat_cycle
+    time_tags: *tt
+    variable_tags:
+      - repeat_cycle_in_day
     variable_name_replacements:
       channel_name:
         - vis_06_hr

satpy/readers/__init__.py

@@ -1,6 +1,4 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-# Copyright (c) 2015-2018 Satpy developers
+# Copyright (c) 2015-2024 Satpy developers
 #
 # This file is part of satpy.
 #
@@ -808,3 +806,31 @@
         except AttributeError:
             f_obj = unknown_file_thing
     return f_obj
+
+
+def create_preloadable_cache(reader_name, filenames):
+    """Create on-disk cache for preloadables.
+
+    Some readers allow on-disk caching of metadata.  This can be used to
+    preload data, creating file handlers and a scene object before data files
+    are available.  This utility function creates the associated cache for
+    multiple filenames.
+
+    Args:
+        reader_name (str):
+            Reader for which to create the cache.
+        filenames (List[str]):
+            Files for which to create the cache.  Typically, this would be
+            the same set of files to create a single scene.
+    """
+    reader_instances = load_readers(filenames, reader_name)
+    for (nm, reader_inst) in reader_instances.items():
+        for (tp, handlers) in reader_inst.file_handlers.items():
+            for handler in handlers:
+                filename = reader_inst._get_cache_filename(
+                        handler.filename,
+                        handler.filename_info,
+                        handler)
+                p = pathlib.Path(filename)
+                p.parent.mkdir(exist_ok=True, parents=True)
+                handler.store_cache(filename)

satpy/readers/fci_l1c_nc.py

@@ -1,6 +1,4 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-# Copyright (c) 2017-2019 Satpy developers
+# Copyright (c) 2017-2024 Satpy developers
 #
 # This file is part of satpy.
 #
@@ -20,12 +18,10 @@
 
 This module defines the :class:`FCIL1cNCFileHandler` file handler, to
 be used for reading Meteosat Third Generation (MTG) Flexible Combined
-Imager (FCI) Level-1c data.  FCI flies
-on the MTG Imager (MTG-I) series of satellites, with the first satellite (MTG-I1)
-launched on the 13th of December 2022.
-For more information about FCI, see `EUMETSAT`_.
+Imager (FCI) Level-1c data.  FCI flies on the MTG Imager (MTG-I) series
+of satellites, with the first satellite (MTG-I1) launched on the 13th
+of December 2022.  For more information about FCI, see `EUMETSAT`_.
 
-For simulated test data to be used with this reader, see `test data releases`_.
 For the Product User Guide (PUG) of the FCI L1c data, see `PUG`_.
 
 .. note::
@@ -130,7 +126,7 @@
 from satpy.readers._geos_area import get_geos_area_naming
 from satpy.readers.eum_base import get_service_mode
 
-from .netcdf_utils import NetCDF4FsspecFileHandler
+from .netcdf_utils import NetCDF4FsspecFileHandler, PreloadableSegments
 
 logger = logging.getLogger(__name__)
 
@@ -183,7 +179,7 @@
     return channel_name
 
 
-class FCIL1cNCFileHandler(NetCDF4FsspecFileHandler):
+class FCIL1cNCFileHandler(PreloadableSegments, NetCDF4FsspecFileHandler):
     """Class implementing the MTG FCI L1c Filehandler.
 
     This class implements the Meteosat Third Generation (MTG) Flexible
@@ -208,12 +204,15 @@
         "MTI3": "MTG-I3",
         "MTI4": "MTG-I4"}
 
-    def __init__(self, filename, filename_info, filetype_info):
+    def __init__(self, filename, filename_info, filetype_info,
+                 *args, **kwargs):
         """Initialize file handler."""
         super().__init__(filename, filename_info,
                          filetype_info,
+                         *args,
                          cache_var_size=0,
-                         cache_handle=True)
+                         cache_handle=True,
+                         **kwargs)
         logger.debug("Reading: {}".format(self.filename))
         logger.debug("Start: {}".format(self.start_time))
         logger.debug("End: {}".format(self.end_time))