Add reader for IASI EPS L2 native format #2457
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Started refactoring the AVHRR L1b EPS reader in preparation for the IASI L2 EPS reader. Replaced the wrong XML format definition by the correct one. Source for the EPS IASI L2 format definition is eugene. After installing eugene with ``mamba install -c eumetsat eugene``, I found the definition file at ``envs/py311/share/eugene/formats/eps_iasil2_9.0.xml``.
Codecov Report
@@ Coverage Diff @@
## main #2457 +/- ##
==========================================
+ Coverage 94.91% 94.96% +0.05%
==========================================
Files 349 353 +4
Lines 50969 51836 +867
==========================================
+ Hits 48375 49225 +850
- Misses 2594 2611 +17
Flags with carried forward coverage won't be shown. Click here to find out more.
|
|
Putting this on hold as I've decided to use the Data Tailor and then read the output from there. See #2518 |
For the XML format specification, add missing type configurations to xmlformat.py. Also uncomment parameter definitions from the XML definition, which are needed to interpret the rest of the file. I don't know why those were commented out.
Change the distribution of responsibility between EPSBaseFileHandler and EPSAVHRRFile. Added a tests for xmlformat.process_array.
In xmlformat, collect a mapping of variable names against dimension names.
Turn xmlformat.parse_format from a module level function to a method on XMLFormat. Remove the module level dictionary VARIABLES and make this an instance attribute on the XMLFormat class.
|
Maybe I should just plunder https://gitlab.eumetsat.int/open-source/data-tailor-plugins/epct_plugin_gis/-/blob/main/epct_plugin_gis/iasisnd02.py (Apache license) |
Take the IASI L2 EPS reader from EUMETSAT and integrate into Satpy. The code was obtained from https://gitlab.eumetsat.int/open-source/data-tailor-plugins/epct_plugin_gis where it is licensed under the Apache License Version 2.0, January 2004. It was developed by B-Open Solutions srl for EUMETSAT under contract EUM/C0/17/4600001943/0PN and is Copyright (c) 2017-2022 EUMETSAT. To the best of my knowledge, the Apache License allows for its integration into GPL code.
|
Sadly, when I use the new XML file with the reading routine for EPS AVHRR native, the result is completely wrong (binary garbage with the record lengths not matching). |
Prior to speeding up the reading, improve tests to test actual values to be read.
When reading grh, support interpreting a memmap.
gerritholl
commented
Jul 12, 2023
Remove more dead code and change the sample data used for test_eps_native_reader. Change the expected results accordilgny.
In the dynamic available datasets for the iasil2snd reader, don't add coordinates to variables that don't have corresponding dimensions.
Add safeguards in the IASI native reader to avoid going into infinite loops
Remove implementation attempts that didn't work, restoring those files.
gerritholl
force-pushed
the
feature-reader-iasi-native
branch
from
9fec89c
to
54c6826
Compare
Pull Request Test Coverage Report for Build 5953974235
💛 - Coveralls |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add a reader for IASI EPS L2 native format "IASI Combined Sounding Products - Metop". The data are available in the EUMETSAT Data Store. User manuals, product description, and more information are available at https://data.eumetsat.int/product/EO:EUM:DAT:METOP:IASSND02.
The core of the reader is copied from the IASI L2 EPS reader from the EUMETSAT package
epct_plugin_gis. I adapted this to make it dask-friendly and to integrate it as a Satpy reader. Specifically, the code was obtained from https://gitlab.eumetsat.int/open-source/data-tailor-plugins/epct_plugin_gis where it is licensed under the Apache License Version 2.0, January 2004. It was developed by B-Open Solutions srl for EUMETSAT under contract EUM/C0/17/4600001943/0PN and is Copyright (c) 2017-2022 EUMETSAT. To the best of my knowledge, the Apache License allows for its integration into GPL code.Status: Code seems to work with memory maps and dask, but overhead remains large. Tests need to run faster, run independently of downloading large data files, and cover non-nominal cases.