Add support for input retrieval from CDSE #101
Conversation
|
Hi Martin, thanks so much for this through work, much appreciated. I think I understand what you are up to with the app and that's fine. Download from CDSP is also fine, I am just relying on ASF webpage for download, which in my opinion does a better job. Regarding the update of the xml files, which version of SNAP do you currently use? Is this then wroking with newer version, and has all functionality been tested against (I still use it for coherence, so that's definitely an important one). Using COG for export is also nice. I actually would like to get rid of GDAL, and only use rasterio, but I think rasterio still does not support vrts the way I am using it. Let me know, |
preprocessing.py now conforms to EO Application Package best practice by interpreting its input not as a direct path to a data file but as the path to a directory containing a STAC catalog containing an item containing an asset giving the path to the actual input. Similarly, after processing, preprocessing.py also writes a STAC catalog describing the output data.
|
Dear Andreas, we still use SNAP 8.0. We did not change that, hoping that this allows to reproduce what you had done in your notebooks. Do you have a hint what may be wrong when we get "products are not consecutive slices" . Is the assumption correct the toolbox makes, that the next granule starts at a time the previous granule stops, without overlap? What can we do to avoid that? Best regards, |
Logging is now done with the standard Python logging library, not with print statements.
preprocessing.py now writes its output to the CWD set on container start-up, as prescribed in OGC EOAP BP §7.2.
Linking between input and output directories doesn't work if they're on different file systems.
Add is_flag=True to the click definitions of the two boolean command-line arguments to conform to CWL and normal *nix behaviour (i.e. flag is now controlled by the argument's presence/absence, not by an additional parameter to the argument).
- Make asset hrefs relative in output STAC. - Set bbox, geometry, and start and end times correctly in output STAC. - Correct some errors in log messages.
- Use custom headers when fetching OTB package to avoid download speed throttling. - Install OST package from version2 branch of bcdev repository. - Add constraints.txt and snap.varfile to build context. - Fetch last commit information via GitHub API to invalidate build cache if package has been updated.
- Add a dry-run parameter. - Update Docker requirement to reference public image on quay.io.
wget progress now uses the dot:giga setting to reduce the amount of noise in the build logs.
Version2
version2 branch has now been merged so we build images from main instead.
get_zip_from_stac has been renamed to get_input_path_from_stac and now also handles STAC catalogues describing unzipped SAFE directories. However, preprocessing.run() still needs to be adapted to deal with these SAFE directories as input.
- Update to version 1.2 to allow use of loadListing and networkAccess options. - Set "loadListing: no_listing" for directory input to prevent errors due to illegally named files in the input directory. - Set "neworkAccess: true" in CommandLineTool requirements to ensure that s1scene can make requests to external services.
This argument controls the branch or commit of the OpenSarToolkit repository which is fetched, making it easier to build locally from a different branch.
Add support for SAFE directory input
proposed changes to CWL - remove cdse and rename workflow ID
Dear developers,
in order to get OpenSarToolkit running I had to make a few updates to the Dockerfile and I had to implement download of input data from CDSE. Neither SciHub nor PEPS exist any more. I also have integrated the pull request by Aleksandr Tulenkov to let it survive a bit longer.
I still struggle with errors reporting "products are not consecutive slices" for more than half of the days of May 2020, the test month in your notebook. Hints how to solve that are welcome.
Best regards,
Martin