expose EuroCrops at granular level

[juansensio]

Posted by @patrick-griffiths

User story: how to expose EuroCrops at granular level, so that queries such as extracting all Winter Wheat Parcels in a AOI can be performed..

[juansensio]

Steps:

1. Generate STAC metadata for EuroCrops (if not already done) and ingest in EOTDL as Q1+ dataset.
2. Optionally, enable STAC query through EOTDL API (which in turn will query geodb or geodb STAC API)

The dataset can be ingested and the entire STAC Catalog can be downloaded, read with the tools and filtered with pandas like queries. Then, the remaining assets can be downloaded using as well the tools (this is demonstrated in a notebook example, will update and link here).

[juansensio]

After some back and forth, we believe the best option is to let the users retrieve the full metadata and perform the queries themselves locally (with some help from eotdl library).

Performing queries directly to geodb is difficult due to the nature of the data stored and its functionality, however we are looking into ways to improve it #274

You can see an example here: https://github.com/earthpulse/eotdl/blob/geodb/tutorials/notebooks/geodb.ipynb

Validate in next progress meeting @Patrick1G

[Patrick1G]

@juansensio lets have a progress meeting on openEO and Eurocrops over the next weeks..

The notebook above uses EuroSat which is a very different TDS as it is just labeled image chips. in EuroCrops we have several shapefiles for each country, consisting of .cpg / .dbf / .prj / .shp / .shx sub file components, within each shapefile 100s of thousands of polygons, each with several attributes most importantly crop type...

this is a very different data structure than the EuroSAT dataset, but this EuroCrops is a TYPICAL CASE in EO, so EOTDL needs to be able to handle this efficiently: users should be able to select/query polygons by attributes or location...

Also given the new STAC approach all of the above shapefile subcomponents will get a STAC catalogue, which does not make much sense.. Probably good to convert shapefile to geojson or geoparquet by default? @jamesemwheeler

[juansensio]

We are working on this in parallel here #190

Also, as mentioned in the past progress meeting, in order to query the database for crop type, country, geometry, etc, requires this information to be stored in the dataset metadata (independently of the underlying format of the data - shapefile, geoparquet, geojson, etc). Otherwise, the full shapefiles have to be loaded to perform the filtering.

From my point of view, these are the options:

1 - stage full dataset locally and filter (dataset is not very large, totally feasible and simple approach)
2 - convert the dataset from shapefiles to STAC metadata (geojsons) with the fields we want to use for queries at individual parcel (one parcel, one STAC item)
3 - store the raw dataset in geodb (instead of the metadata) to perform the queries (but EOTDL is not designed to work like this)