File sources and cache

[kolyanch]

File sources

Currently, we can use the following APIs:

• The OGC API to edit objects and display data.
• MVT and PMTiles to reduce data size and prepare data for fast rendering.
• A combination of MVT and the OGC API to display the current data slice, with two layers output on the client. MVT is used for fast display and the OGC API provides up-to-date data from the moment of latest tile generation.

Current features:

• Data is stored in a database, but in some situations, it is also necessary to display data directly from GeoJSON without uploading it to the database.
• Users can modify data either through the API or directly in the source.
• When using a hybrid approach (Features + Tiles), two requests are made to the server.
• If objects are deleted in database, they will still be in the tile cache.
• With large amounts of spatial data PostGIS may start to slow down.
• Direct data rendering via the OGC API Features is as fast as MVT on a small scale.
• Re-generation of tiles for large territories using e.g. tippecanoe may take a long time.

The proposed approach for processing file sources:

1. We will initially support GeoJSON files only. In the future, we may add support for GeoPackage and Shapefiles.
2. Files will be identified by their direct path on the server or URL. If the files are hosted in S3 buckets, they can be accessed via a direct link.
3. The API will not allow modifying data in files at the initial stage.
4. All file data will be read into an in-memory cache.
5. Only data from the cache will be retrieved through API.
6. Spatial indexing and storing the spatial index in RAM are necessary for querying data within a bbox.

[kolyanch]

Cache

Current data publishing scheme:

The proposed scheme with the cache:

• The use of a standard distributed cache for geodata storage is limited. For example, Redis can only index point objects in WGS84.
• An additional service is required to store and update a spatial index for data.
• It is necessary to implement the ability to configure cache storage settings - in memory or distributed.
• The object identifier in the cache should be a composite string: collection/layer identifier + object identifier.
• The OGC API implementation should support working without a cache.

Options for a caching layer:

• HybridCache
  Supports both primary in-memory cache (L1) and distributed cache (L2)
• FusionCache
  Similar features to HybridCache, but with more customization options

In any case, adding a cache looks extremely simple. The final cache option is selected on the server side that uses the OGC API.

[kolyanch]

Sequence diagrams

Getting a feature by its ID. The feature is present in the cache

The OGC API accesses the cache. If the object is present in the cache, it is immediately returned to the client.

Getting a feature by its ID. The feature is NOT present in the cache

The OGC API checks the cache. If an object is not found in the cache, it makes a request to the data source. From an implementation perspective, this involves passing a delegate to a GetOrCreateAsync method or equivalent in the cache. The information about the object is then added to the cache and the spatial index updated.

Getting features in a bbox

The spatial index is being queried. The result is a set of object IDs. Objects based on the received identifiers are requested from the cache (or if they are missing from the data source).

Data modification (POST/PUT/PATCH/DELETE)

The order in which data is updated is important. It is recommended to update the data source first, followed by the cache, and finally the spatial index.

Complete cache update based on data from the source

This is not a required scenario. The data in the source can be modified not only through the OGC API, but also directly in the database. In this case, the cache can be fully reloaded or only the data since the last load can be updated. It is also possible to set a time after which objects in the cache become obsolete (cache expiration/duration) instead of using this scenario.