Skip to content

Kartoza Map services

Jump to bottom
mazano edited this page May 4, 2023 · 5 revisions

Introduction

Kartoza is one of the leading giants in the provision of GIS Services in Africa and the world. Most of our work is centred on spatial data and we thrive by outsourcing open-source data from public institutions and private companies. We sometimes encounter challenges in sourcing this data on demand due to the following:

  • Public servers crashing and no immediate backups for data services when we need them.
  • Internet connectivity issues from the service providers.
  • Data quality is sometimes poor and does not meet our standards. The list above is not exhaustive but it's an indication of the challenges we encounter when outsourcing data.

Plan of action

We need to source the datasets we need most and store them locally so that they can be availed to our clients and the public in the future. We have always had a map gallery service where we showcase our artistic skills when designing maps. We have recently elevated this to a service that will be serving vector and raster data in accordance with the OGC standards.

Software Stack

The software stack is deployed on rancher2 from https://rancher2.kartoza.com/p/c-qtls8:p-p55lb/workloads

  1. Database and related Services. image

  2. GIS Servers and related Servers. image

Database and related Services

This consists of the following services:

  1. PostgreSQL and PostGIS we utilize PostgreSQL and PostGIS to store our data (vector and raster). PostgreSQL has the ability to store large datasets and can be indexed allowing for fast access. Many web maps we build can interact directly with the spatial database and can render data in various formats eg Geojson. Access to the database is through the username and password which are supplied by the database administrator. It is recommended to access the database through the postgres.conf file.
 [kartoza_gis]
 host=hostname
 port=port
 user=username
 password=password
 dbname=dbname

And when connecting in QGIS you will then connect with the following:

image

  1. Metabase - This is used for business intelligence. It can be accessed from https://postgis.kartoza.com/auth/login?redirect=%2F

image

  1. PgAdmin - A client accessible in the browser to interact with the database, This is useful to run SQL that needs to run for a long time or to debug SQL issues. As the client resides in the same environment with the database, SQL queries generally run faster as they is no latency. This is accessible from https://postgis.kartoza.com/pgadmin/login?next=%2Fpgadmin%2F

  2. Dashboard - This is an interactive pgwatch dashboard which can be used to analyse the database metric. Available from https://postgis.kartoza.com/dashboard

  3. PostgREST - Accessible from https://postgis.kartoza.com/pgrest. This provides an API to interact with the PostgreSQL database. Full documentation is available from https://postgrest.org/en/stable/

GIS Servers

In the open-source GIS world, they are a lot of GIS servers and we decided to use two of the most popular services QGIS and Geoserver.

QGIS Server

QGIS server capabilities and usage have grown exponentially over the past couple of years. QGIS server is easy to use and is tightly integrated with QGIS Desktop and publishing maps is seamless. One limitation of using QGIS Server is the lack of a caching mechanism for its services. Accesible from https://maps.kartoza.com/qgis/ows/?map=Swellendam/Swellendam.qgs

GeoServer

GeoServer provides OGC services and has also inbuilt caching which allows it to render large data sets. Accesible from https://maps.kartoza.com/geoserver/web/

Caching

Because QGIS Server does not have an inbuilt caching mechanism we will use Mapproxy which is a popular caching software and it also allows us to generate XYZ tiles.

Accesible from https://maps.kartoza.com/mapproxy

Data organisation

All QGIS web maps and raster data served by their respective projects are maintained in file browser. If you want to publish a map on the maps.kartoza.com you need the upload it to the file browser folders.

Vector data

All vector data will be stored in a PostGIS database which is accessible from maps.kartoza.com. The structure of the database is centred on the source. Each schema will represent the custodians of the data. For example, data that has been sourced from NGI will be in the ngi schema.

Data that will are not certain of their origins will be maintained in the kartoza schema. The public schema should never be populated with any spatial data.

Example of schemas populated in the database.

  • csir
  • dwa
  • geoscience
  • kartoza
  • ngi
  • sanbi
  • sg
  • statssa

Raster data

All raster data will be shared through filebrowser and in their particular project. For instance, public layers will be in the kartoza folder and Swellendam will be in its own folder.

Access to the database

Currently, Tim, Gavin and Admire, the developers have admin access to the database and other OGC services. So loading of data will be done by these people and all other developers will need to access services (WFS, WMS) etc.

Mapping strategy

QGIS Server vs GeoServer

These two software packages are the same in their offering because they are OGC compliant. Their functionalities can also be enhanced by extensions which are unique to each software. QGIS Server allows users to develop a plugin and use it whilst Geoserver has its own extensions.

QGIS Server

QGIS Server will be used as the primary choice for web map service in most instances. QGIS Server allows us to achieve better and more sophisticated cartography without too much fiddling. GeoServer will only be used in situations where QGIS Server is limited and where performance is greater.

For large layers that do not change unless the data source has been updated, we will utilize Geoserver because it has inbuilt caching that is provided through Geowebcache.

The table below is not meant to be used for benchmarking speed between Geoserver and QGIS but merely to highlight why we might need to use one.

QGIS Server Geoserver
Easy to deploy Not easy to deploy
User-friendly GUI for symbolisation SLD / CSS for map symbolisation
Plugins Geoserver extensions
No inbuilt caching Inbuilt Caching
No RestAPI RestAPI
Limited service endpoints (eg topojson) Multiple endpoints (KML,Geojson,TopoJson,Vector tiles)
Slow on big rasters Can accommodate gdal tiles

Based on each individual project we should consider the above issues and other extras so that we can choose which service to use.

Which data layers are available?

GeoServer is accessible from Kartoza geoserver and performing get capabilities should provide you with a list of all resources available: Geoserver capabilities

QGIS Server is accessible from Kartoza qgis server and performing get capabilities on each project should provide you with the list of resources available.

All public layers are available from the kartoza project the following request kartoza qgis capabilities

Sometimes available layers are not available in the project but are still available in the database we need a way to identify if the layer exists in the database.

Clone this wiki locally