Skip to content

Server architecture

Jump to bottom
Daniel W. Steinbrook edited this page Dec 6, 2023 · 6 revisions

Diagram

flowchart TD
    data[OpenStreetMap planet data] -->|".pbf file (~100GB)"| import
    subgraph Container 1
        import(ingest server)
    end
    import -->|weekly| import_db[(import)]
    subgraph "PostGIS instance (3TB)"
        import_db
        spare[(backup)]
        active[(public)]
        %% import_db --> spare --> active -->|rotated| import_db
    end
    subgraph Container 2
        tile(Tile server)
    end
    active --> tile
    tile -->|"JSON map tiles<br/>1/20th sq. mi."| aoo[iOS app]
Loading

Description

There are three servers running in containers: the ingest server, the tile server, and a PostGIS instance. The ingest server fetches OpenStreetMap data for the entire planet (approx. 100GB) on a weekly basis. The ingest server loads the data into the PostGIS instance.

The PostGIS instance contains three schema namespaces within one database, named import, backup, and public. The three schemas are rotated, i.e. when the import completes, import becomes public, public becomes backup, and backup is overwritten by the next import. This rotation is a feature of the underlying imposm tool. All together, the three schemas take up less than 3TB of space.

The tile server responds to queries from the iOS app by querying the active PostGIS database. The queries from the app are for map tiles in JSON format, each corresponding to a 1/20th square mile area. A user session triggers at least tens and potentially hundreds of map tile requests. The tile data responses are cached by the app.

Single-host setup

The architecture above generally captures what Microsoft provided using Kubernetes on Azure. But all components can also run on a single traditional server using docker-compose.

Specifications

  • CPU: Intel Core i7-8700 (6-core, 3.20GHz)
  • RAM: 4x16GB DDR4
  • HDD: 2x1TB NVMe SSD M.2

As of late 2023, the above can be found fairly cheap in a Hetzner server auction.

We've used these specs to achieve a full-planet import in under 10 hours.

Setup

  • OS: debian bookworm image from Hetzner
  • root partition 50GB mirror on /dev/md0
  • 2GB swap partition on each ssd
  • rest is a striped volume /dev/md/home mounted on /home
  • official docker packages installed
  • soundscape user
  • password logins by ssh disabled (/etc/ssh/sshd_config.d/local.conf)
  • /var/lib/docker is bind mounted from /home/docker
  • in postgresql.conf (data_postgis volume) changed max_wal_size to 10GB and fsync=off to improve import performance

/etc/caddy/Caddyfile

See svcs/data/caddy.

Clone this wiki locally