Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adjust for changes to osm2pgsql RE replication initialization #396

Merged
merged 2 commits into from
Aug 6, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 0 additions & 39 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,6 @@ RAM=2
# to make unit test results visible at the end.
.PHONY: all
all: docker-exec-region docker-exec-input-file \
docker-exec-replication-w-input-file \
docker-exec-default unit-tests

.PHONY: docker-clean
Expand Down Expand Up @@ -115,44 +114,6 @@ docker-exec-input-file: build-run-docker



.PHONE: docker-exec-replication-w-input-file
docker-exec-replication-w-input-file: build-run-docker
# NOTE: This step tests --replication file for an initial load.
# It does **NOT** test the actual replication process for updating data
# using replication mode. Testing actual replication over time in this format
# will not be trivial. The historic file used (2021-01-13) cannot be used
# to seed a replication process, there is a time limit in upstream software
# that requires more recency to the source data. This also cannot simply
# download a file from Geofabrik, as the "latest" file will not have a diff
# to apply so also will not test the actual replication.
#
# Open a PR, Issue, discussion on https://github.com/rustprooflabs/pgosm-flex
# if you have an idea on how to implement this testing functionality.

# copy with arbitrary file name to test --input-file
docker cp tests/data/district-of-columbia-2021-01-13.osm.pbf \
pgosm:/app/output/$(INPUT_FILE_NAME)

# allow files created in later step to be created
docker exec -it pgosm \
chown $(CURRENT_UID):$(CURRENT_GID) /app/output/
# Needed for unit-tests
docker exec -it pgosm \
chown $(CURRENT_UID):$(CURRENT_GID) /app/docker/

# process it, this time without providing the region but directly the filename
docker exec -it \
-e POSTGRES_PASSWORD=mysecretpassword \
-e POSTGRES_USER=postgres \
-u $(CURRENT_UID):$(CURRENT_GID) \
pgosm python3 docker/pgosm_flex.py \
--layerset=minimal \
--ram=$(RAM) \
--replication \
--input-file=/app/output/$(INPUT_FILE_NAME) \
--skip-qgis-style --skip-nested # Make this test run faster


.PHONE: docker-exec-region
docker-exec-region: build-run-docker
# copy for simulating region
Expand Down
26 changes: 26 additions & 0 deletions docs/src/docker-build.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,3 +89,29 @@ docker pull postgis/postgis:16-3.4
docker build --no-cache -t rustprooflabs/pgosm-flex:dev .
```


## Building PgOSM Flex from an `osm2pgsql` feature branch

There are times it is helpful to build the PgOSM Flex Docker image with a
specific feature branch. To do this, change the `OSM2PGSQL_BRANCH`
and/or `OSM2PGSQL_REPO` arguments as necessary at the beginning of the Dockerfile.
The production setup looks like the following example.


```Dockerfile
ARG OSM2PGSQL_BRANCH=master
ARG OSM2PGSQL_REPO=https://github.com/osm2pgsql-dev/osm2pgsql.git
```

To test the feature branch associated with
[osm2pgsql #2212](https://github.com/osm2pgsql-dev/osm2pgsql/pull/2212)
the updated version was set like the following example.
This changes the `OSM2PGSQL_BRANCH` to `check-date-on-replication-init`
and changes the username in the `OSM2PGSQL_REPO` to `lonvia`.


```Dockerfile
ARG OSM2PGSQL_BRANCH=check-date-on-replication-init
ARG OSM2PGSQL_REPO=https://github.com/lonvia/osm2pgsql.git
```

6 changes: 6 additions & 0 deletions docs/src/replication.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,12 @@ your specific database and process.**

----

## Not tested by `make`

The function exposed by `--replication` is not tested via PgOSM's `Makefile`.



## Max connections

The other important change when using replication is to increase Postgres' `max_connections`.
Expand Down
Loading