Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Testing osm2pgsql PR #2134 for reducing conections #376

Merged
merged 3 commits into from
Feb 10, 2024

Conversation

rustprooflabs
Copy link
Owner

@rustprooflabs rustprooflabs commented Feb 4, 2024

This PR started by testing osm2pgsql-dev/osm2pgsql#2134. Those results were encouraging and is now this is a PR addressing a couple other minor improvements.

@rustprooflabs rustprooflabs added the external External project or constraint. Blocked, related or otherwise label Feb 4, 2024
@rustprooflabs rustprooflabs added this to the 1.0.0 milestone Feb 4, 2024
@rustprooflabs
Copy link
Owner Author

Connections before -- 41 idle!

image

Connections after -- 3 idle.

image

@rustprooflabs
Copy link
Owner Author

Remove extra -d from osm2pgsql-replication command after more strict option checking implemented osm2pgsql-dev/osm2pgsql#2115

Sets repo/branch back to production setting. Leaving the new repo URL.

@rustprooflabs rustprooflabs added Technical Debt Not broken, not new, but not right either! Docker Docker image and/or related script(s) labels Feb 10, 2024
@rustprooflabs
Copy link
Owner Author

Adding a few more notes about connection volume with replication. The initial load had a connection pattern similar to a single load w/out replication (sustained 41 idle connections). Where things could get really nasty connection-wise was the replication update steps.

Before the patch the default PgOSM Flex layerset would consume 207 connections for replication updates. It's a relatively short time, but compared to Postgres' default max_connections=100 that can be shocking.

image

After the change to osm2pgsql, the total connection load drops to 17 (-92% diff)

image

@rustprooflabs rustprooflabs merged commit 0afd698 into main Feb 10, 2024
3 checks passed
@rustprooflabs rustprooflabs deleted the reduced-conns-test branch February 10, 2024 16:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Docker Docker image and/or related script(s) external External project or constraint. Blocked, related or otherwise Technical Debt Not broken, not new, but not right either!
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant