-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add pgbouncer #114
feat: add pgbouncer #114
Conversation
9c69644
to
352bea8
Compare
Add option to set up a pgbouncer server that can manage traffic to the actual database BREAKING CHANGE: The `db` argument in the TiPgApiLambda, PgStacApiLambda, and TitilerPgstacApiLambda constructs has been replaced with `connectionTarget`. This object is used to add the Lambdas to the list of acceptable connections and can be either an RDS Instance or an EC2 instance running pgbouncer. also install new pgbouncer-secret-updater package in ci
@vincentsarago I have it set up to work with Lambdas in or out of the VPC. Now the I added some more notes about networking setups over here: #116. |
# [7.4.0](v7.3.0...v7.4.0) (2025-01-24) ### Features * add pgbouncer ([#114](#114)) ([5952858](5952858))
Add option to set up a
PgBouncer
server that can efficiently route traffic to the actual databaseMerge request description
Add option to set up a pgbouncer server that can manage traffic to the actual database.
Adding a connection pooler like PgBouncer to eoAPI drastically improves performance by preventing the database instance from being overwhelmed by connections from eager clients like titiler-pgstac. AWS offers the RDS Proxy as a managed version of this setup, but it is quite expensive and can double the cost of running an RDS instance. By setting up an EC2 instance running PgBouncer, we can get the same impact for a fraction of the RDS Proxy cost (<$10/mo).
Most of the work to get PgBouncer running was done in MAAP-Project/maap-eoapi#39. This PR builds off of that work and does a more complete job of integrating the PgBouncer server with the rest of the eoAPI applications (not just titiler-pgstac).
I did that by creating a new secret that contains the same values as the original PgstacSecret but with the PgBouncer EC2 instance private or public IP address (depending on the networking setup) substituted for the
host
key. If the user specifiesaddPgBouncer: true
in thePgStacDatabase
construct, an EC2 instance running PgBouncer will be created and thepgstac_secret
attribute will be set to the PgBouncer secret instead of the original PgStac secret. This makes it easy to configure client apps to connect to the database via PgBouncer instead of making a direct connection to the database.The PgBouncer infrastructure is slightly more complicated than I want it to be because we cannot create the PgBouncerSecret until the EC2 instance has been created and assigned an IP address. To get around this, I set up a Lambda function to create the PgBouncer secret with nothing in it, then update it with the correct values after the EC2 instance is running.
Breaking changeThedb
argument in the TiPgApiLambda, PgStacApiLambda, and TitilerPgstacApiLambda constructs has been replaced withconnectionTarget
. This object was/is used to add the Lambdas to the list of acceptable connections and now it can be either an RDS Instance or an EC2 instance (running PgBouncer). I made the breaking change because it would be very easy for an existing user to add PgBouncer to thePgStacDatabase
construct then fail to use it when creating the downstream application constructs. By making the breaking change it should force users to at least re-specify the argument name (but since the argument can be either an RDS instance or an EC2 Instance they could easily still mistakenly specify the RDS instance :/). Maybe it is not worth a breaking change - what do you think??I opted to make it a non-breaking change and will make some updates in eoapi-template to show how to use the new
pgbouncer
feature.