-
Notifications
You must be signed in to change notification settings - Fork 5
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
78f0797
commit 0901bd9
Showing
1 changed file
with
5 additions
and
2 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -13,7 +13,7 @@ slug: postgres-cafe-pgstream | |
ogImage: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/[email protected] | ||
--- | ||
|
||
In [episode 3 of Postgres Café](https://www.youtube.com/embed/j1R3a0-jg6c?si=rEPoVI9vyR2MI_e8), we’re talking about pgstream, an open-source tool designed to handle one of PostgreSQL’s biggest gaps: seamlessly replicating schema and data changes. | ||
In [episode 3 of Postgres Café](https://www.youtube.com/embed/j1R3a0-jg6c?si=rEPoVI9vyR2MI_e8), we’re talking about pgstream, an open-source tool designed to handle one of PostgreSQL’s biggest gaps: seamlessly replicating schema and data changes. | ||
|
||
# Episode 3: pgstream | ||
|
||
|
@@ -38,19 +38,22 @@ Schema changes, such as altering table structures or adding columns, can disrupt | |
### Tackling replication lag | ||
|
||
pgstream prevents replication slot lag with two key approaches: | ||
|
||
- **Memory buffering:** Allocates memory to keep consuming replication slots even when downstream systems are slow. | ||
- **Kafka integration:** Decouples upstream and downstream systems by writing changes to Kafka topics, allowing multiple consumers to process data independently and at their own pace. | ||
|
||
### Modular and extensible design | ||
|
||
pgstream’s architecture is built around listeners (reading data changes) and processors (handling those changes). This modular design allows for flexible integrations and customizations. For example: | ||
|
||
- Use a **Postgres-to-OpenSearch processor** for real-time search. | ||
- Build custom processors to integrate with Snowflake, S3, or any other system. | ||
- Leverage its webhook processor to trigger external workflows based on database events. | ||
|
||
### Use cases | ||
|
||
The team shared how pgstream is already in use at Xata: | ||
|
||
- **Search integration:** pgstream powers Postgres-to-OpenSearch replication to enable advanced search features directly from PostgreSQL data. | ||
- **File cleanup workflows:** A custom processor detects when files are deleted in Postgres and triggers corresponding deletions in S3 storage, keeping data consistent across systems. | ||
|
||
|
@@ -62,4 +65,4 @@ For an in-depth exploration of pgstream and its capabilities, watch the full epi | |
|
||
## Stay tuned for more Postgres tools | ||
|
||
This is part of the Postgres Cafe video series! [Subscribe to the playlist](https://www.youtube.com/playlist?list=PLf7KS0svgDP_zJmby3RMzzOVO45qLbruA) for more episodes that feature open-source tools like [pgroll](https://pgroll.com/) for zero downtime schema changes in production, [StatsMgr](https://www.data-bene.io/en/blog/postgres-cafe-expand-monitoring-capabilities-with-statsmgr/) for monitoring and tracking events across PostgreSQL, and more. Watch this space to learn how each tool can make working with Postgres smoother and more efficient. | ||
This is part of the Postgres Cafe video series! [Subscribe to the playlist](https://www.youtube.com/playlist?list=PLf7KS0svgDP_zJmby3RMzzOVO45qLbruA) for more episodes that feature open-source tools like [pgroll](https://pgroll.com/) for zero downtime schema changes in production, [StatsMgr](https://www.data-bene.io/en/blog/postgres-cafe-expand-monitoring-capabilities-with-statsmgr/) for monitoring and tracking events across PostgreSQL, and more. Watch this space to learn how each tool can make working with Postgres smoother and more efficient. |