Support for syncing CDC changes by Unique columns and fall back to re…

…plica identity when present

This way users can still sync a table if it does not have a Primary Key. With Unique columns, composite primary keys and/or replica identities as fallback

.github/workflows/integration.yml

@@ -250,3 +250,41 @@ jobs:
           sleep 10
           ruby ./internal/scripts/e2e_resume_test.rb
           docker-compose -f internal/docker-compose.yml down -v
+
+  postgres_uniqueness_test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - name: Set up Go
+        uses: actions/setup-go@v4
+        with:
+          go-version: "1.21"
+      - name: Build pg_flo
+        run: make build
+      - name: Install dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y postgresql-client jq ruby ruby-dev libpq-dev build-essential
+          sudo gem install pg
+      - name: Set up Docker Compose
+        run: |
+          sudo curl -L "https://github.com/docker/compose/releases/download/v2.17.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
+          sudo chmod +x /usr/local/bin/docker-compose
+      - name: Run test
+        env:
+          PG_HOST: localhost
+          PG_PORT: 5433
+          PG_USER: myuser
+          PG_PASSWORD: mypassword!@#%1234
+          PG_DB: mydb
+          TARGET_PG_HOST: localhost
+          TARGET_PG_PORT: 5434
+          TARGET_PG_USER: targetuser
+          TARGET_PG_PASSWORD: targetpassword!@#1234
+          TARGET_PG_DB: targetdb
+        working-directory: .
+        run: |
+          docker-compose -f internal/docker-compose.yml up -d
+          sleep 10
+          ruby ./internal/scripts/e2e_postgres_uniquness.rb
+          docker-compose -f internal/docker-compose.yml down -v

README.md

@@ -232,6 +232,38 @@ pg_flo worker postgres --group inventory
 - NATS message size: 8MB (configurable)
 - One worker per group recommended
 - PostgreSQL logical replication prerequisites required
+- Tables must have one of the following for replication:
+  - Primary key
+  - Unique constraint with `NOT NULL` columns
+  - `REPLICA IDENTITY FULL` set
+
+Example table configurations:
+
+```sql
+-- Using primary key (recommended)
+CREATE TABLE users (
+  id SERIAL PRIMARY KEY,
+  email TEXT,
+  name TEXT
+);
+
+-- Using unique constraint
+CREATE TABLE orders (
+  order_id TEXT NOT NULL,
+  customer_id TEXT NOT NULL,
+  data JSONB,
+  CONSTRAINT orders_unique UNIQUE (order_id, customer_id)
+);
+ALTER TABLE orders REPLICA IDENTITY USING INDEX orders_unique;
+
+-- Using all columns (higher overhead in terms of performance)
+CREATE TABLE audit_logs (
+  id SERIAL,
+  action TEXT,
+  data JSONB
+);
+ALTER TABLE audit_logs REPLICA IDENTITY FULL;
+```
 
 ## Development