v0.25.3

What's Changed

• refactor(alerting): move drop flow errors to internal error by @iamKunalGupta in #2633
• [dropflow] skip source drop by @heavycrystal in #2635

Full Changelog: v0.25.2...v0.25.3

v0.25.2

What's Changed

• improve error logging for query execution by @iskakaushik in #150
• revert limit 0 by @iskakaushik in #152
• [mysql] refactor to qvalue_convert by @heavycrystal in #2565
• mysql cdc: bounds check pulling schema info by @serprex in #2566
• mysql: don't start batch timeout until first record received by @serprex in #2555
• go 1.24 by @serprex in #2562
• feat(metrics): add gauge and counter counterparts of fetched bytes and records synced by @iamKunalGupta in #2569
• feat(classifier): add too many parts (when pushing to mv) by @iamKunalGupta in #2570
• remove flatbuffers from lua by @serprex in #2571
• mysql: test JSON by @serprex in #2567
• feat(alerting): classify all ssh errors as user errors by @iamKunalGupta in #2573
• [mysql] container with config for testing by @heavycrystal in #2564
• mysql: transactional consistency by @serprex in #2502
• TLS 1.3 by @serprex in #2576
• Change PEERDB_CLICKHOUSE_BINARY_FORMAT default to raw by @serprex in #2554
• [mysql] validate flavor setting by @heavycrystal in #2581
• PEERDB_BIGQUERY_TOAST_MERGE_CHUNKING by @serprex in #2568
• feat(alerting): classify all other mv related error correctly by @iamKunalGupta in #2579
• split out catalog errors as a separate class by @serprex in #2580
• feat: upgrade docker-compose.yml stable image tags by @github-actions in #2582
• chore(deps): update dockerfile dependencies by @renovate in #2585
• chore(deps): update docker-compose dependencies by @renovate in #2584
• chore(deps): update github-actions dependencies by @renovate in #2586
• fix(deps): update npm dependencies by @renovate in #2588
• fix(deps): update cargo dependencies by @renovate in #2587
• fix(deps): update rust crate uuid to v1.13.2 by @renovate in #2590
• Fix LoggerFromCtx to include flowName by @iskakaushik in #2531
• feat(alerting): classify invalid snapshot identifier by @iamKunalGupta in #2577
• chore: update go dependencies by @serprex in #2589
• feat: add alerts on connector setup errors by @iamKunalGupta in #2591
• mysql: pretty size by @serprex in #2583
• feat(flow-context): add current flowStatus and resync information by @iamKunalGupta in #2595
• refactor(alert): reclassify invalid snapshot identifier by @iamKunalGupta in #2599
• feat(maintenance): improvements part 1 by @iamKunalGupta in #2596
• mysql: ServerName by @serprex in #2600
• chore: rename invalid snapshot error by @iamKunalGupta in #2601
• [mysql] move PrettyBytes to mysql directory by @heavycrystal in #2593
• move code relying on generated/protos from shared to internal by @serprex in #2598
• minor lint miss by @heavycrystal in #2603
• merge peerdbenv into internal by @serprex in #2604
• feat(flow-api): add grpc server metrics by @iamKunalGupta in #2606
• cdc flow: increase timeout on repeated errors by @serprex in #2608
• mysql: one connection by @serprex in #2607
• Cleanup connpostgres usage outside connpostgres by @serprex in #2605
• primary key update: skip empty _peerdb_match_data rows by @serprex in #2615
• refactor(metrics): add api metrics renaming and include temporal client metrics by @iamKunalGupta in #2614
• Classify normalization errors specially by @serprex in #2612
• mysql: optimize GetVersion by @serprex in #2616
• feat(classifier): add a new class of retry recoverable error by @iamKunalGupta in #2610
• feat: add more information to metrics by @iamKunalGupta in #2617
• also prefer async function to top level async lambda by @serprex in #2611
• UI: Reflect column and engine values for existing tables by @Amogh-Bharadwaj in #2444
• refactor: conn reset to conn temp and classify err:NetClosed by @iamKunalGupta in #2622
• mysql: avoid interface{} in QValueFromMysqlFieldValue by @serprex in #2624
• [clickhouse] fix couple misses for retryable exec by @heavycrystal in #2625
• fix(deps): update npm dependencies by @renovate in #2629
• chore(deps): update docker-compose dependencies by @renovate in #2626
• fix(deps): update cargo dependencies by @renovate in #2628
• chore(deps): update github-actions dependencies by @renovate in #2627
• chore(deps): update dockerfile dependencies by @renovate in #2630
• fix(deps): update cargo dependencies by @renovate in #2631
• mysql: context by @serprex in #2619
• Enable generic schema changes for pg to clickhouse by @serprex in #2632

Full Changelog: v0.25.1...v0.25.2

v0.25.1

What's Changed

• ui: no session when no password by @serprex in #2542
• feat(disk-spill): add option to disable disk spill based on record count by @iamKunalGupta in #2550
• mysql: map tinyint(1) to bool by @serprex in #2543
• mysql: test more time types by @serprex in #2552
• decouple pg & ssh by @serprex in #2556
• feat(classifier): ignore notifying user for connection resets by @iamKunalGupta in #2559
• mysql: ssh by @serprex in #2558
• feat(classifier): add too many connections by @iamKunalGupta in #2560
• feat(alerts): add chcode to tags by @iamKunalGupta in #2561
• alerting: use ch-go/pgerrcode constants by @serprex in #2563

Full Changelog: v0.25.0...v0.25.1

v0.25.0

What's Changed

• golangci-lint 1.63 by @serprex in #2510
• make pkey error message more informative by @iskakaushik in #2516
• We don't need to compare table structure during resync by @iskakaushik in #2517
• [mysql] disable MariaDB GTID for now by @heavycrystal in #2518
• nexus: remove serde bytes by @serprex in #2511
• chore(deps): update lukemathwalker/cargo-chef:latest-rust-alpine docker digest to c1d8869 by @renovate in #2520
• feat: upgrade docker-compose.yml stable image tags by @github-actions in #2519
• fix(deps): update cargo dependencies by @renovate in #2521
• chore(deps): update github-actions dependencies by @renovate in #2523
• fix(deps): update npm dependencies by @renovate in #2522
• mysql UI by @serprex in #2509
• mysql: SignalAsEmpty/SignalAsNotEmpty by @serprex in #2507
• chore: move oom-mv to oom category by @iamKunalGupta in #2524
• [chore] update dependencies by @serprex in #2526
• feat(record-slot-size): increase frequency to 2 mins by @iamKunalGupta in #2527
• mysql: unsigned integers by @serprex in #2513
• move mirror validation logic into connector interfaces by @serprex in #2525
• give up dropping mirror source or destination after 50 tries, send alert by @serprex in #2528
• QRep Initial Copy Only: set status to completed by @Amogh-Bharadwaj in #2532
• fix: limit partition count to 1000 for temporal platform by @iskakaushik in #2530
• Remove consistency sleep and account for nil env by @Amogh-Bharadwaj in #2529
• ui: less lambda by @serprex in #2533
• mysql column exclusion by @serprex in #2512
• feat(pgcode-28000): classify as error notify connectivity by @iamKunalGupta in #2535
• feat(classifier): add pg insufficient_privilege by @iamKunalGupta in #2536
• schema changes: don't ignore nullable enabled by @serprex in #2537
• mysql: use information schema for schema because decimal info not in mysql.Field by @serprex in #2514
• MaintainTx: Add heartbeat routine by @Amogh-Bharadwaj in #2541
• cdc/qrep workflows: continue as new after setting state to completed by @serprex in #2540

Full Changelog: v0.24.0...v0.25.0

v0.24.0

• ⚡ PeerDB now pulls from Postgres and pushes to ClickHouse in parallel, by default
• 🔧 Introduced knobs to tune memory usage of PeerDB inserts to ClickHouse via chunking #2484 and #2485
• 📊 Added collection of more metrics such as errors emitted and rows synced #2454
• ✅ Improved validation for read replica Postgres instances below PG16 #2483
• 🚀 Re-architected parallelized syncs to ClickHouse and single table sync performance ttps://github.com//pull/2480

What's Changed

• feat: add more metrics by @iamKunalGupta in #2454
• PEERDB_CLICKHOUSE_NORMALIZATION_PARTS by @serprex in #2478
• mysql: GetVersionConnector by @serprex in #2477
• tailwindcss v4 by @serprex in #2481
• Make default channel size for normalize batches as 128 by @iskakaushik in #2482
• Drop mirror: introduce opt-in setting to skip destination dropping by @Amogh-Bharadwaj in #2455
• [validate] check if we're in recovery below PG16 for CDC by @heavycrystal in #2483
• Turn on PEERDB_PARALLEL_SYNC_NORMALIZE by default by @serprex in #2484
• Fix chunking with enabled primary key updates by @serprex in #2480
• PEERDB_CLICKHOUSE_INITIAL_LOAD_PARTS_PER_PARTITION by @serprex in #2485
• cdc flow: only sleep for 10 minutes for panics & application errors by @serprex in #2443
• always spawn heartbeatRoutine asap in activity by @serprex in #2490
• feat: upgrade docker-compose.yml stable image tags by @github-actions in #2486
• remove ureq by @serprex in #2495
• nexus: remove async-recursion by @serprex in #2496
• nexus: consistent binary serialization by @serprex in #2497
• Edit peer button: separate peer type fetch by @Amogh-Bharadwaj in #2499
• fix ui charts by @serprex in #2501
• mysql by @serprex in #2395
• feat: add context propagator and use it to enrich metrics with flow metadata by @iamKunalGupta in #2493
• alerting: tag mysql errors by @serprex in #2508
• feat: add alert classifier by @iamKunalGupta in #2494

Full Changelog: v0.23.0...v0.24.0

v0.23.0

This is a major release which captures all the improvements in the last month. Below are the list of the main features:

📦 Granular formats sync of binary and hex data from Postgres to ClickHouse.
🚀 Improved throughput for CDC by avoiding reconnections to support enterprise workloads.
🔁 Improved performance by using multiple replicas for data ingestion in ClickHouse.
⏳ Drastically reduced slot growth on idle databases by frequent acking on PKMs.
📊 Added internal endpoints for additional telemetry, including bytes moved and so on.
✅ Added additional validation to detect connection poolers.
🗑️ Improved reliability for Resyncs by removing capture of soft-deleted rows.
🛠️ Richer datatype support, including UUID arrays and tstzrange.
🔧 Revamped retry logic to better identify actual errors.
🧪 Enhanced end-to-end tests for various cases, such as partitioned tables.

What's Changed In This Release

Additional changes in this release:

• send flow name with batch id metrics by @serprex in #2451
• LSN Fetch for stats: Make it non-fatal for null LSN by @Amogh-Bharadwaj in #2396
• getCurrentLsn should be able to return null to caller without error by @serprex in #2460
• monitoring: cleanup code, use transactions by @serprex in #2459
• call SyncFlowCleanup during SetupFlow by @heavycrystal in #2464
• Remove slot signal by @serprex in #2452
• e2e: SuiteSource by @serprex in #2473

Full Changelog: v0.22.5...v0.23.0

v0.22.5

What's Changed

• improve error logging for query execution by @iskakaushik in #150
• revert limit 0 by @iskakaushik in #152
• fix clippy errors introduced by latest Rust update by @serprex in #2442
• fail faster when publication does not exist while setting up replication by @serprex in #2446
• nexus: CREATE PEER should not include quotes in name by @serprex in #2445
• normalize: slow down retries by @serprex in #2449

Full Changelog: v0.22.4...v0.22.5

v0.22.4

What's Changed

• changed filename from generate_protos.sh to generate-protos.sh for consistency with readme by @nirnayroy in #2427
• feat: upgrade docker-compose.yml stable image tags by @github-actions in #2431
• chore(deps): update github-actions dependencies by @renovate in #2433
• chore(deps): update dockerfile dependencies by @renovate in #2432
• chore(deps): update postgres:17-alpine docker digest to 0bcc5bb by @renovate in #2434
• fix(deps): update cargo dependencies by @renovate in #2435
• fix(deps): update npm dependencies by @renovate in #2438
• remove usehooks-ts by @serprex in #2436
• refactor: revert to using observables modified to sync gauges by @iamKunalGupta in #2421
• chore(deps): update dependency postcss to v8.5.0 by @renovate in #2439
• chore(deps): update temporalio/ui docker tag to v2.34.0 by @renovate in #2440

New Contributors

• @nirnayroy made their first contribution in #2427

Full Changelog: v0.22.3...v0.22.4

v0.22.3

What's Changed

• support tls_cert/tls_key in nexus by @serprex in #2355
• revive custom sync test by @serprex in #2419
• Edit some logs by @Amogh-Bharadwaj in #2410
• cdc flow: continue as new after state updates by @serprex in #2420
• fix: space pkm response to max once per minute by @iamKunalGupta in #2422
• fix: use condition instead of coalesce for updates on cdc_batches by @iamKunalGupta in #2423
• refactor(pkm-throttle): change ff to throttle seconds and enable by default by @iamKunalGupta in #2425

Full Changelog: v0.22.2...v0.22.3

v0.22.2

What's Changed

• Increase retry initial backoff to 1 minute by @iskakaushik in #2418

Full Changelog: v0.22.1...v0.22.2