v2.0.1

Please be advised that we have withdrawn version v2.0.0 due to a critical bug affecting upgrades of RW clusters from older versions. We sincerely apologize for the inconvenience.

The PRs for the bug fix are:
#18589
#18581

For installation and running instructions, see Get started.

Main changes

SQL features

• Query syntax:
  • Public preview: Supports AS CHANGELOG to convert any stream into an append-only changelog. #17132
  • Supports time travel query to access historical data at a specific point in time. #17665, #17621.
  • Supports CORRESPONDING specification in set operations. #17891
• SQL commands:
  • Breaking change: DECLARE cursor_name SUBSCRIPTION CURSOR is the same as DECLARE cursor_name SUBSCRIPTION CURSOR since now(), which will be consumed from the current time. DECLARE cursor_name SUBSCRIPTION CURSOR FULL will start consuming data from stock. The type of operation has changed to varchar. It is one of Insert, Delete, UpdateInset, or UpdateDelete. #18217
  • Public preview: Supports managing database credentials securely. #17456
  • SHOW CURSORS will return all query cursors in the current session. SHOW SUBSCRIPTION CURSORS will return all subscription cursorS and subscriptions in the current session. #18217
  • Allows ALTER TABLE on tables with generated columns. #17652
  • Allows dropping generated columns from tables created with a schema registry. #17689
  • Supports using scalar functions with list inputs as aggregate functions. #17622
  • Supports altering the backfill rate limit for materialized views. #17911
• SQL functions & operators:
  • Public preview: Supports approx_percentile(). #17814, #17873.
  • Public preview: Supports native map type and related functions for map, and ingesting AVRO MAP type into RisingWave map type. #17986
  • Public preview: Supports scanning a directory of parquet files. #17811
  • Supports pg_index_column_has_property() to query index column properties. #17275
  • Supports continuous timestamp generation in streaming mode. #17371
  • Supports acosd(). #9876
  • Supports function rw_recovery_status() and pg_is_in_recovery() to retrieve the meta node status. #17641
• System catalog:
  • Adds append_only column in RisingWave catalogs rw_tables and rw_materialized_views. #17598
  • Adds RisingWave catalog rw_catalog.rw_secrets. #17726

Connectors

• Public preview: Supports ingesting Avro map type for source connectors. ****#17980
• Public preview: Supports encoding parquet for file source. #17201
• Public preview: Supports batch reading S3 Parquet files. #17625, #17673.
• Supports AWS Glue schema registry with aws.glue.schema_arn parameter. #17605
• Supports creating tables and sources with format upsert encode protobuf. #17624
• Supports ingesting Avro Union type for source connectors. #17485
• Supports reading files compressed in gzip format. #16538
• Adds the option to use a semicolon as the delimiter for CSV encode. #17356
• Uses OpenDAL to connect to S3 object store state backend. #18011
• Public preview: Supports replicating DDL for MySQL CDC source. #17876
• Supports parameter refresh.interval.sec option for S3, GCS, and POSIX sources. #18184
• Supports parameter group.id.prefix for Kafka sources. #18115
• Validates slot name of PostgreSQL CDC sources. #17949
• Supports altering backfill_rate_limit of CDC tables. #17989
• Public preview: Supports sinking data to file systems in parquet format. #17311
• Supports upsert Protobuf type sinks, which requires KEY ENCODE TEXT. #18024
• Adds option jsonb.handling.mode under WITH options for sinks in JSON format. #17693
• Public preview: Supports Azure Blob sinks. #18244
• Public preview: Supports MongoDB sinks. #17102
• Supports Azure Blob file sources. #18295
• Supports glue catalog for iceberg sink and source. #17477
• Adds jdbc.query.timeout for JDBC sinks to set the timeout for queries. #18430
• Changes default Kafka sink message timeout from five seconds to five minutes. #18304
• Adds new parameters retry_on_conflict, batch_size_kb, batch_num_messages, and concurrent_requests for ElasticSearch sink.#17867
• Supports parameter bigquery.retry_times for BigQuery sink. #17237
• Supports parameter bigquery.auto_create_table for BigQuery sink. #17393
• Supports parameter doris.partial_columns for Doris sink. #16821
• Supports ClickHouse sink checkpoint decouple. #17491
• Sets sink decouple as default for all sinks. #18182
• Uses S3's SQS notification to complete the import of data from Snowflake instead of the Snowflake HTTP client. #17627
• Ensures at-least-once delivery semantic and eventual consistency for Kinesis sink. #17983
• Supports backfilling by consuming a fixed snapshot of upstream table and then the upstream data epoch by epoch. #17735

Installation and deployment

• Supports configuring the SQL metastore using username, password, and database separately. #17530
• Supports more seamless scaling-in in Kubernetes deployments. #17802

Cluster configuration changes

• Breaking change: Refactors streaming_rate_limit into source_rate_limit and backfill_rate_limit. #17796
• Breaking change: Adds a default soft and hard limit on actor count per worker parallelism. When the hard limit is reached, streaming workloads will fail. #18383
• Introduces batch.developer.exchange_connection_pool_size and streaming.developer.exchange_connection_pool_size to configure streaming and batch remote exchange between two nodes #17768
• Introduces system parameter license_key used to enable enterprise features. #17396

Fixes

• Deletes related cursors when deleting a subscription. #17232

Full Changelog: v1.10.1...v2.0.1

v1.10.2

release v1.10.2

v2.0.0-rc.1

For installation and running instructions, see Get started.

Main changes

SQL features

• Query syntax:
  • Public preview: Supports AS CHANGELOG to convert any stream into an append-only changelog. #17132
  • Supports time travel query to access historical data at a specific point in time. #17665, #17621.
  • Supports CORRESPONDING specification in set operations. #17891
• SQL commands:
  • Breaking change: DECLARE cursor_name SUBSCRIPTION CURSOR is the same as DECLARE cursor_name SUBSCRIPTION CURSOR since now(), which will be consumed from the current time. DECLARE cursor_name SUBSCRIPTION CURSOR FULL will start consuming data from stock. The type of operation has changed to varchar. It is one of Insert, Delete, UpdateInset, or UpdateDelete. #18217
  • Public preview: Supports managing database credentials securely. #17456
  • SHOW CURSORS will return all query cursors in the current session. SHOW SUBSCRIPTION CURSORS will return all subscription cursorS and subscriptions in the current session. #18217
  • Allows ALTER TABLE on tables with generated columns. #17652
  • Allows dropping generated columns from tables created with a schema registry. #17689
  • Supports using scalar functions with list inputs as aggregate functions. #17622
  • Supports altering the backfill rate limit for materialized views. #17911
• SQL functions & operators:
  • Public preview: Supports approx_percentile(). #17814, #17873.
  • Public preview: Supports native map type and related functions for map, and ingesting AVRO MAP type into RisingWave map type. #17986
  • Public preview: Supports scanning a directory of parquet files. #17811
  • Supports pg_index_column_has_property() to query index column properties. #17275
  • Supports continuous timestamp generation in streaming mode. #17371
  • Supports acosd(). #9876
  • Supports function rw_recovery_status() and pg_is_in_recovery() to retrieve the meta node status. #17641
• System catalog:
  • Adds append_only column in RisingWave catalogs rw_tables and rw_materialized_views. #17598
  • Adds RisingWave catalog rw_catalog.rw_secrets. #17726

Connectors

• Public preview: Supports ingesting Avro map type for source connectors. ****#17980
• Public preview: Supports encoding parquet for file source. #17201
• Public preview: Supports batch reading S3 Parquet files. #17625, #17673.
• Supports AWS Glue schema registry with aws.glue.schema_arn parameter. #17605
• Supports creating tables and sources with format upsert encode protobuf. #17624
• Supports ingesting Avro Union type for source connectors. #17485
• Supports reading files compressed in gzip format. #16538
• Adds the option to use a semicolon as the delimiter for CSV encode. #17356
• Uses OpenDAL to connect to S3 object store state backend. #18011
• Public preview: Supports replicating DDL for MySQL CDC source. #17876
• Supports parameter refresh.interval.sec option for S3, GCS, and POSIX sources. #18184
• Supports parameter group.id.prefix for Kafka sources. #18115
• Validates slot name of PostgreSQL CDC sources. #17949
• Supports altering backfill_rate_limit of CDC tables. #17989
• Public preview: Supports sinking data to file systems in parquet format. #17311
• Supports upsert Protobuf type sinks, which requires KEY ENCODE TEXT. #18024
• Adds option jsonb.handling.mode under WITH options for sinks in JSON format. #17693
• Public preview: Supports Azure Blob sinks. #18244
• Public preview: Supports MongoDB sinks. #17102
• #18295
• Supports glue catalog for iceberg sink and source. #17477
• #18304
• Supports parameter bigquery.retry_times for BigQuery sink connector. #17237
• Supports parameter bigquery.auto_create_table for BigQuery sink. #17393
• Supports parameter doris.partial_columns for Doris sink connector. #16821
• Supports ClickHouse sink checkpoint decouple. #17491
• Sets sink decouple as default for all sinks. #18182
• Uses S3's SQS notification to complete the import of data from Snowflake instead of the Snowflake HTTP client. #17627
• Ensures at-least-once delivery semantic and eventual consistency for Kinesis sink. #17983
• Supports backfilling by consuming a fixed snapshot of upstream table and then the upstream data epoch by epoch. #17735

Installation and deployment

• Supports configuring the SQL metastore using username, password, and database separately. #17530
• Supports more seamless scaling-in in Kubernetes deployments. #17802

Cluster configuration changes

• Breaking change: Refactors streaming_rate_limit into source_rate_limit and backfill_rate_limit. #17796
• Introduces batch.developer.exchange_connection_pool_size and streaming.developer.exchange_connection_pool_size to configure streaming and batch remote exchange between two nodes #17768
• Introduces system parameter license_key used to enable enterprise features. #17396

Fixes

• Deletes related cursors when deleting a subscription. fix(subscription): fix drop subscription not clear cursor

Full Changelog: v1.10.1...v2.0.0-rc.1

v1.10.1

release v1.10.1

v1.10.0

For installation and running instructions, see Get started.

Main changes

SQL features

• SQL commands:
  • Supports specifying the authorization and omitting the schema name when creating a schema. #16806
  • Supports session window in batch and emit-on-window-close mode. #17098
  • Supports fetching multiple rows from subscription cursor. #16764
• SQL functions & operators:
  • Supports has_table_privilege(), has_schema_privilege(), and has_any_column_privilege(). #16674
  • Supports quote_literal() and quote_nullable(). #16807
  • Supports pg_get_keywords(). #17033
  • Supports jsonb_set(). #17124
  • Supports user-defined aggregate functions for embedded Python and JavaScript UDFs. #16874
  • Allows the specified time zone of AT TIME ZONE to be non-literal. #17395
• System catalog:
  • Supports rw_catalog.actor_id_to_ddl and rw_catalog.fragment_id_to_ddl. #17229

Connectors

• Avro schemas with "default": "NaN" and positive and negative infinities, are supported as float and double types. #17309
• Supports ingesting simple AVRO MAP types as JSONB. #16948
• Supports ingesting avro uuid types as varchar. #17069
• Supports ingesting avro with internal Ref types. #17052
• Adds aws prefix to AWS related parameters when creating a source or sink. #16671
• Supports using AWS IAM to connect to Amazon MSK. #16625
• Adds pubsub.parallelism under the WITH option for Google PubSub source. #16733
• Supports INCLUDE TIMESTAMP [AS] clause for MySQL, PostgreSQL, and MongoDB CDC tables.#16833
• Supports additional metadata columns for CDC tables. #17051
• Automatically maps upstream table schema when creating MySQL and PostgreSQL tables. #16986
• Sets a network timeout for JDBC sink connections. #17244
• Enables sink decouple by default for Kafka, Kinesis, Pulsar, Google Pub/Sub, NATS, MQTT, ClickHouse sinks. #17221
• Supports the KEY ENCODE clause when creating a sink. #16377
• Supports FORMAT PLAIN ENCODE AVRO for Kafka sinks. #17216
• Supports DynamoDB sink. #16670
• Supports Microsoft SQL Server sinks for self-hosted SQL Server and Azure SQL. #17154
• Supports OpenSearch sink. #16330
• Supports checkpoint decouple for StarRocks sinks. #16816
• Supports checkpoint decouple for Delta Lake sinks. #16777
• Supports sinking serial types. #16969

Cluster configuration changes

• Sets arrangement backfill as the default. #14846
• Supports spill hash join to avoid OOM issue. #17122
• Supports spill hash aggregation for batch queries. #16771
• Changes the algorithm that calculates the reserve memory size. #16992

Fixes

• Improves error message and location of cursor. #16959
• Improves error message when trying to create CDC source with columns. #16636
• Allows GRANT and REVOKE privileges on views. #16699

Full Changelog: v1.9.1...v1.10.0

v1.10.0-rc.3

release v1.10.0-rc.3

v1.10.0-rc.2

release v1.10.0-rc.2

v1.10.0-rc.1

release v1.10.0-rc.1

v1.9.2

release v1.9.2

v1.9.1

For installation and running instructions, see Get started.

Main changes

SQL features

• Query syntax:
  • Supports non-append-only temporal joins, where the outer side is not required to be append-only. #16286
• SQL commands:
  • Supports DISCARD ALL command for Npgsql compatibility. #16432
  • Supports creating, dropping, altering, and showing subscriptions. #14831
  • Supports cursors for subscription queues. #15180
  • Supports altering stream rate limit for sources and tables with a source. #16399
  • Supports RECOVER command to trigger an ad-hoc recovery. #16259
• SQL functions & operators:
  • Supports using a single ASCII character as an escape character with the LIKE clause. #16057
  • Supports jsonb_populate_record() and jsonb_populate_recordset(). #13421
• System catalog:
  • Supports system table rw_iceberg_files for displaying the files of an Iceberg source or table. #16180
  • Supports system table rw_iceberg_snapshot for listing all snapshots. #16175

Connectors

• Provides stable support for SQLAlchemy 2.0. risingwavelabs/sqlalchemy-risingwave#29
• Deprecates s3 connector. #16337
• Supports generated columns for non-shared CDC tables. #16522
• Supports time travel for Iceberg sources. #15866
• Blocks sink creation until backfill is completed by default. #16249
• Supports Kafka connector parameter properties.request.required.acks. #16482
• Adds connector parameter ssl.mode for PostgreSQL and Neon source connector. #15690
• Adds connector parameter ssl.mode for MySQL source connector. #16579
• Supports parameters snapshot.interval and snapshot.batch_size under WITH options when creating a table from a CDC source. #16426
• Supports implicitly converting numeric types from PostgreSQL sources into rw_int256 or varchar. #16346
• Supports configuring the timeout of CDC sources. #16598
• Supports timestamptz.handling.mode formatting option when creating a source with PLAIN, UPSERT, or DEBEZIUM JSON formats. #16265
• Only uses fragment ID as group ID for Kafka sources. #16111
• Supports cluster URLs for Redis sink connector. #16034
• Supports creating Delta sinks with GCS. #16182
• Supports Snowflake sink connector. #15429
• Supports creating upsert type BigQuery sinks. #15780

Installation and deployment

• Sets PostgreSQL as the default meta store when deploying with Docker Compose. #16724

Cluster configuration changes

• Supports using ALTER SYSTEM to set a system-wide default value for a session parameter. #16062
• Supports setting streaming_rate_limit to zero. #16333
• Supports configuring the reserved memory bytes of the compute node by using RW_RESERVED_MEMORY_BYTES runtime parameter and reserved-memory-bytes startup option. #16433
• Introduce new timeout and retry configurations for ObjectStore and deprecate ambiguous timeout configurations. #16231

Fixes

• Properly convert -inf, +inf, and nan types to null for JDBC sinks. #16230
• Handles sinking -inf, +inf, and nan types for ClickHouse, Doris, and StarRocks sink connectors. #15664
• Fixes an issue where DELETE events could not be sinked if the primary key is uuid type for JDBC sinks.#16447
• Fixes an issue where enum types from PostgreSQL could not be ingested as varchar types. #16423
• Fixes sources with encode avro on decimal ingesting. #16202
• Fixes sources with encode avro on bytes/fixed/decimal default value. #16414

Full Changelog: v1.8.0...v1.9.1