Merge branch 'main' into patch-2

CHANGELOG.md

@@ -1,3 +1,90 @@
+### citus v12.0.0 (July 11, 2023) ###
+
+* Adds support for schema-based sharding.
+  While `citus.enable_schema_based_sharding` GUC allows sharding the database
+  based on newly created schemas, `citus_schema_distribute()` allows doing so
+  for the existing schemas. Distributed schemas used for sharding the database
+  can be listed by using the view `citus_schemas`, monitored by using the view
+  `citus_stat_schemas`, and undistributed by using the udf
+  `citus_schema_undistribute()`
+  (#6866, #6979, #6933, #6936 and many others)
+
+* Supports MERGE command across non-colocated distributed tables/subqueries,
+  reference tables and joins on non-distribution columns (#6927)
+
+* Drops PG13 Support (#7002, #7007)
+
+* Changes default rebalance strategy to by_disk_size (#7033)
+
+* Changes by_disk_size rebalance strategy to have a base size (#7035)
+
+* Improves citus_tables view performance (#7018)
+
+* Improves tenant monitoring performance (#6868)
+
+* Introduces the GUC `citus.stat_tenants_untracked_sample_rate` for sampling in
+  tenant monitoring (#7026)
+
+* Adds CPU usage to citus_stat_tenants (#6844)
+
+* Propagates `ALTER SCHEMA .. OWNER TO ..` commands to worker (#6987)
+
+* Allows `ADD COLUMN` in command string with other commands (#7032)
+
+* Allows `DROP CONSTRAINT` in command string with other commands (#7012)
+
+* Makes sure to properly handle index storage options for `ADD CONSTRAINT
+  `/ COLUMN commands (#7032)
+
+* Makes sure to properly handle `IF NOT EXISTS` for `ADD COLUMN` commands
+  (#7032)
+
+* Allows using generated identity column based on int/smallint when creating
+  a distributed table with the limitation of not being able perform DMLs on
+  identity columns from worker nodes (#7008)
+
+* Supports custom cast from / to timestamptz in time partition management UDFs
+  (#6923)
+
+* Optimizes pushdown planner on memory and cpu (#6945)
+
+* Changes citus_shard_sizes view's table_name column to shard_id (#7003)
+
+* The GUC search_path is now reported when it is updated (#6983)
+
+* Disables citus.enable_non_colocated_router_query_pushdown GUC by default to
+  ensure generating a consistent distributed plan for the queries that
+  reference non-colocated distributed tables (#6909)
+
+* Disallows MERGE with filters that prune down to zero shards (#6946)
+
+* Makes sure to take `shouldhaveshards` setting into account for a node when
+  planning rebalance steps (#6887)
+
+* Improves the compatibility with other extension by forwarding to existing
+  emit_log_hook in our log hook (#6877)
+
+* Fixes wrong result when using `NOT MATCHED` with MERGE command (#6943)
+
+* Fixes querying the view `citus_shard_sizes` when there are too many shards
+  (#7018)
+
+* Fixes a bug related to type casts from other types to text/varchar (#6391)
+
+* Fixes propagating `CREATE SCHEMA AUTHORIZATION ..` with no schema name
+  (#7015)
+
+* Fixes an error when creating a FOREIGN KEY without a name referencing a schema
+  qualified table (#6986)
+
+* Fixes a rare bug which mostly happens with queries that contain both outer
+  join and where clauses  (#6857)
+
+* Fixes a bug related to propagation of schemas when pg_dist_node is empty
+  (#6900)
+
+* Fixes a crash when a query is locally executed with explain analyze (#6892)
+
 ### citus v11.3.0 (May 2, 2023) ###
 
 * Introduces CDC implementation for Citus using logical replication

README.md

@@ -1,4 +1,4 @@
-| **<br/>The Citus database is 100% open source.<br/><img width=1000/><br/>Learn what's new in the [Citus 11.3 release blog](https://www.citusdata.com/blog/2023/05/05/whats-new-in-citus-11-3-multi-tenant-saas/) and the [Citus Updates page](https://www.citusdata.com/updates/).<br/><br/>**|
+| **<br/>The Citus database is 100% open source.<br/><img width=1000/><br/>Learn what's new in the [Citus 12.0 release blog](https://www.citusdata.com/blog/2023/07/18/citus-12-schema-based-sharding-comes-to-postgres/) and the [Citus Updates page](https://www.citusdata.com/updates/).<br/><br/>**|
 |---|
 <br/>
 
@@ -8,7 +8,7 @@
 
 [![Latest Docs](https://img.shields.io/badge/docs-latest-brightgreen.svg)](https://docs.citusdata.com/)
 [![Stack Overflow](https://img.shields.io/badge/Stack%20Overflow-%20-545353?logo=Stack%20Overflow)](https://stackoverflow.com/questions/tagged/citus)
-[Slack](https://citus-public.slack.com/)
+[Slack](https://slack.citusdata.com/)
 [![Code Coverage](https://codecov.io/gh/citusdata/citus/branch/master/graph/badge.svg)](https://app.codecov.io/gh/citusdata/citus)
 [![Twitter](https://img.shields.io/twitter/follow/citusdata.svg?label=Follow%20@citusdata)](https://twitter.com/intent/follow?screen_name=citusdata)
 
@@ -38,6 +38,7 @@ Since Citus is an extension to Postgres, you can use Citus with the latest Postg
 - [Why Citus?](#why-citus)
 - [Getting Started](#getting-started)
 - [Using Citus](#using-citus)
+- [Schema-based sharding](#schema-based-sharding)
 - [Setting up with High Availability](#setting-up-with-high-availability)
 - [Documentation](#documentation)
 - [Architecture](#architecture)
@@ -94,14 +95,14 @@ Install packages on Ubuntu / Debian:
 ```bash
 curl https://install.citusdata.com/community/deb.sh > add-citus-repo.sh
 sudo bash add-citus-repo.sh
-sudo apt-get -y install postgresql-15-citus-11.3
+sudo apt-get -y install postgresql-15-citus-12.0
 ```
 
 Install packages on CentOS / Red Hat:
 ```bash
 curl https://install.citusdata.com/community/rpm.sh > add-citus-repo.sh
 sudo bash add-citus-repo.sh
-sudo yum install -y citus113_15
+sudo yum install -y citus120_15
 ```
 
 To add Citus to your local PostgreSQL database, add the following to `postgresql.conf`:
@@ -347,6 +348,45 @@ When using columnar storage, you should only load data in batch using `COPY` or
 
 To learn more about columnar storage, check out the [columnar storage README](https://github.com/citusdata/citus/blob/master/src/backend/columnar/README.md).
 
+## Schema-based sharding
+
+Available since Citus 12.0, [schema-based sharding](https://docs.citusdata.com/en/stable/get_started/concepts.html#schema-based-sharding) is the shared database, separate schema model, the schema becomes the logical shard within the database. Multi-tenant apps can a use a schema per tenant to easily shard along the tenant dimension. Query changes are not required and the application usually only needs a small modification to set the proper search_path when switching tenants. Schema-based sharding is an ideal solution for microservices, and for ISVs deploying applications that cannot undergo the changes required to onboard row-based sharding.
+
+### Creating distributed schemas
+
+You can turn an existing schema into a distributed schema by calling `citus_schema_distribute`:
+
+```sql
+SELECT citus_schema_distribute('user_service');
+```
+
+Alternatively, you can set `citus.enable_schema_based_sharding` to have all newly created schemas be automatically converted into distributed schemas:
+
+```sql
+SET citus.enable_schema_based_sharding TO ON;
+
+CREATE SCHEMA AUTHORIZATION user_service;
+CREATE SCHEMA AUTHORIZATION time_service;
+CREATE SCHEMA AUTHORIZATION ping_service;
+```
+
+### Running queries
+
+Queries will be properly routed to schemas based on `search_path` or by explicitly using the schema name in the query.
+
+For [microservices](https://docs.citusdata.com/en/stable/get_started/tutorial_microservices.html) you would create a USER per service matching the schema name, hence the default `search_path` would contain the schema name. When connected the user queries would be automatically routed and no changes to the microservice would be required.
+
+```sql
+CREATE USER user_service;
+CREATE SCHEMA AUTHORIZATION user_service;
+```
+
+For typical multi-tenant applications, you would set the search path to the tenant schema name in your application:
+
+```sql
+SET search_path = tenant_name, public;
+```
+
 ## Setting up with High Availability
 
 One of the most popular high availability solutions for PostgreSQL, [Patroni 3.0](https://github.com/zalando/patroni), has [first class support for Citus 10.0 and above](https://patroni.readthedocs.io/en/latest/citus.html#citus), additionally since Citus 11.2 ships with improvements for smoother node switchover in Patroni.
@@ -414,6 +454,8 @@ Citus is uniquely capable of scaling both analytical and transactional workloads
 
   Example multi-tenant SaaS users: [Copper](https://www.citusdata.com/customers/copper), [Salesloft](https://fivetran.com/case-studies/replicating-sharded-databases-a-case-study-of-salesloft-citus-data-and-fivetran), [ConvertFlow](https://www.citusdata.com/customers/convertflow)
 
+- **[Microservices](https://docs.citusdata.com/en/stable/get_started/tutorial_microservices.html)**: Citus supports schema based sharding, which allows distributing regular database schemas across many machines. This sharding methodology fits nicely with typical Microservices architecture, where storage is fully owned by the service hence can’t share the same schema definition with other tenants. Citus allows distributing horizontally scalable state across services, solving one of the [main problems](https://stackoverflow.blog/2020/11/23/the-macro-problem-with-microservices/) of microservices.
+
 - **Geospatial**:
   Because of the powerful [PostGIS](https://postgis.net/) extension to Postgres that adds support for geographic objects into Postgres, many people run spatial/GIS applications on top of Postgres. And since spatial location information has become part of our daily life, well, there are more geospatial applications than ever. When your Postgres database needs to scale out to handle an increased workload, Citus is a good fit.
 

configure

@@ -1,6 +1,6 @@
 #! /bin/sh
 # Guess values for system-dependent variables and create Makefiles.
-# Generated by GNU Autoconf 2.69 for Citus 12.0devel.
+# Generated by GNU Autoconf 2.69 for Citus 12.1devel.
 #
 #
 # Copyright (C) 1992-1996, 1998-2012 Free Software Foundation, Inc.
@@ -579,8 +579,8 @@ MAKEFLAGS=
 # Identity of this package.
 PACKAGE_NAME='Citus'
 PACKAGE_TARNAME='citus'
-PACKAGE_VERSION='12.0devel'
-PACKAGE_STRING='Citus 12.0devel'
+PACKAGE_VERSION='12.1devel'
+PACKAGE_STRING='Citus 12.1devel'
 PACKAGE_BUGREPORT=''
 PACKAGE_URL=''
 
@@ -1262,7 +1262,7 @@ if test "$ac_init_help" = "long"; then
   # Omit some internal or obsolete options to make the list less imposing.
   # This message is too long to be a string in the A/UX 3.1 sh.
   cat <<_ACEOF
-\`configure' configures Citus 12.0devel to adapt to many kinds of systems.
+\`configure' configures Citus 12.1devel to adapt to many kinds of systems.
 
 Usage: $0 [OPTION]... [VAR=VALUE]...
 
@@ -1324,7 +1324,7 @@ fi
 
 if test -n "$ac_init_help"; then
   case $ac_init_help in
-     short | recursive ) echo "Configuration of Citus 12.0devel:";;
+     short | recursive ) echo "Configuration of Citus 12.1devel:";;
    esac
   cat <<\_ACEOF
 
@@ -1429,7 +1429,7 @@ fi
 test -n "$ac_init_help" && exit $ac_status
 if $ac_init_version; then
   cat <<\_ACEOF
-Citus configure 12.0devel
+Citus configure 12.1devel
 generated by GNU Autoconf 2.69
 
 Copyright (C) 2012 Free Software Foundation, Inc.
@@ -1912,7 +1912,7 @@ cat >config.log <<_ACEOF
 This file contains any messages produced by compilers while
 running configure, to aid debugging if configure makes a mistake.
 
-It was created by Citus $as_me 12.0devel, which was
+It was created by Citus $as_me 12.1devel, which was
 generated by GNU Autoconf 2.69.  Invocation command line was
 
   $ $0 $@
@@ -5393,7 +5393,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
 # report actual input values of CONFIG_FILES etc. instead of their
 # values after options handling.
 ac_log="
-This file was extended by Citus $as_me 12.0devel, which was
+This file was extended by Citus $as_me 12.1devel, which was
 generated by GNU Autoconf 2.69.  Invocation command line was
 
   CONFIG_FILES    = $CONFIG_FILES
@@ -5455,7 +5455,7 @@ _ACEOF
 cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
 ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`"
 ac_cs_version="\\
-Citus config.status 12.0devel
+Citus config.status 12.1devel
 configured by $0, generated by GNU Autoconf 2.69,
   with options \\"\$ac_cs_config\\"
 

configure.ac

@@ -5,7 +5,7 @@
 # everyone needing autoconf installed, the resulting files are checked
 # into the SCM.
 
-AC_INIT([Citus], [12.0devel])
+AC_INIT([Citus], [12.1devel])
 AC_COPYRIGHT([Copyright (c) Citus Data, Inc.])
 
 # we'll need sed and awk for some of the version commands

src/backend/distributed/citus.control

@@ -1,6 +1,6 @@
 # Citus extension
 comment = 'Citus distributed database'
-default_version = '12.0-1'
+default_version = '12.1-1'
 module_pathname = '$libdir/citus'
 relocatable = false
 schema = pg_catalog