From 31bc927c0a10a3fe280febb1098b16496262c6f6 Mon Sep 17 00:00:00 2001 From: Rich Loveland Date: Mon, 9 Dec 2024 16:28:19 -0500 Subject: [PATCH] Add zone config troubleshooting guide v1 Fixes DOC-9210 Summary of changes: - Add Zone Config troubleshooting guide - a.k.a. Chapter 3 of _The ZoneConfigonomicon (tm)_ (the existing 'Replication Controls' page is Chapter 1, and 'Zone Config Extensions' is Chapter 2) - Update 'Replication controls' page with more detailed info re: zone config inheritance hierarchy and behavior - Fix incorrect ALTER RANGE statements since they're needed to map range IDs from critical nodes endpoint (mentioned in troubleshooting guide) to actual schema objects - Add links from various zone config-related pages to the new troubleshooting guide --- .../see-zone-config-troubleshooting-guide.md | 1 + .../v24.3/sidebar-data/troubleshooting.json | 6 ++ src/current/v24.3/alter-database.md | 10 +++ src/current/v24.3/alter-index.md | 4 + src/current/v24.3/alter-partition.md | 8 ++ src/current/v24.3/alter-range.md | 14 +-- src/current/v24.3/alter-table.md | 2 + src/current/v24.3/backup.md | 1 + src/current/v24.3/cluster-api.md | 2 +- .../v24.3/cluster-setup-troubleshooting.md | 6 +- src/current/v24.3/common-errors.md | 3 +- .../v24.3/configure-replication-zones.md | 43 ++++++++-- ...emo-low-latency-multi-region-deployment.md | 4 +- .../v24.3/migrate-to-multiregion-sql.md | 1 + src/current/v24.3/monitoring-and-alerting.md | 3 +- src/current/v24.3/multiregion-overview.md | 1 + src/current/v24.3/node-shutdown.md | 2 +- src/current/v24.3/partitioning.md | 8 +- .../v24.3/query-replication-reports.md | 3 + src/current/v24.3/show-backup.md | 2 +- src/current/v24.3/show-create.md | 2 +- src/current/v24.3/show-zone-configurations.md | 1 + ...take-and-restore-locality-aware-backups.md | 2 +- .../v24.3/troubleshoot-replication-zones.md | 86 +++++++++++++++++++ src/current/v24.3/troubleshooting-overview.md | 4 +- src/current/v24.3/ui-debug-pages.md | 2 +- src/current/v24.3/zone-config-extensions.md | 1 + 27 files changed, 191 insertions(+), 31 deletions(-) create mode 100644 src/current/_includes/v24.3/see-zone-config-troubleshooting-guide.md create mode 100644 src/current/v24.3/troubleshoot-replication-zones.md diff --git a/src/current/_includes/v24.3/see-zone-config-troubleshooting-guide.md b/src/current/_includes/v24.3/see-zone-config-troubleshooting-guide.md new file mode 100644 index 00000000000..a3773ced626 --- /dev/null +++ b/src/current/_includes/v24.3/see-zone-config-troubleshooting-guide.md @@ -0,0 +1 @@ +For instructions showing how to troubleshoot replication zones, see [Troubleshoot Replication Zone Configurations]({% link {{ page.version.version}}/troubleshoot-replication-zones.md %}). diff --git a/src/current/_includes/v24.3/sidebar-data/troubleshooting.json b/src/current/_includes/v24.3/sidebar-data/troubleshooting.json index 739e9cfa5ef..e3373e8dc1b 100644 --- a/src/current/_includes/v24.3/sidebar-data/troubleshooting.json +++ b/src/current/_includes/v24.3/sidebar-data/troubleshooting.json @@ -56,6 +56,12 @@ "/${VERSION}/query-replication-reports.html" ] }, + { + "title": "Troubleshoot Replication Zone Configurations", + "urls": [ + "/${VERSION}/troubleshoot-replication-zones.html" + ] + }, { "title": "Benchmarking", "items": [ diff --git a/src/current/v24.3/alter-database.md b/src/current/v24.3/alter-database.md index 5f09b5d0cee..f13cc0e5dc3 100644 --- a/src/current/v24.3/alter-database.md +++ b/src/current/v24.3/alter-database.md @@ -715,6 +715,10 @@ You cannot `DISCARD` any zone configurations on multi-region tables, indexes, or ALTER DATABASE movr CONFIGURE ZONE DISCARD; ~~~ +#### Troubleshoot replication zones + +{% include {{ page.version.version }}/see-zone-config-troubleshooting-guide.md %} + ### Use Zone Config Extensions The following examples show: @@ -1078,6 +1082,12 @@ When you discard a zone configuration, the objects it was applied to will then i However, this statement will not remove any configuration created by the [multi-region abstractions]({% link {{ page.version.version }}/multiregion-overview.md %}). {{site.data.alerts.end}} +#### Troubleshoot Zone Config Extensions + +The process for troubleshooting Zone Config Extensions is the same as troubleshooting any other changes to zone configs. + +{% include {{ page.version.version }}/see-zone-config-troubleshooting-guide.md %} + ### Change database owner {% include {{page.version.version}}/sql/movr-statements.md %} diff --git a/src/current/v24.3/alter-index.md b/src/current/v24.3/alter-index.md index ff2106056e7..43e45e8a29d 100644 --- a/src/current/v24.3/alter-index.md +++ b/src/current/v24.3/alter-index.md @@ -225,6 +225,10 @@ You cannot `DISCARD` any zone configurations on multi-region tables, indexes, or ALTER INDEX vehicles@vehicles_auto_index_fk_city_ref_users CONFIGURE ZONE DISCARD; ~~~ +#### Troubleshoot replication zones + +{% include {{ page.version.version }}/see-zone-config-troubleshooting-guide.md %} + ### Define partitions #### Define a list partition on an index diff --git a/src/current/v24.3/alter-partition.md b/src/current/v24.3/alter-partition.md index d8b0c1b3479..1899441e103 100644 --- a/src/current/v24.3/alter-partition.md +++ b/src/current/v24.3/alter-partition.md @@ -9,6 +9,8 @@ docs_area: reference.sql To view details about existing replication zones, use [`SHOW ZONE CONFIGURATIONS`]({% link {{ page.version.version }}/show-zone-configurations.md %}). For more information about replication zones, see [Replication Controls]({% link {{ page.version.version }}/configure-replication-zones.md %}). +{% include {{ page.version.version }}/see-zone-config-troubleshooting-guide.md %} + You can use *replication zones* to control the number and location of replicas for specific sets of data, both when replicas are first added and when they are rebalanced to maintain cluster equilibrium. @@ -44,3 +46,9 @@ The user must have the [`CREATE`]({% link {{ page.version.version }}/grant.md %} ### Create a replication zone for a partition {% include {{ page.version.version }}/zone-configs/create-a-replication-zone-for-a-table-partition.md hide-enterprise-warning="true" %} + +## See also + +- [Table partitioning]({% link {{page.version.version}}/partitioning.md %}) +- [`SHOW PARTITIONS`]({% link {{page.version.version}}/show-partitions.md %}) +- [Troubleshoot Replication Zone Configurations]({% link {{ page.version.version}}/troubleshoot-replication-zones.md %}) diff --git a/src/current/v24.3/alter-range.md b/src/current/v24.3/alter-range.md index 051468f3da9..c3c0dfc6bca 100644 --- a/src/current/v24.3/alter-range.md +++ b/src/current/v24.3/alter-range.md @@ -34,9 +34,9 @@ Additional parameters are documented for the respective [subcommands](#subcomman ### `CONFIGURE ZONE` -`ALTER RANGE ... CONFIGURE ZONE` is used to add, modify, reset, or remove replication zones for a range. To view details about existing replication zones, see [`SHOW ZONE CONFIGURATIONS`]({% link {{ page.version.version }}/show-zone-configurations.md %}). +`ALTER RANGE ... CONFIGURE ZONE` is used to add, modify, reset, or remove [replication zones]({% link {{ page.version.version }}/configure-replication-zones.md %}) for a range. To view details about existing replication zones, see [`SHOW ZONE CONFIGURATIONS`]({% link {{ page.version.version }}/show-zone-configurations.md %}). -You can use *replication zones* to control the number and location of replicas for specific sets of data, both when replicas are first added and when they are rebalanced to maintain cluster equilibrium. +You can use replication zones to control the number and location of replicas for specific sets of data, both when replicas are first added and when they are rebalanced to maintain cluster equilibrium. #### Required privileges @@ -121,7 +121,7 @@ For example, to get all range IDs, leaseholder store IDs, and leaseholder locali {% include_cached copy-clipboard.html %} ~~~ sql -WITH user_info AS (SHOW RANGES FROM TABLE users) SELECT range_id, lease_holder, lease_holder_locality FROM user_info; +WITH user_info AS (SHOW RANGES FROM TABLE users WITH DETAILS) SELECT range_id, lease_holder, lease_holder_locality FROM user_info; ~~~ ~~~ @@ -163,7 +163,7 @@ To move the leases for all data in the [`movr.users`]({% link {{ page.version.ve {% include_cached copy-clipboard.html %} ~~~ sql -ALTER RANGE RELOCATE LEASE TO 2 FOR SELECT range_id from crdb_internal.ranges where table_name = 'users' +ALTER RANGE RELOCATE LEASE TO 2 FOR SELECT range_id from [SHOW RANGES FROM TABLE users WITH DETAILS]; ~~~ ~~~ @@ -205,7 +205,7 @@ To move the replicas for all data in the [`movr.users`]({% link {{ page.version. {% include_cached copy-clipboard.html %} ~~~ sql -ALTER RANGE RELOCATE FROM 2 TO 7 FOR SELECT range_id from crdb_internal.ranges where table_name = 'users'; +ALTER RANGE RELOCATE FROM 2 TO 7 FOR SELECT range_id from [SHOW RANGES FROM TABLE users WITH DETAILS]; ~~~ ~~~ @@ -231,7 +231,7 @@ To move all of a range's voting replicas from one store to another store: {% include_cached copy-clipboard.html %} ~~~ sql -ALTER RANGE RELOCATE VOTERS FROM 7 TO 2 FOR SELECT range_id from crdb_internal.ranges where table_name = 'users'; +ALTER RANGE RELOCATE VOTERS FROM 7 TO 2 FOR SELECT range_id from [SHOW RANGES FROM TABLE users WITH DETAILS]; ~~~ ~~~ @@ -261,7 +261,7 @@ This statement will only have an effect on clusters that have non-voting replica {% include_cached copy-clipboard.html %} ~~~ sql -ALTER RANGE RELOCATE NONVOTERS FROM 7 TO 2 FOR SELECT range_id from crdb_internal.ranges where table_name = 'users'; +ALTER RANGE RELOCATE NONVOTERS FROM 7 TO 2 FOR SELECT range_id from [SHOW RANGES FROM TABLE users WITH DETAILS]; ~~~ ~~~ diff --git a/src/current/v24.3/alter-table.md b/src/current/v24.3/alter-table.md index dc183d5a518..b94259ba8a8 100644 --- a/src/current/v24.3/alter-table.md +++ b/src/current/v24.3/alter-table.md @@ -223,6 +223,8 @@ You can use *replication zones* to control the number and location of replicas f For examples, see [Replication Controls](#configure-replication-zones). +{% include {{ page.version.version }}/see-zone-config-troubleshooting-guide.md %} + #### Required privileges The user must be a member of the [`admin` role]({% link {{ page.version.version }}/security-reference/authorization.md %}#admin-role) or have been granted [`CREATE`]({% link {{ page.version.version }}/security-reference/authorization.md %}#supported-privileges) or [`ZONECONFIG`]({% link {{ page.version.version }}/security-reference/authorization.md %}#supported-privileges) privileges. To configure [`system` objects]({% link {{ page.version.version }}/configure-replication-zones.md %}#for-system-data), the user must be a member of the `admin` role. diff --git a/src/current/v24.3/backup.md b/src/current/v24.3/backup.md index 7d5b004a881..b388d776ba3 100644 --- a/src/current/v24.3/backup.md +++ b/src/current/v24.3/backup.md @@ -378,3 +378,4 @@ To use an external connection URI to back up to cloud storage with an associated - [`CREATE SCHEDULE FOR BACKUP`]({% link {{ page.version.version }}/create-schedule-for-backup.md %}) - [`RESTORE`]({% link {{ page.version.version }}/restore.md %}) - [Replication Controls]({% link {{ page.version.version }}/configure-replication-zones.md %}) +- [Troubleshoot Replication Zone Configurations]({% link {{ page.version.version}}/troubleshoot-replication-zones.md %}) diff --git a/src/current/v24.3/cluster-api.md b/src/current/v24.3/cluster-api.md index 740e94f519a..e10016abd8a 100644 --- a/src/current/v24.3/cluster-api.md +++ b/src/current/v24.3/cluster-api.md @@ -21,7 +21,7 @@ Endpoint | Name | Description | Support [`/databases/{database}`](https://cockroachlabs.com/docs/api/cluster/v2.html#operation/databaseDetails) | Get database details | Get the descriptor ID of a specified database. | Stable [`/databases/{database}/grants`](https://cockroachlabs.com/docs/api/cluster/v2.html#operation/databaseGrants) | List database grants | List all [privileges](security-reference/authorization.html#managing-privileges) granted to users for a specified database. | Stable [`/databases/{database}/tables`](https://cockroachlabs.com/docs/api/cluster/v2.html#operation/databaseTables) | List database tables | List all tables in a specified database. | Stable -[`/databases/{database}/tables/{table}`](https://cockroachlabs.com/docs/api/cluster/v2.html#operation/tableDetails) | Get table details | Get details on a specified table, including schema, grants, indexes, range count, and zone configuration. | Stable +[`/databases/{database}/tables/{table}`](https://cockroachlabs.com/docs/api/cluster/v2.html#operation/tableDetails) | Get table details | Get details on a specified table, including schema, grants, indexes, range count, and [zone configurations]({% link {{ page.version.version }}/configure-replication-zones.md %}). | Stable [`/events`](https://cockroachlabs.com/docs/api/cluster/v2.html#operation/listEvents) | List events | List the latest [events](eventlog.html) on the cluster, in descending order. | Unstable [`/health`](https://cockroachlabs.com/docs/api/cluster/v2.html#operation/health) | Check node health | Determine if the node is running and ready to accept SQL connections. | Stable [`/nodes`](https://cockroachlabs.com/docs/api/cluster/v2.html#operation/listNodes) | List nodes | Get details on all nodes in the cluster, including node IDs, software versions, and hardware. | Stable diff --git a/src/current/v24.3/cluster-setup-troubleshooting.md b/src/current/v24.3/cluster-setup-troubleshooting.md index 80c3fa2b930..8bfa5b8f79e 100644 --- a/src/current/v24.3/cluster-setup-troubleshooting.md +++ b/src/current/v24.3/cluster-setup-troubleshooting.md @@ -635,15 +635,15 @@ Even with `server.eventlog.enabled` set to `false`, notable log events are still ## Check for under-replicated or unavailable data -To see if any data is under-replicated or unavailable in your cluster, follow the steps described in [Replication Reports]({% link {{ page.version.version }}/query-replication-reports.md %}). +To see if any data is under-replicated or unavailable in your cluster, follow the steps described in [Critical nodes endpoint]({% link {{ page.version.version }}/monitoring-and-alerting.md %}#critical-nodes-endpoint). ## Check for replication zone constraint violations -To see if any of your cluster's [data placement constraints]({% link {{ page.version.version }}/configure-replication-zones.md %}#replication-constraints) are being violated, follow the steps described in [Replication Reports]({% link {{ page.version.version }}/query-replication-reports.md %}). +To see if any of your cluster's [data placement constraints]({% link {{ page.version.version }}/configure-replication-zones.md %}#replication-constraints) are being violated, follow the steps described in [Troubleshoot Replication Zone Configurations]({% link {{ page.version.version}}/troubleshoot-replication-zones.md %}). ## Check for critical localities -To see which of your [localities]({% link {{ page.version.version }}/cockroach-start.md %}#locality) (if any) are critical, follow the steps described in [Replication Reports]({% link {{ page.version.version }}/query-replication-reports.md %}). A locality is "critical" for a range if all of the nodes in that locality becoming [unreachable](#node-liveness-issues) would cause the range to become unavailable. In other words, the locality contains a majority of the range's replicas. +To see which of your [localities]({% link {{ page.version.version }}/cockroach-start.md %}#locality) (if any) are critical, follow the steps described in the [Critical nodes endpoint documentation]({% link {{ page.version.version }}/monitoring-and-alerting.md %}#critical-nodes-endpoint). A locality is "critical" for a range if all of the nodes in that locality becoming [unreachable](#node-liveness-issues) would cause the range to become unavailable. In other words, the locality contains a majority of the range's replicas. ## Something else? diff --git a/src/current/v24.3/common-errors.md b/src/current/v24.3/common-errors.md index 5b6961044b1..559ca90c076 100644 --- a/src/current/v24.3/common-errors.md +++ b/src/current/v24.3/common-errors.md @@ -106,7 +106,7 @@ When running a single-node CockroachDB cluster, an error about replicas failing E160407 09:53:50.337328 storage/queue.go:511 [replicate] 7 replicas failing with "0 of 1 store with an attribute matching []; likely not enough nodes in cluster" ~~~ -This happens because CockroachDB expects three nodes by default. If you do not intend to add additional nodes, you can stop this error by using [`ALTER RANGE ... CONFIGURE ZONE`]({% link {{ page.version.version }}/alter-range.md %}#configure-zone) to update your default zone configuration to expect only one node: +This happens because CockroachDB expects three nodes by default. If you do not intend to add additional nodes, you can stop this error by using [`ALTER RANGE ... CONFIGURE ZONE`]({% link {{ page.version.version }}/alter-range.md %}#configure-zone) to update your default [zone configuration]({% link {{ page.version.version }}/configure-replication-zones.md %}) to expect only one node: {% include_cached copy-clipboard.html %} ~~~ shell @@ -222,3 +222,4 @@ Try searching the rest of our docs for answers or using our other [support resou - [StackOverflow](http://stackoverflow.com/questions/tagged/cockroachdb) - [CockroachDB Support Portal](https://support.cockroachlabs.com) - [Transaction retry error reference]({% link {{ page.version.version }}/transaction-retry-error-reference.md %}) +- [Troubleshoot Replication Zone Configurations]({% link {{ page.version.version}}/troubleshoot-replication-zones.md %}) diff --git a/src/current/v24.3/configure-replication-zones.md b/src/current/v24.3/configure-replication-zones.md index 1a61cf26308..59418914387 100644 --- a/src/current/v24.3/configure-replication-zones.md +++ b/src/current/v24.3/configure-replication-zones.md @@ -65,13 +65,41 @@ System Range | CockroachDB comes with pre-configured replication zones for impor ### Level priorities +[XXX](): EDIT THIS SECTION MORE + When replicating data, whether table or system, CockroachDB always uses the most granular replication zone available. For example, for a piece of user data: -1. If there's a replication zone for the row, CockroachDB uses it. -1. If there's no applicable row replication zone and the row is from a secondary index, CockroachDB uses the secondary index replication zone. -1. If the row isn't from a secondary index or there is no applicable secondary index replication zone, CockroachDB uses the table replication zone. -1. If there's no applicable table replication zone, CockroachDB uses the database replication zone. -1. If there's no applicable database replication zone, CockroachDB uses the `default` cluster-wide replication zone. +1. If there's a replication zone [for the row](#create-a-replication-zone-for-a-partition) (a.k.a. [partition]({% link {{ page.version.version }}/partitioning.md %})), CockroachDB uses it. +1. If there's no applicable row replication zone and the row is from a secondary index, CockroachDB uses the [secondary index replication zone](#create-a-replication-zone-for-a-secondary-index). +1. If the row isn't from a secondary index or there is no applicable secondary index replication zone, CockroachDB uses the [table replication zone](#create-a-replication-zone-for-a-table). +1. If there's no applicable table replication zone, CockroachDB uses the [database replication zone](#create-a-replication-zone-for-a-database). +1. If there's no applicable database replication zone, CockroachDB uses [the `default` cluster-wide replication zone](#view-the-default-replication-zone). + +The hierarchy of inheritance for zone configs can be visualized as follows: + +``` +- default + - database + - table + - index + - partition (row) A + - (sub)partition A.1 (DOES NOT INHERIT, SEE NOTE BELOW) + - (sub)partition A.1.1 (DOES NOT INHERIT) + - (sub)partition A.2 (DOES NOT INHERIT) + - ... etc. +``` + +Put differently, the system does a depth-first search (DFS) down the inheritance tree and always takes the most specific modified zone configuration at the current node of the tree, unless or until it finds a different modifed value for that zone configuration further down the tree. In cases where it doesn't find a modified value, it inherits from the current node's parent, which inherits from its parent, and on and on all the way back up the tree. + +{{site.data.alerts.callout_info}} +The exception to the inheritance behavior described above is that sub-partitions do not inherit their values from their parent partitions (rows). Instead, they inherit their fields from the parent table. For more information, see [cockroachdb/cockroach#75862](https://github.com/cockroachdb/cockroach/issues/75862). +{{site.data.alerts.end}} + +Each zone config inherits all of its initial values from its parent object. Any changes to a zone configuration's initial values are therefore supplied by the user. + +A zone config only stores the values that differ from its parent. CockroachDB then looks up the values for any unset fields in the parent object’s zone configuration. This continues recursively up the inheritance tree all the way to the default zone config. In practice, most values are cached to avoid performance impacts. + +All configurations will be modified versions of the [`default` range](#view-the-default-replication-zone). ## Manage replication zones @@ -136,7 +164,7 @@ See [Cluster Topology]({% link {{ page.version.version }}/recommended-production ### Troubleshooting zone constraint violations -To see if any of the data placement constraints defined in your replication zone configurations are being violated, use the `system.replication_constraint_stats` report as described in [Replication Reports]({% link {{ page.version.version }}/query-replication-reports.md %}). +{% include {{ page.version.version }}/see-zone-config-troubleshooting-guide.md %} ## View replication zones @@ -690,4 +718,5 @@ There's no need to make zone configuration changes; by default, the cluster is c - [`SHOW PARTITIONS`]({% link {{ page.version.version }}/show-partitions.md %}) - [SQL Statements]({% link {{ page.version.version }}/sql-statements.md %}) - [Table Partitioning]({% link {{ page.version.version }}/partitioning.md %}) -- [Replication Reports]({% link {{ page.version.version }}/query-replication-reports.md %}) +- [Critical nodes endpoint]({% link {{ page.version.version }}/monitoring-and-alerting.md %}#critical-nodes-endpoint) +- [Troubleshoot Replication Zone Configurations]({% link {{ page.version.version }}/troubleshoot-replication-zones.md %}) diff --git a/src/current/v24.3/demo-low-latency-multi-region-deployment.md b/src/current/v24.3/demo-low-latency-multi-region-deployment.md index 97102a0b8ef..2025a5d8c49 100644 --- a/src/current/v24.3/demo-low-latency-multi-region-deployment.md +++ b/src/current/v24.3/demo-low-latency-multi-region-deployment.md @@ -68,7 +68,7 @@ To determine which nodes are in which regions, you will need to refer to two (2) Here is the output of `\demo ls` from the SQL shell. {% include_cached copy-clipboard.html %} -~~~ sql +~~~ > \demo ls ~~~ @@ -145,7 +145,7 @@ Follow these steps to start 3 instances of MovR. Each instance is pointed at a n {% include_cached copy-clipboard.html %} ~~~ sql - CREATE DATABASE movr; + CREATE DATABASE IF NOT EXISTS movr; ~~~ 1. Open a second terminal and run the command below to populate the MovR data set. The options are mostly self-explanatory. We limit the application to 1 thread because using multiple threads quickly overloads this small demo cluster's ability to ingest data. As a result, loading the data takes about 90 seconds on a fast laptop. diff --git a/src/current/v24.3/migrate-to-multiregion-sql.md b/src/current/v24.3/migrate-to-multiregion-sql.md index bd5838a1480..e757cff4850 100644 --- a/src/current/v24.3/migrate-to-multiregion-sql.md +++ b/src/current/v24.3/migrate-to-multiregion-sql.md @@ -307,3 +307,4 @@ SHOW ZONE CONFIGURATION FROM TABLE promo_codes; - [Low Latency Reads and Writes in a Multi-Region Cluster]({% link {{ page.version.version }}/demo-low-latency-multi-region-deployment.md %}) - [Configure Replication Zones]({% link {{ page.version.version }}/configure-replication-zones.md %}) - [Non-voting replicas]({% link {{ page.version.version }}/architecture/replication-layer.md %}#non-voting-replicas) +- [Troubleshoot Replication Zone Configurations]({% link {{ page.version.version}}/troubleshoot-replication-zones.md %}) diff --git a/src/current/v24.3/monitoring-and-alerting.md b/src/current/v24.3/monitoring-and-alerting.md index a7880f7c02a..788b401f1ed 100644 --- a/src/current/v24.3/monitoring-and-alerting.md +++ b/src/current/v24.3/monitoring-and-alerting.md @@ -195,7 +195,7 @@ The critical nodes status endpoint is used to: - Check if any of your nodes are in a critical state. A node is _critical_ if that node becoming unreachable would cause [replicas to become unavailable]({% link {{ page.version.version }}/ui-cluster-overview-page.md %}#replication-status). - Check if any ranges are [under-replicated or unavailable]({% link {{ page.version.version }}/ui-cluster-overview-page.md %}#replication-status). This is useful when determining whether a node is ready for [decommissioning]({% link {{ page.version.version }}/node-shutdown.md %}#decommissioning). -- Check if any of your cluster's data placement constraints (set via [multi-region SQL]({% link {{ page.version.version }}/multiregion-overview.md %}) or direct [configuration of replication zones]({% link {{ page.version.version }}/configure-replication-zones.md %})) are being violated. This is useful when implementing [data domiciling]({% link {{ page.version.version }}/data-domiciling.md %}). +- Check if any of your cluster's data placement constraints (set via [multi-region SQL]({% link {{ page.version.version }}/multiregion-overview.md %}) or direct [configuration of replication zones]({% link {{ page.version.version }}/configure-replication-zones.md %})) are being violated. This is useful when implementing [data domiciling]({% link {{ page.version.version }}/data-domiciling.md %}) or [troubleshooting zone configurations]({% link {{ page.version.version }}/troubleshoot-replication-zones.md %}) generally. {{site.data.alerts.callout_info}} This HTTP status endpoint supersedes the deprecated [Replication Reports]({% link {{ page.version.version }}/query-replication-reports.md %}) SQL API. Due to architectural changes in CockroachDB, the SQL queries described on that page will not result in correct output. @@ -1216,3 +1216,4 @@ Currently, not all events listed have corresponding alert rule definitions avail - [Local Deployment]({% link {{ page.version.version }}/start-a-local-cluster.md %}) - [Third-Party Monitoring Integrations]({% link {{ page.version.version }}/third-party-monitoring-tools.md %}) - [Metrics]({% link {{ page.version.version }}/metrics.md %}) +- [Troubleshoot Replication Zone Configurations]({% link {{ page.version.version }}/troubleshoot-replication-zones.md %}) diff --git a/src/current/v24.3/multiregion-overview.md b/src/current/v24.3/multiregion-overview.md index 297c76404dd..baa49242d75 100644 --- a/src/current/v24.3/multiregion-overview.md +++ b/src/current/v24.3/multiregion-overview.md @@ -180,3 +180,4 @@ For more information, see [Zone Config Extensions]({% link {{ page.version.versi - [`SET SECONDARY REGION`]({% link {{ page.version.version }}/alter-database.md %}#set-secondary-region) - [`ALTER DATABASE ... DROP SECONDARY REGION`]({% link {{ page.version.version }}/alter-database.md %}#drop-secondary-region) - [Zone Config Extensions]({% link {{ page.version.version }}/zone-config-extensions.md %}) +- [Troubleshoot Replication Zone Configurations]({% link {{ page.version.version}}/troubleshoot-replication-zones.md %}) diff --git a/src/current/v24.3/node-shutdown.md b/src/current/v24.3/node-shutdown.md index 4c549df1edc..59556b95075 100644 --- a/src/current/v24.3/node-shutdown.md +++ b/src/current/v24.3/node-shutdown.md @@ -301,7 +301,7 @@ To determine an appropriate termination grace period: ### Size and replication factor -Before decommissioning a node, make sure other nodes are available to take over the range replicas from the node. If fewer nodes are available than the replication factor, CockroachDB will automatically reduce the replication factor (for example, from 5 to 3) to try to allow the decommission to succeed. However, the replication factor will not be reduced lower than 3. If three nodes are not available, the decommissioning process will hang indefinitely until nodes are added or you update the zone configurations to use a replication factor of 1. +Before decommissioning a node, make sure other nodes are available to take over the range replicas from the node. If fewer nodes are available than the replication factor, CockroachDB will automatically reduce the replication factor (for example, from 5 to 3) to try to allow the decommission to succeed. However, the replication factor will not be reduced lower than 3. If three nodes are not available, the decommissioning process will hang indefinitely until nodes are added or you update the [zone configurations]({% link {{ page.version.version }}/configure-replication-zones.md %}#num_replicas) to use a replication factor of 1. Note that when you decommission a node and immediately add another node, CockroachDB does **not** simply move all of the replicas from the decommissioned node to the newly added node. Instead, replicas are placed across all nodes in the cluster. This speeds up the decommissioning process by spreading the load. The new node will eventually "catch up" with the rest of the cluster. diff --git a/src/current/v24.3/partitioning.md b/src/current/v24.3/partitioning.md index a313c97134a..3e57265f9d6 100644 --- a/src/current/v24.3/partitioning.md +++ b/src/current/v24.3/partitioning.md @@ -15,8 +15,8 @@ CockroachDB allows you to define table partitions, giving you row-level control Table partitioning helps you reduce latency and cost: -- **Geo-partitioning** allows you to keep user data close to the user, which reduces the distance that the data needs to travel, thereby **reducing latency**. To geo-partition a table, define location-based partitions while creating a table, create location-specific zone configurations, and apply the zone configurations to the corresponding partitions. -- **Archival-partitioning** allows you to store infrequently-accessed data on slower and cheaper storage, thereby **reducing costs**. To archival-partition a table, define frequency-based partitions while creating a table, create frequency-specific zone configurations with appropriate storage devices constraints, and apply the zone configurations to the corresponding partitions. +- **Geo-partitioning** allows you to keep user data close to the user, which reduces the distance that the data needs to travel, thereby **reducing latency**. To geo-partition a table, define location-based partitions while creating a table, create location-specific [zone configurations]({% link {{ page.version.version }}/configure-replication-zones.md %}), and apply the zone configurations to the corresponding partitions. +- **Archival-partitioning** allows you to store infrequently-accessed data on slower and cheaper storage, thereby **reducing costs**. To archival-partition a table, define frequency-based partitions while creating a table, create frequency-specific [zone configurations]({% link {{ page.version.version }}/configure-replication-zones.md %}) with appropriate storage devices constraints, and apply the zone configurations to the corresponding partitions. ## How it works @@ -109,6 +109,7 @@ On their own, partitions are inert and simply apply a label to the rows of the t CockroachDB uses the most granular zone config available. Zone configs that target a partition are considered more granular than those that target a table or index, which in turn are considered more granular than those that target a database. +For more information about how zone config inheritance works, see [Replication Controls]({% link {{ page.version.version }}/configure-replication-zones.md %}#level-priorities). {% include {{page.version.version}}/sql/querying-partitions.md %} @@ -688,7 +689,7 @@ Other databases use partitioning for three additional use cases: secondary index - **Changes to secondary indexes:** CockroachDB solves these changes through online schema changes. Online schema changes are a superior feature to partitioning because they require zero-downtime and eliminate the potential for consistency problems. - **Sharding:** CockroachDB automatically shards data as a part of its distributed database architecture. - **Bulk Loading & Deleting:** CockroachDB does not have a feature that supports this use case as of now. -- **Logical structure of partitions:** CockroachDB uses the partitioning concept to allow users to logically subdivide a single physical table to enable independent configuration of those partitions using zone configurations. Other databases sometimes implement partitioning as the logical union of physically separate tables. This difference means that CockroachDB is able to permit inexpensive repartitioning in contrast to other databases. Consequently, CockroachDB is unable to provide the same bulk data deletion operations over table partitions that other databases achieve by physically dropping the underlying table represented by the partition. +- **Logical structure of partitions:** CockroachDB uses the partitioning concept to allow users to logically subdivide a single physical table to enable independent configuration of those partitions using [zone configurations]({% link {{ page.version.version }}/configure-replication-zones.md %}). Other databases sometimes implement partitioning as the logical union of physically separate tables. This difference means that CockroachDB is able to permit inexpensive repartitioning in contrast to other databases. Consequently, CockroachDB is unable to provide the same bulk data deletion operations over table partitions that other databases achieve by physically dropping the underlying table represented by the partition. ## Known limitations @@ -701,3 +702,4 @@ Other databases use partitioning for three additional use cases: secondary index - [`CREATE TABLE`]({% link {{ page.version.version }}/create-table.md %}) - [`ALTER TABLE`]({% link {{ page.version.version }}/alter-table.md %}) - [Computed Columns]({% link {{ page.version.version }}/computed-columns.md %}) +- [Troubleshoot Replication Zone Configurations]({% link {{ page.version.version}}/troubleshoot-replication-zones.md %}) diff --git a/src/current/v24.3/query-replication-reports.md b/src/current/v24.3/query-replication-reports.md index 475624517b8..7dbd43ec52e 100644 --- a/src/current/v24.3/query-replication-reports.md +++ b/src/current/v24.3/query-replication-reports.md @@ -10,6 +10,8 @@ docs_area: manage The SQL API described on this page is deprecated and will be removed in a future release. **Due to architectural changes in CockroachDB, the SQL queries described here will not result in correct output.** To check the status of your cluster's data replication, data placement, and zone constraint conformance, use the [Critical nodes status endpoint]({% link {{ page.version.version }}/monitoring-and-alerting.md %}#critical-nodes-endpoint). {{site.data.alerts.end}} +[XXX](): DELETE ME (slash redirect me) + Several new and updated tables (listed below) are available to help you query the status of your cluster's data replication, data placement, and zone constraint conformance. For example, you can: - See what data is under-replicated or unavailable. @@ -522,3 +524,4 @@ To give another example, let's say your cluster were similar to the one shown ab - [`PARTITION BY`]({% link {{ page.version.version }}/partitioning.md %}) - [`CONFIGURE ZONE`]({% link {{ page.version.version }}/alter-table.md %}#configure-zone) - [Start a node]({% link {{ page.version.version }}/cockroach-start.md %}) +- [Troubleshoot Replication Zone Configurations]({% link {{ page.version.version }}/troubleshoot-replication-zones.md %}) diff --git a/src/current/v24.3/show-backup.md b/src/current/v24.3/show-backup.md index d7ecd78a5d9..fe1849f69e1 100644 --- a/src/current/v24.3/show-backup.md +++ b/src/current/v24.3/show-backup.md @@ -77,7 +77,7 @@ Field | Value | Description `rows` | [`INT`]({% link {{ page.version.version }}/int.md %}) | Number of rows in tables that are part of the backup. `create_statement` | [`STRING`]({% link {{ page.version.version }}/string.md %}) | The `CREATE` statement used to create [table(s)]({% link {{ page.version.version }}/create-table.md %}), [view(s)]({% link {{ page.version.version }}/create-view.md %}), or [sequence(s)]({% link {{ page.version.version }}/create-sequence.md %}) that are stored within the backup. This displays when `SHOW BACKUP SCHEMAS` is used. Note that tables with references to [foreign keys]({% link {{ page.version.version }}/foreign-key.md %}) will only display foreign key constraints if the table to which the constraint relates to is also included in the backup. `is_full_cluster` | [`BOOL`]({% link {{ page.version.version }}/bool.md %}) | Whether the backup is of a full cluster or not. -`regions` | [`STRING`]({% link {{ page.version.version }}/string.md %}) | The [`ALTER DATABASE`]({% link {{ page.version.version }}/alter-database.md %}) statement(s) used to configure the database with [multi-region zone configurations]({% link {{ page.version.version }}/multiregion-overview.md %}#database-regions), if they exist. If the database is a part of a single region cluster configuration, `NULL` will show. +`regions` | [`STRING`]({% link {{ page.version.version }}/string.md %}) | The [`ALTER DATABASE`]({% link {{ page.version.version }}/alter-database.md %}) statement(s) used to configure [multiple database regions]({% link {{ page.version.version }}/multiregion-overview.md %}#database-regions), if they exist. If the database is a part of a single region cluster configuration, `NULL` will show. `file_bytes` | [`INT`]({% link {{ page.version.version }}/int.md %}) | (With the `check_files` option only) The estimated bytes in external storage for a particular table object. This is the physical bytes that a given table object is taking up. For example, when the files are written to disk in storage they could be compressed. If you total all file bytes, the result is the physical bytes in your storage location. Note that for smaller tables the byte size in `file_bytes` may be larger than `size_bytes` because of the overhead required to create an SST file. `path` | [`STRING`]({% link {{ page.version.version }}/string.md %}) | The list of the [full backup]({% link {{ page.version.version }}/take-full-and-incremental-backups.md %}#full-backups)'s subdirectories. This field is returned for `SHOW BACKUPS IN collectionURI` only. The path format is `//-`. diff --git a/src/current/v24.3/show-create.md b/src/current/v24.3/show-create.md index 8ae0c86f859..c3b340f0802 100644 --- a/src/current/v24.3/show-create.md +++ b/src/current/v24.3/show-create.md @@ -96,7 +96,7 @@ To return just the `create_statement` value: ~~~ {{site.data.alerts.callout_info}} -`SHOW CREATE TABLE` also lists any partitions and zone configurations defined on primary and secondary indexes of a table. If partitions are defined, but no zones are configured, the `SHOW CREATE TABLE` output includes a warning. +`SHOW CREATE TABLE` also lists any partitions and [zone configurations]({% link {{ page.version.version }}/configure-replication-zones.md %}) defined on primary and secondary indexes of a table. If partitions are defined, but no zones are configured, the `SHOW CREATE TABLE` output includes a warning. {{site.data.alerts.end}} ### Show the `CREATE TABLE` statement for a table with a hidden column diff --git a/src/current/v24.3/show-zone-configurations.md b/src/current/v24.3/show-zone-configurations.md index 1df6ab75243..a16e1ca93ff 100644 --- a/src/current/v24.3/show-zone-configurations.md +++ b/src/current/v24.3/show-zone-configurations.md @@ -94,3 +94,4 @@ CONFIGURE ZONE 1 - [`ALTER RANGE ... CONFIGURE ZONE`]({% link {{ page.version.version }}/alter-range.md %}#configure-zone) - [`ALTER TABLE ... CONFIGURE ZONE`]({% link {{ page.version.version }}/alter-table.md %}#configure-zone) - [SQL Statements]({% link {{ page.version.version }}/sql-statements.md %}) +- [Troubleshoot Replication Zone Configurations]({% link {{ page.version.version}}/troubleshoot-replication-zones.md %}) diff --git a/src/current/v24.3/take-and-restore-locality-aware-backups.md b/src/current/v24.3/take-and-restore-locality-aware-backups.md index bd921f2674b..6c4558383a3 100644 --- a/src/current/v24.3/take-and-restore-locality-aware-backups.md +++ b/src/current/v24.3/take-and-restore-locality-aware-backups.md @@ -92,7 +92,7 @@ Specifying both locality tier pairs (e.g., `region=us-east,az=az1`) from the out Given a list of URIs that together contain the locations of all of the files for a single [locality-aware backup](#create-a-locality-aware-backup), [`RESTORE`]({% link {{ page.version.version }}/restore.md %}) can read in that backup. Note that the list of URIs passed to [`RESTORE`]({% link {{ page.version.version }}/restore.md %}) may be different from the URIs originally passed to [`BACKUP`]({% link {{ page.version.version }}/backup.md %}). This is because it's possible to move the contents of one of the parts of a locality-aware backup (i.e., the files written to that destination) to a different location, or even to consolidate all the files for a locality-aware backup into a single location. -When restoring a [full backup]({% link {{ page.version.version }}/take-full-and-incremental-backups.md %}#full-backups), the cluster data is restored first, then the system table data "as is." This means that the restored zone configurations can point to regions that do not have active nodes in the new cluster. For example, if your full backup has the following [zone configurations]({% link {{ page.version.version }}/alter-partition.md %}#create-a-replication-zone-for-a-partition): +When restoring a [full backup]({% link {{ page.version.version }}/take-full-and-incremental-backups.md %}#full-backups), the cluster data is restored first, then the system table data "as is." This means that the restored [zone configurations]({% link {{ page.version.version }}/alter-partition.md %}#create-a-replication-zone-for-a-partition) can point to regions that do not have active nodes in the new cluster. For example, if your full backup has the following zone configurations: ~~~ sql ALTER PARTITION europe_west OF INDEX movr.public.rides@rides_pkey \ diff --git a/src/current/v24.3/troubleshoot-replication-zones.md b/src/current/v24.3/troubleshoot-replication-zones.md new file mode 100644 index 00000000000..ffad1cb716d --- /dev/null +++ b/src/current/v24.3/troubleshoot-replication-zones.md @@ -0,0 +1,86 @@ +--- +title: Troubleshoot Replication Zone Configurations +summary: Troubleshooting guide for replication zones, which control the number and location of replicas for specific sets of data. +keywords: ttl, time to live, availability zone +toc: true +docs_area: manage +--- + +This page has instructions showing how to troubleshoot scenarios where you believe replicas are not behaving as specified by your zone configurations. + +## Prerequisites + +This page assumes you have read and understood the following materials: + +- [Replication controls > Replication zone levels]({% link {{ page.version.version }}/configure-replication-zones.md %}#replication-zone-levels), which describes how the hierarchy of inheritance of replication zones works. This is critical to understand for troubleshooting. +- [Monitoring and alerting > Critical nodes endpoint]({% link {{ page.version.version }}/monitoring-and-alerting.md %}#critical-nodes-endpoint), which is used to monitor if any of your cluster's ranges are under-replicated, or if your data placement constraints are being violated. +- [`SHOW ZONE CONFIGURATIONS`]({% link {{ page.version.version }}/show-zone-configurations.md %}), which is the SQL statement used to view details about the replication zone configuration of various schema objects. + +## Types of problems + +There are several classes of common problems users encounter when [manually configuring replication zones]({% link {{ page.version.version }}/configure-replication-zones.md %}#manage-replication-zones). Cockroach Labs does not recommend adding zone configurations manually, since it is easy to introduce logic errors. It's also difficult to do proper change management and auditing of manually tweaked zone configurations. Most users should use [Multi-region SQL statements]({% link {{ page.version.version }}/multiregion-overview.md %}) instead; if more control is needed, [Zone config extensions]({% link {{ page.version.version }}/zone-config-extensions.md %}) can be used to augment the multi-region SQL statements. + +Generally, the problems tend to fall into one of the following categories: + +- "the system isn't sending replicas where I told it to" +- "the system isn't managing replicas how I told it to" + +This behavior is almost always caused by a replication zone **misconfiguration**, but it can be difficult to see what the error is or how it was introduced. Zone configurations do not have much observability beyond `SHOW ZONE CONFIGURATIONS`, nor is there much built-in validation to prevent logic errors. It's easy to put the system in a state where you've told it to do two mutually incompatible things. + +The most common class of logic error occurs because of the way inheritance works for replication zone configurations. As discussed in [Replication Controls]({% link {{ page.version.version }}/configure-replication-zones.md %}#level-priorities), CockroachDB always uses the most granular replication zone available in a "bottom-up" fashion. + +When you manually set a field at, say, the table level, it overrides the value that was already set at the next level up, in the parent database. If you later change something at the database level and find that it isn't working as expected for all tables, it may be that the more-specific settings you applied at the table level are overriding the database-level settings. In other words, the system is doing what it was told, because it is respecting the table-level change you already applied. However, this may not be what you _intended_. + +As noted previously, the problems tend to fall into one of the following general categories: + +- "the system isn't sending replicas where I told it to" +- "the system isn't doing what I told it to with the replica configuration" + +Specifically: + +- If the problem is "the system isn't sending replicas where I told it to", see: + - `constraints = '{+region=europe-west1: 1, +region=us-east1: 1, +region=us-west1: 1}',` + - `voter_constraints = '[+region=us-east1]',` + - `lease_preferences = '[[+region=us-east1]]'` +- If the problem is "the system isn't doing what I told it to with the replica configuration", see: + - `num_replicas = 5,` + - `range_min_bytes = 134217728,` + - `range_max_bytes = 536870912,` + - `gc.ttlseconds = 14400,` + - `num_voters = 3,` + +The most common reason why "the thing isn't going where I told it to go" or "the thing isn't doing what I told it to do" is misconfiguration. + +[XXX](): ADD THE THING ABOUT BACKUPS OVERWRITING ZONE CONFIGS ON RESTORE? (via backup.md search for 'overwritten') + +## Troubleshooting steps + +Troubleshooting zone configs is difficult because it requires running a mix of one or more of the following statements, peering at their output, and deducing what went wrong. Specifically, running a mix of: + +- [`SHOW ZONE CONFIGURATIONS`]({% link {{ page.version.version }}/show-zone-configurations.md %}) for different levels of objects in the inheritance hierarchy and checking where they differ. +- [`SHOW ALL ZONE CONFIGURATIONS`]({% link {{ page.version.version }}/show-zone-configurations.md %}#view-all-replication-zones) and parsing the output into a tree-like format that lets you see what has changed. ([XXX](): is this really what we want to say?) + +[XXX](): WRITE ME + +### Run SQL statements + +## Considerations + +### Replication system priorities + +A further wrinkle is that the [Replication Layer]({% link {{ page.version.version }}/architecture/replication-layer.md %})'s top priority is avoiding data loss, _not_ putting replicas exactly where you told it to. For more information about this limitation, see [the data domiciling docs]({% link {{ page.version.version }}/data-domiciling.md %}#known-limitations). + +That said, the replication logic takes user-supplied zone configurations into account when allocating replicas. + +### Replication lag + +Sometimes the changes you make to a zone configuration are not reflected in the running system for a few minutes. Depending on the size of the cluster, this is expected behavior. It can take several minutes for changes to replica locality you specify in a changed zone config to propagate across a cluster. In general, the larger the cluster, the longer this process may take, due to the amount of data shuffling that occurs. In general, it's better to avoid making big changes to replica constraints on a large cluster unless you are prepared for it to take some time. + +For more information about how to troubleshoot replication issues, especially under-replicated ranges, see [Troubleshoot Self-Hosted Setup > Replication issues]({% link {{ page.version.version }}/cluster-setup-troubleshooting.md %}#replication-issues). + +## See also + +- [Troubleshooting overview]({% link {{ page.version.version }}/troubleshooting-overview.md %}) +- [Troubleshoot Self-Hosted Setup > Replication issues]({% link {{ page.version.version }}/cluster-setup-troubleshooting.md %}#replication-issues) +- [Replication Controls]({% link {{ page.version.version }}/configure-replication-zones.md %}) +- [Critical nodes status endpoint]({% link {{ page.version.version }}/monitoring-and-alerting.md %}#critical-nodes-endpoint): check the status of your cluster's data replication, data placement, and zone constraint conformance. diff --git a/src/current/v24.3/troubleshooting-overview.md b/src/current/v24.3/troubleshooting-overview.md index 9b21e2725f0..e0f4302a7df 100644 --- a/src/current/v24.3/troubleshooting-overview.md +++ b/src/current/v24.3/troubleshooting-overview.md @@ -20,7 +20,9 @@ If you experience an issue when using CockroachDB, try these steps to resolve th - If you see discrepancies in metrics, refer to [Differences in Metrics between Third-Party Monitoring Integrations and DB Console]({% link {{ page.version.version }}/differences-in-metrics-between-third-party-monitoring-integrations-and-db-console.md %}). -- [Replication Reports]({% link {{ page.version.version }}/query-replication-reports.md %}) let you query the status of your cluster's data replication, data placement, and zone constraint conformance. +- The [critical nodes endpoint]({% link {{ page.version.version }}/monitoring-and-alerting.md %}#critical-nodes-endpoint) lets you query the status of your cluster's data replication, data placement, and zone constraint conformance. + +- If you believe [replicas]({% link {{ page.version.version }}/architecture/overview.md %}#architecture-replica) are not behaving as specified by your zone configurations, whether in terms of data placement or zone constraint conformance, see [Troubleshoot Replication Zone Configurations]({% link {{ page.version.version}}/troubleshoot-replication-zones.md %}). - If you cannot resolve the issue yourself, the following tools can help you move forward: - [Support Resources]({% link {{ page.version.version }}/support-resources.md %}) identify ways you can get help with troubleshooting. diff --git a/src/current/v24.3/ui-debug-pages.md b/src/current/v24.3/ui-debug-pages.md index c75bcf60a0f..0758d18d462 100644 --- a/src/current/v24.3/ui-debug-pages.md +++ b/src/current/v24.3/ui-debug-pages.md @@ -32,7 +32,7 @@ Report | Description | Access level --------|-----|-------- [Custom Time Series Chart]({% link {{ page.version.version }}/ui-custom-chart-debug-page.md %}) | Create a custom chart of time series data. | All users. Problem Ranges | View ranges in your cluster that are unavailable, under-replicated, slow, paused, or have other problems. | On secure clusters, [`admin` user]({% link {{ page.version.version }}/security-reference/authorization.md %}#admin-role) or a SQL user with the [`VIEWCLUSTERMETADATA`]({% link {{ page.version.version }}/security-reference/authorization.md %}#viewclustermetadata) [system privilege]({% link {{ page.version.version }}/security-reference/authorization.md %}#supported-privileges). -Data Distribution and Zone Configs | View the distribution of table data across nodes and verify zone configuration. | On secure clusters, [`admin` user]({% link {{ page.version.version }}/security-reference/authorization.md %}#admin-role) or a SQL user with the [`VIEWCLUSTERMETADATA`]({% link {{ page.version.version }}/security-reference/authorization.md %}#viewclustermetadata) [system privilege]({% link {{ page.version.version }}/security-reference/authorization.md %}#supported-privileges). +Data Distribution and Zone Configs | View the distribution of table data across nodes and verify [zone configurations]({% link {{ page.version.version }}/configure-replication-zones.md %}). | On secure clusters, [`admin` user]({% link {{ page.version.version }}/security-reference/authorization.md %}#admin-role) or a SQL user with the [`VIEWCLUSTERMETADATA`]({% link {{ page.version.version }}/security-reference/authorization.md %}#viewclustermetadata) [system privilege]({% link {{ page.version.version }}/security-reference/authorization.md %}#supported-privileges). Statement Diagnostics History | Diagnostic bundles for all statements executed on the cluster. | On secure clusters, [`admin` user]({% link {{ page.version.version }}/security-reference/authorization.md %}#admin-role) only. ## Configuration diff --git a/src/current/v24.3/zone-config-extensions.md b/src/current/v24.3/zone-config-extensions.md index 980eae68074..a59888331bf 100644 --- a/src/current/v24.3/zone-config-extensions.md +++ b/src/current/v24.3/zone-config-extensions.md @@ -44,3 +44,4 @@ For instructions showing how to use Zone Config Extensions, see [Use Zone Config - [`SET LOCALITY`]({% link {{ page.version.version }}/alter-table.md %}#set-locality) - [`ADD REGION`]({% link {{ page.version.version }}/alter-database.md %}#add-region) - [`SET PRIMARY REGION`]({% link {{ page.version.version }}/alter-database.md %}#set-primary-region) +- [Troubleshoot Replication Zone Configurations]({% link {{ page.version.version}}/troubleshoot-replication-zones.md %})