diff --git a/developers/weaviate/concepts/replication-architecture/consistency.md b/developers/weaviate/concepts/replication-architecture/consistency.md index 8ac241ad8..4e4e7a2c7 100644 --- a/developers/weaviate/concepts/replication-architecture/consistency.md +++ b/developers/weaviate/concepts/replication-architecture/consistency.md @@ -166,7 +166,7 @@ Repair-on-read works well with one or two isolated repairs. Async replication is Async replication supplements the repair-on-read mechanism. If a node becomes inconsistent between sync checks, the repair-on-read mechanism catches the problem at read time. -To activate async replication, set `asyncEnabled` to true in the [`replicationConfig` section of your collection definition](../../manage-data/collections.mdx#replication-settings). +To activate async replication, set `asyncEnabled` to true in the [`replicationConfig` section of your collection definition](../../manage-data/collections.mdx#replication-settings). Visit the [How-to: Replication](/developers/weaviate/configuration/replication#configuring-async-replication) page to learn more about all the available async replication settings. ### Deletion resolution strategies diff --git a/developers/weaviate/config-refs/env-vars.md b/developers/weaviate/config-refs/env-vars.md index f3e272d49..375525bc2 100644 --- a/developers/weaviate/config-refs/env-vars.md +++ b/developers/weaviate/config-refs/env-vars.md @@ -124,6 +124,16 @@ Role-based access control (RBAC) is added `v1.28` as a **technical preview**. Th | Variable | Description | Type | Example Value | | --- | --- | --- | --- | +| `ASYNC_REPLICATION_DISABLED` | Disable async replication. Default: `false` | `boolean` | `false` | +| `ASYNC_REPLICATION_HASHTREE_HEIGHT` | Height of the hashtree used for data comparison between nodes. Default: `16`, Min: `1`, Max: `20` | `string - number` | `10` | +| `ASYNC_REPLICATION_FREQUENCY` | Frequency of periodic data comparison between nodes in seconds. Default: `5` | `string - number` | `60` | +| `ASYNC_REPLICATION_FREQUENCY_WHILE_PROPAGATING` | Frequency of data comparison between nodes after a node has been synced in milliseconds. Default: `10` | `string - number` | `20` | +| `ASYNC_REPLICATION_ALIVE_NODES_CHECKING_FREQUENCY` | Frequency of how often the background process checks for changes in the availability of nodes in seconds. Default: `1` | `string - number` | `20` | +| `ASYNC_REPLICATION_LOGGING_FREQUENCY` | Frequency of how often the background process logs any events in seconds. Default: default `3` | `string - number` | `7` | +| `ASYNC_REPLICATION_DIFF_PER_NODE_TIMEOUT` | Defines the time limit a node has to provide a comparison response in seconds. Default: `10` | `string - number` | `30` | +| `ASYNC_REPLICATION_PROPAGATION_TIMEOUT` | Defines the time limit a node has to provide a propagation response in seconds. Default: `30` | `string - number` | `60` | +| `ASYNC_REPLICATION_PROPAGATION_LIMIT` | The maximum number of objects propagated per iteration. Default: `10000` | `string - number` | `5000` | +| `ASYNC_REPLICATION_BATCH_SIZE` | The maximum size of the batch propagated. Default: `100` |`string - number` | `200` | | `CLUSTER_DATA_BIND_PORT` | Port for exchanging data. | `string - number` | `7103` | | `CLUSTER_GOSSIP_BIND_PORT` | Port for exchanging network state information. | `string - number` | `7102` | | `CLUSTER_HOSTNAME` | Hostname of a node. Always set this value if the default OS hostname might change over time. | `string` | `node1` | diff --git a/developers/weaviate/configuration/replication.md b/developers/weaviate/configuration/replication.md index 3d34522c2..d7057370b 100644 --- a/developers/weaviate/configuration/replication.md +++ b/developers/weaviate/configuration/replication.md @@ -51,6 +51,36 @@ import ReplicationConfigWithAsyncRepair from '/_includes/code/configuration/repl +### Configuring Async Replication + +:::info Added in `v1.29` +Async replication support has been added in `v1.26`while the environment variables for configuring async replication (`ASYNC_*`) have been introduced in `v1.29`. +::: + +Async Replication is a background synchronization process in Weaviate that ensures eventual consistency across nodes storing the same shard. When a collection is partitioned into multiple shards, each shard is replicated across several nodes (as defined by the replication factor `REPLICATION_MINIMUM_FACTOR`). Async replication guarantees that all nodes holding the same shard remain in sync by periodically comparing and propagating data. + +#### 1. Periodic data comparison + +Each node runs a background process that periodically compares its locally stored data with other nodes holding the same shard. This comparison is triggered either: +- At regular intervals (`ASYNC_REPLICATION_FREQUENCY`). +- When a change in availability of a node is detected (`ASYNC_REPLICATION_ALIVE_NODES_CHECKING_FREQUENCY`). + +To efficiently detect differences, Weaviate uses a **hashtree**. Instead of checking entire datasets, it compares hash digests at multiple levels, narrowing down differences to specific objects. The size of this hashtree can be defined via `ASYNC_REPLICATION_HASHTREE_HEIGHT`. + +If a node is unresponsive, Weaviate applies a timeout (`ASYNC_REPLICATION_DIFF_PER_NODE_TIMEOUT`) to avoid delays in the replication process. + +#### 2. Data synchronization + +When differences are detected, the outdated or missing data is propagated to the affected nodes. This propagation process: +- Sends data in batches of a custom size (`ASYNC_REPLICATION_BATCH_SIZE`). +- Limits each propagation step object limit (`ASYNC_REPLICATION_PROPAGATION_LIMIT`). +- Enforces a time-bound for updates (`ASYNC_REPLICATION_PROPAGATION_TIMEOUT`). +- Uses a different comparison frequency right after completing synchronization on a node (`ASYNC_REPLICATION_FREQUENCY_WHILE_PROPAGATING`). + +:::tip Replication settings +You can find a complete list of the environment variables related to async replication on the page [Reference: Environment variables](/developers/weaviate/config-refs/env-vars#multi-node-instances). +::: + ## How to use: Queries When you add (write) or query (read) data, one or more replica nodes in the cluster will respond to the request. How many nodes need to send a successful response and acknowledgement to the coordinator node depends on the `consistency_level`. Available [consistency levels](../concepts/replication-architecture/consistency.md) are `ONE`, `QUORUM` (replication_factor / 2 + 1) and `ALL`. diff --git a/developers/weaviate/manage-data/collections.mdx b/developers/weaviate/manage-data/collections.mdx index ed3c44e09..f3ecaff35 100644 --- a/developers/weaviate/manage-data/collections.mdx +++ b/developers/weaviate/manage-data/collections.mdx @@ -842,7 +842,7 @@ import RaftRFChangeWarning from '/_includes/1-25-replication-factor.mdx'; -Configure replication settings, such as [async replication](../concepts/replication-architecture/consistency.md#async-replication) and [deletion resolution strategy](../concepts/replication-architecture/consistency.md#deletion-resolution-strategies). +Configure replication settings, such as [async replication](/developers/weaviate/configuration/replication#configuring-async-replication) and [deletion resolution strategy](../concepts/replication-architecture/consistency.md#deletion-resolution-strategies).