Skip to content

Commit

Permalink
Add async replication docs for v1.29
Browse files Browse the repository at this point in the history
  • Loading branch information
g-despot committed Feb 6, 2025
1 parent 1ae85f7 commit 5882607
Show file tree
Hide file tree
Showing 4 changed files with 42 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ Repair-on-read works well with one or two isolated repairs. Async replication is

Async replication supplements the repair-on-read mechanism. If a node becomes inconsistent between sync checks, the repair-on-read mechanism catches the problem at read time.

To activate async replication, set `asyncEnabled` to true in the [`replicationConfig` section of your collection definition](../../manage-data/collections.mdx#replication-settings).
To activate async replication, set `asyncEnabled` to true in the [`replicationConfig` section of your collection definition](../../manage-data/collections.mdx#replication-settings). Visit the [How-to: Replication](/developers/weaviate/configuration/replication#configuring-async-replication) page to learn more about all the available async replication settings.

### Deletion resolution strategies

Expand Down
10 changes: 10 additions & 0 deletions developers/weaviate/config-refs/env-vars.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,16 @@ Role-based access control (RBAC) is added `v1.28` as a **technical preview**. Th

| Variable | Description | Type | Example Value |
| --- | --- | --- | --- |
| `ASYNC_REPLICATION_DISABLED` | Disable async replication. Default: `false` | `boolean` | `false` |
| `ASYNC_REPLICATION_HASHTREE_HEIGHT` | Height of the hashtree used for data comparison between nodes. Default: `16`, Min: `1`, Max: `20` | `string - number` | `10` |
| `ASYNC_REPLICATION_FREQUENCY` | Frequency of periodic data comparison between nodes in seconds. Default: `5` | `string - number` | `60` |
| `ASYNC_REPLICATION_FREQUENCY_WHILE_PROPAGATING` | Frequency of data comparison between nodes after a node has been synced in milliseconds. Default: `10` | `string - number` | `20` |
| `ASYNC_REPLICATION_ALIVE_NODES_CHECKING_FREQUENCY` | Frequency of how often the background process checks for changes in the availability of nodes in seconds. Default: `1` | `string - number` | `20` |
| `ASYNC_REPLICATION_LOGGING_FREQUENCY` | Frequency of how often the background process logs any events in seconds. Default: default `3` | `string - number` | `7` |
| `ASYNC_REPLICATION_DIFF_PER_NODE_TIMEOUT` | Defines the time limit a node has to provide a comparison response in seconds. Default: `10` | `string - number` | `30` |
| `ASYNC_REPLICATION_PROPAGATION_TIMEOUT` | Defines the time limit a node has to provide a propagation response in seconds. Default: `30` | `string - number` | `60` |
| `ASYNC_REPLICATION_PROPAGATION_LIMIT` | The maximum number of objects propagated per iteration. Default: `10000` | `string - number` | `5000` |
| `ASYNC_REPLICATION_BATCH_SIZE` | The maximum size of the batch propagated. Default: `100` |`string - number` | `200` |
| `CLUSTER_DATA_BIND_PORT` | Port for exchanging data. | `string - number` | `7103` |
| `CLUSTER_GOSSIP_BIND_PORT` | Port for exchanging network state information. | `string - number` | `7102` |
| `CLUSTER_HOSTNAME` | Hostname of a node. Always set this value if the default OS hostname might change over time. | `string` | `node1` |
Expand Down
30 changes: 30 additions & 0 deletions developers/weaviate/configuration/replication.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,36 @@ import ReplicationConfigWithAsyncRepair from '/_includes/code/configuration/repl

<ReplicationConfigWithAsyncRepair />

### Configuring Async Replication

:::info Added in `v1.29`
Async replication support has been added in `v1.26`while the environment variables for configuring async replication (`ASYNC_*`) have been introduced in `v1.29`.
:::

Async Replication is a background synchronization process in Weaviate that ensures eventual consistency across nodes storing the same shard. When a collection is partitioned into multiple shards, each shard is replicated across several nodes (as defined by the replication factor `REPLICATION_MINIMUM_FACTOR`). Async replication guarantees that all nodes holding the same shard remain in sync by periodically comparing and propagating data.

#### 1. Periodic data comparison

Each node runs a background process that periodically compares its locally stored data with other nodes holding the same shard. This comparison is triggered either:
- At regular intervals (`ASYNC_REPLICATION_FREQUENCY`).
- When a change in availability of a node is detected (`ASYNC_REPLICATION_ALIVE_NODES_CHECKING_FREQUENCY`).

To efficiently detect differences, Weaviate uses a **hashtree**. Instead of checking entire datasets, it compares hash digests at multiple levels, narrowing down differences to specific objects. The size of this hashtree can be defined via `ASYNC_REPLICATION_HASHTREE_HEIGHT`.

If a node is unresponsive, Weaviate applies a timeout (`ASYNC_REPLICATION_DIFF_PER_NODE_TIMEOUT`) to avoid delays in the replication process.

#### 2. Data synchronization

When differences are detected, the outdated or missing data is propagated to the affected nodes. This propagation process:
- Sends data in batches of a custom size (`ASYNC_REPLICATION_BATCH_SIZE`).
- Limits each propagation step object limit (`ASYNC_REPLICATION_PROPAGATION_LIMIT`).
- Enforces a time-bound for updates (`ASYNC_REPLICATION_PROPAGATION_TIMEOUT`).
- Uses a different comparison frequency right after completing synchronization on a node (`ASYNC_REPLICATION_FREQUENCY_WHILE_PROPAGATING`).

:::tip Replication settings
You can find a complete list of the environment variables related to async replication on the page [Reference: Environment variables](/developers/weaviate/config-refs/env-vars#multi-node-instances).
:::

## How to use: Queries

When you add (write) or query (read) data, one or more replica nodes in the cluster will respond to the request. How many nodes need to send a successful response and acknowledgement to the coordinator node depends on the `consistency_level`. Available [consistency levels](../concepts/replication-architecture/consistency.md) are `ONE`, `QUORUM` (replication_factor / 2 + 1) and `ALL`.
Expand Down
2 changes: 1 addition & 1 deletion developers/weaviate/manage-data/collections.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -842,7 +842,7 @@ import RaftRFChangeWarning from '/_includes/1-25-replication-factor.mdx';

<RaftRFChangeWarning/>

Configure replication settings, such as [async replication](../concepts/replication-architecture/consistency.md#async-replication) and [deletion resolution strategy](../concepts/replication-architecture/consistency.md#deletion-resolution-strategies).
Configure replication settings, such as [async replication](/developers/weaviate/configuration/replication#configuring-async-replication) and [deletion resolution strategy](../concepts/replication-architecture/consistency.md#deletion-resolution-strategies).

<Tabs groupId="languages">
<TabItem value="py" label="Python Client v4">
Expand Down

0 comments on commit 5882607

Please sign in to comment.