From a1f8635b7bc0ddf0c4c83ecdb21ed897b23e76b5 Mon Sep 17 00:00:00 2001 From: "Heverson B. Ribeiro" Date: Mon, 26 Jun 2023 15:10:09 +0200 Subject: [PATCH 01/10] issue-650: added a 1st version of discovery authority --- docs/chap-networking.md | 76 +++++++++++++++++++++++++++++++++++++++-- docs/chap-overview.md | 2 +- 2 files changed, 75 insertions(+), 3 deletions(-) diff --git a/docs/chap-networking.md b/docs/chap-networking.md index 87cf4e44c..b85521993 100644 --- a/docs/chap-networking.md +++ b/docs/chap-networking.md @@ -32,7 +32,7 @@ Complete specification of the Polkadot networking protocol relies on the followi Each Polkadot Host node maintains an ED25519 key pair which is used to identify the node. The public key is shared with the rest of the network allowing the nodes to establish secure communication channels. -Each node must have its own unique ED25519 key pair. If two or more nodes use the same key, the network will interpret those nodes as a single node, which will result in unspecified behavior. Furthermore, the node’s *PeerId* as defined in [Definition -def-num-ref-](chap-networking#defn-peer-id) is derived from its public key. *PeerId* is used to identify each node when they are discovered in the course of the discovery mechanism described in [Section -sec-num-ref-](chap-networking#sect-discovery-mechanism). +Each node must have its own unique ED25519 key pair. If two or more nodes use the same key, the network will interpret those nodes as a single node, which will result in unspecified behavior. Furthermore, the node’s *PeerId* as defined in [Definition -def-num-ref-](chap-networking#defn-peer-id) is derived from its public key. *PeerId* is used to identify each node when they are discovered in the course of the discovery mechanism described in [Section -sec-num-ref-](chap-networking#sect-network-bootstrap). ###### Definition -def-num- PeerId {#defn-peer-id} :::definition @@ -74,7 +74,7 @@ $$ - ${b}_{{4}}$, ${b}_{{5}}$ and ${b}_{{{6}.{.37}}}$ are a protobuf encoded field-value pair where ${b}_{{5}}$ indicates the length of the public key followed by the the raw ED25519 public key itself, which varies for each Polkadot Host and is always 32 bytes (field ${2}$ contains the public key, which has a field value length prefix). ::: -## -sec-num- Discovery mechanism {#sect-discovery-mechanism} +## -sec-num- Network bootstrap and discovery {#sect-network-bootstrap} The Polkadot Host uses various mechanisms to find peers within the network, to establish and maintain a list of peers and to share that list with other peers from the network as follows: @@ -214,6 +214,78 @@ The Polkadot Host must actively communicate with the network in order to partici The Polkadot network originally only used SCALE encoding for all message formats. Meanwhile, Protobuf has been adopted for certain messages. The encoding of each listed message is always SCALE encoded unless Protobuf is explicitly mentioned. Encoding and message formats are subject to change. ::: +dskjfsldkfjgsldfjgs + + + + +### -sec-num- Discovering authorities {#sect-authority-discovery} + + +The authority discovery mechanism enables Polkadot nodes to both publish their local addresses and learn about other nodes' local addresses by means of the undelayed Kademlia DHT. Note that, the authority discovery differs from the bootstrap mechanism, described in [Section -sec-num-ref-](chap-networking#sect-network-bootstrap), in that it restricts the discovery output to nodes that are currently holding the authority role (i.e., validators). The authority discovery mechanism consists of two components: the discovery `worker` and the discovery `service`. + + + +The discovery `worker` is in charge of publishing local nodes addresses and discovering other nodes addresses via Kademlia operations. The discovery `service` enables Polkadot nodes to interact with the `worker` by sending `ServicetoWorkerMsg` messages through the `NetworkDHTProvider`. + + +The `ServicetoWorkerMsg` message has two different implementations (or types): `GetAuthorityIdsByPeerId` and `GetAddressesByAuthorityId`. + +#### -sec-num- Requesting authority addresses {#sect-auth-discovery-addresses} +By using authorityIds, Polkadot nodes can find the addresses of other authorities. To do so, a `ServicetoWorkerMsg` of type `GetAddressesByAuthorityId` must be sent to the worker. + +###### Definition -def-num- Authority Addresses Request {#defn-msg-authority-address-request} +:::definition + +An **authority addresses request** is sent to the `worker` to request the addresses of an authority node with a given `AuthorityId`. An `AuthorityId` is the 256-bit identifier that is sent as a message's parameter. A `Multiaddr` data structure is returned if an entry is found, None otherwise. +::: + +#### -sec-num- Requesting Authority Ids {#sect-auth-discovery-authorityId} +By using the `PeerId` of authorities, Polkadot nodes can find their associated `AuthorityId`. To achieve that, a `ServicetoWorkerMsg` of type `GetAuthorityIdsByPeerId` must be sent to the worker. + + +###### Definition -def-num- Authority Id Request {#defn-msg-authority-authid-request} +:::definition + +An **authority id request** is sent to the `worker` to request the `AuthorityId` of an authority node for a given `PeerId`. +An `PeerId` the node’s PeerId as defined in [Definition -def-num-ref-](chap-networking#defn-peer-id). An `AuthorityId` is returned if an entry is found, None otherwise. +::: + +#### -sec-num- Worker requests {#sect-auth-discovery-worker} +The `worker` component of the authority discovery has an internal data structure called `AddrCache` . This data structure is used to cache the addresses and authority identifiers found on Kademlia. `AddrCache` maps `AuthorityId` $\to$ `Multiaddr` and `PeerId` $\to$ `AuthorityId`. The `AddrCache` is used to handle the [Authority Addresses Request](chap-networking#defn-msg-authority-address-request) and [Authority Id Request](chap-networking#defn-msg-authority-authid-request) above. + +The `worker` periodically sends `put_value()` messages to the DHT to publish a `SignedAuthorityRecord` of authorities it knows about. + + +###### Definition -def-num- Signed Authority Record {#defn-msg-signed-authority-record} +:::definition + +The `SignedAuthorityRecord` is a Protobuf serialized structure representing the authority records and signature to send over the wire. It is defined in the following format: + +| Type | Id | Description | +|--------------------|-----|-------------------------------------------------------| +| *AuthorityRecord* | 1 | Serialized authority record | +| `bytes` | 2 | An Schnorrkel/Ristretto x25519 ("sr25519") signature | +| *PeerSignature* | 3 | Serialized peer signature | +**where** + +**AuthorityRecord** is a Protobuf serialized structure indicating the addresses that will be signed. + +| Type | Id | Description | +|-------------------|-----|-------------------------------------------------------------------------| +| `repeated bytes` | 1 | Possibly multiple `MultiAddress`es through which a node can be connected | + + +**PeerSignature** is a Protobuf serialized structure indicating the authority record signature and public key used to verify the record. + +| Type | Id | Description | +|----------|-----|---------------------------------------------------| +| `bytes` | 1 | An sr25519 signature | +| `bytes` | 1 | A sr25519 public key used to verify the signature | +::: + + + ### -sec-num- Announcing blocks {#sect-msg-block-announce} When the node creates or receives a new block, it must be announced to the network. Other nodes within the network will track this announcement and can request information about this block. The mechanism for tracking announcements and requesting the required data is implementation-specific. diff --git a/docs/chap-overview.md b/docs/chap-overview.md index 3776c9eb7..7d66247ac 100644 --- a/docs/chap-overview.md +++ b/docs/chap-overview.md @@ -16,7 +16,7 @@ While the full node is still a mostly passive participant of the protocol, they 1. The node must populate the state storage with the official genesis state, elaborated further in [Section -sec-num-ref-](id-cryptography-encoding#section-genesis). - 2. The node should maintain a set of around 50 active peers at any time. New peers can be found using the discovery protocols ([Section -sec-num-ref-](chap-networking#sect-discovery-mechanism)) + 2. The node should maintain a set of around 50 active peers at any time. New peers can be found using the discovery protocols ([Section -sec-num-ref-](chap-networking#sect-network-bootstrap)) 3. The node should open and maintain the various required streams ([Section -sec-num-ref-](chap-networking#sect-protocols-substreams)) with each of its active peers. From 978d853604fece4630d4ece0c3e4c56b357eb092 Mon Sep 17 00:00:00 2001 From: "Heverson B. Ribeiro" Date: Mon, 3 Jul 2023 22:23:43 +0200 Subject: [PATCH 02/10] review text to focus on the worker exchange with dht --- docs/chap-networking.md | 92 ++++++++++++++++++++++++----------------- 1 file changed, 54 insertions(+), 38 deletions(-) diff --git a/docs/chap-networking.md b/docs/chap-networking.md index b85521993..f6c092c9f 100644 --- a/docs/chap-networking.md +++ b/docs/chap-networking.md @@ -74,7 +74,7 @@ $$ - ${b}_{{4}}$, ${b}_{{5}}$ and ${b}_{{{6}.{.37}}}$ are a protobuf encoded field-value pair where ${b}_{{5}}$ indicates the length of the public key followed by the the raw ED25519 public key itself, which varies for each Polkadot Host and is always 32 bytes (field ${2}$ contains the public key, which has a field value length prefix). ::: -## -sec-num- Network bootstrap and discovery {#sect-network-bootstrap} +## -sec-num- Network bootstrap and peer discovery {#sect-network-bootstrap} The Polkadot Host uses various mechanisms to find peers within the network, to establish and maintain a list of peers and to share that list with other peers from the network as follows: @@ -222,68 +222,84 @@ dskjfsldkfjgsldfjgs ### -sec-num- Discovering authorities {#sect-authority-discovery} -The authority discovery mechanism enables Polkadot nodes to both publish their local addresses and learn about other nodes' local addresses by means of the undelayed Kademlia DHT. Note that, the authority discovery differs from the bootstrap mechanism, described in [Section -sec-num-ref-](chap-networking#sect-network-bootstrap), in that it restricts the discovery output to nodes that are currently holding the authority role (i.e., validators). The authority discovery mechanism consists of two components: the discovery `worker` and the discovery `service`. +A discovery mechanim enables Polkadot nodes to both publish their local addresses and learn about other nodes' local addresses by means of the undelayed Kademlia DHT. +The Authority discovery mechanism differs from the bootstrap mechanism, described in [Section -sec-num-ref-](chap-networking#sect-network-bootstrap), in that it restricts the discovery output to nodes that are currently holding the authority role (i.e., validators). The authority discovery mechanism consists of two components: the discovery `worker` and the discovery `service`. -The discovery `worker` is in charge of publishing local nodes addresses and discovering other nodes addresses via Kademlia operations. The discovery `service` enables Polkadot nodes to interact with the `worker` by sending `ServicetoWorkerMsg` messages through the `NetworkDHTProvider`. +The discovery `worker` is in charge of publishing local node addresses and looking up for the addresses of other authority nodes. The discovery `service` enables Polkadot nodes to interact with the `worker`, which in turn send and receive messages to the Kademlia DHT via `NetworkDHTProvider`. -The `ServicetoWorkerMsg` message has two different implementations (or types): `GetAuthorityIdsByPeerId` and `GetAddressesByAuthorityId`. -#### -sec-num- Requesting authority addresses {#sect-auth-discovery-addresses} -By using authorityIds, Polkadot nodes can find the addresses of other authorities. To do so, a `ServicetoWorkerMsg` of type `GetAddressesByAuthorityId` must be sent to the worker. - -###### Definition -def-num- Authority Addresses Request {#defn-msg-authority-address-request} -:::definition - -An **authority addresses request** is sent to the `worker` to request the addresses of an authority node with a given `AuthorityId`. An `AuthorityId` is the 256-bit identifier that is sent as a message's parameter. A `Multiaddr` data structure is returned if an entry is found, None otherwise. -::: - -#### -sec-num- Requesting Authority Ids {#sect-auth-discovery-authorityId} -By using the `PeerId` of authorities, Polkadot nodes can find their associated `AuthorityId`. To achieve that, a `ServicetoWorkerMsg` of type `GetAuthorityIdsByPeerId` must be sent to the worker. - - -###### Definition -def-num- Authority Id Request {#defn-msg-authority-authid-request} -:::definition - -An **authority id request** is sent to the `worker` to request the `AuthorityId` of an authority node for a given `PeerId`. -An `PeerId` the node’s PeerId as defined in [Definition -def-num-ref-](chap-networking#defn-peer-id). An `AuthorityId` is returned if an entry is found, None otherwise. -::: - -#### -sec-num- Worker requests {#sect-auth-discovery-worker} -The `worker` component of the authority discovery has an internal data structure called `AddrCache` . This data structure is used to cache the addresses and authority identifiers found on Kademlia. `AddrCache` maps `AuthorityId` $\to$ `Multiaddr` and `PeerId` $\to$ `AuthorityId`. The `AddrCache` is used to handle the [Authority Addresses Request](chap-networking#defn-msg-authority-address-request) and [Authority Id Request](chap-networking#defn-msg-authority-authid-request) above. - -The `worker` periodically sends `put_value()` messages to the DHT to publish a `SignedAuthorityRecord` of authorities it knows about. +#### -sec-num- Publishing local addresses {#sect-auth-discovery-worker-publishing} + +The `worker` periodically sends `put_value()` messages to the DHT in order to publish a `SignedAuthorityRecord` of addresses of authorities it knows from the current authority sets. The payload of the `put_value()` message to send to Kademlia DHT is created as follows. + ###### Definition -def-num- Signed Authority Record {#defn-msg-signed-authority-record} :::definition -The `SignedAuthorityRecord` is a Protobuf serialized structure representing the authority records and signature to send over the wire. It is defined in the following format: +The `SignedAuthorityRecord` is a Protobuf serialized structure representing the authority records and signature to send over the wire. +It is defined in the following format: | Type | Id | Description | |--------------------|-----|-------------------------------------------------------| | *AuthorityRecord* | 1 | Serialized authority record | | `bytes` | 2 | An Schnorrkel/Ristretto x25519 ("sr25519") signature | | *PeerSignature* | 3 | Serialized peer signature | + **where** -**AuthorityRecord** is a Protobuf serialized structure indicating the addresses that will be signed. +**AuthorityRecord** is a serialized Protobuf structure that lists the addresses of authority nodes that are currently part of the authority set. -| Type | Id | Description | -|-------------------|-----|-------------------------------------------------------------------------| -| `repeated bytes` | 1 | Possibly multiple `MultiAddress`es through which a node can be connected | +| Type | Id | Description | +|------------------|----|--------------------------------------------------------------------------| +| `repeated bytes` | 1 | Possibly multiple `MultiAddress`es through which a node can be connected | +**PeerSignature** is a Protobuf serialized structure indicating the signature and public key used to sign and verify the `AuthorityRecord`. +This is the protobuf structure used to exchange the signature with other nodes. -**PeerSignature** is a Protobuf serialized structure indicating the authority record signature and public key used to verify the record. +| Type | Id | Description | +|---------|-----|---------------------------------------------------| +| `bytes` | 1 | An sr25519 signature | +| `bytes` | 2 | A sr25519 public key used to verify the signature | -| Type | Id | Description | -|----------|-----|---------------------------------------------------| -| `bytes` | 1 | An sr25519 signature | -| `bytes` | 1 | A sr25519 public key used to verify the signature | ::: + +###### Definition -def-num- Publish authority addresses message {#defn-msg-auth-discovery-publish} +:::definition + +For each authority node $i$ in the current authority set, the local node sends a `put_value()` message to the Kademlia DHT in the following format: + +$$ +\text{PUT}_{\left({KademliaKey}_{i} , {Sig}_{AR}\right)} +$$ + +**where** +- ${KademliaKey}_{i}$ is the $Sha256$ hash of the authorityID of node $i$. + +- ${Sig}_{AR}$ is the [SignedAuthorityRecord](chap-networking#defn-msg-signed-authority-record) described above. + +::: + +#### -sec-num- Discovering authority addresses {#sect-auth-discovery-worker-discover} + +The `worker` periodically sends `get_value()` messages to the Kedemlia DHT in order to discover addresses of authority nodes it knows about. + +###### Definition -def-num- Discover authority addresses message {#defn-msg-auth-discovery-lookup} +:::definition + +At each cycle, nodes `worker` creates a bounded number of `get_value()` messages in the following format: +$$ +\text{GET}_{\left({KademliaKey}_{i}\right)} +$$ + +**where** +- ${KademliaKey}_{i}$ is the $Sha256$ hash of the `authorityID` of node $i$ selected from the current authority set. +::: + ### -sec-num- Announcing blocks {#sect-msg-block-announce} From 89e15f3037d76b906c396166981ed0f4e28e7c65 Mon Sep 17 00:00:00 2001 From: "Heverson B. Ribeiro" Date: Tue, 4 Jul 2023 09:33:03 +0200 Subject: [PATCH 03/10] minor correction --- docs/chap-networking.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/chap-networking.md b/docs/chap-networking.md index f6c092c9f..c57381010 100644 --- a/docs/chap-networking.md +++ b/docs/chap-networking.md @@ -74,7 +74,7 @@ $$ - ${b}_{{4}}$, ${b}_{{5}}$ and ${b}_{{{6}.{.37}}}$ are a protobuf encoded field-value pair where ${b}_{{5}}$ indicates the length of the public key followed by the the raw ED25519 public key itself, which varies for each Polkadot Host and is always 32 bytes (field ${2}$ contains the public key, which has a field value length prefix). ::: -## -sec-num- Network bootstrap and peer discovery {#sect-network-bootstrap} +## -sec-num- Network bootstrap and discovery {#sect-network-bootstrap} The Polkadot Host uses various mechanisms to find peers within the network, to establish and maintain a list of peers and to share that list with other peers from the network as follows: @@ -280,7 +280,9 @@ $$ **where** - ${KademliaKey}_{i}$ is the $Sha256$ hash of the authorityID of node $i$. -- ${Sig}_{AR}$ is the [SignedAuthorityRecord](chap-networking#defn-msg-signed-authority-record) described above. +- ${Sig}_{AR}$ is the [SignedAuthorityRecord](chap-networking#defn-msg-signed-authority-record) +([Definition -def-num-ref-](chap-networking#defn-msg-signed-authority-record)) + described above. ::: From 60ce6bdee9ceec39db90c5e518da5672ecd610d7 Mon Sep 17 00:00:00 2001 From: "Heverson B. Ribeiro" Date: Wed, 5 Jul 2023 13:55:30 +0200 Subject: [PATCH 04/10] improved version --- docs/chap-networking.md | 110 +++++++++++++++++++++++++++++++--------- 1 file changed, 87 insertions(+), 23 deletions(-) diff --git a/docs/chap-networking.md b/docs/chap-networking.md index c57381010..2df238316 100644 --- a/docs/chap-networking.md +++ b/docs/chap-networking.md @@ -222,19 +222,83 @@ dskjfsldkfjgsldfjgs ### -sec-num- Discovering authorities {#sect-authority-discovery} -A discovery mechanim enables Polkadot nodes to both publish their local addresses and learn about other nodes' local addresses by means of the undelayed Kademlia DHT. -The Authority discovery mechanism differs from the bootstrap mechanism, described in [Section -sec-num-ref-](chap-networking#sect-network-bootstrap), in that it restricts the discovery output to nodes that are currently holding the authority role (i.e., validators). The authority discovery mechanism consists of two components: the discovery `worker` and the discovery `service`. +A discovery mechanism enables Polkadot nodes to both publish their local addresses and learn about other nodes identifiers and addresses by means of the undelayed Kademlia DHT. +The Authority discovery mechanism differs from the bootstrap mechanism, described in [Section -sec-num-ref-](chap-networking#sect-network-bootstrap), in that it restricts the discovery output to nodes currently holding the authority role (e.g., validators). +The authority discovery mechanism consists of two main components: discovery `service` and discovery `worker`. The `service` provides an interface to Polkadot nodes that exposes two functions that allow Polkadot nodes to request authority identifiers or authority addresses. The `worker` interacts with the Kademlia DHT in order to publish local node addresses and discover the addresses of other authority nodes. - -The discovery `worker` is in charge of publishing local node addresses and looking up for the addresses of other authority nodes. The discovery `service` enables Polkadot nodes to interact with the `worker`, which in turn send and receive messages to the Kademlia DHT via `NetworkDHTProvider`. + +#### -sec-num- Requesting authority identifier and addresses {#sect-auth-discovery-service-requests} + +The following requests must be exposed by the discovery authority mechanism to Polkadot nodes. - +###### Definition -def-num- Authority addresses request {#defn-auth-discovery-address-request} +:::definition + +An **authority addresses request** is a request that Polkadot nodes can send to the authority discovery mechanism in order to request the addresses of an authority node. The request has the following format: + +$$ +\texttt{get\_addresses\_by\_authority\_id}{\left(\texttt{authorityId}\right)} +$$ + +**where** +- $\texttt{authorityId}$ is the `authorityId` 256-bit identifier representing the public key of the targeted authority node. + + +**expected response** + +The response to the previous query includes an enum with one of the following values: + + +| Value | Description | +|----------------------|--------------------------------------------------------| +| None | A type representing `no value` | +| HashSet[Multiaddr] | An unordered collection of unique `Multiaddr` elements | + + +**with** +- `Multiaddr` a [Multiaddr](https://github.com/libp2p/specs/blob/master/addressing/README.md#multiaddr-in-libp2p) data structure. + +::: + + + +###### Definition -def-num- Authority identifier request {#defn-auth-discovery-auth-identifier-request} +:::definition + +An **authority identifier request** is a request that polkadot nodes can send to the authority discovery mechanism in order to request the `AuthorityId` of an authority node. The request has the following format: +$$ +\texttt{get\_authority\_ids\_by\_peer\_id}{\left(\texttt{PeerId}\right)} +$$ + +**where** +- $\texttt{PeerId}$ is the Polkadot node’s PeerId ([Definition -def-num-ref-](chap-networking#defn-peer-id)). + + +**expected response** + +The response to the previous query includes an enum with one of the following values: + + +| Value | Description | +|------------------------|--------------------------------------------------------| +| None | A type representing `no value` | +| HashSet[authorityId] | An unordered collection of unique `authorityId` elements | + + +**with** +- `authorityId` the 256-bit identifier representing the public key of the requested `PeerId`. + +::: + + + #### -sec-num- Publishing local addresses {#sect-auth-discovery-worker-publishing} - + -The `worker` periodically sends `put_value()` messages to the DHT in order to publish a `SignedAuthorityRecord` of addresses of authorities it knows from the current authority sets. The payload of the `put_value()` message to send to Kademlia DHT is created as follows. +The authority discovery mechanism periodically sends `put_value` messages to the DHT in order to publish a `SignedAuthorityRecord` of the addresses of authorities it knows from its current authority sets. The `put_value` message to be sent to Kademlia DHT is created as follows. ###### Definition -def-num- Signed Authority Record {#defn-msg-signed-authority-record} @@ -253,9 +317,9 @@ It is defined in the following format: **AuthorityRecord** is a serialized Protobuf structure that lists the addresses of authority nodes that are currently part of the authority set. -| Type | Id | Description | -|------------------|----|--------------------------------------------------------------------------| -| `repeated bytes` | 1 | Possibly multiple `MultiAddress`es through which a node can be connected | +| Type | Id | Description | +|------------------|----|-----------------------------------------------------------------------| +| `repeated bytes` | 1 | Possibly multiple `MultiAddr`ess through which a node can be connected | **PeerSignature** is a Protobuf serialized structure indicating the signature and public key used to sign and verify the `AuthorityRecord`. This is the protobuf structure used to exchange the signature with other nodes. @@ -271,38 +335,38 @@ This is the protobuf structure used to exchange the signature with other nodes. ###### Definition -def-num- Publish authority addresses message {#defn-msg-auth-discovery-publish} :::definition -For each authority node $i$ in the current authority set, the local node sends a `put_value()` message to the Kademlia DHT in the following format: +For each authority node $i$ in the current authority set, the local node sends a `put_value` message to the Kademlia DHT in the following format: $$ -\text{PUT}_{\left({KademliaKey}_{i} , {Sig}_{AR}\right)} +\texttt{put\_value}{\left(\texttt{KademliaKey}_{i} , \texttt{Sig}_{AR}\right)} $$ -**where** -- ${KademliaKey}_{i}$ is the $Sha256$ hash of the authorityID of node $i$. -- ${Sig}_{AR}$ is the [SignedAuthorityRecord](chap-networking#defn-msg-signed-authority-record) -([Definition -def-num-ref-](chap-networking#defn-msg-signed-authority-record)) - described above. +**where** +- $\texttt{KademliaKey}_{i}$ is the $Sha256$ hash of the authorityId of node $i$. +- $\texttt{Sig}_{AR}$ is the `SignedAuthorityRecord` + described above([Definition -def-num-ref-](chap-networking#defn-msg-signed-authority-record)). ::: -#### -sec-num- Discovering authority addresses {#sect-auth-discovery-worker-discover} -The `worker` periodically sends `get_value()` messages to the Kedemlia DHT in order to discover addresses of authority nodes it knows about. + +#### -sec-num- Discovering authority addresses {#sect-auth-discovery-worker-discover} + +The authority discovery mechanism periodically sends `get_value` messages to the Kedemlia DHT in order to discover addresses of authority nodes it knows about. ###### Definition -def-num- Discover authority addresses message {#defn-msg-auth-discovery-lookup} :::definition -At each cycle, nodes `worker` creates a bounded number of `get_value()` messages in the following format: +At each cycle, nodes `worker` creates a bounded number of `get_value` messages to send in the following format: $$ -\text{GET}_{\left({KademliaKey}_{i}\right)} +\texttt{get\_value}{\left(\texttt{KademliaKey}_{i}\right)} $$ **where** -- ${KademliaKey}_{i}$ is the $Sha256$ hash of the `authorityID` of node $i$ selected from the current authority set. +- $\texttt{KademliaKey}_{i}$ is the $Sha256$ hash of the `authorityId` of node $i$ selected from the current authority set. ::: - ### -sec-num- Announcing blocks {#sect-msg-block-announce} From 7ec2b1c11cf53d26fd8b4b53c6198bb0bff4ac36 Mon Sep 17 00:00:00 2001 From: "Heverson B. Ribeiro" Date: Wed, 5 Jul 2023 17:27:28 +0200 Subject: [PATCH 05/10] removed typing mistake --- docs/chap-networking.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/chap-networking.md b/docs/chap-networking.md index 2df238316..75697c876 100644 --- a/docs/chap-networking.md +++ b/docs/chap-networking.md @@ -214,8 +214,6 @@ The Polkadot Host must actively communicate with the network in order to partici The Polkadot network originally only used SCALE encoding for all message formats. Meanwhile, Protobuf has been adopted for certain messages. The encoding of each listed message is always SCALE encoded unless Protobuf is explicitly mentioned. Encoding and message formats are subject to change. ::: -dskjfsldkfjgsldfjgs - From 3378354e91e77a99fdd5ae7516c2c27050abefc8 Mon Sep 17 00:00:00 2001 From: David Hawig Date: Mon, 24 Jul 2023 10:48:28 +0200 Subject: [PATCH 06/10] Update chap-networking.md Spelling fixes --- docs/chap-networking.md | 76 ++++++++++++++++++++--------------------- 1 file changed, 38 insertions(+), 38 deletions(-) diff --git a/docs/chap-networking.md b/docs/chap-networking.md index 75697c876..81a20d447 100644 --- a/docs/chap-networking.md +++ b/docs/chap-networking.md @@ -3,12 +3,12 @@ title: -chap-num- Networking --- :::info -This chapter in its current form is incomplete and considered work in progress. Authors appreciate receiving request for clarification or any reports regarding deviation from the current Polkadot network protocol. This can be done through filing an issue in [Polkadot Specification repository](https://github.com/w3f/polkadot-spec). +This chapter, in its current form, is incomplete and considered work in progress. Authors appreciate receiving requests for clarification or any reports regarding deviation from the current Polkadot network protocol. This can be done by filing an issue in [Polkadot Specification repository](https://github.com/w3f/polkadot-spec). ::: ## -sec-num- Introduction {#id-introduction-2} -The Polkadot network is decentralized and does not rely on any central authority or entity for achieving its fullest potential of provided functionality. The networking protocol is based on a family of open protocols, including protocol implemented *libp2p* e.g. the distributed Kademlia hash table which is used for peer discovery. +The Polkadot network is decentralized and does not rely on any central authority or entity to achieve its fullest potential of provided functionality. The networking protocol is based on a family of open protocols, including protocol implemented *libp2p*, e.g., the distributed Kademlia hash table, which is used for peer discovery. This chapter walks through the behavior of the networking implementation of the Polkadot Host and defines the network messages. The implementation details of the *libp2p* protocols used are specified in external sources as described in [Section -sec-num-ref-](chap-networking#sect-networking-external-docs) @@ -16,7 +16,7 @@ This chapter walks through the behavior of the networking implementation of the Complete specification of the Polkadot networking protocol relies on the following external protocols: -- [libp2p](https://github.com/libp2p/specs) - *libp2p* is a modular peer-to-peer networking stack composed of many modules and different parts. includes the multiplexing protocols and . +- [libp2p](https://github.com/libp2p/specs) - *libp2p* is a modular peer-to-peer networking stack composed of many modules and different parts. includes the multiplexing protocols and - [libp2p addressing](https://docs.libp2p.io/concepts/addressing/) - The Polkadot Host uses the *libp2p* addressing system to identify and connect to peers. @@ -32,7 +32,7 @@ Complete specification of the Polkadot networking protocol relies on the followi Each Polkadot Host node maintains an ED25519 key pair which is used to identify the node. The public key is shared with the rest of the network allowing the nodes to establish secure communication channels. -Each node must have its own unique ED25519 key pair. If two or more nodes use the same key, the network will interpret those nodes as a single node, which will result in unspecified behavior. Furthermore, the node’s *PeerId* as defined in [Definition -def-num-ref-](chap-networking#defn-peer-id) is derived from its public key. *PeerId* is used to identify each node when they are discovered in the course of the discovery mechanism described in [Section -sec-num-ref-](chap-networking#sect-network-bootstrap). +Each node must have its own unique ED25519 key pair. If two or more nodes use the same key, the network will interpret those nodes as a single node, which will result in unspecified behavior. Furthermore, the node’s *PeerId*, as defined in [Definition -def-num-ref-](chap-networking#defn-peer-id), is derived from its public key. *PeerId* is used to identify each node when they are discovered in the course of the discovery mechanism described in [Section -sec-num-ref-](chap-networking#sect-network-bootstrap). ###### Definition -def-num- PeerId {#defn-peer-id} :::definition @@ -71,12 +71,12 @@ $$ - ${b}_{{2}}$ and ${b}_{{3}}$ are a protobuf encoded field-value pair [indicating the used key type](https://github.com/libp2p/specs/blob/master/peer-ids/peer-ids.md#keys) (field ${1}$ of value ${1}$ implies *ED25519*). -- ${b}_{{4}}$, ${b}_{{5}}$ and ${b}_{{{6}.{.37}}}$ are a protobuf encoded field-value pair where ${b}_{{5}}$ indicates the length of the public key followed by the the raw ED25519 public key itself, which varies for each Polkadot Host and is always 32 bytes (field ${2}$ contains the public key, which has a field value length prefix). +- ${b}_{{4}}$, ${b}_{{5}}$ and ${b}_{{{6}.{.37}}}$ are a protobuf encoded field-value pair where ${b}_{{5}}$ indicates the length of the public key followed by the raw ED25519 public key itself, which varies for each Polkadot Host and is always 32 bytes (field ${2}$ contains the public key, which has a field value length prefix). ::: ## -sec-num- Network bootstrap and discovery {#sect-network-bootstrap} -The Polkadot Host uses various mechanisms to find peers within the network, to establish and maintain a list of peers and to share that list with other peers from the network as follows: +The Polkadot Host uses various mechanisms to find peers within the network, to establish and maintain a list of peers, and to share that list with other peers from the network as follows: - **Bootstrap nodes** are hard-coded node identities and addresses provided by the genesis state ([Section -sec-num-ref-](id-cryptography-encoding#section-genesis)). @@ -108,7 +108,7 @@ The Polkadot Host can establish a connection with any peer of which it knows the - **TCP/IP** with addresses in the form of `/ip4/1.2.3.4/tcp/30333` to establish a TCP connection and negotiate encryption and a multiplexing layer. -- **WebSocket** with addresses in the form of `/ip4/1.2.3.4/tcp/30333/ws` to establish a TCP connection and negotiate the WebSocket protocol within the connection. Additionally, encryption and multiplexing layer is negotiated within the WebSocket connection. +- **WebSocket** with addresses in the form of `/ip4/1.2.3.4/tcp/30333/ws` to establish a TCP connection and negotiate the WebSocket protocol within the connection. Additionally, the encryption and multiplexing layer are negotiated within the WebSocket connection. - **DNS** addresses in form of `/dns/example.com/tcp/30333` and `/dns/example.com/tcp/30333/ws`. @@ -120,11 +120,11 @@ Polkadot protocol uses the *libp2p* Noise framework to build an encryption proto Polkadot nodes use the [XX handshake pattern](https://noiseexplorer.com/patterns/XX/) to establish a connection between peers. The three following steps are required to complete the handshake process: -1. The initiator generates a keypair and sends the public key to the responder. The [Noise specification](https://github.com/libp2p/specs/tree/master/noise) and the [libp2p PeerId specification](https://github.com/libp2p/specs/blob/master/peer-ids/peer-ids.md) describe keypairs in more detail. +1. The initiator generates a key pair and sends the public key to the responder. The [Noise specification](https://github.com/libp2p/specs/tree/master/noise) and the [libp2p PeerId specification](https://github.com/libp2p/specs/blob/master/peer-ids/peer-ids.md) describe keypairs in more detail. -2. The responder generates its own key pair and sends its public key back to the initiator. After that, the responder derives a shared secret and uses it to encrypt all further communication. The responder now sends its static Noise public key (which may change anytime and does not need to be persisted on disk), its *libp2p* public key and a signature of the static Noise public key signed with the *libp2p* public key. +2. The responder generates its own key pair and sends its public key back to the initiator. After that, the responder derives a shared secret and uses it to encrypt all further communication. The responder now sends its static Noise public key (which may change anytime and does not need to be persisted on disk), its *libp2p* public key, and a signature of the static Noise public key signed with the *libp2p* public key. -3. The initiator derives a shared secret and uses it to encrypt all further communication. It also sends its static Noise public key, *libp2p* public key and signature to the responder. +3. The initiator derives a shared secret and uses it to encrypt all further communication. It also sends its static Noise public key, *libp2p* public key, and signature to the responder. After these three steps, both the initiator and responder derive a new shared secret using the static and session-defined Noise keys, which are used to encrypt all further communication. @@ -140,11 +140,11 @@ The prefixes on those substreams are known as protocol identifiers and are used - `/ipfs/ping/1.0.0` - Open a standardized substream *libp2p* to a peer and initialize a ping to verify if a connection is still alive. If the peer does not respond, the connection is dropped. This is a *Request-Response substream*. - Further specification and reference implementation are available in the [libp2p documentation](https://docs.libp2p.io/concepts/protocols/#ping). + Further specification and reference implementations are available in the [libp2p documentation](https://docs.libp2p.io/concepts/protocols/#ping). - `/ipfs/id/1.0.0` - Open a standardized *libp2p* substream to a peer to ask for information about that peer. This is a *Request-Response substream*, but the initiator does **not** send any message to the responder and only waits for the response. - Further specification and reference implementation are available in the [libp2p documentation](https://docs.libp2p.io/concepts/protocols/#identify). + Further specification and reference implementations are available in the [libp2p documentation](https://docs.libp2p.io/concepts/protocols/#identify). - `/dot/kad` - Open a standardized substream for Kademlia `FIND_NODE` requests. This is a *Request-Response substream*, as defined by the *libp2p* standard. @@ -155,7 +155,7 @@ The prefixes on those substreams are known as protocol identifiers and are used The messages are specified in [Section -sec-num-ref-](sect-lightclient#sect-light-msg). :::info - For backwards compatibility reasons, `/dot/light/2` is also a valid substream for those messages. + For backward compatibility reasons, `/dot/light/2` is also a valid substream for those messages. ::: - `/91b171bb158e2d3848fa23a9f1c25182fb8e20313b2c1eb49219da7a70ce90c3/block-announces/1` - a substream/notification protocol which sends blocks to connected peers. This is a *Notification substream*. @@ -163,7 +163,7 @@ The prefixes on those substreams are known as protocol identifiers and are used The messages are specified in [Section -sec-num-ref-](chap-networking#sect-msg-block-announce). :::info - For backwards compatibility reasons, `/dot/block-announces/1` is also a valid substream for those messages. + For backward compatibility reasons, `/dot/block-announces/1` is also a valid substream for those messages. ::: - `/91b171bb158e2d3848fa23a9f1c25182fb8e20313b2c1eb49219da7a70ce90c3/sync/2` - a request and response protocol that allows the Polkadot Host to request information about blocks. This is a *Request-Response substream*. @@ -171,7 +171,7 @@ The prefixes on those substreams are known as protocol identifiers and are used The messages are specified in [Section -sec-num-ref-](chap-networking#sect-msg-block-request). :::info - For backwards compatibility reasons, `/dot/sync/2` is also a valid substream for those messages. + For backward compatibility reasons, `/dot/sync/2` is also a valid substream for those messages. ::: - `/91b171bb158e2d3848fa23a9f1c25182fb8e20313b2c1eb49219da7a70ce90c3/sync/warp` - a request and response protocol that allows the Polkadot Host to perform a warp sync request. This is a *Request-Response substream*. @@ -179,7 +179,7 @@ The prefixes on those substreams are known as protocol identifiers and are used The messages are specified in [Section -sec-num-ref-](chap-networking#sect-msg-warp-sync). :::info - For backwards compatibility reasons, `/dot/sync/warp` is also a valid substream for those messages. + For backward compatibility reasons, `/dot/sync/warp` is also a valid substream for those messages. ::: - `/91b171bb158e2d3848fa23a9f1c25182fb8e20313b2c1eb49219da7a70ce90c3/transactions/1` - a substream/notification protocol which sends transactions to connected peers. This is a *Notification substream*. @@ -187,7 +187,7 @@ The prefixes on those substreams are known as protocol identifiers and are used The messages are specified in [Section -sec-num-ref-](chap-networking#sect-msg-transactions). :::info - For backwards compatibility reasons, `/dot/transactions/1` is also a valid substream for those messages. + For backward compatibility reasons, `/dot/transactions/1` is also a valid substream for those messages. ::: - `/91b171bb158e2d3848fa23a9f1c25182fb8e20313b2c1eb49219da7a70ce90c3/grandpa/1` - a substream/notification protocol that sends GRANDPA votes to connected peers. This is a *Notification substream*. @@ -195,7 +195,7 @@ The prefixes on those substreams are known as protocol identifiers and are used The messages are specified in [Section -sec-num-ref-](chap-networking#sect-msg-grandpa). :::info - For backwards compatibility reasons, `/paritytech/grandpa/1` is also a valid substream for those messages. + For backward compatibility reasons, `/paritytech/grandpa/1` is also a valid substream for those messages. ::: - `/91b171bb158e2d3848fa23a9f1c25182fb8e20313b2c1eb49219da7a70ce90c3/beefy/1` - a substream/notification protocol which sends signed BEEFY statements, as described in [Section -sec-num-ref-](sect-finality#sect-grandpa-beefy), to connected peers. This is a *Notification* substream. @@ -203,7 +203,7 @@ The prefixes on those substreams are known as protocol identifiers and are used The messages are specified in [Section -sec-num-ref-](chap-networking#sect-msg-grandpa-beefy). :::info - For backwards compatibility reasons, `/paritytech/beefy/1` is also a valid substream for those messages. + For backward compatibility reasons, `/paritytech/beefy/1` is also a valid substream for those messages. ::: ## -sec-num- Network Messages {#sect-network-messages} @@ -265,7 +265,7 @@ The response to the previous query includes an enum with one of the following va ###### Definition -def-num- Authority identifier request {#defn-auth-discovery-auth-identifier-request} :::definition -An **authority identifier request** is a request that polkadot nodes can send to the authority discovery mechanism in order to request the `AuthorityId` of an authority node. The request has the following format: +An **authority identifier request** is a request that Polkadot nodes can send to the authority discovery mechanism in order to request the `AuthorityId` of an authority node. The request has the following format: $$ \texttt{get\_authority\_ids\_by\_peer\_id}{\left(\texttt{PeerId}\right)} @@ -351,7 +351,7 @@ $$ #### -sec-num- Discovering authority addresses {#sect-auth-discovery-worker-discover} -The authority discovery mechanism periodically sends `get_value` messages to the Kedemlia DHT in order to discover addresses of authority nodes it knows about. +The authority discovery mechanism periodically sends `get_value` messages to the Kedemlia DHT in order to discover the addresses of authority nodes it knows about. ###### Definition -def-num- Discover authority addresses message {#defn-msg-auth-discovery-lookup} :::definition @@ -370,7 +370,7 @@ $$ When the node creates or receives a new block, it must be announced to the network. Other nodes within the network will track this announcement and can request information about this block. The mechanism for tracking announcements and requesting the required data is implementation-specific. -Block announcements, requests and responses are sent over the substream as described in [Definition -def-num-ref-](chap-networking#defn-block-announce-handshake). +Block announcements, requests, and responses are sent over the substream as described in [Definition -def-num-ref-](chap-networking#defn-block-announce-handshake). ###### Definition -def-num- Block Announce Handshake {#defn-block-announce-handshake} :::definition @@ -404,7 +404,7 @@ $$ The `BlockAnnounce` message is sent to the specified substream and indicates to remote peers that the node has either created or received a new block. -The message is a structure of the following format: +The message is structured in the following format: $$ {B}{A}=\text{Enc}_{{\text{SC}}}{\left(\text{Head}{\left({B}\right)},{b}\right)} @@ -486,18 +486,18 @@ where *BlockData* is a Protobuf structure containing the requested blocks. Do no ::: ### -sec-num- Requesting States {#sect-msg-state-request} -The Polkadot Host can request the state in form of a key/value list at a specified block. +The Polkadot Host can request the state in the form of a key/value list at a specified block. -When receiving state entries from the state response messages ([Definition -def-num-ref-](chap-networking#defn-msg-state-response)), the node can verify the entries with the entry proof (id *1* in *KeyValueStorage*) against the merkle root in the block header (of the block specified in [Definition -def-num-ref-](chap-networking#defn-msg-state-request)). Once the state response message claims that all entries have been sent (id *3* in *KeyValueStorage*), the node can use all collected entry proofs and validate it against the merkle root to confirm that claim. +When receiving state entries from the state response messages ([Definition -def-num-ref-](chap-networking#defn-msg-state-response)), the node can verify the entries with the entry proof (id *1* in *KeyValueStorage*) against the Merkle root in the block header (of the block specified in [Definition -def-num-ref-](chap-networking#defn-msg-state-request)). Once the state response message claims that all entries have been sent (id *3* in *KeyValueStorage*), the node can use all collected entry proofs and validate them against the Merkle root to confirm that claim. -See the the synchronization chapter for more information ([Chapter -chap-num-ref-](chap-sync)). +See the synchronization chapter for more information ([Chapter -chap-num-ref-](chap-sync)). ###### Definition -def-num- State Request {#defn-msg-state-request} :::definition A **state request** is sent to a peer to request the state at a specified block. The message is a single 32-byte Blake2 hash which indicates the block from which the sync should start. -Depending on what substream is used, he remote peer either sends back a state response ([Definition -def-num-ref-](chap-networking#defn-msg-state-response)) on the `/dot/sync/2` substream or a warp sync proof ([Definition -def-num-ref-](chap-networking#defn-warp-sync-proof)) on the `/dot/sync/warp`. +Depending on what substream is used, the remote peer either sends back a state response ([Definition -def-num-ref-](chap-networking#defn-msg-state-response)) on the `/dot/sync/2` substream or a warp sync proof ([Definition -def-num-ref-](chap-networking#defn-warp-sync-proof)) on the `/dot/sync/warp`. ::: ###### Definition -def-num- State Response {#defn-msg-state-response} @@ -514,7 +514,7 @@ where *KeyValueStateEntry* is of the following format: | Type | Id | Description | |-----------------------|-----|---------------------------------------------------| -| `bytes` | 1 | Root of the entry, empty if top level | +| `bytes` | 1 | Root of the entry, empty if top-level | | `repeated StateEntry` | 2 | Collection of key/values | | `bool` | 3 | Equal 'true' if there are no more keys to return. | @@ -528,9 +528,9 @@ and *StateEntry*: ::: ### -sec-num- Warp Sync {#sect-msg-warp-sync} -The warp sync protocols allows nodes to retrieve blocks from remote peers where authority set changes occurred. This can be used to speed up synchronization to the latest state. +The warp sync protocols allow nodes to retrieve blocks from remote peers where authority set changes occurred. This can be used to speed up synchronization to the latest state. -See the the synchronization chapter for more information ([Chapter -chap-num-ref-](chap-sync)). +See the synchronization chapter for more information ([Chapter -chap-num-ref-](chap-sync)). ###### Definition -def-num- Warp Sync Proof {#defn-warp-sync-proof} :::definition @@ -577,7 +577,7 @@ Transactions are sent over the `/dot/transactions/1` substream. ::: ### -sec-num- GRANDPA Messages {#sect-msg-grandpa} -The exchange of GRANDPA messages is conducted on the substream. The process for the creation and distributing these messages is described in [Chapter -chap-num-ref-](sect-finality). The underlying messages are specified in this section. +The exchange of GRANDPA messages is conducted on the substream. The process for the creation and distribution of these messages is described in [Chapter -chap-num-ref-](sect-finality). The underlying messages are specified in this section. ###### Definition -def-num- Grandpa Gossip Message {#defn-gossip-message} :::definition @@ -671,7 +671,7 @@ This message is the sub-component of the GRANDPA gossip message ([Definition -de ::: #### -sec-num- GRANDPA Neighbor Messages {#sect-grandpa-neighbor-msg} -Neighbor messages are sent to all connected peers but they are not repropagated on reception. A message should be send whenever the messages values change and at least every 5 minutes. The sender should take the recipients state into account and avoid sending messages to peers that are using a different voter sets or are in a different round. Messages received from a future voter set or round can be dropped and ignored. +Neighbor messages are sent to all connected peers but they are not repropagated on reception. A message should be sent whenever the values of the message change and at least every 5 minutes. The sender should take the recipient's state into account and avoid sending messages to peers that are using different voter sets or are in a different round. Messages received from a future voter set or round can be dropped and ignored. ###### Definition -def-num- GRANDPA Neighbor Message {#defn-grandpa-neighbor-msg} :::definition @@ -729,7 +729,7 @@ This message is the sub-component of the GRANDPA Gossip message ([Definition -de ### -sec-num- GRANDPA BEEFY {#sect-msg-grandpa-beefy} :::caution -The BEEFY protocol is currently in early development and subject to change. +The BEEFY protocol is currently in early development and is subject to change. ::: This section defines the messages required for the GRANDPA BEEFY protocol ([Section -sec-num-ref-](sect-finality#sect-grandpa-beefy)). Those messages are sent over the `/paritytech/beefy/1` substream. @@ -737,7 +737,7 @@ This section defines the messages required for the GRANDPA BEEFY protocol ([Sect ###### Definition -def-num- Commitment {#defn-grandpa-beefy-commitment} :::definition -A **commitment**, ${C}$, contains the information extracted from the finalized block at height ${H}_{{i}}{\left({B}_{{\text{last}}}\right)}$ as specified in the message body and a datastructure of the following format: +A **commitment**, ${C}$, contains the information extracted from the finalized block at height ${H}_{{i}}{\left({B}_{{\text{last}}}\right)}$ as specified in the message body and a data structure of the following format: $$ {C}={\left({R}_{{h}},{H}_{{i}}{\left({B}_{{\text{last}}}\right)},\text{id}_{{{\mathbb{{V}}}}}\right)} @@ -754,7 +754,7 @@ $$ ###### Definition -def-num- Vote Message {#defn-msg-beefy-gossip} :::definition -A **vote message**, ${M}_{{v}}$, is direct vote created by the Polkadot Host on every BEEFY round and is gossiped to its peers. The message is a datastructure of the following format: +A **vote message**, ${M}_{{v}}$, is a direct vote created by the Polkadot Host on every BEEFY round and is gossiped to its peers. The message is a data structure of the following format: $$ {M}_{{v}}=\text{Enc}_{{\text{SC}}}{\left({C},{{A}_{{\text{id}}}^{{\text{bfy}}}},{A}_{{\text{sig}}}\right)} @@ -785,7 +785,7 @@ $$ - ${S}_{{n}}$ is an array where its exact size matches the number of validators in the current authority set as specified by $\text{id}_{{{\mathbb{{V}}}}}$ ([Definition -def-num-ref-](sect-finality#defn-authority-set-id)) in ${C}$. Individual items are of the type *Option* ([Definition -def-num-ref-](id-cryptography-encoding#defn-option-type)) which can contain a signature of a validator which signed the same statement (${R}_{{h}}$ in ${C}$) and is active in the current authority set. It’s critical that the signatures are sorted based on their corresponding public key entry in the authority set. - For example, the signature of the validator at index 3 in the authority set must be placed at index *3* in ${S}_{{n}}$. If not signature is available for that validator, then the *Option* variant is *None* inserted ([Definition -def-num-ref-](id-cryptography-encoding#defn-option-type)). This sorting allows clients to map public keys to their corresponding signatures. + For example, the signature of the validator at index 3 in the authority set must be placed at index *3* in ${S}_{{n}}$. If no signature is available for that validator, then the *Option* variant is *None* inserted ([Definition -def-num-ref-](id-cryptography-encoding#defn-option-type)). This sorting allows clients to map public keys to their corresponding signatures. ::: ###### Definition -def-num- Signed Commitment Witness {#defn-grandpa-beefy-signed-commitment-witness} @@ -793,7 +793,7 @@ $$ A **signed commitment witness**, ${{M}_{{\text{SC}}}^{{w}}}$, is a light version of the signed BEEFY commitment ([Definition -def-num-ref-](chap-networking#defn-grandpa-beefy-signed-commitment)). Instead of containing the entire list of signatures, it only claims which validator signed the statement. -The message is a datastructure of the following format: +The message is a data structure of the following format: $$ {{M}_{{\text{SC}}}^{{w}}}=\text{Enc}_{{\text{SC}}}{\left({C},{V}_{{{0},\ldots{n}}},{R}_{{\text{sig}}}\right)} @@ -802,7 +802,7 @@ $$ **where** - ${C}$ is the BEEFY commitment ([Definition -def-num-ref-](chap-networking#defn-grandpa-beefy-commitment)). -- ${V}_{{{0},\ldots{n}}}$ is an array where its exact size matches the number of validators in the current authority set as specified by $\text{id}_{{{\mathbb{{V}}}}}$ in ${C}$. Individual items are booleans which indicate whether the validator has signed the statement (*true*) or not (*false*). It’s critical that the boolean indicators are sorted based on their corresponding public key entry in the authority set. +- ${V}_{{{0},\ldots{n}}}$ is an array where its exact size matches the number of validators in the current authority set as specified by $\text{id}_{{{\mathbb{{V}}}}}$ in ${C}$. Individual items are booleans that indicate whether the validator has signed the statement (*true*) or not (*false*). It’s critical that the boolean indicators are sorted based on their corresponding public key entry in the authority set. For example, the boolean indicator of the validator at index 3 in the authority set must be placed at index *3* in ${V}_{{n}}$. This sorting allows clients to map public keys to their corresponding boolean indicators. From 22bb29080f8b5bfa7722506d2cf8c597f48bfc2d Mon Sep 17 00:00:00 2001 From: "Heverson B. Ribeiro" Date: Mon, 24 Jul 2023 17:19:21 +0200 Subject: [PATCH 07/10] update after review --- docs/chap-networking.md | 49 ++++++++++++++++++++++++++--------------- 1 file changed, 31 insertions(+), 18 deletions(-) diff --git a/docs/chap-networking.md b/docs/chap-networking.md index 75697c876..43d789bfd 100644 --- a/docs/chap-networking.md +++ b/docs/chap-networking.md @@ -220,16 +220,30 @@ The Polkadot Host must actively communicate with the network in order to partici ### -sec-num- Discovering authorities {#sect-authority-discovery} -A discovery mechanism enables Polkadot nodes to both publish their local addresses and learn about other nodes identifiers and addresses by means of the undelayed Kademlia DHT. +A discovery mechanism enables Polkadot nodes to both publish their local addresses and learn about other nodes identifiers and addresses. The Authority discovery mechanism differs from the bootstrap mechanism, described in [Section -sec-num-ref-](chap-networking#sect-network-bootstrap), in that it restricts the discovery output to nodes currently holding the authority role (e.g., validators). -The authority discovery mechanism consists of two main components: discovery `service` and discovery `worker`. The `service` provides an interface to Polkadot nodes that exposes two functions that allow Polkadot nodes to request authority identifiers or authority addresses. The `worker` interacts with the Kademlia DHT in order to publish local node addresses and discover the addresses of other authority nodes. +The authority discovery mechanism consists of two main interfaces. A `service` interface that exposes two functions that allow Polkadot nodes to request `identifiers` or `addresses` of current authority nodes. A `worker` interface that provides means of interacting with the Kademlia DHT in order to publish local node addresses and discover the addresses of other authority nodes. + +``` + ------- + | Node | + ------- + | + --------------------- +| Discovery Authority | + --------------------- + | + ------- + | DHT | + ------- +``` #### -sec-num- Requesting authority identifier and addresses {#sect-auth-discovery-service-requests} -The following requests must be exposed by the discovery authority mechanism to Polkadot nodes. +The following requests are exposed by the discovery authority mechanism to Polkadot nodes. ###### Definition -def-num- Authority addresses request {#defn-auth-discovery-address-request} :::definition @@ -293,13 +307,15 @@ The response to the previous query includes an enum with one of the following va -#### -sec-num- Publishing local addresses {#sect-auth-discovery-worker-publishing} - +#### -sec-num- Publishing addresses {#sect-auth-discovery-worker-publishing} -The authority discovery mechanism periodically sends `put_value` messages to the DHT in order to publish a `SignedAuthorityRecord` of the addresses of authorities it knows from its current authority sets. The `put_value` message to be sent to Kademlia DHT is created as follows. - -###### Definition -def-num- Signed Authority Record {#defn-msg-signed-authority-record} +The authority discovery mechanism ensures up-to-date addresses for authority nodes by periodically publishing and discovering addresses into the DHT. + +In order to publish on the DHT, the authority discovery mechanism periodically triggers a `put_value` operation that stores a `SignedAuthorityRecord` of the addresses of authorities it knows from its current and next authority sets into the DHT. The `put_value` operation is created as follows. + + +###### Definition -def-num- Signed Authority Record {#defn-signed-authority-record} :::definition The `SignedAuthorityRecord` is a Protobuf serialized structure representing the authority records and signature to send over the wire. @@ -317,7 +333,7 @@ It is defined in the following format: | Type | Id | Description | |------------------|----|-----------------------------------------------------------------------| -| `repeated bytes` | 1 | Possibly multiple `MultiAddr`ess through which a node can be connected | +| `repeated bytes` | 1 | Binary representation of zero or more multiaddresses through which a node is reachable | **PeerSignature** is a Protobuf serialized structure indicating the signature and public key used to sign and verify the `AuthorityRecord`. This is the protobuf structure used to exchange the signature with other nodes. @@ -330,33 +346,30 @@ This is the protobuf structure used to exchange the signature with other nodes. ::: -###### Definition -def-num- Publish authority addresses message {#defn-msg-auth-discovery-publish} +###### Definition -def-num- Publish addresses operation {#defn-auth-discovery-publish} :::definition -For each authority node $i$ in the current authority set, the local node sends a `put_value` message to the Kademlia DHT in the following format: +For each authority node $i$ in the current authority set, the local node invokes a `put_value` operation that triggers a store operation into the DHT with the following format: $$ \texttt{put\_value}{\left(\texttt{KademliaKey}_{i} , \texttt{Sig}_{AR}\right)} $$ - - **where** - $\texttt{KademliaKey}_{i}$ is the $Sha256$ hash of the authorityId of node $i$. - $\texttt{Sig}_{AR}$ is the `SignedAuthorityRecord` - described above([Definition -def-num-ref-](chap-networking#defn-msg-signed-authority-record)). + described above([Definition -def-num-ref-](chap-networking#defn-signed-authority-record)). ::: - #### -sec-num- Discovering authority addresses {#sect-auth-discovery-worker-discover} -The authority discovery mechanism periodically sends `get_value` messages to the Kedemlia DHT in order to discover addresses of authority nodes it knows about. +The authority discovery mechanism periodically invokes `get_value` operation on the DHT in order to discover the addresses of authority nodes it knows about. -###### Definition -def-num- Discover authority addresses message {#defn-msg-auth-discovery-lookup} +###### Definition -def-num- Discover addresses operation {#defn-msg-auth-discovery-lookup} :::definition -At each cycle, nodes `worker` creates a bounded number of `get_value` messages to send in the following format: +Periodically, the authority discover performs a bounded number of `get_value` operations in the following format: $$ \texttt{get\_value}{\left(\texttt{KademliaKey}_{i}\right)} $$ From 607adf25b5ef1d03cc6338dceb4f7d10ed3a5630 Mon Sep 17 00:00:00 2001 From: "Heverson B. Ribeiro" Date: Wed, 26 Jul 2023 12:15:52 +0200 Subject: [PATCH 08/10] shortened version --- docs/chap-networking.md | 60 ++++++++++------------------------------- 1 file changed, 14 insertions(+), 46 deletions(-) diff --git a/docs/chap-networking.md b/docs/chap-networking.md index 613941183..d27774e9a 100644 --- a/docs/chap-networking.md +++ b/docs/chap-networking.md @@ -215,34 +215,15 @@ The Polkadot Host must actively communicate with the network in order to partici ::: - + ### -sec-num- Discovering authorities {#sect-authority-discovery} - -A discovery mechanism enables Polkadot nodes to both publish their local addresses and learn about other nodes identifiers and addresses. +The discovery mechanism enables Polkadot nodes to both publish their local addresses and learn about other nodes identifiers and addresses. The Authority discovery mechanism differs from the bootstrap mechanism, described in [Section -sec-num-ref-](chap-networking#sect-network-bootstrap), in that it restricts the discovery output to nodes currently holding the authority role (e.g., validators). -The authority discovery mechanism consists of two main interfaces. A `service` interface that exposes two functions that allow Polkadot nodes to request `identifiers` or `addresses` of current authority nodes. A `worker` interface that provides means of interacting with the Kademlia DHT in order to publish local node addresses and discover the addresses of other authority nodes. - -``` - ------- - | Node | - ------- - | - --------------------- -| Discovery Authority | - --------------------- - | - ------- - | DHT | - ------- -``` - - #### -sec-num- Requesting authority identifier and addresses {#sect-auth-discovery-service-requests} - -The following requests are exposed by the discovery authority mechanism to Polkadot nodes. + +The following requests are exposed by the discovery authority to Polkadot nodes. ###### Definition -def-num- Authority addresses request {#defn-auth-discovery-address-request} :::definition @@ -256,25 +237,19 @@ $$ **where** - $\texttt{authorityId}$ is the `authorityId` 256-bit identifier representing the public key of the targeted authority node. - **expected response** The response to the previous query includes an enum with one of the following values: - | Value | Description | |----------------------|--------------------------------------------------------| | None | A type representing `no value` | | HashSet[Multiaddr] | An unordered collection of unique `Multiaddr` elements | - **with** - `Multiaddr` a [Multiaddr](https://github.com/libp2p/specs/blob/master/addressing/README.md#multiaddr-in-libp2p) data structure. - ::: - - ###### Definition -def-num- Authority identifier request {#defn-auth-discovery-auth-identifier-request} :::definition @@ -289,9 +264,7 @@ $$ **expected response** - The response to the previous query includes an enum with one of the following values: - | Value | Description | |------------------------|--------------------------------------------------------| @@ -304,15 +277,10 @@ The response to the previous query includes an enum with one of the following va ::: +#### -sec-num- Publishing and discovering addresses {#sect-auth-discovery-publishing} - -#### -sec-num- Publishing addresses {#sect-auth-discovery-worker-publishing} - - -The authority discovery mechanism ensures up-to-date addresses for authority nodes by periodically publishing and discovering addresses into the DHT. - -In order to publish on the DHT, the authority discovery mechanism periodically triggers a `put_value` operation that stores a `SignedAuthorityRecord` of the addresses of authorities it knows from its current and next authority sets into the DHT. The `put_value` operation is created as follows. - + +The authority discovery mechanism triggers operations to publish a `SignedAuthorityRecord` of the addresses of authorities it knows from its current and next authority sets into the DHT. The `SignedAuthorityRecord` and the publish operation are created as follows: ###### Definition -def-num- Signed Authority Record {#defn-signed-authority-record} :::definition @@ -345,10 +313,10 @@ This is the protobuf structure used to exchange the signature with other nodes. ::: -###### Definition -def-num- Publish addresses operation {#defn-auth-discovery-publish} +###### Definition -def-num- Publishing addresses {#defn-auth-discovery-publish} :::definition -For each authority node $i$ in the current authority set, the local node invokes a `put_value` operation that triggers a store operation into the DHT with the following format: +For each authority node $i$ in the current authority set, the local node invokes a `put_value` operation that triggers the publishing operation into the DHT with the following format: $$ \texttt{put\_value}{\left(\texttt{KademliaKey}_{i} , \texttt{Sig}_{AR}\right)} @@ -361,14 +329,14 @@ $$ ::: -#### -sec-num- Discovering authority addresses {#sect-auth-discovery-worker-discover} - -The authority discovery mechanism periodically invokes `get_value` operation on the DHT in order to discover the addresses of authority nodes it knows about. + + +The authority discovery mechanism also invokes operations on the DHT to discover the addresses of authority nodes, as follows: -###### Definition -def-num- Discover addresses operation {#defn-msg-auth-discovery-lookup} +###### Definition -def-num- Discovering addresses {#defn-msg-auth-discovery-lookup} :::definition -Periodically, the authority discover performs a bounded number of `get_value` operations in the following format: +Periodically, the authority discovery performs a number of `get_value` operations in the following format: $$ \texttt{get\_value}{\left(\texttt{KademliaKey}_{i}\right)} $$ From 41722df2dfc8101609547d4804e4343e6225b33d Mon Sep 17 00:00:00 2001 From: David Hawig Date: Wed, 9 Aug 2023 11:30:01 +0200 Subject: [PATCH 09/10] Update chap-networking.md small spelling fixes --- docs/chap-networking.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/chap-networking.md b/docs/chap-networking.md index 6d06e6427..104473bab 100644 --- a/docs/chap-networking.md +++ b/docs/chap-networking.md @@ -210,7 +210,7 @@ The Polkadot Host must actively communicate with the network in order to partici ### -sec-num- Discovering authorities {#sect-authority-discovery} -The discovery mechanism enables Polkadot nodes to both publish their local addresses and learn about other nodes identifiers and addresses. +The discovery mechanism enables Polkadot nodes to both publish their local addresses and learn about other nodes' identifiers and addresses. The Authority discovery mechanism differs from the bootstrap mechanism, described in [Section -sec-num-ref-](chap-networking#sect-network-bootstrap), in that it restricts the discovery output to nodes currently holding the authority role (e.g., validators). #### -sec-num- Requesting authority identifier and addresses {#sect-auth-discovery-service-requests} @@ -265,7 +265,7 @@ The response to the previous query includes an enum with one of the following va **with** -- `authorityId` the 256-bit identifier representing the public key of the requested `PeerId`. +- `authorityId` is the 256-bit identifier representing the public key of the requested `PeerId`. ::: @@ -431,7 +431,7 @@ The `BlockRequest` message is a Protobuf serialized structure of the following f | 0 | Enumerate in ascending order (from child to parent) | | 1 | Enumerate in descending order (from parent to canonical child) | -- ${B}_{{m}}$ is the number of blocks to be returned. An implementation defined maximum is used when unspecified. +- ${B}_{{m}}$ is the number of blocks to be returned. An implementation-defined maximum is used when unspecified. ::: ###### Definition -def-num- Block Response {#defn-msg-block-response} @@ -453,7 +453,7 @@ where *BlockData* is a Protobuf structure containing the requested blocks. Do no | `bytes` | 4 | Block receipt (optional) | | | `bytes` | 5 | Block message queue (optional) | | | `bytes` | 6 | Justification (optional) | [Definition -def-num-ref-](sect-finality#defn-grandpa-justification) | -| `bool` | 7 | Indicates whether the justification is empty (i.e. should be ignored) | | +| `bool` | 7 | Indicates whether the justification is empty (i.e., should be ignored) | | ::: ### -sec-num- Requesting States {#sect-msg-state-request} @@ -507,7 +507,7 @@ See the synchronization chapter for more information ([Chapter -chap-num-ref-](c ###### Definition -def-num- Warp Sync Proof {#defn-warp-sync-proof} :::definition -The **warp sync proof** message, ${P}$, is sent to the peer that initialized the state request ([Definition -def-num-ref-](chap-networking#defn-msg-state-request)) on the `/dot/sync/warp` substream and contains accumulated proof of multiple authority set changes ([Section -sec-num-ref-](chap-sync#sect-consensus-message-digest)). It’s a datastructure of the following format: +The **warp sync proof** message, ${P}$, is sent to the peer that initialized the state request ([Definition -def-num-ref-](chap-networking#defn-msg-state-request)) on the `/dot/sync/warp` substream and contains accumulated proof of multiple authority set changes ([Section -sec-num-ref-](chap-sync#sect-consensus-message-digest)). It’s a data structure of the following format: $$ {P}={\left({{f}_{{x}}\ldots}{{f}_{{y}},}{c}\right)} From 9eccea1b43c30ff3359d5358fd30e3eaf10c8220 Mon Sep 17 00:00:00 2001 From: David Hawig Date: Wed, 9 Aug 2023 11:31:37 +0200 Subject: [PATCH 10/10] Update chap-overview.md small fixes --- docs/chap-overview.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/chap-overview.md b/docs/chap-overview.md index dc77de74a..22c9c1d0d 100644 --- a/docs/chap-overview.md +++ b/docs/chap-overview.md @@ -6,11 +6,11 @@ The Polkadot Protocol differentiates between different classes of Polkadot Hosts ## -sec-num- Light Client {#sect-client-light} -The light client is a mostly passive participant in the protocol. Light clients are designed to work in resource-constrained environments like browsers, mobile devices, or even on-chain. Its main objective is to follow the chain, make queries to the full node on specific information on the recent state of the blockchain, and add extrinsics (transactions). It does not maintain the full state, but rather queries the full node on the latest finalized state and verifies the authenticity of the responses trustlessly. Details of specifications focused on Light Clients can be found in [Chapter -chap-num-ref-](sect-lightclient). +The light client is a mostly passive participant in the protocol. Light clients are designed to work in resource-constrained environments like browsers, mobile devices, or even on-chain. Its main objective is to follow the chain, make queries to the full node on specific information on the recent state of the blockchain, and add extrinsics (transactions). It does not maintain the full state but rather queries the full node on the latest finalized state and verifies the authenticity of the responses trustlessly. Details of specifications focused on Light Clients can be found in [Chapter -chap-num-ref-](sect-lightclient). ## -sec-num- Full Node {#sect-node-full} -While the full node is still a mostly passive participant of the protocol, they follow the chain by receiving and verifying every block in the chain. It maintains a full state of the blockchain by executing the extrinsics in blocks. Their role in the consesus mechanism is limited to following the chain and not producing the blocks. +While the full node is still a mostly passive participant of the protocol, they follow the chain by receiving and verifying every block in the chain. It maintains a full state of the blockchain by executing the extrinsics in blocks. Their role in the consensus mechanism is limited to following the chain and not producing the blocks. - **Functional Requirements:**