Merge remote-tracking branch 'origin/main' into integrate-subnetevm-0…

….5.8
ava-labs · Oct 31, 2023 · 5b548b8 · 5b548b8
2 parents 8daa4eb + d79025f
commit 5b548b8
Show file tree

Hide file tree

Showing 10 changed files with 307 additions and 132 deletions.
diff --git a/README.md b/README.md
@@ -1,18 +1,18 @@
 # awm-relayer
 
-Standalone relayer for cross-chain Avalanche Warp Message delivery.
+Reference relayer implementation for cross-chain Avalanche Warp Message delivery.
 
 ## Usage
 
 ### Building
 
-Build the relayer by running the included build script:
+Build the relayer by running the script:
 
 ```bash
 ./scripts/build.sh
 ```
 
-Build a Docker image by running the included build script:
+Build a Docker image by running the script:
 ```
 ./scripts/build-local-image.sh
 ```
@@ -25,19 +25,102 @@ The relayer binary accepts a path to a JSON configuration file as the sole argum
 ./build/awm-relayer --config-file path-to-config
 ```
 
-## Architecture
+### Configuration
+
+The relayer is configured via a JSON file, the path to which is passed in via the `--config-file` command line argument. The following configuration options are available:
+
+`"log-level": "debug" | "info" | "warn" | "error" | "fatal" | "panic"` 
+- The log level for the relayer. Defaults to `info`.
+
+`"network-id": integer`
+- The ID of the Avalanche network to which the relayer will connect. Defaults to `1` (Mainnet).
+
+`"p-chain-api-url": string`
+- The URL of the Avalanche P-Chain API node to which the relayer will connect. This API node needs to have the following methods enabled:
+  - info.peers
+  - platform.getHeight
+  - platform.validatedBy
+  - platform.getValidatorsAt
+
+`"encrypt-connection": boolean`
+- Whether or not to encrypt the connection to the P-Chain API node. Defaults to `true`.
+
+`"storage-location": string`
+- The path to the directory in which the relayer will store its state. Defaults to `./awm-relayer-storage`.
+
+`"source-subnets": []SourceSubnets`
+- The list of source subnets to support. Each `SourceSubnet` has the following configuration:
+
+  `"subnet-id": string` 
+  - cb58-encoded Subnet ID
+
+  `"blockchain-id": string` 
+  - cb58-encoded Blockchain ID
+
+  `"vm": string` 
+  - The VM type of the source subnet.
+
+  `"api-node-host": string` 
+  - The host of the source subnet's API node.
+
+  `"api-node-port": integer` 
+  - The port of the source subnet's API node.
+
+  `"encrypt-connection": boolean` 
+  - Whether or not to encrypt the connection to the source subnet's API node.
 
-**Note:** The relayer in its current state supports Teleporter messages between `subnet-evm` instances. A handful of abstractions have been added to make the relayer extensible to other Warp message formats and VM types, but this work is ongoing.
+  `"rpc-endpoint": string` 
+  - The RPC endpoint of the source subnet's API node. Used in favor of `api-node-host`, `api-node-port`, and `encrypt-connection` when constructing the endpoint
+
+  `"ws-endpoint": string` 
+  - The WebSocket endpoint of the source subnet's API node. Used in favor of `api-node-host`, `api-node-port`, and `encrypt-connection` when constructing the endpoint
+
+  `"message-contracts": map[string]MessageProtocolConfig` 
+  - Map of contract addresses to the config options of the protocol at that address. Each `MessageProtocolConfig` consists of a unique `message-format` name, and the raw JSON `settings`
+
+  `"supported-destinations": []string` 
+  - List of destination subnet IDs that the source subnet supports. If empty, then all destinations are supported.
+
+`"destination-subnets": []DestinationSubnets`
+- The list of destination subnets to support. Each `DestinationSubnet` has the following configuration:
+
+  `"subnet-id": string`
+  - cb58-encoded Subnet ID
+
+  `"blockchain-id": string` 
+  - cb58-encoded Blockchain ID
+
+  `"vm": string` 
+  - The VM type of the source subnet.
+
+  `"api-node-host": string` 
+  - The host of the source subnet's API node.
+
+  `"api-node-port": integer` 
+  - The port of the source subnet's API node.
+
+  `"encrypt-connection": boolean` 
+  - Whether or not to encrypt the connection to the source subnet's API node.
+
+  `"rpc-endpoint": string` 
+  - The RPC endpoint of the destination subnet's API node. Used in favor of `api-node-host`, `api-node-port`, and `encrypt-connection` when constructing the endpoint
+
+  `"account-private-key": string` 
+  - The hex-encoded private key to use for signing transactions on the destination subnet. May be provided by the environment variable `ACCOUNT_PRIVATE_KEY`. Each `destination-subnet` may use a separate private key by appending the blockchain ID to the private key environment variable name, e.g. `ACCOUNT_PRIVATE_KEY_11111111111111111111111111111111LpoYY` 
+
+## Architecture
 
 ### Components
 
 The relayer consists of the following components:
 
 - At the global level:
-    - *P2P App Network*: issues signature `AppRequests`
+    - *P2P app network*: issues signature `AppRequests`
     - *P-Chain client*: gets the validators for a subnet
+    - *JSON database*: stores latest processed block for each source subnet
 - Per Source subnet
     - *Subscriber*: listens for logs pertaining to cross-chain message transactions
+    - *Source RPC client*: queries for missed blocks on startup
 - Per Destination subnet
     - *Destination RPC client*: broadcasts transactions to the destination
 
@@ -51,7 +134,7 @@ The relayer consists of the following components:
 
 ### Unit tests
 
-Unit tests can be ran locally by running the command in root of the project:
+Unit tests can be ran locally by running the command in the root of the project:
 
 ```bash
 ./scripts/test.sh
@@ -61,13 +144,13 @@ Unit tests can be ran locally by running the command in root of the project:
 
 E2E tests are ran as part of CI, but can also be ran locally with the `--local` flag. To run the E2E tests locally, you'll need to install Gingko following the intructions [here](https://onsi.github.io/ginkgo/#installing-ginkgo)
 
-Next, provide the path to the `subnet-evm` repository and the path to a writeable data directory (here we use the `~/subnet-evm` and `~/tmp/e2e-test`) to use for the tests:
+Next, provide the path to the `subnet-evm` repository and the path to a writeable data directory (this example uses `~/subnet-evm` and `~/tmp/e2e-test`) to use for the tests:
 ```bash
 ./scripts/e2e_test.sh --local --subnet-evm ~/subnet-evm --data-dir ~/tmp/e2e-test
 ```
 ### Generate Mocks
 
-We use [gomock](https://pkg.go.dev/go.uber.org/mock/gomock) to generate mocks for testing. To generate mocks, run the following command at the root of the project:
+[Gomock](https://pkg.go.dev/go.uber.org/mock/gomock) is used to generate mocks for testing. To generate mocks, run the following command at the root of the project:
 
 ```bash
 go generate ./...

diff --git a/config/config.go b/config/config.go
@@ -37,15 +37,19 @@ type MessageProtocolConfig struct {
 	Settings      map[string]interface{} `mapstructure:"settings" json:"settings"`
 }
 type SourceSubnet struct {
-	SubnetID          string                           `mapstructure:"subnet-id" json:"subnet-id"`
-	ChainID           string                           `mapstructure:"chain-id" json:"chain-id"`
-	VM                string                           `mapstructure:"vm" json:"vm"`
-	APINodeHost       string                           `mapstructure:"api-node-host" json:"api-node-host"`
-	APINodePort       uint32                           `mapstructure:"api-node-port" json:"api-node-port"`
-	EncryptConnection bool                             `mapstructure:"encrypt-connection" json:"encrypt-connection"`
-	RPCEndpoint       string                           `mapstructure:"rpc-endpoint" json:"rpc-endpoint"`
-	WSEndpoint        string                           `mapstructure:"ws-endpoint" json:"ws-endpoint"`
-	MessageContracts  map[string]MessageProtocolConfig `mapstructure:"message-contracts" json:"message-contracts"`
+	SubnetID              string                           `mapstructure:"subnet-id" json:"subnet-id"`
+	ChainID               string                           `mapstructure:"chain-id" json:"chain-id"`
+	VM                    string                           `mapstructure:"vm" json:"vm"`
+	APINodeHost           string                           `mapstructure:"api-node-host" json:"api-node-host"`
+	APINodePort           uint32                           `mapstructure:"api-node-port" json:"api-node-port"`
+	EncryptConnection     bool                             `mapstructure:"encrypt-connection" json:"encrypt-connection"`
+	RPCEndpoint           string                           `mapstructure:"rpc-endpoint" json:"rpc-endpoint"`
+	WSEndpoint            string                           `mapstructure:"ws-endpoint" json:"ws-endpoint"`
+	MessageContracts      map[string]MessageProtocolConfig `mapstructure:"message-contracts" json:"message-contracts"`
+	SupportedDestinations []string                         `mapstructure:"supported-destinations" json:"supported-destinations"`
+
+	// convenience field to access the supported destinations after initialization
+	supportedDestinations set.Set[ids.ID]
 }
 
 type DestinationSubnet struct {
@@ -67,12 +71,15 @@ type Config struct {
 	StorageLocation    string              `mapstructure:"storage-location" json:"storage-location"`
 	SourceSubnets      []SourceSubnet      `mapstructure:"source-subnets" json:"source-subnets"`
 	DestinationSubnets []DestinationSubnet `mapstructure:"destination-subnets" json:"destination-subnets"`
+
+	// convenience fields to access the source subnet and chain IDs after initialization
+	sourceSubnetIDs []ids.ID
+	sourceChainIDs  []ids.ID
 }
 
 func SetDefaultConfigValues(v *viper.Viper) {
 	v.SetDefault(LogLevelKey, logging.Info.String())
 	v.SetDefault(NetworkIDKey, constants.MainnetID)
-	v.SetDefault(PChainAPIURLKey, "https://api.avax.network")
 	v.SetDefault(EncryptConnectionKey, true)
 	v.SetDefault(StorageLocationKey, "./.awm-relayer-storage")
 }
@@ -165,17 +172,8 @@ func (c *Config) Validate() error {
 	if _, err := url.ParseRequestURI(c.PChainAPIURL); err != nil {
 		return err
 	}
-	sourceChains := set.NewSet[string](len(c.SourceSubnets))
-	for _, s := range c.SourceSubnets {
-		if err := s.Validate(); err != nil {
-			return err
-		}
-		if sourceChains.Contains(s.ChainID) {
-			return fmt.Errorf("configured source subnets must have unique chain IDs")
-		}
-		sourceChains.Add(s.ChainID)
-	}
 
+	// Validate the destination chains
 	destinationChains := set.NewSet[string](len(c.DestinationSubnets))
 	for _, s := range c.DestinationSubnets {
 		if err := s.Validate(); err != nil {
@@ -187,30 +185,47 @@ func (c *Config) Validate() error {
 		destinationChains.Add(s.ChainID)
 	}
 
-	return nil
-}
-
-// GetSourceIDs returns the Subnet and Chain IDs of all subnets configured as a source
-func (cfg *Config) GetSourceIDs() ([]ids.ID, []ids.ID, error) {
+	// Validate the source chains and store the source subnet and chain IDs for future use
+	sourceChains := set.NewSet[string](len(c.SourceSubnets))
 	var sourceSubnetIDs []ids.ID
 	var sourceChainIDs []ids.ID
-	for _, s := range cfg.SourceSubnets {
+	for _, s := range c.SourceSubnets {
+		// Validate configuration
+		if err := s.Validate(&destinationChains); err != nil {
+			return err
+		}
+		// Verify uniqueness
+		if sourceChains.Contains(s.ChainID) {
+			return fmt.Errorf("configured source subnets must have unique chain IDs")
+		}
+		sourceChains.Add(s.ChainID)
+
+		// Save IDs for future use
 		subnetID, err := ids.FromString(s.SubnetID)
 		if err != nil {
-			return nil, nil, fmt.Errorf("invalid subnetID in configuration. error: %v", err)
+			return fmt.Errorf("invalid subnetID in configuration. error: %v", err)
 		}
 		sourceSubnetIDs = append(sourceSubnetIDs, subnetID)
 
 		chainID, err := ids.FromString(s.ChainID)
 		if err != nil {
-			return nil, nil, fmt.Errorf("invalid subnetID in configuration. error: %v", err)
+			return fmt.Errorf("invalid subnetID in configuration. error: %v", err)
 		}
 		sourceChainIDs = append(sourceChainIDs, chainID)
 	}
-	return sourceSubnetIDs, sourceChainIDs, nil
+
+	c.sourceSubnetIDs = sourceSubnetIDs
+	c.sourceChainIDs = sourceChainIDs
+
+	return nil
 }
 
-func (s *SourceSubnet) Validate() error {
+func (s *SourceSubnet) GetSupportedDestinations() set.Set[ids.ID] {
+	return s.supportedDestinations
+}
+
+// Validates the source subnet configuration, including verifying that the supported destinations are present in destinationChainIDs
+func (s *SourceSubnet) Validate(destinationChainIDs *set.Set[string]) error {
 	if _, err := ids.FromString(s.SubnetID); err != nil {
 		return fmt.Errorf("invalid subnetID in source subnet configuration. Provided ID: %s", s.SubnetID)
 	}
@@ -244,6 +259,21 @@ func (s *SourceSubnet) Validate() error {
 		}
 	}
 
+	// Validate and store the allowed destinations for future use
+	s.supportedDestinations = set.Set[ids.ID]{}
+	for _, blockchainIDStr := range s.SupportedDestinations {
+		blockchainID, err := ids.FromString(blockchainIDStr)
+		if err != nil {
+			return fmt.Errorf("invalid chainID in configuration. error: %v", err)
+		}
+		if !destinationChainIDs.Contains(blockchainIDStr) {
+			return fmt.Errorf("configured source subnet %s has a supported destination blockchain ID %s that is not configured as a destination blockchain",
+				s.SubnetID,
+				blockchainID)
+		}
+		s.supportedDestinations.Add(blockchainID)
+	}
+
 	return nil
 }
 
@@ -373,3 +403,12 @@ func (s *DestinationSubnet) GetRelayerAccountInfo() (*ecdsa.PrivateKey, common.A
 	pkBytes = append(pkBytes, pk.PublicKey.Y.Bytes()...)
 	return pk, common.BytesToAddress(crypto.Keccak256(pkBytes)), nil
 }
+
+//
+// Top-level config getters
+//
+
+// GetSourceIDs returns the Subnet and Chain IDs of all subnets configured as a source
+func (c *Config) GetSourceIDs() ([]ids.ID, []ids.ID) {
+	return c.sourceSubnetIDs, c.sourceChainIDs
+}
diff --git a/go.mod b/go.mod
@@ -10,7 +10,7 @@ require (
 	github.com/onsi/ginkgo/v2 v2.13.0
 	github.com/onsi/gomega v1.29.0
 	github.com/pkg/errors v0.9.1
-	github.com/prometheus/client_golang v1.16.0
+	github.com/prometheus/client_golang v1.17.0
 	github.com/spf13/pflag v1.0.5
 	github.com/spf13/viper v1.16.0
 	github.com/stretchr/testify v1.8.4
@@ -110,9 +110,9 @@ require (
 	github.com/olekukonko/tablewriter v0.0.5 // indirect
 	github.com/pires/go-proxyproto v0.6.2 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
-	github.com/prometheus/client_model v0.3.0 // indirect
-	github.com/prometheus/common v0.42.0 // indirect
-	github.com/prometheus/procfs v0.10.1 // indirect
+	github.com/prometheus/client_model v0.4.1-0.20230718164431-9a2bf3000d16 // indirect
+	github.com/prometheus/common v0.44.0 // indirect
+	github.com/prometheus/procfs v0.11.1 // indirect
 	github.com/rs/cors v1.7.0 // indirect
 	github.com/shirou/gopsutil v3.21.11+incompatible // indirect
 	github.com/status-im/keycard-go v0.2.0 // indirect

diff --git a/go.sum b/go.sum
@@ -478,15 +478,15 @@ github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINE
 github.com/pkg/sftp v1.13.1/go.mod h1:3HaPG6Dq1ILlpPZRO0HVMrsydcdLt6HRDccSgb87qRg=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
-github.com/prometheus/client_golang v1.16.0 h1:yk/hx9hDbrGHovbci4BY+pRMfSuuat626eFsHb7tmT8=
-github.com/prometheus/client_golang v1.16.0/go.mod h1:Zsulrv/L9oM40tJ7T815tM89lFEugiJ9HzIqaAx4LKc=
+github.com/prometheus/client_golang v1.17.0 h1:rl2sfwZMtSthVU752MqfjQozy7blglC+1SOtjMAMh+Q=
+github.com/prometheus/client_golang v1.17.0/go.mod h1:VeL+gMmOAxkS2IqfCq0ZmHSL+LjWfWDUmp1mBz9JgUY=
 github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
-github.com/prometheus/client_model v0.3.0 h1:UBgGFHqYdG/TPFD1B1ogZywDqEkwp3fBMvqdiQ7Xew4=
-github.com/prometheus/client_model v0.3.0/go.mod h1:LDGWKZIo7rky3hgvBe+caln+Dr3dPggB5dvjtD7w9+w=
-github.com/prometheus/common v0.42.0 h1:EKsfXEYo4JpWMHH5cg+KOUWeuJSov1Id8zGR8eeI1YM=
-github.com/prometheus/common v0.42.0/go.mod h1:xBwqVerjNdUDjgODMpudtOMwlOwf2SaTr1yjz4b7Zbc=
-github.com/prometheus/procfs v0.10.1 h1:kYK1Va/YMlutzCGazswoHKo//tZVlFpKYh+PymziUAg=
-github.com/prometheus/procfs v0.10.1/go.mod h1:nwNm2aOCAYw8uTR/9bWRREkZFxAUcWzPHWJq+XBB/FM=
+github.com/prometheus/client_model v0.4.1-0.20230718164431-9a2bf3000d16 h1:v7DLqVdK4VrYkVD5diGdl4sxJurKJEMnODWRJlxV9oM=
+github.com/prometheus/client_model v0.4.1-0.20230718164431-9a2bf3000d16/go.mod h1:oMQmHW1/JoDwqLtg57MGgP/Fb1CJEYF2imWWhWtMkYU=
+github.com/prometheus/common v0.44.0 h1:+5BrQJwiBB9xsMygAB3TNvpQKOwlkc25LbISbrdOOfY=
+github.com/prometheus/common v0.44.0/go.mod h1:ofAIvZbQ1e/nugmZGz4/qCb9Ap1VoSTIO7x0VV9VvuY=
+github.com/prometheus/procfs v0.11.1 h1:xRC8Iq1yyca5ypa9n1EZnWZkt7dwcoRPQwX/5gwaUuI=
+github.com/prometheus/procfs v0.11.1/go.mod h1:eesXgaPo1q7lBpVMoMy0ZOFTth9hBn4W/y0/p/ScXhY=
 github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ=
 github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
 github.com/rogpeppe/go-internal v1.6.1/go.mod h1:xXDCJY+GAPziupqXw64V24skbSoqbTEfhy4qGm1nDQc=

diff --git a/main/main.go b/main/main.go
@@ -86,14 +86,7 @@ func main() {
 
 	// Initialize the global app request network
 	logger.Info("Initializing app request network")
-	sourceSubnetIDs, sourceChainIDs, err := cfg.GetSourceIDs()
-	if err != nil {
-		logger.Error(
-			"Failed to get source IDs",
-			zap.Error(err),
-		)
-		return
-	}
+	sourceSubnetIDs, sourceChainIDs := cfg.GetSourceIDs()
 
 	// Initialize metrics gathered through prometheus
 	gatherer, registerer, err := initMetrics()

diff --git a/messages/teleporter/message_manager.go b/messages/teleporter/message_manager.go
@@ -113,6 +113,7 @@ func (m *messageManager) ShouldSendMessage(warpMessageInfo *vmtypes.WarpMessageI
 	if !ok {
 		return false, ids.ID{}, fmt.Errorf("relayer not configured to deliver to destination. destinationChainID=%s", destinationChainID.String())
 	}
+
 	senderAddress := destinationClient.SenderAddress()
 	if !isAllowedRelayer(teleporterMessage.AllowedRelayerAddresses, senderAddress) {
 		m.logger.Info(