Lighthouse errors connecting to eth1 nodes while Catalyst or Nethermind running.

[anuzis]

Running into issues with Lighthouse via Docker failing to connect to eth1 nodes. First tried running Catalyst, switched to Nethermind, both result in the same error (below).

Additional context about my configuration:

• Minimal changes from the tutorial. Adjusted genesis_validators.yml count to 75, and generated 75 validator keystores
• teku running fine in client mode (omitted --validator-keys )
• start_validator.sh running fine via Docker
• catalyst client successfully binds to 127.0.0.1:8545 when running via command line (not at same time as nethermind)
• nethermind successfully binds to 0.0.0.0:8545 when running via docker (not at same time as catalyst)

Any ideas what may be off?

Error logs from /lh_scripts/start_beacon.sh docker:

rayonism: Pulling from sigp/lighthouse
Digest: sha256:65e61d4a57a2a64989d8bc322dc363567d556eb2e97f897bf73d85cebb75a172
Status: Image is up to date for sigp/lighthouse:rayonism
docker.io/sigp/lighthouse:rayonism
Apr 28 02:48:01.902 INFO Lighthouse started                      version: Lighthouse/v1.3.0-6401a63
Apr 28 02:48:01.902 INFO Configured for network                  name: custom loaded from mynetwork/public/eth2_config.yaml
Apr 28 02:48:01.902 INFO Data directory initialised              datadir: /mynetwork/nodes/lighthouse_docker
Apr 28 02:48:01.902 WARN Running HTTP server on port 5052
Apr 28 02:48:01.902 INFO Deposit contract                        address: 0x4242424242424242424242424242424242424242, deploy_block: 0
Apr 28 02:48:01.921 INFO Starting beacon chain                   method: resume, service: beacon
Apr 28 02:48:01.935 INFO Block production enabled                method: json rpc via http, endpoints: ["http://localhost:8545"]
Apr 28 02:48:01.935 INFO Beacon chain initialized                head_slot: 0, head_block: 0x84c6…ce8b, head_state: 0x87c6…3907, service: beacon
Apr 28 02:48:01.935 INFO Timer service started                   service: node_timer
Apr 28 02:48:01.935 INFO UPnP Attempting to initialise routes    service: UPnP
Apr 28 02:48:01.937 INFO Libp2p Service                          peer_id: 16Uiu2HAmK2cK6EZc7MpJfe3HrU3GysSeEj33rmLY8X2WWNvrwj22, service: libp2p
Apr 28 02:48:01.937 INFO ENR Initialised                         tcp: Some(9000), udp: None, ip: None, id: 0xc14f..2597, seq: 1, enr: enr:-KO4QPeF9chLLL-3e7r4guqos77cAvUsLVfO7XT7XJm5JJOqZFQhyusZY8jJ1osXH1dmrwNzhTUmNjZ4Io0ZYPj8Le8Bh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB6RgHmAAAHAP__________gmlkgnY0iXNlY3AyNTZrMaEDXq_7I6Wzxi8WkK1nhD1teUrYRyZElospotMlvR2r8YODdGNwgiMo, service: libp2p
Apr 28 02:48:01.938 INFO Listening established                   address: /ip4/0.0.0.0/tcp/9000/p2p/16Uiu2HAmK2cK6EZc7MpJfe3HrU3GysSeEj33rmLY8X2WWNvrwj22, service: libp2p
Apr 28 02:48:01.938 INFO HTTP API started                        listen_address: 127.0.0.1:5052
Apr 28 02:48:01.952 WARN Eth1 endpoint is not synced             action: trying fallback, last_seen_block_unix_timestamp: 1619574826, endpoint: http://localhost:8545, service: eth1_rpc
Apr 28 02:48:01.952 CRIT Couldn't connect to any eth1 node. Please ensure that you have an eth1 http server running locally on http://localhost:8545 or specify one or more (remote) endpoints using `--eth1-endpoints <COMMA-SEPARATED-SERVER-ADDRESSES>`. Also ensure that `eth` and `net` apis are enabled on the eth1 http server, warning: BLOCK PROPOSALS WILL FAIL WITHOUT VALID, SYNCED ETH1 CONNECTION, service: eth1_rpc
Apr 28 02:48:01.952 ERRO Failed to update eth1 cache             error: Failed to update Eth1 service: "All fallback errored: http://localhost:8545 => EndpointError(FarBehind)", retry_millis: 7000, service: eth1_rpc
Apr 28 02:48:07.001 WARN Low peer count                          peer_count: 0, service: slot_notifier
Apr 28 02:48:07.001 INFO Searching for peers                     current_slot: 443, head_slot: 0, finalized_epoch: 0, finalized_root: 0x0000…0000, peers: 0, service: slot_notifier
Apr 28 02:48:07.001 WARN Syncing eth1 block cache                est_blocks_remaining: initializing deposits, msg: sync can take longer when using remote eth1 nodes, service: slot_notifier
Apr 28 02:48:08.949 WARN Eth1 endpoint is not synced             action: trying fallback, last_seen_block_unix_timestamp: 1619574826, endpoint: http://localhost:8545, service: eth1_rpc
Apr 28 02:48:08.949 CRIT Couldn't connect to any eth1 node. Please ensure that you have an eth1 http server running locally on http://localhost:8545 or specify one or more (remote) endpoints using `--eth1-endpoints <COMMA-SEPARATED-SERVER-ADDRESSES>`. Also ensure that `eth` and `net` apis are enabled on the eth1 http server, warning: BLOCK PROPOSALS WILL FAIL WITHOUT VALID, SYNCED ETH1 CONNECTION, service: eth1_rpc
Apr 28 02:48:08.949 ERRO Failed to update eth1 cache             error: Failed to update Eth1 service: "All fallback errored: http://localhost:8545 => EndpointError(FarBehind)", retry_millis: 7000, service: eth1_rpc
Apr 28 02:48:11.981 INFO UPnP not available                      error: IO error: Resource temporarily unavailable (os error 11), service: UPnP

[anuzis]

Couple quick followups:

• Confirming I've tried running catalyst via command line & docker, both run but get the same error from lighthouse
• Regarding the error hint: Also ensure that ethandnet apis are enabled on the eth1 http server -- confirming catalyst is run with --http --http.api net,eth,consensus, which appears to be the suggestion from the error.

[rahulxero]

any update I've facing same error