Added support for LH v0.3.0

CHANGELOG.md

@@ -0,0 +1,38 @@
+# Changelog for eth2-docker project
+
+## Updating the project
+
+To update the components of the project, run from within the project
+directoy (`cd ~/eth2-docker` by default):
+
+* `git pull`
+* `cp .env .env.bak && cp default.env .env`
+* Adjust contents of new `.env`, use `.env.bak` for guidance (LOCAL_UID
+  and COMPOSE_FILE are the most common variables that may need to be adjusted)
+* `sudo docker-compose build --no-cache beacon` (go for a 30 minute walk)
+* `sudo docker-compose down`
+* !! If coming from Lighthouse v0.2.x, make changes as per notes for [v0.1.6](#v016-2020-10-09)
+* `sudo docker-compose up -d eth2`
+
+## v0.1.6 2020-10-09
+
+* Support for Lighthouse v0.3.0, drop support for v0.2.x
+  * Please note that Lighthouse v0.3.x makes a breaking change to the beacon
+    db. You will need to sync again from scratch, after building the new v0.3.0
+    beacon image. You can force this with 
+    `sudo docker-compose down`, `sudo docker volume rm eth2-docker_lhbeacon-data`
+    (adjust to your directory path if you are not in `eth2-docker`, see
+    `sudo docker volume ls` for a list).
+  * Likewise, the location of the validator keystore has changed. The fastest way
+    to resolve this involves importing the keystore from scratch:
+    `sudo docker volume rm eth2-docker_lhvalidator-data` (as before, adjust for
+    your directory), and then import the keystore(s) again with
+    `sudo docker-compose run validator-import`. Your keystore(s) need to be in
+    `.eth2/validator_keys` inside the project directory for that.
+  * When you have completed the above steps, bring up Lighthouse with
+    `sudo docker-compose up -d eth2` and verify that the beacon started syncing
+    and the validator found its public key(s) by observing logs:<br />
+    `sudo docker-compose logs -f beacon` and `sudo docker-compose logs validator | head -30`,
+    and if you wish to see ongoing validator logs, `sudo docker-compose logs -f validator`.
+  * The beacon will sync from scratch, which will take about half a day. Your
+    validator will be marked offline for that duration.

README.md

@@ -1,4 +1,4 @@
-# eth2-docker v0.1.5
+# eth2-docker v0.1.6
 Unofficial and experimental docker build instructions for eth2 clients
 
 ## Acknowledgements
@@ -24,7 +24,7 @@ Currently supported optional components:
   your eth2 beacon node, so you can [propose](https://ethos.dev/beacon-chain/) blocks.
 - Grafana dashboard
 
-**Note**: As of 9/30/2020, Nimbus requires an "archive" source of eth1 chain data.
+**Note**: As of 10/09/2020, Nimbus requires an "archive" source of eth1 chain data.
 This can be a local geth in archive mode via ws://, or a 3rd-party provider via wss://. 
 Please see [SETUP](SETUP.md) for details.
 
@@ -205,6 +205,8 @@ This project does not monitor client versions. It is up to you to decide that yo
 are going to update a component. When you are ready to do so, the below instructions
 show you how to.
 
+Please see the [changelog](CHANGELOG.md) to see what changed between versions.
+
 ### The project itself
 
 Inside the project directory, run:<br />
@@ -237,7 +239,7 @@ If you did not store the wallet password with the validator, come up
 
 # Addendum: Troubleshooting
 
-A few useful commands if you run into issues. As always, `sudo` is a Linux-ism and not needed on Windows 10.
+A few useful commands if you run into issues. As always, `sudo` is a Linux-ism and may not be needed on MacOS.
 
 `sudo docker-compose stop servicename` brings a service down, for example `docker-compose stop validator`.<br />
 `sudo docker-compose down` will stop the entire stack.<br />

RECOMMENDATIONS.md

@@ -101,9 +101,9 @@ For more on withdrawal key security, read this article: https://www.attestant.io
 See the client team recommendations. Generally, however, 8 GiB of RAM is considered
 a minimum, and 16 GiB is recommended. 2 or 4 CPU cores, and an SSD for storage
 because the node databases are so IOPS-heavy. The eth1 node (geth) would require
-around 300GB of storage by itself. The beacon node database is small, around 11GB,
+around 310GiB of storage by itself. The beacon node database is small, around 11GiB,
 but we don't know what growth will look like from phase 1.5 and 2 on. If
-you are running a slasher, that might be another 100 to 300GB by itself.
+you are running a slasher, that might be another 100 to 300GiB by itself.
 
 Two home server builds that I like and am happy to recommend are below. Both support
 IPMI, which means they can be managed and power-cycled remotely and need neither
@@ -137,7 +137,7 @@ options are Node 304 (mITX) and Node 804 (uATX) with Seasonic PSUs,
 but really any quality case that won't cook your components will do.
 
 On SSD size, 1TB is very, very conservative and assumes you are running
-an eth1 node as well, which currently takes about 230GB and keeps
+an eth1 node as well, which currently takes about 310GiB and keeps
 growing. The eth2 db is expected to be far smaller, though exact figures
 won't be seen until Phase 1.5 and 2.
 

SETUP.md

@@ -45,7 +45,7 @@ Please choose:
   * Lighthouse
   * Prysm
   * Teku
-  * Nimbus - as of 9/28 requires an archive-mode geth or remote archive like goerli.infura.io
+  * Nimbus - as of 10/09 requires an archive-mode geth or remote archive like goerli.infura.io
 * Your source of eth1 data
   * geth
   * 3rd-party
@@ -78,13 +78,9 @@ has that functionality built-in.
 - Set the `COMPOSE_FILE` entry depending on the client you are going to run,
 and with which options. See below for available compose files.
 - If you are going to use a 3rd-party provider as your eth1 chain source, set `ETH1_NODE` to that URL.
+  This is most relevant to Nimbus, see [how to create your own Infura account](https://status-im.github.io/nimbus-eth2/infura-guide).
 - Adjust ports if you are going to need custom ports instead of the defaults. These are the ports
-exposed to the Internet via your firewall/router.
-
-Note that the Prysm client will find its external IP, but this project currently assumes
-that IP is static. You can restart the container, possibly via crontab, with
-`docker-compose restart beacon` if your IP is dynamic.<br />
-Work to support dynamic DNS would be welcome.
+exposed to the host, and for everything but Grafana to the Internet via your firewall/router.
 
 ### Client compose files
 
@@ -96,12 +92,13 @@ geth with `:` between the file names.
 - `nimbus-base.yml` - Nimbus
 - `geth.yml` - local geth eth1 chain node
 - `geth-archive.yml` - local geth node in full archive mode, required for Nimbus
-- `lh-prysm-grafana.yml` - grafana dashboard for Lighthouse or Prysm
+- `lh-grafana.yml` - grafana dashboard for Lighthouse
+- `prysm-grafana.yml` - grafana dashboard for Prysm
 - `nimbus-grafana.yml` - grafana dashboard for Nimbus
 - `teku-grafana.yml` - grafana dashboard for Teku
 
 For example, Lighthouse with local geth and grafana:
-`COMPOSE_FILE=lh-base.yml:geth.yml:lh-prysm-grafana.yml`
+`COMPOSE_FILE=lh-base.yml:geth.yml:lh-grafana.yml`
 
 In this setup, clients are isolated from each other. Each run their own validator, and if geth
 is in use, their own geth. This is perfect for running a single client, or multiple isolated
@@ -110,9 +107,9 @@ clients each in their own directory.
 If you want to run multiple isolated clients, just clone this project into a new directory for
 each. This is great for running medalla and zinken in parallel, for example.
 
-> Nimbus: Nimbus as of 9/30/2020 requires a "full archive" source of eth1 chain data.
-> It should work with a 3rd party via wss://, though possibly not https://, and a local
-> archive node via ws://, though possibly not http://. Nimbus' capabilities will evolve,
+> Nimbus: Nimbus as of 10/09/2020 requires a "full archive" source of eth1 chain data.
+> It will work with a 3rd party via wss://, though not https://, and a local
+> archive node via ws://, though not http://. Nimbus' capabilities will evolve,
 > check with the Nimbus Discord for its current state.<br />
 > Note that a "full archive" geth takes ~60GB for goerli testnet and ~1.3TB for mainnet.<br />
 > As of the same date, Nimbus does not detect its external IP via P2P protocol. It will

default.env

@@ -2,7 +2,7 @@
 # This is vital for key import to work. You can find your UID with "echo $UID".
 LOCAL_UID=1000
 # Client choice: See SETUP.md for available options
-COMPOSE_FILE=lh-base.yml:geth.yml:lh-prysm-grafana.yml
+COMPOSE_FILE=lh-base.yml:geth.yml:lh-grafana.yml
 # ETH1 endpoint / chain source. This default uses the geth container
 # For Nimbus with geth, use ws://geth:8546, and the geth-archive.yml file
 # in COMPOSE_FILE

lh-base.yml

@@ -16,7 +16,7 @@ services:
       - ${LH_PORT}:${LH_PORT}/tcp
       - ${LH_PORT}:${LH_PORT}/udp
     expose:
-      - 5052
+      - 5052/tcp
     entrypoint: 
       - lighthouse
       - bn
@@ -31,8 +31,7 @@ services:
       - ${LH_PORT}
       - --eth1-endpoint
       - ${ETH1_NODE}
-      - --debug-level
-      - ${LOG_LEVEL}
+      - --debug-level=${LOG_LEVEL}
   validator:
     restart: "no"
     user: ${LOCAL_UID}:${LOCAL_UID}
@@ -50,14 +49,13 @@ services:
       - validator
       - --datadir
       - /var/lib/lighthouse
-      - --server
+      - --beacon-node
       - http://beacon:5052
       - --graffiti
       - ${GRAFFITI}
       - --testnet
       - ${TESTNET}
-      - --debug-level
-      - ${LOG_LEVEL}
+      - --debug-level=${LOG_LEVEL}
     depends_on:
       - beacon
   validator-import:
@@ -83,10 +81,9 @@ services:
       - /var/lib/lighthouse
       - --directory
       - /var/lib/lighthouse/validator_keys
-      - --validator-dir
-      - /var/lib/lighthouse
       - --testnet
       - ${TESTNET}
+      - --debug-level=${LOG_LEVEL}
   deposit-cli:
     restart: "no"
     user: ${LOCAL_UID}:${LOCAL_UID}

lh-grafana.yml

@@ -0,0 +1,46 @@
+version: "3"
+services:
+  beacon:
+    expose:
+      - 5054/tcp
+    command:
+      - --metrics
+      - --metrics-address
+      - 0.0.0.0
+  prometheus:
+    restart: "no"
+    build:
+      context: ./prometheus
+    image: prometheus
+    volumes:
+      - prom-data:/prometheus
+    environment:
+      - CLIENT=${COMPOSE_FILE}
+    expose:
+      - 9090/tcp
+    entrypoint: choose-config.sh
+    command: ["/bin/prometheus", "--storage.tsdb.path=/prometheus", "--web.console.libraries=/usr/share/prometheus/console_libraries", "--web.console.templates=/usr/share/prometheus/consoles"]
+    depends_on:
+      - beacon
+      - validator
+      - node-exporter
+  node-exporter:
+    restart: "no"
+    image: prom/node-exporter
+    expose:
+      - 9100/tcp
+  grafana:
+    restart: "no"
+    image: grafana/grafana
+    volumes:
+      - grafana-data:/var/lib/grafana
+    depends_on:
+      - prometheus
+    ports: 
+      - ${GRAFANA_PORT}:3000/tcp
+  eth2:
+    depends_on:
+      - grafana
+volumes:
+  grafana-data:
+  prom-data:

nimbus-grafana.yml

@@ -2,7 +2,7 @@ version: "3"
 services:
   beacon:
     expose:
-      - 8008
+      - 8008/tcp
     command:
       - --metrics
       - --metrics-port=8008
@@ -17,7 +17,7 @@ services:
     environment:
       - CLIENT=${COMPOSE_FILE}
     expose:
-      - 9090
+      - 9090/tcp
     entrypoint: choose-config.sh
     command: ["/bin/prometheus", "--storage.tsdb.path=/prometheus", "--web.console.libraries=/usr/share/prometheus/console_libraries", "--web.console.templates=/usr/share/prometheus/consoles"]
     depends_on:
@@ -27,7 +27,7 @@ services:
     restart: "no"
     image: prom/node-exporter
     expose:
-      - 9100
+      - 9100/tcp
   grafana:
     restart: "no"
     image: grafana/grafana

prometheus/lh-prom.yml

@@ -16,7 +16,7 @@ rule_files:
 scrape_configs:
   - job_name: 'nodes'
     static_configs:
-      - targets: ['beacon:5052']
+      - targets: ['beacon:5054']
   - job_name: 'node_exporter'
     static_configs:
       - targets: ['node-exporter:9100']

lh-prysm-grafana.yml → prysm-grafana.yml

@@ -10,7 +10,7 @@ services:
     environment:
       - CLIENT=${COMPOSE_FILE}
     expose:
-      - 9090
+      - 9090/tcp
     entrypoint: choose-config.sh
     command: ["/bin/prometheus", "--storage.tsdb.path=/prometheus", "--web.console.libraries=/usr/share/prometheus/console_libraries", "--web.console.templates=/usr/share/prometheus/consoles"]
     depends_on:
@@ -21,7 +21,7 @@ services:
     restart: "no"
     image: prom/node-exporter
     expose:
-      - 9100
+      - 9100/tcp
   grafana:
     restart: "no"
     image: grafana/grafana

teku-grafana.yml

@@ -2,7 +2,7 @@ version: "3"
 services:
   beacon:
     expose:
-      - 8008
+      - 8008/tcp
     command:
       - --metrics-enabled=true
       - --metrics-port=8008
@@ -18,7 +18,7 @@ services:
     environment:
       - CLIENT=${COMPOSE_FILE}
     expose:
-      - 9090
+      - 9090/tcp
     entrypoint: choose-config.sh
     command: ["/bin/prometheus", "--storage.tsdb.path=/prometheus", "--web.console.libraries=/usr/share/prometheus/console_libraries", "--web.console.templates=/usr/share/prometheus/consoles"]
     depends_on:
@@ -28,7 +28,7 @@ services:
     restart: "no"
     image: prom/node-exporter
     expose:
-      - 9100
+      - 9100/tcp
   grafana:
     restart: "no"
     image: grafana/grafana