Deposit-cli main net support

CHANGELOG.md

@@ -15,6 +15,12 @@ directoy (`cd ~/eth2-docker` by default):
 * !! If coming from Prysm alpha.29 or earlier, make changes as per notes for [v0.1.7](#v017-2020-10-15)
 * `sudo docker-compose up -d eth2`
 
+## v0.1.8 2020-11-04
+
+* eth2.0-deposit-cli 1.0.0 for Ethereum 2.0 main net
+* First stab at Lighthouse voluntary exit
+* More conservative build targets for Lighthouse, Prysm, Teku, and Geth: Latest release tag instead of `master`
+
 ## v0.1.7.5 2020-10-29
 
 * validator-import for Teku now understands Prysm export

README.md

@@ -1,4 +1,4 @@
-# eth2-docker v0.1.7.5
+# eth2-docker v0.1.8
 Unofficial and experimental docker build instructions for eth2 clients
 
 ## Acknowledgements
@@ -24,10 +24,6 @@ Currently supported optional components:
   your eth2 beacon node, so you can [propose](https://ethos.dev/beacon-chain/) blocks.
 - Grafana dashboard
 
-**Note**: As of 10/09/2020, Nimbus requires an "archive" source of eth1 chain data.
-This can be a local geth in archive mode via ws://, or a 3rd-party provider via wss://. 
-Please see [SETUP](SETUP.md) for details.
-
 Please see [WEB](WEB.md) for experimental Web UI support on Prysm, and use the Web instead
 of validator-import to import keys.
 
@@ -90,7 +86,7 @@ Edit the `.env` file to set the number of validators you wish to run. The defaul
 is just one (1) validator.
 
 This command will get you ready to deposit eth:<br />
-`sudo docker-compose run deposit-cli`
+`sudo docker-compose run --rm deposit-cli`
 
 The created files will be in the directory `.eth2/validator_keys` in this project.
 This is also where you'd place your own keystore files if you already have some for import.
@@ -108,7 +104,7 @@ They go into `.eth2/validator_keys` in this project directory, not directly unde
 
 Import the validator key(s) to the validator client:
 
-`sudo docker-compose run validator-import`
+`sudo docker-compose run --rm validator-import`
 
 > #### Prysm-specific
 > - You will be asked to provide a wallet directory. Use `/var/lib/prysm`.
@@ -144,12 +140,13 @@ from the running container.
 
 ## Step 7: Depositing
 
-Optional: You may wish to wait until the beacon node is fully synchronized before you deposit. Check
+**Caution**: You may wish to wait until the beacon node is fully synchronized before you deposit. Check
 its logs with `sudo docker-compose logs -f beacon`. This safe-guards against the validator being
 marked offline if your validator is activated before the beacon syncs.
 
 Once you are ready, you can send eth to the deposit contract by using
-the `.eth2/validator_keys/deposit_data-TIMESTAMP.json` file at the [Medalla launchpad](https://medalla.launchpad.ethereum.org/).
+the `.eth2/validator_keys/deposit_data-TIMESTAMP.json` file at the [Medalla launchpad](https://medalla.launchpad.ethereum.org/)
+or [Mainnet launchpad](https://launchpad.ethereum.org).
 
 ## Step 8: Grafana Dashboards
 
@@ -247,14 +244,36 @@ Then restart the client:<br />
 If you did not store the wallet password with the validator, come up 
 [more manually](#start-the-client) instead.
 
+## Addendum: Add or recover validators
+
+You can use eth2.0-deposit-cli to either recover validator signing keys or add
+additional ones, if you wish to deposit more validators against the same mnemonic.
+
+In order to recover all your validator signing keys, edit `.env`, set `NUMVALS` to the number
+of validators you had created previously, then run `sudo docker-compose run --rm deposit-cli-add-recover`
+and provide your mnemonic.
+
+In order to add additional validator signing keys, edit `.env`, and set `VAL_START_INDEX`
+to the number of validator keys you had created previously, for example, `4`. New validators
+will be created after this point. Set `NUMVALS` to the number of new validators you wish to
+create and deposit, then run `sudo docker-compose run --rm deposit-cli-add-recover`
+and provide your mnemonic. You will receive new `keystore-m` signing keys and a new `deposit_data` JSON.
+
+> Please triple-check your work here. You want to be sure the new validator keys are created after
+> the existing ones. Launchpad will likely safeguard you against depositing twice, but don't rely
+> on it. Verify that the public keys in `deposit_data` are new and you did not deposit for them
+> previously.
+ 
 ## Addendum: Voluntary client exit
 
 Ethereum 2.0 has a concept of "voluntary client exit", which will remove the
 validator from attesting duties. Locked eth could be withdrawn in phase 2,
 and not sooner.
 
-Currently, Prysm supports voluntary exit. This requires a fully synced
-Prysm client.
+Currently, Prysm and Lighthouse support voluntary exit. This requires a fully synced
+beacon node.
+
+### Prysm
 
 To exit, run `sudo docker-compose run validator-voluntary-exit` and follow the
 prompts.
@@ -277,6 +296,19 @@ as follows:
 - Check logs until the beacon is synced: `sudo docker-compose logs -f beacon`
 - Initiate voluntary exit and follow the prompts: `sudo docker-compose run validator-voluntary-exit`
 
+
+### Lighthouse
+
+The exit procedure for lighthouse is not very refined, yet.
+
+- Copy the `keystore-m` JSON files into `.eth2/validator_keys` in this project
+  directory.
+- Run `sudo docker-compose run --rm validator-voluntary-exit /var/lib/lighthouse/validator_keys/<name-of-keystore-file>`,
+  once for each keystore (validator) you wish to exit.
+- Follow prompts.
+
+### Avoid penalties
+
 > Note you will need to continue running your validator until the exit
 > has been processed by the chain, if you wish to avoid incurring offline
 > penalties. You can check the status of your validator with tools such

SETUP.md

@@ -53,7 +53,8 @@ Please choose:
 * Whether to run a grafana dashboard for monitoring
 
 > Note: Teku is written in Java, which makes it memory-hungry. In its default configuration, you may
-> want a machine with 16 GiB of RAM. See `.env` for a parameter to restrict Teku to 4 GiB of RAM.
+> want a machine with 16 GiB of RAM. See `.env` for a parameter to restrict Teku to 6 GiB of heap. It
+> may still take more than 6 GiB of RAM in total.
 
 First, copy the environment file.<br />
 `cp default.env .env`
@@ -63,7 +64,6 @@ First, copy the environment file.<br />
 > which this project does not use.
  
 Then, adjust the contents of `.env`. On Ubuntu Linux, you can run `nano .env`.
-- Set the `GRAFFITI` string if you want a POAP or just a specific string
 - If you are on Linux, **adjust `LOCAL_UID` to the UID of the logged-in user**. 
 `echo $UID` will show it to you. It is highly recommended to run as a non-root
 user on Linux. On [Debian](https://devconnected.com/how-to-add-a-user-to-sudoers-on-debian-10-buster/)
@@ -76,11 +76,15 @@ has that functionality built-in.
 > permissions errors during use.
 
 - Set the `COMPOSE_FILE` entry depending on the client you are going to run,
-and with which options. See below for available compose files.
+and with which options. See below for available compose files
 - If you are going to use a 3rd-party provider as your eth1 chain source, set `ETH1_NODE` to that URL.
-  This is most relevant to Nimbus, see [how to create your own Infura account](https://status-im.github.io/nimbus-eth2/infura-guide).
+  See [how to create your own Infura account](https://status-im.github.io/nimbus-eth2/infura-guide)
 - Adjust ports if you are going to need custom ports instead of the defaults. These are the ports
-exposed to the host, and for everything but Grafana to the Internet via your firewall/router.
+exposed to the host, and for everything but Grafana to the Internet via your firewall/router
+- Set the `NETWORK` variable to either "mainnet" or a test network such as "medalla"
+- Comment out the `ETH_NETWORK` variable, to use the main net, or set it to a test network such as "--goerli",
+  with the two dashes.
+- Set the `GRAFFITI` string if you want a specific string
 
 ### Client compose files
 
@@ -91,9 +95,9 @@ geth with `:` between the file names.
 - `teku-base.yml` - Teku
 - `nimbus-base.yml` - Nimbus
 - `geth.yml` - local geth eth1 chain node
-- `geth-archive.yml` - local geth node in full archive mode, required for Nimbus
 - `lh-grafana.yml` - grafana dashboard for Lighthouse
-- `prysm-grafana.yml` - grafana dashboard for Prysm, as well as experimental Web UI
+- `prysm-grafana.yml` - grafana dashboard for Prysm. Not encrypted, do not expose to Internet.
+- `prysm-web.yml` - Prysm experimental Web UI. Not encrypted, do not expose to Internet.
 - `nimbus-grafana.yml` - grafana dashboard for Nimbus
 - `teku-grafana.yml` - grafana dashboard for Teku
 
@@ -109,28 +113,6 @@ clients each in their own directory.
 If you want to run multiple isolated clients, just clone this project into a new directory for
 each. This is great for running medalla and zinken in parallel, for example.
 
-> Nimbus: Nimbus as of 10/09/2020 requires a "full archive" source of eth1 chain data.
-> It will work with a 3rd party via wss://, though not https://, and a local
-> archive node via ws://, though not http://. Nimbus' capabilities will evolve,
-> check with the Nimbus Discord for its current state.<br />
-> Note that a "full archive" geth takes ~60GB for goerli testnet and ~1.3TB for mainnet.<br />
-> As of the same date, Nimbus does not detect its external IP via P2P protocol. It will
-> still work, but may take (much) longer to connect to peers.
-
-### Optional: Advanced setup with multiple beacons, shared geth and Vouch client
-
-**Work in progress**, this is not currently functional
-
-In this setup, local eth1 node(s) and local beacons would run each in their own directory,
-and connect via a shared network or the Internet. beacon nodes could be configured to use their own validators
-or a shared validator-client like Vouch. This is very much a work-in-progress and not ready even
-for testing in this release. It will be supported on Linux only, as docker-compose's host network
-behavior differs between MacOS/Windows and Linux, and an advanced setup like this is not likely to run
-on a local user's MacOS/Windows machine.
-
-- `geth-shared.yml` - local geth node, sharable between multiple beacons
-- `geth-archive-shared.yml` - local geth node in full archive mode, sharable between multiple beacons
-
 ## Firewalling
 
 You'll want to forward ports to the services of your eth2 node, and on Linux, enable a host firewall.

default.env

@@ -4,24 +4,26 @@ LOCAL_UID=1000
 # Client choice: See SETUP.md for available options
 COMPOSE_FILE=lh-base.yml:geth.yml:lh-grafana.yml
 # ETH1 endpoint / chain source. This default uses the geth container
-# For Nimbus with geth, use ws://geth:8546, and the geth-archive.yml file
-# in COMPOSE_FILE
 ETH1_NODE=http://geth:8545
 # Graffiti to use for validator, in order to receive POAP
 GRAFFITI=eth2-docker
 # Number of validators to create with deposit-cli
 NUMVAL=1
+# Advanced: Start index for recovery of validators or adding validators
+VAL_START_INDEX=0
 # Restart policy: Set to no if you do not want services to automatically restart
 RESTART=unless-stopped
 # If you want debug logs, set this to debug instead of info
 LOG_LEVEL=info
-# Uncomment this if you need to restrict teku to just 4 GiB of RAM
-#TEKU_LOW_MEM=-Xmx4G
+# Uncomment this if you need to restrict teku to use 6 GiB of heap
+#TEKU_LOW_MEM=-Xmx6G
 # Leave this as-is to compile nimbus with support for metrics via grafana. 
 # Change to empty (nothing after first =) to disable that functionality
 NIM_METRICS=NIMFLAGS="-d:insecure"
-# Testnet to use for eth2.
-TESTNET=medalla
+# Network to use for eth2. Set to mainnet for main net.
+NETWORK=medalla
+# Network to use for geth. Comment out for main net.
+ETH_NETWORK=--goerli
 
 # Ports you will forward to your staking node. Adjust here if you are
 # going to use something other than defaults.
@@ -33,32 +35,25 @@ NIM_PORT=9000
 TEKU_PORT=9000
 # Local grafana dashboard port. Do not expose to Internet, it is insecure http
 GRAFANA_PORT=3000
-# Geth ports are for advanced config, not functional yet
-# Local geth http port if you are using geth-shared.yml. Do not expose to Internet, it is insecure http
-GETH_HTTP_PORT=8545 
-# Local geth ws port if you are using geth-archive-shared.yml. Do not expose to Internet, it is insecure
-GETH_WS_PORT=8546
 
 # These variables are not likely to require adjustment
-# Network to use for geth.
-ETH_NETWORK=goerli
 
-GETH_BUILD_TARGET=release/1.9
+GETH_BUILD_TARGET=$(git describe --tags $(git rev-list --tags --max-count=1))
 GETH_USER=geth
 
-LH_BUILD_TARGET=master
+LH_BUILD_TARGET=$(git describe --tags $(git rev-list --tags --max-count=1))
 LH_USER=lighthouse
 
-PRYSM_BUILD_TARGET=master
+PRYSM_BUILD_TARGET=$(git describe --tags $(git rev-list --tags --max-count=1))
 PRYSM_USER=prysm
 
-NIM_BUILD_TARGET=devel
-NIM_USER=nimbus
 
-TEKU_BUILD_TARGET=master
+TEKU_BUILD_TARGET=$(git describe --tags $(git rev-list --tags --max-count=1))
 TEKU_USER=teku
 
-DEPCLI_BUILD_TARGET=master
+NIM_BUILD_TARGET=devel
+NIM_USER=nimbus
+
+DEPCLI_BUILD_TARGET=$(git describe --tags $(git rev-list --tags --max-count=1))
 DEPCLI_USER=depcli
-DEPCLI_ENTRYPOINT=./deposit.sh --folder /var/lib/depcli-data
 DEPCLI_LOCALDIR=./.eth2

eth2.0-deposit-cli/Dockerfile

@@ -1,4 +1,4 @@
-from python:buster
+from python:buster as builder
 
 ARG USER
 ARG UID
@@ -19,14 +19,21 @@ RUN mkdir -p /var/lib/depcli-data && chown ${USER}:${USER} /var/lib/depcli-data
 
 WORKDIR /var/lib/depcli
 USER ${USER}
-RUN git clone https://github.com/ethereum/eth2.0-deposit-cli.git && cd eth2.0-deposit-cli && git config advice.detachedHead false && git fetch --all --tags && git checkout ${BUILD_TARGET}
-USER root
-RUN cd eth2.0-deposit-cli && pip3 install -r requirements.txt && python3 setup.py install
+RUN bash -c "git clone https://github.com/ethereum/eth2.0-deposit-cli.git && cd eth2.0-deposit-cli && git config advice.detachedHead false && git fetch --all --tags && git checkout ${BUILD_TARGET}"
 
-USER ${USER}
-WORKDIR /var/lib/depcli/eth2.0-deposit-cli
+FROM python:3.7-alpine
+
+WORKDIR /app
+
+COPY --from=builder /var/lib/depcli/eth2.0-deposit-cli/requirements.txt /var/lib/depcli/eth2.0-deposit-cli/setup.py ./
+COPY --from=builder /var/lib/depcli/eth2.0-deposit-cli/eth2deposit ./eth2deposit
+
+RUN apk add --update gcc libc-dev linux-headers
 
-ENV numvals 1
-ENV chain medalla
+RUN pip3 install -r requirements.txt
+
+RUN python3 setup.py install
+
+USER ${USER}
 
-ENTRYPOINT ./deposit.sh --folder /var/lib/depcli-data --num_validators $numvals --chain $chain 
+ENTRYPOINT [ "python3", "./eth2deposit/deposit.py" ]

geth.yml

@@ -17,6 +17,7 @@ services:
       - ${GETH_PORT}:${GETH_PORT}/udp
     expose:
       - 8545
+      - 8546
     entrypoint:
       - geth
       - --nousb
@@ -29,9 +30,9 @@ services:
       - --ipcdisable
       - --datadir
       - /var/lib/goethereum
-      - --${ETH_NETWORK}
       - --port
       - ${GETH_PORT}
+    command: ${ETH_NETWORK}
   beacon:
     depends_on:
       - geth

geth/Dockerfile

@@ -5,7 +5,7 @@ ARG BUILD_TARGET
 ARG USER
 ARG UID
 
-RUN apk update && apk add --no-cache make gcc musl-dev linux-headers git ca-certificates
+RUN apk update && apk add --no-cache make gcc musl-dev linux-headers git ca-certificates bash
 
 # See https://stackoverflow.com/a/55757473/12429735RUN
 RUN adduser \
@@ -18,7 +18,7 @@ RUN adduser \
     "${USER}"
 
 RUN go get -d github.com/ethereum/go-ethereum 
-RUN cd $GOPATH/src/github.com/ethereum/go-ethereum && git config advice.detachedHead false && git fetch --all --tags && git checkout ${BUILD_TARGET} && GO111MODULE=on go install -ldflags="-linkmode external -extldflags -static" ./cmd/geth
+RUN bash -c "cd $GOPATH/src/github.com/ethereum/go-ethereum && git config advice.detachedHead false && git fetch --all --tags && git checkout ${BUILD_TARGET} && GO111MODULE=on go install -ldflags='-linkmode external -extldflags -static' ./cmd/geth"
 RUN ldd $GOPATH/bin/geth 2>&1 | grep -q "Not a valid dynamic program"
 RUN mkdir -p /var/lib/goethereum && chown ${USER}:${USER} /var/lib/goethereum