Skip to content

Commit

Permalink
Add support for nimbus client
Browse files Browse the repository at this point in the history
  • Loading branch information
yorickdowne committed Sep 27, 2020
1 parent be2fdce commit bef3524
Show file tree
Hide file tree
Showing 14 changed files with 282 additions and 22 deletions.
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# eth2-docker v0.1.1
# eth2-docker v0.1.2
Unofficial and experimental docker build instructions for eth2 clients

## Acknowledgements
Expand All @@ -14,6 +14,7 @@ binary images is a TODO, as long as it does not duplicate work
by client teams.

Currently supported clients:
- Nimbus
- Lighthouse
- Prysm

Expand Down
36 changes: 27 additions & 9 deletions RECOMMENDATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,9 @@ For more on withdrawal key security, read this article: https://www.attestant.io
See the client team recommendations. Generally, however, 8 GiB of RAM is considered
a minimum, and 16 GiB is recommended. 2 or 4 CPU cores, and an SSD for storage
because the node databases are so IOPS-heavy. The eth1 node (geth) would require
around 300GB of storage by itself. The beacon node database is small, around 11GB.
around 300GB of storage by itself. The beacon node database is small, around 11GB,
but we don't know what growth will look like from phase 1.5 and 2 on. If
you are running a slasher, that might be another 100 to 300GB by itself.

Two home server builds that I like and am happy to recommend are below. Both support
IPMI, which means they can be managed and power-cycled remotely and need neither
Expand All @@ -110,17 +112,25 @@ was unable to report ECC errors via IPMI, only OS-level reporting worked.

**Intel**

SuperMicro X11SCL-IF(-O) or X11SCL-F(-O)
Intel i3-9100F or Intel Xeon E-2xxx (i5/7 do not support ECC)
16 GiB of Micron or Samsung DDR4 UDIMM ECC RAM (unbuffered, **not** registered)
1TB M.2 NVMe SSD, e.g. Samsung 970 EVO
mITX:
- SuperMicro X11SCL-IF(-O)
uATX:
- SuperMicro X11SCL-F(-O)
Common components:
- Intel i3-9100F or Intel Xeon E-2xxx (i5/7 do not support ECC)
- 16 GiB of Micron or Samsung DDR4 UDIMM ECC RAM (unbuffered, **not** registered)
- 1TB M.2 NVMe SSD or SATA SSD, e.g. Samsung 970 EVO or Samsung 860 EVO

**AMD**

AsRock Rack X470D4U or X570D4U (if you prefer mITX, X570D4I-2T)
AMD Ryzen CPU, but not APU (APUs do not support ECC)
16 GiB of Micron or Samsung DDR4 UDIMM ECC RAM (unbuffered, **not** registered)
1TB M.2 NVMe SSD, e.g. Samsung 970 EVO
mITX:
- AsRock Rack X570D4I-2T
uATX:
- AsRock Rack X470D4U (SATA only) or X570D4U (NVMe)
Common components:
- AMD Ryzen CPU, but not APU (APUs do not support ECC)
- 16 GiB of Micron or Samsung DDR4 UDIMM ECC RAM (unbuffered, **not** registered)
- 1TB M.2 NVMe SSD or SATA SSD, e.g. Samsung 970 EVO or Samsung 860 EVO

Plus, obviously, a case, PSU, case fans. Pick your own. Well-liked
options are Node 304 (mITX) and Node 804 (uATX) with Seasonic PSUs,
Expand All @@ -131,6 +141,14 @@ an eth1 node as well, which currently takes about 230GB and keeps
growing. The eth2 db is expected to be far smaller, though exact figures
won't be seen until Phase 1.5 and 2.

You'll want decent write endurance. The two models mentioned here have 600TB
write endurance each. Intel is also well-liked, their data center SSDs
are quite reliable, if a bit pricey.

You may also consider getting two SSDs and running them in a software mirror
(RAID-1) setup, in the OS. That way, data loss becomes less likely for the
chain databases, reducing potential down time because of hardware issues.

Why ECC? This is a personal preference. The cost difference is minimal,
and the potential time savings huge. An eth2 client does not require
ECC RAM; I maintain it is very nice to have regardless.
Expand Down
31 changes: 21 additions & 10 deletions SETUP.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,13 +42,14 @@ cd eth2-docker

Please choose:
- The eth2 client you wish to run
- Nimbus
- Lighthouse
- Prysm
- Your source of eth1 data
- geth
- 3rd-party
- Whether to run a slasher (not yet implemented)
- Whether to run a grafana dashboard for monitoring (not yet implemented)
- Whether to run a grafana dashboard for monitoring

First, copy the environment file.<br />
`cp default.env .env`
Expand All @@ -75,10 +76,12 @@ Work to support dynamic DNS would be welcome.

Set the `COMPOSE_FILE` string depending on which client you are going to use. Add optional services like
geth with `:` between the file names.
- `nimbus-base.yml` - Nimbus
- `lh-base.yml` - Lighthouse
- `prysm-base.yml` - Prysm
- `geth.yml` - local geth eth1 chain node
- `grafana.yml` - grafana dashboard
- `grafana.yml` - grafana dashboard for Lighthouse or Prysm
- `nim-grafana.yml` - grafana dashboard for Nimbus

For example, Lighthouse with local geth and grafana:
`COMPOSE_FILE=lh-base.yml:geth.yml:grafana.yml`
Expand All @@ -91,21 +94,29 @@ Ports that I mention should be "Open to Internet" need to be either forwarded
to your node if behind a home router, or allowed in via the VPS firewall.

- 30303 tcp/udp - local eth1 node, geth or openethereum. Open to Internet.
- 9000 tcp/udp - lighthouse beacon node. Open to Internet.
- 19000 tcp/udp - Nimbus beacon node. Open to Internet.
- 9000 tcp/udp - Lighthouse beacon node. Open to Internet.
- 13000/tcp - Prysm beacon node. Open to Internet.
- 12000/udp - Prysm beacon node. Open to Internet.
- 3000/tcp - Grafana. **Not** open to Internet, allow locally only. It is insecure http.
- 22/tcp - SSH. Only open to Internet if this is a remote server (VPS). If open to Internet, configure
SSH key authentication.

On Ubuntu, the host firewall `ufw` can be used to only allow specific ports inbound.
- Document the ports you need, then allow them to come in. Some examples below.
- `sudo ufw allow OpenSSH` will allow ssh inbound
- `sudo ufw allow 30303` will allow traffic to port 30303, both tcp and udp.
- `sudo ufw allow 13000/tcp` will allow traffic to port 13000, tcp only
- Enable the firewall and check the rules you created
- `sudo ufw enable`
- `sudo ufw status numbered`
* Document the ports you need, then allow them to come in. Adjust as needed if you are
not using default ports.
* `sudo ufw allow OpenSSH` will allow ssh inbound
* `sudo ufw allow 30303` will allow traffic for geth to port 30303, both tcp and udp.
* `sudo ufw allow 3000/tcp` will allow traffic to the Grafana dashboard
* Nimbus
* `sudo ufw allow 19000` will allow Nimbus beacon traffic, both tcp and udp.
* Lighthouse
* `sudo ufw allow 9000` will allow Lighthouse beacon traffic, both tcp and udp
* Prysm
* `sudo ufw allow 13000/tcp && sudo ufw allow 12000/udp` will allow Prysm beacon traffic
* Enable the firewall and check the rules you created
* `sudo ufw enable`
* `sudo ufw status numbered`

## Time synchronization on Linux

Expand Down
2 changes: 1 addition & 1 deletion TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ In the absence of a proper test script, a few quick notes on a test sequence tha
should show functionality.

Prep that's not client specific:
- `cp default.env .env`, adjust LOCAL_UID and ports as needed
- `cp default.env .env`, adjust LOCAL_UID and ports as needed, set `NUMVAL` to `2`
- `sudo docker volume rm $(sudo docker volume ls -q | grep eth2-docker)`, wipe volumes from last pass,
assuming that `eth2-docker` is the directory we are testing in.

Expand Down
9 changes: 9 additions & 0 deletions default.env
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@ ETH_NETWORK=--goerli
TESTNET=medalla
# If you want debug logs, set this to debug instead of info
LOG_LEVEL=info
# Leave this as-is to compile nimbus with support for metrics via grafana.
# Change to empty (nothing after first =) to disable that functionality
NIM_METRICS=NIMFLAGS="-d:insecure"

# Ports you will forward to your staking node. Adjust here if you are
# going to use something other than defaults.
Expand All @@ -25,6 +28,7 @@ LH_PORT=9000
PRYSM_PORT=13000
PRYSM_UDP_PORT=12000
GRAFANA_PORT=3000
NIM_PORT=19000

# These variables are not likely to require adjustment
GETH_BUILD_TARGET=release/1.9
Expand All @@ -43,6 +47,11 @@ PRYSM_BN_ENTRYPOINT=start-prysm-beacon-extip.sh
PRYSM_VAL_ENTRYPOINT=validator --datadir /var/lib/prysm --wallet-dir /var/lib/prysm/
PRYSM_VAL_IMPORT_ENTRYPOINT=create-prysm-validator-wallet.sh

NIM_BUILD_TARGET=devel
NIM_USER=nimbus
NIM_ENTRYPOINT=["/usr/local/bin/beacon_node", "--data-dir=/var/lib/nimbus", "--non-interactive", "--log-file=/var/lib/nimbus/beacon_node.log"]
NIM_IMPORT_ENTRYPOINT=["/usr/local/bin/beacon_node", "deposits", "import", "--data-dir=/var/lib/nimbus", "--log-file=/var/lib/nimbus/beacon_node.log", "/var/lib/nimbus/validator_keys"]

DEPCLI_BUILD_TARGET=master
DEPCLI_USER=depcli
DEPCLI_ENTRYPOINT=./deposit.sh --folder /var/lib/depcli-data
Expand Down
2 changes: 1 addition & 1 deletion lh-base.yml
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ services:
- lhvalidator-data:/var/lib/lighthouse
- ${DEPCLI_LOCALDIR}/validator_keys:/var/lib/lighthouse/validator_keys
entrypoint: ${LH_VAL_ENTRYPOINT}
command: ["account", "validator", "import", "--directory", "/var/lib/lighthouse/validator_keys", "--validator-dir", "/var/lib/lighthouse", "--testnet", "{$TESTNET}"]
command: ["account", "validator", "import", "--directory", "/var/lib/lighthouse/validator_keys", "--validator-dir", "/var/lib/lighthouse", "--testnet", "${TESTNET}", "--reuse-password"]
deposit-cli:
restart: "no"
user: ${LOCAL_UID}:${LOCAL_UID}
Expand Down
45 changes: 45 additions & 0 deletions nim-grafana.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
version: "3"
services:
beacon:
expose:
- 8008
command:
- --metrics
- --metrics-port=8008
- --metrics-address=0.0.0.0
prometheus:
restart: "no"
build:
context: ./prometheus
image: prometheus
volumes:
- prom-data:/prometheus
environment:
- CLIENT=${COMPOSE_FILE}
expose:
- 9090
entrypoint: choose-config.sh
command: ["/bin/prometheus", "--storage.tsdb.path=/prometheus", "--web.console.libraries=/usr/share/prometheus/console_libraries", "--web.console.templates=/usr/share/prometheus/consoles"]
depends_on:
- beacon
- node-exporter
node-exporter:
restart: "no"
image: prom/node-exporter
expose:
- 9100
grafana:
restart: "no"
image: grafana/grafana
volumes:
- grafana-data:/var/lib/grafana
depends_on:
- prometheus
ports:
- ${GRAFANA_PORT}:3000/tcp
eth2:
depends_on:
- grafana
volumes:
grafana-data:
prom-data:
67 changes: 67 additions & 0 deletions nimbus-base.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
version: "3"
services:
beacon:
restart: "no"
user: ${LOCAL_UID}:${LOCAL_UID}
build:
context: ./nimbus
args:
- BUILD_TARGET=${NIM_BUILD_TARGET}
- USER=${NIM_USER}
- UID=${LOCAL_UID}
- METRICS=${NIM_METRICS}
image: nimbus
volumes:
- nimbus-data:/var/lib/nimbus
ports:
- ${NIM_PORT}:19000/tcp
- ${NIM_PORT}:19000/udp
entrypoint:
- /usr/local/bin/beacon_node
- --data-dir=/var/lib/nimbus
- --non-interactive
- --log-file=/var/lib/nimbus/beacon_node.log
- --network=${TESTNET}
- --web3-url=${ETH1_NODE}
- --graffiti=${GRAFFITI}
- --log-level=${LOG_LEVEL}
validator-import:
restart: "no"
user: ${LOCAL_UID}:${LOCAL_UID}
image: nimbus
build:
context: ./nimbus
args:
- BUILD_TARGET=${NIM_BUILD_TARGET}
- USER=${NIM_USER}
- UID=${LOCAL_UID}
- METRICS=${NIM_METRICS}
volumes:
- nimbus-data:/var/lib/nimbus
- ${DEPCLI_LOCALDIR}/validator_keys:/var/lib/nimbus/validator_keys
entrypoint: ["/usr/local/bin/beacon_node", "deposits", "import", "--data-dir=/var/lib/nimbus", "--log-file=/var/lib/nimbus/beacon_node.log", "--network=${TESTNET}", "/var/lib/nimbus/validator_keys"]
deposit-cli:
restart: "no"
user: ${LOCAL_UID}:${LOCAL_UID}
build:
context: ./eth2.0-deposit-cli
args:
- BUILD_TARGET=${DEPCLI_BUILD_TARGET}
- USER=${DEPCLI_USER}
- UID=${LOCAL_UID}
image: eth2.0-deposit-cli
volumes:
- ${DEPCLI_LOCALDIR}:/var/lib/depcli-data
entrypoint: ${DEPCLI_ENTRYPOINT}
command:
- --num_validators
- ${NUMVAL}
- --chain
- ${DEPOSIT_CHAIN}
eth2:
image: tianon/true
restart: "no"
depends_on:
- beacon
volumes:
nimbus-data:
41 changes: 41 additions & 0 deletions nimbus/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Build Nimbus in a stock debian container
FROM debian:buster-slim as builder

ARG BUILD_TARGET
ARG METRICS

RUN apt-get update && apt-get install -y build-essential git libpcre3-dev ca-certificates

WORKDIR /usr/src
RUN git clone https://github.com/status-im/nim-beacon-chain && cd nim-beacon-chain && git config advice.detachedHead false && git fetch --all --tags && git checkout ${BUILD_TARGET} && make ${METRICS} beacon_node

# Pull all binaries into a second stage deploy debian container
FROM debian:buster-slim

ARG USER
ARG UID

RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*

# See https://stackoverflow.com/a/55757473/12429735RUN
RUN adduser \
--disabled-password \
--gecos "" \
--home "/nonexistent" \
--shell "/usr/sbin/nologin" \
--no-create-home \
--uid "${UID}" \
"${USER}"

RUN mkdir -p /var/lib/nimbus && chown ${USER}:${USER} /var/lib/nimbus

# Copy executable
COPY --from=builder /usr/src/nim-beacon-chain/build/beacon_node /usr/local/bin/beacon_node

# Use an unprivileged user.
USER ${USER}:${USER}

ENTRYPOINT ["beacon_node", "--data-dir=/var/lib/nimbus", "--web3-url=http://geth:8545", "--network=medalla", "--log-file=/var/lib/nimbus/beacon_node.log"]
46 changes: 46 additions & 0 deletions nimbus/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
Docker container for Nimbus eth2 client

This creates a compiled nimbus, in a debian-slim container

Pass BUILD_TARGET, USER and UID during build if you are not using docker-compose, as well as METRICS
which is the compile-time flag for the http metrics server

Firewalling: You want 19000 tcp/udp to be exposed to "the Internet", port-forwarded if
this runs behind NAT. Do NOT expose 8008/tcp to world, it is http and meant only for the dashboard
to interface with to get metrics

A few notes on compilation and runtime options

The git branch to build should be `devel` as of Sept 2020
The validator is included with the beacon node by default. They can be separated, but the code is still experimental.
Max peers default is 79, you will not likely have to adjust this.
Metrics require `NIMFLAGS="-d:insecure"` during compile. This option is exposed to the user via .env so they can
decide on whether to include metrics.

This Dockerfile compiles for the machine it is being executed on, the resulting executable is **not** portable to
other architectures. `make NIMFLAGS="-d:disableMarchNative" beacon_node` is the way to build portable binaries,
but that is outside the purview of this project. See https://github.com/status-im/nim-beacon-chain/tree/devel#makefile-tips-and-tricks-for-developers

A static binary can be compiled with `make NIMFLAGS="--passL:-static" beacon_node`, but running in scratch was
not successful, therefore I am not doing a static compile.

The following assumes Ubuntu, hence sudo to run docker. If that's not necessary in your environment,
just leave sudo off the command and run directly as the logged-in user.

You'd run this from the docker-compose one level up. To test build and run here, while mapping to default ports:

sudo docker build -t nimbus --build-arg BUILD_TARGET=devel --build-arg USER=nimbus --build-arg UID=$UID .
sudo docker volume create nimbus-data
sudo docker run -d --name nimbus -v nimbus:/var/lib/nimbus -p 19000:19000 -p 19000:19000/udp nimbus

Example of running on port 19010 to world:

sudo docker run -d --name nimbus -v nimbus:/var/lib/nimbus -p 19010:19000 -p 19010:19000/udp nimbus

Watch logs:

sudo docker logs -f nimbus

Prune build images - saves space if no further builds are likely:

sudo docker system prune -f
Loading

0 comments on commit bef3524

Please sign in to comment.