Add Run Codex page (#7)

* Add Testnet contracts information * Add Run Codex page * Add information how to run Codex storage node * Update Testnet Verifier/Marketplace contracts codex-storage/infra-codex#223
codex-storage · Sep 24, 2024 · 153e200 · 153e200
1 parent de3f5b9
commit 153e200
Show file tree

Hide file tree

Showing 3 changed files with 361 additions and 1 deletion.
diff --git a/.vitepress/config.mts b/.vitepress/config.mts
@@ -79,7 +79,8 @@ export default defineConfig({
           { text: 'Architecture', link: '/learn/architecture' },
           { text: 'Marketplace', link: '/learn/marketplace' },
           { text: 'Whitepaper', link: '/learn/whitepaper' },
-          { text: 'Build Codex', link: '/learn/build' }
+          { text: 'Build Codex', link: '/learn/build' },
+          { text: 'Run Codex', link: '/learn/run' }
         ]
       },
       {

diff --git a/learn/run.md b/learn/run.md
@@ -0,0 +1,351 @@
+# Run Codex
+
+As for now, Codex is implemented only in [Nim](https://nim-lang.org) and can be found in [nim-codex](https://github.com/codex-storage/nim-codex) repository.
+
+It is a command-line application which may be run in a different ways:
+ - [Using binary](#using-binary)
+ - [Run as a daemon in Linux](#run-as-a-daemon-in-linux) (not supported yet)
+ - [Run as a service in Windows](#run-as-a-service-in-windows) (not supported yet)
+ - [Using Docker](#using-docker)
+ - [Using Docker Compose](#using-docker-compose)
+ - [Using Ansible](#using-ansible)
+ - [On Kubernetes](#on-kubernetes)
+
+During the run, it is required to pass [configuration](#configuration) option to the application, which can be done in a different ways.
+
+## Configuration
+
+It is possible to configure Codex node in several ways:
+ 1. [CLI options](#cli-options)
+ 2. [Environment variables](#environment-variables)
+ 3. [Configuration file](#configuration-file)
+
+The order of priority is the same as above:  
+[CLI options](#cli-options) --> [Environment variables](#environment-variables) --> [Configuration file](#configuration-file).
+
+### Common information
+
+#### Units
+
+For some configuration options, we can pass values in common units like:
+```shell
+--storage-quota=2m/2M/2mb/2MB
+--cache-size
+
+--block-ttl=2s/2S/2m/2M/2h/2H/2d/2D/2w/2W
+--block-mi
+```
+
+#### Logging
+
+Codex uses [Chronicles](https://github.com/status-im/nim-chronicles) logging library, which allows great flexibility in working with logs.
+Chronicles has the concept of topics, which categorize log entries into semantic groups.
+
+Using the `log-level` parameter, you can set the top-level log level like `--log-level="trace"`, but more importantly,
+you can set log levels for specific topics like `--log-level="info; trace: marketplace,node; error: blockexchange"`,
+which sets the top-level log level to `info` and then for topics `marketplace` and `node` sets the level to `trace` and so on.
+
+### CLI options
+
+```shell
+codex --help
+Usage:
+
+codex [OPTIONS]... command
+
+The following options are available:
+
+     --config-file          Loads the configuration from a TOML file [=none].
+     --log-level            Sets the log level [=info].
+     --metrics              Enable the metrics server [=false].
+     --metrics-address      Listening address of the metrics server [=127.0.0.1].
+     --metrics-port         Listening HTTP port of the metrics server [=8008].
+ -d, --data-dir             The directory where codex will store configuration and data
+                            [=/root/.cache/codex].
+ -i, --listen-addrs         Multi Addresses to listen on [=/ip4/0.0.0.0/tcp/0].
+ -a, --nat                  IP Addresses to announce behind a NAT [=127.0.0.1].
+ -e, --disc-ip              Discovery listen address [=0.0.0.0].
+ -u, --disc-port            Discovery (UDP) port [=8090].
+     --net-privkey          Source of network (secp256k1) private key file path or name [=key].
+ -b, --bootstrap-node       Specifies one or more bootstrap nodes to use when connecting to the network.
+     --max-peers            The maximum number of peers to connect to [=160].
+     --agent-string         Node agent string which is used as identifier in network [=Codex].
+     --api-bindaddr         The REST API bind address [=127.0.0.1].
+ -p, --api-port             The REST Api port [=8080].
+     --api-cors-origin      The REST Api CORS allowed origin for downloading data. '*' will allow all
+                            origins, '' will allow none. [=Disallow all cross origin requests to download
+                            data].
+     --repo-kind            Backend for main repo store (fs, sqlite, leveldb) [=fs].
+ -q, --storage-quota        The size of the total storage quota dedicated to the node [=$DefaultQuotaBytes].
+ -t, --block-ttl            Default block timeout in seconds - 0 disables the ttl [=$DefaultBlockTtl].
+     --block-mi             Time interval in seconds - determines frequency of block maintenance cycle: how
+                            often blocks are checked for expiration and cleanup
+                            [=$DefaultBlockMaintenanceInterval].
+     --block-mn             Number of blocks to check every maintenance cycle [=1000].
+ -c, --cache-size           The size of the block cache, 0 disables the cache - might help on slow hardrives
+                            [=0].
+
+Available sub-commands:
+
+codex persistence [OPTIONS]... command
+
+The following options are available:
+
+     --eth-provider         The URL of the JSON-RPC API of the Ethereum node [=ws://localhost:8545].
+     --eth-account          The Ethereum account that is used for storage contracts.
+     --eth-private-key      File containing Ethereum private key for storage contracts.
+     --marketplace-address  Address of deployed Marketplace contract.
+     --validator            Enables validator, requires an Ethereum node [=false].
+     --validator-max-slots  Maximum number of slots that the validator monitors [=1000].
+
+Available sub-commands:
+
+codex persistence prover [OPTIONS]...
+
+The following options are available:
+
+     --circom-r1cs          The r1cs file for the storage circuit
+                            [=/root/.cache/codex/circuits/proof_main.r1cs].
+     --circom-wasm          The wasm file for the storage circuit
+                            [=/root/.cache/codex/circuits/proof_main.wasm].
+     --circom-zkey          The zkey file for the storage circuit
+                            [=/root/.cache/codex/circuits/proof_main.zkey].
+     --circom-no-zkey       Ignore the zkey file - use only for testing! [=false].
+     --proof-samples        Number of samples to prove [=5].
+     --max-slot-depth       The maximum depth of the slot tree [=32].
+     --max-dataset-depth    The maximum depth of the dataset tree [=8].
+     --max-block-depth      The maximum depth of the network block merkle tree [=5].
+     --max-cell-elements    The maximum number of elements in a cell [=67].
+```
+
+### Environment variables
+
+In order to set a configuration option using environment variables, first find the desired [CLI option](#cli-options)
+and then transform it in the following way:
+
+ 1. prepend it with `CODEX_`
+ 2. make it uppercase
+ 3. replace `-` with `_`
+
+For example, to configure `--log-level`, use `CODEX_LOG_LEVEL` as the environment variable name.
+
+> [!WARNING]
+> Some options can't be configured via environment variables for now. Please read [Known issues](#known-issues) for more details.
+
+### Configuration file
+
+A [TOML](https://toml.io/en/) configuration file can also be used to set configuration values. Configuration option names and corresponding values are placed in the file, separated by `=`. Configuration option names can be obtained from the [`codex --help`](#cli-options) command, and should not include the `--` prefix. For example, a node's log level (`--log-level`) can be configured using TOML as follows:
+
+```toml
+log-level = "trace"
+```
+
+For option, like `bootstrap-node` and `listen-addrs` which accept multiple values we can specify data as an array
+```toml
+listen-addrs = [
+  "/ip4/0.0.0.0/tcp/7777",
+  "/ip4/0.0.0.0/tcp/8888"
+]
+```
+
+The Codex node can then read the configuration from this file using the `--config-file` CLI parameter, like `codex --config-file=/path/to/your/config.toml`.
+
+## Run
+
+Basically, we can run Codex in three different modes
+ - [Codex node](#codex-node) - useful for local testing/development and basic/files sharing
+ - [Codex node with marketplace support](#codex-node-with-marketplace-support) - main mode and should be used by the end users
+ - [Codex storage node](#codex-storage-node) - should be used by storage providers or if you would like to sell your local storage
+
+### Using binary
+
+#### Codex node
+
+We can run Codex in a simple way like following:
+```shell
+codex
+```
+
+But, it will use a default `data-dir` value and we can pass a custom one:
+```shell
+codex --data-dir=./datadir
+```
+
+This will run Codex as an isolated instance, and if we would like to join an existing network, it is required to pass a [bootstrap node](architecture#network-architecture). We can pass multiple nodes as well:
+```shell
+codex \
+  --data-dir=./datadir \
+  --bootstrap-node=spr:CiUIAhIhAiJvIcA_ZwPZ9ugVKDbmqwhJZaig5zKyLiuaicRcCGqLEgIDARo8CicAJQgCEiECIm8hwD9nA9n26BUoNuarCEllqKDnMrIuK5qJxFwIaosQ3d6esAYaCwoJBJ_f8zKRAnU6KkYwRAIgM0MvWNJL296kJ9gWvfatfmVvT-A7O2s8Mxp8l9c8EW0CIC-h-H-jBVSgFjg3Eny2u33qF7BDnWFzo7fGfZ7_qc9P \
+  --bootstrap-node=spr:CiUIAhIhAyUvcPkKoGE7-gh84RmKIPHJPdsX5Ugm_IHVJgF-Mmu_EgIDARo8CicAJQgCEiEDJS9w-QqgYTv6CHzhGYog8ck92xflSCb8gdUmAX4ya78QoemesAYaCwoJBES39Q2RAnVOKkYwRAIgLi3rouyaZFS_Uilx8k99ySdQCP1tsmLR21tDb9p8LcgCIG30o5YnEooQ1n6tgm9fCT7s53k6XlxyeSkD_uIO9mb3
+```
+
+> [!IMPORTANT]
+> Make sure you are using a proper value for the [network](/networks/networks) you would like to join.
+
+Also, to make your Codex node accessible for other network participants, it is required to specify a public IP address which can be used to access your node:
+```shell
+codex \
+  --data-dir=./datadir \
+  --bootstrap-node=spr:CiUIAhIhAiJvIcA_ZwPZ9ugVKDbmqwhJZaig5zKyLiuaicRcCGqLEgIDARo8CicAJQgCEiECIm8hwD9nA9n26BUoNuarCEllqKDnMrIuK5qJxFwIaosQ3d6esAYaCwoJBJ_f8zKRAnU6KkYwRAIgM0MvWNJL296kJ9gWvfatfmVvT-A7O2s8Mxp8l9c8EW0CIC-h-H-jBVSgFjg3Eny2u33qF7BDnWFzo7fGfZ7_qc9P \
+  --nat=<your public IP>
+```
+
+> [!TIP]
+> We can set public IP using curl and IP lookup service, like [ip.codex.storage](https://ip.codex.storage).
+
+After that, node will announce itself using your public IP and dynamic TCP port for [libp2p transport](https://docs.libp2p.io/concepts/transports/overview/) (data transfer), which can be adjusted in the following way:
+```shell
+codex \
+  --data-dir=./datadir \
+  --bootstrap-node=spr:CiUIAhIhAiJvIcA_ZwPZ9ugVKDbmqwhJZaig5zKyLiuaicRcCGqLEgIDARo8CicAJQgCEiECIm8hwD9nA9n26BUoNuarCEllqKDnMrIuK5qJxFwIaosQ3d6esAYaCwoJBJ_f8zKRAnU6KkYwRAIgM0MvWNJL296kJ9gWvfatfmVvT-A7O2s8Mxp8l9c8EW0CIC-h-H-jBVSgFjg3Eny2u33qF7BDnWFzo7fGfZ7_qc9P \
+  --nat=`curl -s https://ip.codex.storage` \
+  --listen-addrs=/ip4/0.0.0.0/tcp/8070
+```
+
+In that way, node will announce itself using the following [multiaddress](https://docs.libp2p.io/concepts/fundamentals/addressing/):
+
+`/ip4/<your public IP>/tcp/8070`
+
+and we can check that via [API](https://api.codex.storage/#tag/Debug/operation/getDebugInfo) call:
+```shell
+curl -s localhost:8080/api/codex/v1/debug/info | jq -r '.announceAddresses'
+```
+```json
+[
+  "/ip4/<your public IP>/tcp/8070"
+]
+```
+Basically, for P2P communication we should set/adjust two ports:
+| # | Protocol function                                                        | CLI option       | Example                                |
+| - | ------------------------------------------------------------------------ | ---------------- | -------------------------------------- |
+| 1 | [Discovery](https://docs.libp2p.io/concepts/discovery-routing/overview/) | `--disc-port`    | `--disc-port=8090`                     |
+| 2 | [Transport](https://docs.libp2p.io/concepts/transports/overview/)        | `--listen-addrs` | `--listen-addrs=/ip4/0.0.0.0/tcp/8070` |
+
+And, also it is required to setup port-forwarding on your Internet device, please read [Known issues](#known-issues) for more details.
+
+So, a fully working basic configuration will looks like following:
+```shell
+codex \
+  --data-dir=./datadir \
+  --bootstrap-node=spr:CiUIAhIhAiJvIcA_ZwPZ9ugVKDbmqwhJZaig5zKyLiuaicRcCGqLEgIDARo8CicAJQgCEiECIm8hwD9nA9n26BUoNuarCEllqKDnMrIuK5qJxFwIaosQ3d6esAYaCwoJBJ_f8zKRAnU6KkYwRAIgM0MvWNJL296kJ9gWvfatfmVvT-A7O2s8Mxp8l9c8EW0CIC-h-H-jBVSgFjg3Eny2u33qF7BDnWFzo7fGfZ7_qc9P \
+  --nat=`curl -s https://ip.codex.storage` \
+  --disc-port=8090 \
+  --listen-addrs=/ip4/0.0.0.0/tcp/8070
+```
+
+After node is up and running and port-forwarding configurations was done, we should be able to [Upload a file](https://github.com/codex-storage/codex-testnet-starter/blob/master/USINGCODEX.md#upload-a-file)/[Download a file](https://github.com/codex-storage/codex-testnet-starter/blob/master/USINGCODEX.md#download-a-file) in the network. And to be able to purchase storage, we should run [Codex node with marketplace support](#codex-node-with-marketplace-support).
+
+#### Codex node with marketplace support
+
+[Marketplace](marketplace) support permits to purchase the storage in Codex network. Basically, we should add just a `persistence` sub-command and required CLI options to the [previous run](#codex-node).
+
+> [!NOTE]
+> Please ignore `--eth-account` CLI option, read [Known issues](#known-issues) for more details.
+
+1. For a daily use, we should consider to run a local blockchain node based on the [network](/networks/networks) you would like to join. That process is described in the [Join Testnet](/networks/testnet)(:construction:), but for quick start we can use a public RPC endpoint.
+
+2. Create a file with ethereum private key and set a proper permissions.
+   ```shell
+   key=$(curl -s https://key.codex.storage)
+   echo $key | awk -F ':' '/address/ {print $2}' >eth.address
+   echo $key | awk -F ':' '/private/ {print $2}' >eth.key
+   chmod 600 eth.key
+   ```
+   > [!CAUTION]
+   > Please use mentioned key generation service for demo purpose only.
+
+3. Specify bootstrap nodes and marketplace address based on the [network](/networks/networks) you would like to join.
+
+4. Run the node
+   ```shell
+   codex \
+     persistence \
+     --data-dir=./datadir \
+     --bootstrap-node=spr:CiUIAhIhAiJvIcA_ZwPZ9ugVKDbmqwhJZaig5zKyLiuaicRcCGqLEgIDARo8CicAJQgCEiECIm8hwD9nA9n26BUoNuarCEllqKDnMrIuK5qJxFwIaosQ3d6esAYaCwoJBJ_f8zKRAnU6KkYwRAIgM0MvWNJL296kJ9gWvfatfmVvT-A7O2s8Mxp8l9c8EW0CIC-h-H-jBVSgFjg3Eny2u33qF7BDnWFzo7fGfZ7_qc9P \
+     --nat=`curl -s https://ip.codex.storage` \
+     --disc-port=8090 \
+     --listen-addrs=/ip4/0.0.0.0/tcp/8070 \
+     --eth-provider=https://rpc.testnet.codex.storage \
+     --eth-private-key=eth.key \
+     --marketplace-address=0xB119d28d3A1bFD281b23A0890B4c1B626EE8F6F0
+   ```
+
+After node is up and running, you just need to fill-up your ethereum address (`cat eth.address`) with the tokens and then you should be able to [Purchase storage](https://github.com/codex-storage/codex-testnet-starter/blob/master/USINGCODEX.md#purchase-storage).
+
+#### Codex storage node
+
+Codex [storage node](architecture#network-architecture) should be run by storage providers or in case you would like to sell your local storage. For that, we should use configuration for [Codex node with marketplace support](#codex-node-with-marketplace-support) and additionally use `prover` sub-command and required CLI options.
+
+That sub-command will make Codex to listen for a proof request on the blockchain and answer them. To compute an answer for the proof request, Codex will use stored data and circuit files generated by the code in the [codex-storage-proofs-circuits](https://github.com/codex-storage/codex-storage-proofs-circuits) repository.
+
+Every [network](/networks/networks) uses its own generated set of the files which are stored in the [codex-contracts-eth](https://github.com/codex-storage/codex-contracts-eth/tree/master/verifier/networks) repository and also uploaded to the CDN. Hash of the set is also known by the marketplace smart contract.
+
+To download circuit files and make them available to Codex app, we have a stand-alone utility - `cirdl`. It can be compiled from the sources (`make cirdl`) or downloaded from the [GitHub release page](https://github.com/codex-storage/nim-codex/releases) (work in progress - [Rework circuit downloader #882](https://github.com/codex-storage/nim-codex/pull/882)).
+
+You would need to pass a bootstrap nodes, blockchain RPC endpoint and marketplace address based on the [network](/networks/networks) you would like to join.
+
+1. Download circuit files
+   ```shell
+   # Create circuit files folder
+   mkdir -p ./datadir/circuit
+
+   # Download circuit files
+   cirdl \
+     ./datadir/circuit \
+     https://rpc.testnet.codex.storage \
+     0xB119d28d3A1bFD281b23A0890B4c1B626EE8F6F0
+   ```
+
+2. Start Codex storage node
+   ```shell
+     codex \
+       persistence \
+       prover \
+       --data-dir=./datadir \
+       --bootstrap-node=spr:CiUIAhIhAiJvIcA_ZwPZ9ugVKDbmqwhJZaig5zKyLiuaicRcCGqLEgIDARo8CicAJQgCEiECIm8hwD9nA9n26BUoNuarCEllqKDnMrIuK5qJxFwIaosQ3d6esAYaCwoJBJ_f8zKRAnU6KkYwRAIgM0MvWNJL296kJ9gWvfatfmVvT-A7O2s8Mxp8l9c8EW0CIC-h-H-jBVSgFjg3Eny2u33qF7BDnWFzo7fGfZ7_qc9P \
+       --nat=`curl -s https://ip.codex.storage` \
+       --disc-port=8090 \
+       --listen-addrs=/ip4/0.0.0.0/tcp/8070 \
+       --eth-provider=https://rpc.testnet.codex.storage \
+       --eth-private-key=eth.key \
+       --marketplace-address=0xB119d28d3A1bFD281b23A0890B4c1B626EE8F6F0 \
+       --circom-r1cs=./datadir/circuits/proof_main.r1cs \
+       --circom-wasm=./datadir/circuits/proof_main.wasm \
+       --circom-zkey=./datadir/circuits/proof_main.zkey
+   ```
+
+After node is up and running, you just need to fill-up your ethereum address (`cat eth.address`) with the tokens and then you should be able to [Create storage availability](https://github.com/codex-storage/codex-testnet-starter/blob/master/USINGCODEX.md#create-storage-availability).
+
+### Run as a daemon in Linux
+
+This functionality is not supported yet :construction:
+
+### Run as a service in Windows
+
+This functionality is not supported yet :construction:
+
+### Using Docker
+
+To be added :construction:
+
+### Using Docker Compose
+
+To be added :construction:
+
+### Using Ansible
+
+Planning :construction:
+
+### On Kubernetes
+
+To be added :construction:
+
+Helm chart code is available in [helm-charts](https://github.com/codex-storage/helm-charts) repository, but chart was not published yet.
+
+
+## Known issues
+
+1. Environment variables like `CODEX_BOOTSTRAP_NODE` and `CODEX_LISTEN_ADDRS` does not support multiple values. Please check [[Feature request] Support multiple SPR records via environment variable #525](https://github.com/codex-storage/nim-codex/issues/525), for more information.
+2. Sub-commands configuration like `persistence` and `persistence prover` can't be done via environment variables for now.
+3. [NAT traversal #753](https://github.com/codex-storage/nim-codex/issues/753) is not implemented yet and we would need to setup port-forwarding for discovery and transport protocols.
+4. Please ignore `--eth-account` CLI option - [Drop support for --eth-account #727](https://github.com/codex-storage/nim-codex/issues/727).
diff --git a/networks/testnet.md b/networks/testnet.md
@@ -30,6 +30,14 @@ enode://6eeb3b3af8bef5634b47b573a17477ea2c4129ab3964210afe3b93774ce57da832eb110f
 enode://6ba0e8b5d968ca8eb2650dd984cdcf50acc01e4ea182350e990191aadd79897801b79455a1186060aa3818a6bc4496af07f0912f7af53995a5ddb1e53d6f31b5@209.38.160.40:40070
 ```
 
+#### Marketplace
+
+| Contract    | Address                                                                                                                                   |
+| ----------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
+| Token       | [`0x34a22f3911De437307c6f4485931779670f78764`](https://explorer.testnet.codex.storage/address/0x34a22f3911De437307c6f4485931779670f78764) |
+| Verifier    | [`0x02dd582726F7507D7d0F8bD8bf8053d3006F9092`](https://explorer.testnet.codex.storage/address/0x02dd582726F7507D7d0F8bD8bf8053d3006F9092) |
+| Marketplace | [`0xCDef8d6884557be4F68dC265b6bB2E3e52a6C9d6`](https://explorer.testnet.codex.storage/address/0xCDef8d6884557be4F68dC265b6bB2E3e52a6C9d6) |
+
 #### Endpoints
 
 | # | Service         | URL                                                                          |