Lotus-Provider WindowPoSt Test Setup Instructions

[Reiers]

Steps

Yugabyte

1. Run Yugabyte. The default user, password, and database is presumed by the lotus-provider code but can be changed. It also looks for yugabyte as a hostname by default.
2. Make the basic binaries

   git checkout feat/sturdypost
   git pull
   make all

Miner

• Fix the config to point to Yugabyte
• Run the Miner
• For features not provided by Lotus-Provider
• To upload the sectorIndex to the database
• There should be no observable changes for Miner’s behavior

Lotus-Provider Configuration

Generate the default configuration file:

./lotus-provider --db-host=localhost config default > base.toml

• Note how the DB hostname can be set. Other settings can be set there too.
• Update base.toml to include all the settings you want.
• Any same-named settings as Miner work the same.Note how the DB hostname can be set. Other settings can be set there too.

⚠️ Important configs that must be set to test:

[Subsystems]
EnableWindowPost = true
# env var: LOTUS_SUBSYSTEMS_ENABLESECTORINDEXDB
EnableSectorIndexDB = true

[Addresses]
MinerAddresses = ["MyMinerID"]

[Apis]
FULLNODE_API_INFO = ["myJWT"]

• To get your Miner Address:

lotus-miner info

• To get your Fullnode API Info:

lotus auth api-info --perm=admin

• Upload your settings with:

./lotus-provider --db-host=localhost config set base.toml

❗ This creates the "base" layer which is always loaded if it exists.

• Run Lotus-Provider:

./lotus-provider --db-host=localhost run

Additional layers can give different machines different settings with --layers=my1,my2,my3.

🗒️ Note: The Lotus-Provider binary is the only local file needed to run. However, storage.json (if defined by CLI or default) will be used and should provide file access. There is no way to make this file other than to borrow it.

Explore Data

You can see the would-be posts using ysql, a Yugabyte bin tool:

SELECT * FROM wdpost_proofs;

Explore all the tables:

• wdpost_partition_tasks
• harmony_task
• harmony_task_history

Status 🟠

As of Nov 1, this implementation does not persist to chain. This is expected to change soon.

[Reiers]

Test Checklist for Lotus-Provider WindowPoSt Implementation

First run should be on calibration ⚠️ ❗

Configuration Setup

• Check the Lotus-Miner config file to ensure it points to the YugabyteDB instance.

Miner Operations

• Start the Miner and confirm it is operational.
• Validate that the sectorIndex is uploaded to the database without any observable changes in the Miner's behavior.

Lotus-Provider Configuration

• Generate a base.toml configuration file with the default settings.
• Update base.toml with necessary configurations, including the MinerAddresses and FULLNODE_API_INFO.
• Upload the base.toml configuration settings to the Lotus-Provider.

Functional Tests

• Test the EnableWindowPost functionality in the Subsystems section.
• Retrieve the Miner Address using the Lotus-Miner info command and verify it's correct.
• Execute the Lotus-Provider with the updated configuration and confirm it is running without errors.

Data Exploration and Validation

• Access the wdpost_proofs table using ysql and verify the output.
• Explore all related tables like wdpost_partition_tasks, harmony_task, and harmony_task_history for correct data entry.

Logging and Feedback

• Identify any error messages during the test and document them.
• Enable additional logging for detailed feedback if required or if errors are encountered.
• Gather feedback on the ease of use, any unexpected behaviors, and potential improvements.

Error Handling and Reporting

• Ensure proper error messages are logged for all failure scenarios.

Documentation

• Validate that all steps and configurations are clearly documented.
• Ensure that the test checklist is available and easy to follow for other testers.

[Reiers]

• Missing placeholder in the base.toml for the things that actually is needed to make it run

Miner config.toml

[Subsystems]
EnableWindowPost = true
# env var: LOTUS_SUBSYSTEMS_ENABLESECTORINDEXDB
EnableSectorIndexDB = true

provider.toml

[Addresses]
MinerAddresses = ["MyMinerID"]

[Apis]
FULLNODE_API_INFO = ["myJWT"]
NEW NAME !
#ChainApiInfo =
Apis.ChainApiInfo

• Missing Documentation. Updating... ⌛
• This should be in the documents, or it wont start
  cat ~/.lotusminer/keystore/MF2XI2BNNJ3XILLQOJUXMYLUMU | jq -r .PrivateKey
• This needs to be configured in the .lotusminer/config.toml before start:

[HarmonyDB]
  # HOSTS is a list of hostnames to nodes running YugabyteDB
  # in a cluster. Only 1 is required
  #
  # type: []string
  # env var: LOTUS_HARMONYDB_HOSTS
   Hosts = ["127.0.1.1"]

  # The Yugabyte server's username with full credentials to operate on Lotus' Database. Blank for default.
  #
  # type: string
  # env var: LOTUS_HARMONYDB_USERNAME
  #Username = "yugabyte"

  # The password for the related username. Blank for default.
  #
  # type: string
  # env var: LOTUS_HARMONYDB_PASSWORD
  #Password = "yugabyte"

  # The database (logical partition) within Yugabyte. Blank for default.
  #
  # type: string
  # env var: LOTUS_HARMONYDB_DATABASE
  #Database = "yugabyte"

  # The port to find Yugabyte. Blank for default.
  #
  # type: string
  # env var: LOTUS_HARMONYDB_PORT
  #Port = "5433"

• default config lands in the lotus repo, this will lead a .dirty build - could be fixed by adding it to "sudo make install" /// Workaround for now: sudo cp lotus-provider /usr/local/bin

• lotus-miner doesn't start without lotus-provider/YugabytesDB after switching to this branch.

ERROR: creating node: starting node: could not build arguments for function "reflect".makeFuncStub (/snap/go/10426/src/reflect/asm_amd64.s:28): failed to build *paths.Local: could not build arguments for function "reflect".makeFuncStub (/snap/go/10426/src/reflect/asm_amd64.s:28): failed to build paths.SectorIndex: could not build arguments for function "reflect".makeFuncStub (/snap/go/10426/src/reflect/asm_amd64.s:28): failed to build *paths.IndexProxy: could not build arguments for function "reflect".makeFuncStub (/snap/go/10426/src/reflect/asm_amd64.s:28): failed to build *harmonydb.DB: received non-nil error from function "reflect".makeFuncStub (/snap/go/10426/src/reflect/asm_amd64.s:28): unable to connect to db: host=127.0.0.1 dbname=yugabyte port=5433 user=yugabyte password=yugabyte , err: failed to connect to `host=127.0.0.1 user=yugabyte database=yugabyte`: dial error (dial tcp 127.0.0.1:5433: connect: connection refused)

• FULLNODE_API_INFO = ["myJWT"] example should be -> FULLNODE_API_INFO = ["JWTtoken:/ipfornode/"] - if not set you will get this error:

2023-11-09T12:57:41.699Z	WARN	cliutil	util/apiinfo.go:94	API Token not set and requested, capabilities might be limited.
2023-11-09T12:57:41.699Z	WARN	cliutil	util/api.go:448	Not able to establish connection to node with addr: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJBbGxvdyI6WyJyZWFkIiwid3JpdGUiLCJzaWduIiwiYWRtaW4iXX0.ztT0N8dOyVXv_JeWwqGbs0EEwBToNAF15Mb_ca_WPqc/rpc/v1
panic: runtime error: integer divide by zero
goroutine 1 [running]:
github.com/filecoin-project/lotus/cli/util.FullNodeProxy[...].func1.1()
	/home/reiers/lotus/cli/util/api.go:300 +0x1bd
github.com/filecoin-project/lotus/lib/retry.Retry[...]({0x5fa14c0?, 0x9cc0320}, 0x5, 0x6, {0xc00093ea78?, 0x2, 0x2}, 0xc00093ea98)
	/home/reiers/lotus/lib/retry/retry.go:21 +0x19f
github.com/filecoin-project/lotus/cli/util.FullNodeProxy[...].func1()
	/home/reiers/lotus/cli/util/api.go:299 +0x3af
github.com/filecoin-project/lotus/api.(*CommonStruct).Version(0x5f7fe60?, {0x5fa14c0?, 0x9cc0320?})
	/home/reiers/lotus/api/proxy_gen.go:1437 +0x30
github.com/filecoin-project/lotus/cli/util.GetFullNodeAPIV1LotusProvider(0xc0002c4480, {0xc000317dd0, 0x1, 0x1}, {0x0, 0x0, 0x6?})
	/home/reiers/lotus/cli/util/api.go:470 +0x725
main.glob..func7(0xc0002c4480?)
	/home/reiers/lotus/cmd/lotus-provider/run.go:199 +0xaed
github.com/urfave/cli/v2.(*Command).Run(0x991dca0, 0xc0002c4480, {0xc000451040, 0x1, 0x1})
	/home/reiers/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:274 +0x998
github.com/urfave/cli/v2.(*Command).Run(0xc0002e5e40, 0xc0002c40c0, {0xc0000525d0, 0x3, 0x3})
	/home/reiers/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:267 +0xbe5
github.com/urfave/cli/v2.(*App).RunContext(0xc000252000, {0x5fa14c0?, 0x9cc0320}, {0xc0000525d0, 0x3, 0x3})
	/home/reiers/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:332 +0x5b7
github.com/urfave/cli/v2.(*App).Run(...)
	/home/reiers/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:309
github.com/filecoin-project/lotus/cli.RunApp(0x47ec1c0?)
	/home/reiers/lotus/cli/helper.go:48 +0xfa
main.main()
	/home/reiers/lotus/cmd/lotus-provider/main.go:154 +0xbbd

• It should be a similar command to test windowpost like lotus-miner proving compute window-post

• If the end-user is using the same worker/repo for multiple minerIDs - lotus-miner is sending both those sectors to the DB eventhough those sectors doesn't belong to that miner. In this case, it's sending mainnet sectors into calibration DB.

2023-11-09T13:53:06.829Z	WARN	stores	paths/db_index.go:395	sector {22352 67828} redeclared in 79e96904-b308-4fa7-9137-10dc920077df
2023-11-09T13:53:06.831Z	WARN	stores	paths/db_index.go:395	sector {22352 67838} redeclared in 79e96904-b308-4fa7-9137-10dc920077df
2023-11-09T13:53:06.832Z	WARN	stores	paths/db_index.go:395	sector {22352 67840} redeclared in 79e96904-b308-4fa7-9137-10dc920077df
2023-11-09T13:53:06.833Z	WARN	stores	paths/db_index.go:395	sector {22352 67842} redeclared in 79e96904-b308-4fa7-9137-10dc920077df
2023-11-09T13:53:06.834Z	WARN	stores	paths/db_index.go:395	sector {22352 67846} redeclared in 79e96904-b308-4fa7-9137-10dc920077df
2023-11-09T13:53:06.835Z	WARN	stores	paths/db_index.go:395	sector {22352 67855} redeclared in 79e96904-b308-4fa7-9137-10dc920077df
2023-11-09T13:53:06.837Z	WARN	stores	paths/db_index.go:395	sector {22352 67857} redeclared in 79e96904-b308-4fa7-9137-10dc920077df

[Reiers]

🔍 🤔

2023-11-09T14:10:24.716Z	WARN	lpwindow	lpwindow/faults_simple.go:142	CheckProvable Sector FAULT: generating vanilla proof	{"sector": {"ID":{"Miner":1013,"Number":41},"ProofType":13}, "err": "do request: Post \"http://192.168.100.70:2345/remote/vanilla/single\": context deadline exceeded", "errVerbose": "do request:\n    github.com/filecoin-project/lotus/storage/paths.(*Remote).GenerateSingleVanillaProof\n        /home/tippyflits/lotus/storage/paths/remote.go:801\n  - Post \"http://192.168.100.70:2345/remote/vanilla/single\": context deadline exceeded"}
2023-11-09T14:10:24.716Z	WARN	lpwindow	lpwindow/compute_do.go:284	Checked sectors{"checked": 32, "good": 0}
2023-11-09T14:10:24.717Z	ERROR	lpwindow	lpwindow/compute_task.go:155	WdPostTask.Do() failed to doPartition: no sectors to prove
2023-11-09T14:10:24.717Z	ERROR	harmonytask	harmonytask/task_type_handler.go:148	Do() returned error	{"type": "WdPost", "id": "1", "error": "no sectors to prove", "errorVerbose": "no sectors to prove:\n    github.com/filecoin-project/lotus/provider/lpwindow.(*WdPostTask).doPartition\n        /home/tippyflits/lotus/provider/lpwindow/compute_do.go:127"}

[Reiers]

Storage paths are created, trying to find what causes it 🔍

Miner log:

2023-11-09T14:59:12.782Z	WARN	rpc	[email protected]/handler.go:444	error in RPC call to 'Filecoin.StorageStat': fsstat: path not found:
    github.com/filecoin-project/lotus/storage/paths.init
        /home/reiers/lotus/storage/paths/local.go:738
2023-11-09T14:59:12.785Z	WARN	rpc	[email protected]/handler.go:444	error in RPC call to 'Filecoin.StorageStat': fsstat: path not found:
    github.com/filecoin-project/lotus/storage/paths.init
        /home/reiers/lotus/storage/paths/local.go:738
2023-11-09T14:59:13.865Z	WARN	rpc	[email protected]/handler.go:444	error in RPC call to 'Filecoin.StorageStat': fsstat: path not found:
    github.com/filecoin-project/lotus/storage/paths.init
        /home/reiers/lotus/storage/paths/local.go:738
2023-11-09T14:59:13.867Z	WARN	rpc	[email protected]/handler.go:444	error in RPC call to 'Filecoin.StorageStat': fsstat: path not found:
    github.com/filecoin-project/lotus/storage/paths.init
        /home/reiers/lotus/storage/paths/local.go:738
2023-11-09T14:59:13.871Z	WARN	rpc	[email protected]/handler.go:444	error in RPC call to 'Filecoin.StorageStat': fsstat: path not found:
    github.com/filecoin-project/lotus/storage/paths.init
        /home/reiers/lotus/storage/paths/local.go:738
2023-11-09T14:59:13.873Z	WARN	rpc	[email protected]/handler.go:444	error in RPC call to 'Filecoin.StorageStat': fsstat: path not found:
    github.com/filecoin-project/lotus/storage/paths.init
        /home/reiers/lotus/storage/paths/local.go:738
2023-11-09T14:59:13.887Z	WARN	rpc	[email protected]/handler.go:444	error in RPC call to 'Filecoin.StorageStat': fsstat: path not found:
    github.com/filecoin-project/lotus/storage/paths.init
        /home/reiers/lotus/storage/paths/local.go:738
2023-11-09T14:59:13.889Z	WARN	rpc	[email protected]/handler.go:444	error in RPC call to 'Filecoin.StorageStat': fsstat: path not found:
    github.com/filecoin-project/lotus/storage/paths.init
        /home/reiers/lotus/storage/paths/local.go:738
2023-11-09T14:59:13.891Z	WARN	rpc	[email protected]/handler.go:444	error in RPC call to 'Filecoin.StorageStat': fsstat: path not found:
    github.com/filecoin-project/lotus/storage/paths.init
        /home/reiers/lotus/storage/paths/local.go:738
2023-11-09T14:59:13.893Z	WARN	rpc	[email protected]/handler.go:444	error in RPC call to 'Filecoin.StorageStat': fsstat: path not found:
    github.com/filecoin-project/lotus/storage/paths.init
        /home/reiers/lotus/storage/paths/local.go:738
2023-11-09T14:59:14.965Z	WARN	rpc	[email protected]/handler.go:444	error in RPC call to 'Filecoin.StorageStat': fsstat: path not found:
    github.com/filecoin-project/lotus/storage/paths.init
        /home/reiers/lotus/storage/paths/local.go:738

1e5d2871-bb43-4b42-b903-26e46333ed8b:
        Error: fsstat: path not found:
4575cd85-b49d-43ad-bb0c-4e799892198a:
        Error: fsstat: path not found:
79e96904-b308-4fa7-9137-10dc920077df:
        Error: fsstat: path not found:
ff164f63-a5e0-4a18-884b-9c1a8858fa3e:
        Error: fsstat: path not found:

[snadrus]

RESPONSES (Comprehensive so far)
Additions mentioned below are in fix-sturdy-tests

1. now make install will install lotus-provider.
2. The placeholders are in lotus-provider config default already, but I have now added them to the migrator's base.toml writer and to the view command. I didn't want it applied in get/set because that would clobber any user comments.
3. lotus-miner uses Yugabyte, the sector index gets setup for lotus-provider. If we plan to call this a bug, we need to document enabling the Yugabyte sector index in miner as part of the migration. Please file a bug as this is a nuanced matter
4. Here's a PR to Magik's winningPost branch code for the migrator to build ChainApiInfo correctly (formerly FULLNODE_API_INFO). Please ensure docs include this detail.
5. lotus-provider proving compute window-post sounds useful TODO
6. Yes, the database is intended to absorb all sector info including those it is not currently posting. Multi-miner will come out later to benefit from that data.
7. I think the storagepath issue was debugged with Magic.
8. I understand now there are issues with windowpost not finding any sectors to prove. Please ensure the lotus-provider config default field names are filled-in correctly for miner-id. If the problem persists, talk to Magik for more hours of debug round-trip.