How do you persist the keys and thus identity of remote harvester only?

[CharlieTemplar]

This used to work by just setting keys to "persistent" but somewhere along the way various devs have "messed about" with this and now despite my keys being persisted in a harvester unique host bound volume, the keys get delete, then later the check for persistent says of, we don't need to touch these, they are persisted, but it's already TOO LATE and the harvester has a new identity.

Please fix properly or document properly, then leave it the heck alone, please.

CHIA_ROOT is set to /root/.chia/mainnet
Chia directory /root/.chia/mainnet
SSL file permissions are correct
No keys are present in the keychain. Generate them with 'chia keys generate'
/root/.chia/mainnet already exists, no migration action taken
Deleting your OLD CA in /root/.chia/mainnet/config/ssl/ca
Copying your CA from /ca to /root/.chia/mainnet/config/ssl/ca
/root/.chia/mainnet/config/ssl/ca/private_ca.crt
Attempting to set permissions 0644 on /root/.chia/mainnet/config/ssl/ca/private_ca.crt
/root/.chia/mainnet/config/ssl/ca/chia_ca.crt
Attempting to set permissions 0644 on /root/.chia/mainnet/config/ssl/ca/chia_ca.crt
/root/.chia/mainnet/config/ssl/ca/chia_ca.key
Attempting to set permissions 0600 on /root/.chia/mainnet/config/ssl/ca/chia_ca.key
/root/.chia/mainnet/config/ssl/ca/private_ca.key
Attempting to set permissions 0600 on /root/.chia/mainnet/config/ssl/ca/private_ca.key
Found private CA in /root/.chia/mainnet, using it to generate TLS certificates
Not touching key directories, key directory likely mounted by volume

[Starttoaster]

Hi @CharlieTemplar could you clarify more about your setup? But to provide some quick points of clarification, remote harvesters have never needed your 24 word mnemonic keys, they need the CA from your farmer. From the logs you provided, it looks like you are copying your CA over from /ca, which is correct, presumably the contents of that directory match your farmer machine's CA. With that said, is your remote harvester working? If your harvester is configured with your farmer's CA and pointed towards its IP address, it seems like it should be.

[Starttoaster]

This might be good to read for more information in case you haven't seen it https://docs.chia.net/farming-on-many-machines

It's not specific to chia-docker but explains how it would work on a baremetal setup, which helps me conceptually with how it should work in containers.

[CharlieTemplar]

Hi @CharlieTemplar could you clarify more about your setup? But to provide some quick points of clarification, remote harvesters have never needed your 24 word mnemonic keys, they need the CA from your farmer. From the logs you provided, it looks like you are copying your CA over from /ca, which is correct, presumably the contents of that directory match your farmer machine's CA. With that said, is your remote harvester working? If your harvester is configured with your farmer's CA and pointed towards its IP address, it seems like it should be.

To clarify, I'm not talking about mnemonic keys, only the key pairs generated for the harverster to remotely communicate with the farmer. The farmer's /ca keys are indeed used to generate the ssl keys for the harvester to successfully communicate with the farmer. Yes this works as it should.
The problem is that every time the container is restarted after maintenance or image update update a NEW set of SSL keys are generated for the harvester. Yes the harvester can successfully communicate with the farmer using these new keys, this in turn give the harvester a new harvester ID (node_id)
I use the harvester's ID in monitoring and pooling statistics and but getting a new identity everytime the container(s) restarts breaks all this. Setting -e keys="persistent" use help keep the same identity in between restarts, but now no longer does.

[Starttoaster]

Ah I see, the confusion stems from the keys verbiage (in this context) typically referring to the mnemonic. Anyway, with that added information and now that we're on the same page, thanks for the report. You're correct that people are constantly iterating on this project and things do/will change :) In this case, the keys environment variable used to have an option that ran the command for chia to initialize a new CA, which didn't make any sense and needed to be changed because the main purpose of the variable was to determine whether or not to generate a new mnemonic on container startup.

So there are probably a couple of solutions to this. One that you can do right now, and one that somebody needs to work on and add to the script :) If you remove the ca environment variable from your config it shouldn't run chia init -c ${ca} again, which means your harvester's ssl cert/key should be persisted in your volume mount. The solution we will try to implement in the future is to check to see if certs already exist, and if they do, check to make sure they're signed by the currently provided CA, and only regenerate them if they are not. You're welcome to contribute this yourself as well, if you're interested in solving this problem for others. I'm sure other people are using monitoring solutions that rely on a somewhat consistent harvester identity.

[CharlieTemplar]

If you remove the ca environment variable from your config it shouldn't run chia init -c ${ca} again, which means your harvester's ssl cert/key should be persisted in your volume mount.

without the ca env variable,the harvester won't start, just loops around with...
A farmer peer address, port, and ca path are required.

[Starttoaster]

If you remove the ca environment variable from your config it shouldn't run chia init -c ${ca} again, which means your harvester's ssl cert/key should be persisted in your volume mount.

without the ca env variable,the harvester won't start, just loops around with... A farmer peer address, port, and ca path are required.

Hey Charlie, you're absolutely correct about that, my apologies. I was looking for a quick workaround for you, and the if statement that blocked you with that message was further down in a different portion of the script.

Anyway, in #218 we added a check to verify the current private harvester cert against the provided CA, and if they match, it skips the extra chia init. This change would be in the :latest and :2.1.1 tags. Testing locally, it seems to solve the problem for me, let me know if it helps! And thanks again for the report.

[CharlieTemplar]

Anyway, in #218 we added a check to verify the current private harvester cert against the provided CA, and if they match, it skips the extra chia init. This change would be in the :latest and :2.1.1 tags. Testing locally, it seems to solve the problem for me, let me know if it helps! And thanks again for the report.

Great, just tried this and indeed seems to be working again, finally I can retain the identity of my harvesters.
Please keep it this way ;)

[Starttoaster]

Thanks for confirming the fix! We absolutely do our best to retain functionality like this when contributing. In this instance we refactored out some technical debt and this fairly innocuous regression was missed. But we're always open to hearing that we could do better, so we will. Have a great weekend :)