Add PGP key retrieval endpoint

[michel-laterman]

What is the problem this PR solves?

Fleet-server must be able to have a request path where a valid PGP key will be served to requesting agents.
This endpoint must only be valid when the server runs HTTPS.
An operator must be able to change the key that is served.

How does this PR solve the problem?

Add a new endpoint GET /api/agents/upgrades/{major}.{minor}.{patch}/pgp-public-key that will return a PGP key.
The endpoint needs a valid API key and is rate limited.
Currently the major, minor, and patch values are ignored.

The endpoint will server a PGP key from it's cache if one exists, otherwise it will serve the contents of a PGP key found on disk at FLEETSERVER_BINARY_DIR/elastic-agent-upgrade-keys/default.pgp. Note that the directory can be changed through configuration.
The key value is added to cache if it exists.

If this file does not exist the fleet-server will retrieve the key from https://artifacts.elastic.co/GPG-KEY-elastic-agent and serve it's contents (writing them to cache and disk). The upstream URL can also be changed through configuration.

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

• Closes Allow Fleet Server to serve the PGP key when stored on disk #2887

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-09-25T14:34:38.622+0000

• Duration: 39 min 41 sec

Test stats 🧪

Test	Results
Failed	0
Passed	827
Skipped	1
Total	828

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[nchaulet]

🎉 nice to add some doc on how to run tests

[nchaulet]

I am wondering if we should read the key on start as we do for the the other filer we read from the config like (SecretTokenPath , ...) it will be less confusing for the user to change the key it will have to restart fleet-server and not rely on a cache expiring no?

[nchaulet]

If we write the upstream key to disk, this mean if that key is updated user will have to manually update it on their fleet-server? does the cache is not good enough here?

[michel-laterman]

Yes, if the upstream key is updated the user will need to manually update the key on disk (or remove it)

edit: the issue is to have fleet-server serve a local key with the fallback being an upstream source.
I added caching because if this endpoint is ever used it is likely to serve very bursty traffic, caching the key from disk or from the network would reduce the work fleet-server does

[michel-laterman]

Currently pgp key config can be specified through the fleet-server integration.
If we changed it to be read on startup, then it should be specified similar to the others and need a command line/env var option in the elastic-agent.
This would change it (from a user's perspective) to a setting that they can specify through config* to one that is only specified on install.

I think we should keep it as is, it may be easier to do an emergency key replacement with the current behaviour (if upstream has changed and an old key is on disk), the user would just need to change pgp.dir and restart the process (to clear the cache) instead of doing any on-host operations if we were to change it.

[juliaElastic]

LGTM

[nchaulet]

Thanks for addressing the cache changes 🚀

[elastic-sonarqube]

SonarQube Quality Gate

0 Bugs
0 Vulnerabilities
0 Security Hotspots
7 Code Smells

82.9% Coverage
0.0% Duplication

[michalpristas]

after merge comment: was testing this with agent. everything seems fine