Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add elastic-agent-wolfi-complete Docker image #5475

Merged
merged 10 commits into from
Sep 11, 2024

Conversation

rdner
Copy link
Member

@rdner rdner commented Sep 9, 2024

What does this PR do?

The PR adds a new Wolfi-based elastic-agent-wolfi-complete Docker image that contains Synthetics.

Why is it important?

It's useful, for example, if a configured integration is running Heartbeat. It also completes the migration of all images to Wolfi.

How to test this PR locally

PLATFORMS=linux/arm64 PACKAGES=docker mage package

Before:

REPOSITORY                                       TAG          IMAGE ID       CREATED         SIZE
docker.elastic.co/beats-ci/elastic-agent-cloud   8.16.0       3d78b08a1856   6 minutes ago   939MB
docker.elastic.co/beats-dev/golang-crossbuild    1.22.6-arm   eae46a040730   11 days ago     752MB
docker.elastic.co/beats/elastic-agent            8.16.0       c522ff44933d   6 minutes ago   647MB
docker.elastic.co/beats/elastic-agent-complete   8.16.0       0f72cb8295c2   5 minutes ago   2.16GB
docker.elastic.co/beats/elastic-agent-ubi        8.16.0       09c24222756b   6 minutes ago   644MB
docker.elastic.co/beats/elastic-agent-wolfi      8.16.0       48fe4922082a   6 minutes ago   573MB

After:

REPOSITORY                                             TAG          IMAGE ID       CREATED         SIZE
docker.elastic.co/beats-ci/elastic-agent-cloud         8.16.0       8d38473ba140   9 minutes ago   939MB
docker.elastic.co/beats-dev/golang-crossbuild          1.22.6-arm   eae46a040730   11 days ago     752MB
docker.elastic.co/beats/elastic-agent                  8.16.0       734a01ade09b   9 minutes ago   647MB
docker.elastic.co/beats/elastic-agent-complete         8.16.0       fadd98e9ca69   8 minutes ago   2.16GB
docker.elastic.co/beats/elastic-agent-ubi              8.16.0       d80f9c55d596   9 minutes ago   644MB
docker.elastic.co/beats/elastic-agent-wolfi            8.16.0       fc91b6e99adc   9 minutes ago   573MB
docker.elastic.co/beats/elastic-agent-wolfi-complete   8.16.0       f7bf606c8fea   7 minutes ago   1.96GB

I enrolled an agent running on the new elastic-agent-wolfi-complete image with the flags from here https://www.elastic.co/guide/en/observability/current/synthetics-private-location.html#synthetics-private-location-connect:
Screenshot 2024-09-10 at 16 44 02

Created a private location in Synthetics:
Screenshot 2024-09-10 at 16 44 13

I created a lightweight monitor (HTTP ping) for https://www.elastic.co/docs/current/integrations and a browser monitor (multistep) for this script:

step('Load the System integration page', async () => {
  await page.goto('https://www.elastic.co/docs/current/integrations');
});

Both monitors succeeded:

Screenshot 2024-09-11 at 10 12 12

Related issues

So, we have a Wolfi-based image with Synthetics.
@rdner rdner added Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team Team:Elastic-Agent Label for the Agent team labels Sep 9, 2024
@rdner rdner self-assigned this Sep 9, 2024
@mergify mergify bot added the backport-skip label Sep 9, 2024
@elastic elastic deleted a comment from mergify bot Sep 9, 2024
@rdner rdner marked this pull request as ready for review September 9, 2024 15:34
@rdner rdner requested a review from a team as a code owner September 9, 2024 15:34
@elasticmachine
Copy link
Contributor

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

@rdner rdner requested a review from leehinman September 9, 2024 15:34
Copy link
Contributor

@blakerouse blakerouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good and I don't see any issues with this PR, except for all the switches of '' to ". That really pollutes this PR and when doing a git show once this is merged it is going to show many line changes that are unrelated to creating wolfi-complete. Could you please split out the ' to " change to its own PR?

@rdner
Copy link
Member Author

rdner commented Sep 9, 2024

@blakerouse sorry about that, I'll clean it up.

@cmacknz
Copy link
Member

cmacknz commented Sep 9, 2024

How'd we test this? Can you create a Synethics private location using the agent container? https://www.elastic.co/guide/en/observability/current/synthetics-settings.html#synthetics-settings-private-locations

CC @emilioalvap @vigneshshanmugam

Copy link
Contributor

mergify bot commented Sep 9, 2024

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b elastic-agent-wolfi-complete upstream/elastic-agent-wolfi-complete
git merge upstream/main
git push upstream elastic-agent-wolfi-complete

Copy link
Member

@vigneshshanmugam vigneshshanmugam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes looks good to me.

I would test this by packaging and adding it as Synthetics private location as @cmacknz commented out. Steps would be

  1. Package it up and run the elastic agent with specific flags
  2. Start the Kibana and add the agent as Private location
  3. Now you can go the Synthetics App and create both Lightweight and browser monitors to confirm they are running as expected

{{ $beatHome }}/.synthetics \
{{ $beatHome }}/.npm \
{{ $beatHome }}/.cache \
| xargs -IDIR sh -c 'mkdir -m 0770 -p DIR && chown -R {{ .user }} DIR'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we can keep the permission in sync with the other PR - https://github.com/elastic/beats/pull/40689/files#r1750771988

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I have not seen the original comment https://github.com/elastic/beats/pull/40689/files#r1750771988 and thought it was a typo which I "fixed" in elastic/beats@b06f7ce

I think I need more context on how switching permissions 0770 to 0775 reflects removing the agent from the root group. The user created for this container is the owner of the entire directory tree and the last 0 in the bitmask denies access to "other" users and groups. Why should it be "read and execute" instead?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think @emilioalvap would be the right person to ask in this case 🙂

@rdner
Copy link
Member Author

rdner commented Sep 10, 2024

I'll go through the testing steps from #5475 (review) and will update the description of the PR once done.

Copy link
Contributor

@blakerouse blakerouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rdner Thanks for cleaning up the diff.

@rdner
Copy link
Member Author

rdner commented Sep 10, 2024

@vigneshshanmugam @emilioalvap I might need some help here.

I've created a browser (muiltistep) monitor with the following script:

step('Load the System integration page', async () => {
  await page.goto('https://www.elastic.co/docs/current/integrations');
});

And I get this error on its execution:

⚠ Throttling may not be active when the tests run - see
https://github.com/elastic/synthetics/blob/main/docs/throttling.md for more details
browserType.launch: Executable doesn't exist at /usr/share/elastic-agent/.cache/ms-playwright/chromium-1124/chrome-linux/chrome
╔═════════════════════════════════════════════════════════════════════════╗
║ Looks like Playwright Test or Playwright was just installed or updated. ║
║ Please run the following command to download new browsers: ║
║ ║
║ npx playwright install ║
║ ║
║ <3 Playwright Team ║
╚═════════════════════════════════════════════════════════════════════════╝
at Function.setupDriver (/usr/share/elastic-agent/.npm/lib/node_modules/@elastic/synthetics/src/core/gatherer.ts:56:43)
at Function.createContext (/usr/share/elastic-agent/.npm/lib/node_modules/@elastic/synthetics/src/core/runner.ts:83:35)
at Runner.runJourney (/usr/share/elastic-agent/.npm/lib/node_modules/@elastic/synthetics/src/core/runner.ts:367:34)
at Runner.run (/usr/share/elastic-agent/.npm/lib/node_modules/@elastic/synthetics/src/core/runner.ts:496:22)
at Command.<anonymous> (/usr/share/elastic-agent/.npm/lib/node_modules/@elastic/synthetics/src/cli.ts:143:23) {
name: 'Error'
}

And, indeed, I don't see Chromium installation in the original PR for Heartbeat elastic/beats#40689

Unlike the Ubuntu section here https://github.com/elastic/beats/blob/88c84a09ef35f4c5c5548562f7e1b796eec8fe8f/dev-tools/packaging/templates/docker/Dockerfile.tmpl#L217 the new code for Wolfi does not have Chromium.

I see that the Heartbeat PR was well-tested (see description) and I wonder how it worked if the same setup with Elastic Agent does not?

Can you help me with the missing parts of the installation script?

@vigneshshanmugam
Copy link
Member

@rdner Good catch, Indeed we dont have the deps. I was thinking the installation of synthetics would auto-install the playwright browsers but this has been changed recently. We need to fix also on the HB layer.

Can you add this to the wolfi image

$NODE_PATH/node/lib/node_modules/@elastic/synthetics/node_modules/.bin/playwright install chromium

This should install the chromium browser that we need to run the browser monitors.

@rdner
Copy link
Member Author

rdner commented Sep 10, 2024

@vigneshshanmugam if I add this line I get:

 => ERROR [stage-1 14/15] RUN $NODE_PATH/node/lib/node_modules/@elastic/synthetics/node_modules/.bin/playwright install chromium   0.3s
------                                                                                                                                  
 > [stage-1 14/15] RUN $NODE_PATH/node/lib/node_modules/@elastic/synthetics/node_modules/.bin/playwright install chromium:
0.107 /bin/sh: /node/lib/node_modules/@elastic/synthetics/node_modules/.bin/playwright: not found
------

Looks like we need to install playwright manually as well.

@vigneshshanmugam
Copy link
Member

@rdner My bad, the issue here is that @elastic/synthetics is installed on a different directory than the one I sent (it was on ubuntu). If you can exec in to the container and find the NPM bin paths, we can get the correct path by appending the relevant bits.

which elastic-synthetics // will give the relevant bin path for the Elastic synthetics library

// replace the bin path with lib and invoke the Playwright cli
<extacted path till .bin>/lib/node_modules/@elastic/synthetics/node_modules/.bin/playwright

Copy link
Contributor

mergify bot commented Sep 10, 2024

backport-v8.x has been added to help with the transition to the new branch 8.x.

@rdner
Copy link
Member Author

rdner commented Sep 11, 2024

@emilioalvap @vigneshshanmugam my bad, in the Dockerfile I switched to the root user before installing synthetics and all the dependencies got installed in a wrong directory.

Fixed it in 0f354a7 and everything worked as expected (see the description of this PR).

Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

Copy link
Contributor

mergify bot commented Sep 11, 2024

backport-v8.x has been added to help with the transition to the new branch 8.x.

@mergify mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Sep 11, 2024
@rdner rdner merged commit f8fd7a7 into elastic:main Sep 11, 2024
14 checks passed
@rdner rdner deleted the elastic-agent-wolfi-complete branch September 11, 2024 12:26
mergify bot pushed a commit that referenced this pull request Sep 11, 2024
So, we have a Wolfi-based image with Synthetics.

(cherry picked from commit f8fd7a7)
ycombinator pushed a commit that referenced this pull request Sep 12, 2024
So, we have a Wolfi-based image with Synthetics.

(cherry picked from commit f8fd7a7)

Co-authored-by: Denis <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-8.x Automated backport to the 8.x branch with mergify backport-skip skip-changelog Team:Elastic-Agent Label for the Agent team Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants