Elastic Agent is not attempting to restart APM Server

[simitt]

We observed that the Elastic Agent is not aiming to restart the APM Server when the server stops, due to a sourcemap issue. There is a dedicated bug issue filed to avoid the APM Server being stopped in the first place elastic/kibana#149660 (comment)). However, this also uncovered that the Elastic Agent does not attempt to restart the APM Server, which is confirmed by @ycombinator as unexpected behaviour. We would expect the Agent to try restarting the APM Server.

If possible, it would also be helpful if Elastic Agent logs contained information about subprocess uptime to help with investigations of failures.

• Version: reproduced for 8.12 and 8.13, might also be available in other versions
• System: reproducible on ESS
• Steps to Reproduce:
  This is easily reproducible in ESS for everyone; following the steps in [APM] Uploading a source map with ${foo} in the service_version param prevents APM Server from working kibana#149660 (comment)

Logs:

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[cmacknz]

Seeing the reproduction attached to the use of service_version=${VERSION} which I assume can't be substituted, this makes me think this is #2261 and we are dropping apm-server from the set of running components.

[blakerouse]

I think this has two related issues:

The main one is #2177 which needs to be solved to allow passing in $${foo} which will then become ${foo}. The second issue would be to determine if Fleet or the integration should escape it automatically so the user doesn't need to do the escape themselves.

[ycombinator]

Thanks for the investigation, @blakerouse. Let's put this issue on hold for now as it depends on #2177 and work on #2177 first instead. Accordingly, I've brought #2177 into the current sprint and moved this issue into the next one. If #2177 turns out to be a somewhat quick fix, we can consider bringing this issue back into the current sprint.

[blakerouse]

I was able to reproduce this issue on 8.15. The issue is that once a source map is uploaded, Kibana creates a new policy revision with the follow in it:

rum:
        enabled: true
        allow_headers: null
        allow_origins:
          - '*'
        exclude_from_grouping: ^/webpack
        library_pattern: node_modules|bower_components|~
        response_headers: null
        source_mapping:
          metadata:
            - service.name: foo
              service.version: '${VERSION}'
              bundle.filepath: /test/e2e/general-usecase/bundle.js
              sourcemap.url: >-
                /api/fleet/artifacts/foo-${VERSION}/9824f1486e9362e9b083a979ec4cc462eea6fafff53e2511503ec96dc9bb2ba7
          elasticsearch:
            api_key: 'PgTw6pABgMVqePbnEstb:uo8CHX5jToSMZjDdoYiYrA'

See the source.version: '${VERSION}'. That needs to be escaped by Kibana before being placed into the policy as source.version: '$${VERSION}'.

[ycombinator]

Thanks @blakerouse for the investigation.

That needs to be escaped by Kibana before being placed into the policy as source.version: '$${VERSION}'.

This appears to be elastic/kibana#188377. cc: @kpollich

[ycombinator]

With elastic/kibana#188377 resolved, I'm now closing this issue as well.