Define value for all constant-keyword fields in the apm package

[simitt]

The apm package defines fields of type constant_keyword in several places. Some of these fields have a fixed value set already in the template definition; others do not. This is generally supported by ES, but causes problems in combination with how Fleet handles mapping updates and rollovers, see details in elastic/elasticsearch#102829. Solving this in Fleet would add undesired complexity, therefore it was suggested to instead require more restrictive behaviour via the package spec, by requiring a value to be defined for constant keyword types (elastic/kibana#178528).

We need to update the apm-package to define values for all of these fields. This includes changing the package generation logic for the interval based templates (https://github.com/elastic/apm-server/blob/main/apmpackage/cmd/genpackage/main.go#L62).

[carsonip]

TODO: port the change to https://github.com/elastic/integrations/tree/main/packages/apm after testing in 8.12.1

The change isn't going out in 8.12.1. It is now planned for 8.12.2.

[carsonip]

integrations PR: elastic/integrations#9055

[carsonip]

For an equiv change in ES apm-data plugin, since the component template mappings are shared between intervals e.g. for service_summary, to specify the value will require duplicating the mappings to 1m, 10m and 60m versions of component templates for each of the aggregated metrics. I'm not familiar with the implications in terms of plugin upgrade when we add / remove component templates. Let me know if we want to proceed and I'll make the changes.

cc @axw @simitt

[axw]

The issue that prompted us to specify constant_keyword values is not relevant to the apm-data plugin; it is only relevant to Fleet, due to the way it attempts to update mappings for backing indices. The ES plugin will just update the component template and perform a rollover.

[carsonip]

The issue that prompted us to specify constant_keyword values is not relevant to the apm-data plugin

Sorry for not making it clear enough. Yes, that also came up in the discussion with @simitt , and we wanted to align the ES plugin and the integrations package, i.e. bringing back changes. @simitt do you still think it is worth it to do so?

[simitt]

If it is not required in the plugin for full functionality, we can also keep the plugin as is and consider this done with the changes merged and bundled for the Fleet apm package.

[carsonip]

Thanks for confirming.

Closing as #12219 and elastic/integrations#9055 are merged.