[APM] 8.0 breaking changes

[sorenlouv]

TODO: Make list of breaking changes in apm ui, and whether they should be migrated via the upgrade assistant.

Decided

• Deprecate and remove unused xpack.apm.* config settings ([APM] Deprecate obsolete xpack.apm.serviceMapEnabled config options #96278)
• Deprecate apm_oss app ([APM] Deprecate apm_oss app #108160)
• Prepare for unsampled transactions being dropped ([APM] Support for dropping unsampled transactions (meta) #108466)
  • This will cause incorrect doc_count (must be multiplied by sampling rate)
  • tail-based sampling will cause metrics like avg to be incorrect
  • Alerting, Correlations and ML jobs still use transaction documents
  • Observability Overview doesn't show a callout when falling back to transactions
• Drop apm_user ([APM] Replace apm_user with stack-wide roles (editor / viewer) #116760)

Up for discussion
These can be tabled for now:

• Use/require aggregate_metric_double for span/transaction breakdown metrics: Store various metrics (sum,count,etc) as aggregate_metric_double apm-server#4612
• consider creating a single identifier for a service (combination of service.name, service.environment, tranasction.type)
• consider making service.environment required in agent (Set default service.environment  apm-server#4468)
• Consider removing the ALL option in agent configuration

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[dgieselaar]

Some other fields we should consider making required:

• span.subtype (used in breakdown data)
• transaction.result (used in transaction result charts)
• service.node.name (used for system metrics)

Simply put, we should look into making all fields that we aggregate on via a terms or composite aggregation required, as using missing or missing_bucket means that we miss out on specific ES optimisations.

[sorenlouv]

Thanks @dgieselaar. I'll talk to the server team about this.

[dgieselaar]

Controversial, but we might want to consider to require one and only one environment per service. Supporting multiple environments per service makes both our application code and our UX more complicated. The workaround is adding the environment of a service in a title which seems acceptable (to me).

[formgeist]

@dgieselaar Understood that it has created way more complexity in our application code than probably necessary, but I think that mostly stems from not having a default environment to begin with. IMO if we remove the ability to separate environment data but group under the same service name, there's no need for environments in the first place (also controversial). Users will be instructed to separate by name easing the burden of maintaining environment names and for us the complexity of service name, transaction type, and environment combinations. Alternatively, we could consider only supporting a set of environment names (production, default, dev, and staging) to make it easier to users to ensure consistency but have enough flexibility to separate their service data per environment.

[dgieselaar]

If we don't use environment to separate data (ie, remove the environment filter entirely) that's even better. The user can still use custom visualisations that do separate by service.environment.

[dgieselaar]

• Use/require aggregate_metric_double for span/transaction breakdown metrics: Store various metrics (sum,count,etc) as aggregate_metric_double apm-server#4612

[dgieselaar]

• Require service destination metrics (with upstream paths) to generate a service map: Store upstream paths in transactions/spans for service maps apm#364

[alisonelizabeth]

Hi 👋! I wanted to point out that there is now a deprecations service (#94845) provided by core to register deprecations within Kibana. All registered deprecations will be displayed in the Upgrade Assistant (to be implemented via #97159). Feel free to reach out to myself or the core team with any questions!