Missing global labels for aggregated metrics for RUM

[b2ronn]

Issue Description:

With the release of version 8.7.0, APM-Server introduced several new data streams, including:

apm.service_destination.10m
apm.service_summary.10m
apm.service_transaction.10m
apm.transaction.10m
apm.service_destination.1m
apm.service_summary.1m
apm.service_transaction.1m
apm.transaction.1m

However, it has been observed that in version 8.7.1, specifically in the following data streams:
apm.service_transaction.1m
apm.service_destination.1m
apm.transaction.1m
apm.service_summary.1m
the global labels are missing. Global labels play a crucial role in providing additional context and metadata to the data, allowing for more comprehensive analysis and correlation across different data streams.

The absence of global labels in these specific data streams limits their usefulness and hinders the ability to effectively utilize and explore the data. Including global labels would greatly enhance the value and flexibility of these data streams, enabling users to gain deeper insights and perform more advanced analyses.

Expected Behavior:

It is expected that in version 8.7.1, the aforementioned data streams (apm.service_transaction.1m, apm.service_destination.1m, apm.transaction.1m, apm.service_summary.1m) should continue to support global labels, similar to their presence in version 8.7.0. The availability of global labels would allow users to enrich the data with relevant metadata, enabling more comprehensive analysis and correlation.

Steps to Reproduce:

Upgrade to APM-Server version 8.7.1.
Access the specific data streams mentioned above (apm.service_transaction.1m, apm.service_destination.1m, apm.transaction.1m, apm.service_summary.1m).
Observe that global labels are missing within these data streams.
Additional Information:

The inclusion of global labels in these specific data streams would greatly enhance the usability and value of APM-Server for users relying on these data sets. Having access to a wider range of metadata through global labels would enable more advanced analysis, correlation, and visualization capabilities.

This issue is affecting users who heavily rely on global labels for their monitoring, observability, and analytical workflows. Your attention and prompt resolution of this matter would be highly appreciated.

Thank you for your attention to this issue.

[b2ronn]

I saw a commit Remove global labels from aggregations for RUM services 10624, but also some commits on Fixing service assignment labels regression
whether to return global labels in the new version?

[lahsivjar]

@b2ronn Thanks for raising the issue.

Are you facing the missing global labels issue for RUM or for normal intake too?

[b2ronn]

on the environment where it was updated to 8.7.1 there was only RUM. problem with missing global labels for RUM

[lahsivjar]

@b2ronn As you correctly figured out, #10624 intentionally drops global labels from aggregated metrics for RUM. The main objective for this is to limit the impact of cardinality explosion due to RUM services. We have plans to address this more elegantly by having overflow limits for all our aggregated metrics however, we don't have any fixed date for resolution on this.

May I know how you are using the global labels from the aggregated metrics so I can understand the situation from your perspective and suggest any alternatives if available?

[b2ronn]

Since the APM server does not support dynamic values for data_streams.namespace, we have added labels.namespace to all our applications(rum, java, .net,.....). Then, in the Ingest Pipeline, we overwrite data_streams.namespace and the index name. Based on these namespaces, we grant permissions to the indexes.
However, since you removed global labels in version 8.7.1, we are unable to upgrade from version 8.7.0 to a newer version. It is critical for us to have the ability to differentiate permissions.

[lahsivjar]

@b2ronn Thanks for the explanation. Unfortunately, currently, there are no alternatives available for your use case. We are working on introducing different protections against RUM global label overflow, however, we don't have a timeline for the resolution to be available.

[lahsivjar]

Keeping the issue open as an enhancement request for adding global labels to RUM.

[b2ronn]

I understand that it may not be entirely correct to use the "ingest pipeline" for this purpose. It would be better if there was something like the following in the APM Server configuration:

apm-server.default_data_streams.namespace: default
apm-server.data_streams_from_field.namespace: labels.namespace

However, the APM Server does not have the capability to use a dynamic namespace.

[axw]

@b2ronn you are correct that APM Server doesn't currently support dynamic namespaces. There has been some work going on behind the scenes to enable this in the hopefully not too distant future:

• [Fleet] Add support for dynamic_namespace and dynamic_dataset kibana#154732 was added to Fleet to enable integrations to index data into arbitrary namespaces and/or datasets.
• Add reroute processor elasticsearch#76511 was added to Elasticsearch to enable ingest pipelines to route documents to different data streams

The current thinking is that we'll:

• update the APM data streams to allow dynamic namespaces
• introduce something along the lines of a service.namespace attribute, which could be specified by agents (I guess by environment variable?) which will explicitly control the data_stream.namespace field

Alternatively to the second point, users may use the reroute ingest processor to a different namespace based on some other properties of the data.

That approach is not set in stone, but it sounds like it would address the needs you described in #10848 (comment).

(Independently of all that, we'll also be looking into how we can safely support including global labels in aggregation keys for RUM.)

[simitt]

Closing this in favor of #11037, which is purely focused on adding global labels to RUM.