[BUG] trace-agent consuming more memory on average since 7.61.0

[relu]

Agent Environment

# agent version
Agent 7.61.0 - Commit: 202f54bcf5 - Serialization version: v5.0.137 - Go version: go1.22.8

Describe what happened:

Observing higher average memory utilization for the trace-agent after upgrading to v7.61.0.

Describe what you expected:

Average memory utilization should not have increased after the upgrade.

Steps to reproduce the issue:

Observe the memory utilization metrics before and after upgrading to 7.61.0

Additional environment details (Operating System, Cloud provider, etc):

Datadog Agent running on AWS EKS, Kubernetes v1.31.1, Cilium v1.16.4.

[ajgajg1134]

Hello! Thanks for opening this ticket, can you open a support ticket here? This will give us a lot more visibility into other metrics, grab agent flares etc so we can narrow down exactly the cause of the memory increase here.

[ajgajg1134]

Ahh sorry for directing you to the support endpoint so quickly but I believe I've found a likely culprit:

Enables a cache for SQL and MongoDB obfuscation. This cache is enabled by default but can be disabled by setting apm_config.obfuscation.cache.enabled to false.

This feature can significantly improve the throughput of the trace-agent by reducing the CPU cost of obfuscating SQL and MongoDB queries however it does add a bit of additional memory overhead. I still recommend opening a support ticket so we can confirm for sure if this is the issue.

If you don't want the memory overhead / you aren't seeing CPU improvements or throughput improvements (which is possible depending on your exact workload) you can disable the cache using apm_config.obfuscation.cache.enabled.

[relu]

Thank you @ajgajg1134, that sounds like a reasonable explanation and given the information I suspect this would have been an expected outcome so not really a bug. I will check to see if the respective setting will affect the memory utilization.

[relu]

As you can see, there doesn't appear to be any visible positive impact on CPU utilization (average), thus it might make sense for us to disable the setting.