Can we expand the Grafana stack....

[ewassef]

It would be a better fit to replace promtail and the loki ingester stack with grafana agent and to add Tempo into the MLA https://grafana.com/docs/tempo/latest/grafana-agent/.

This would reduce the processing and configuration impact on the MLA stack and add a new feature for distributed tracing. I would imagine the swap wouldnt be too difficult with the MLA gateway already configured for promtail to be modified to accept the agent and then to install the tempo chart as well

[ewassef]

@toschneck any one you know we can discuss this with?

[csengerszabo]

@ewassef, can you please elaborate on:

• your use case where this was helpful
• how do you mean it would reduce the processing and configuration impact on the MLA stack
• also on the new feature for distributed tracing, why this would be beneficial

[ewassef]

Hi @csengerszabo

The MLA stack is built on the Grafana stack and the utilizes both the promtail configurations on the user cluster and the loki ingesters on the seed. The functionality of both of these can be replaced by the single grafana agent AND include open telemetry information as well. Furthermore, the agent provides these additional benefits:

Provides a one-stop solution for collecting metrics, logs, and traces.
Collects out-of-the-box telemetry from popular projects like MySQL through integrations
Works seamlessly with the Grafana Stack. Alternatively, metrics can be sent to any Prometheus-compatible endpoint, and traces can be sent to any OTLP-compatible endpoint.
Offers new solutions to help scale metrics collection like host_filtering and sharding
Provides the Grafana Agent Operator, which enables individual teams to manage their configurations through PodMonitors, ServiceMonitors, and Probes.
see: https://grafana.com/docs/agent/latest/

So, it would be helpful in more fine-grained log, metric and telemetry filtering (which we havent been able to do today), a CRD based way to add monitors etc vs using the KKP ui and, finally, adding open telementry onto the stack.

Because its a single agent, the load would be halved right now since we are only collecting logs and metrics, but would be a third once telemetry is added.

Finally, Distributed tracing is becoming a requirement with these microservices and distributed systems as a way to determine where items are failing and how to recover. These three components would rival any commercial product and be a no-brainer to adopt in the MLA stack.
see https://github.com/grafana/tempo