Create a grafana cloud account (maybe start cloud trial). Generate an admin api token (it is scoped across all products).
Save your generated values (for your own sanity):
user:
prometheus: 53715
loki: 25836
tempo: 22348
password: MY_TOKEN=
urls:
prometheus_remote_write: https://prometheus-us-central1.grafana.net/api/prom/push
loki_hostname: logs-prod-us-central1.grafana.net
tempo_endpoint: tempo-us-central1.grafana.net:443
On k3d
4.2.0, using kubernetes 1.20.1
mkdir ~/k3d/storage
k3d cluster create cluxfana \
--servers 1 --agents 1 \
-v $HOME/k3d/storage/:/var/lib/k3d/agent-k3d/storage/ \
-v /etc/machine-id:/etc/machine-id
k3d kubeconfig get cluxfana > ~/.kube/k3d
k create ns monitoring
k apply -f agent.yml
k apply -f loki.yml
k apply -f tempo.yml
Go to expore
on clux.grafana.net, and verify:
- prometheus input:
topk(10, count by (__name__)({__name__=~".+"}))
has output - loki input:
{namespace="monitoring"}
has correctly formatted output - tempo: run an app configured against the collector
Update the snapshot by utilizing scripts from grafana agent installation giving it #user-values when prompted.
NAMESPACE="monitoring" /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/grafana/agent/release/production/kubernetes/install.sh)" > agent.yml
NAMESPACE="monitoring" /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/grafana/agent/release/production/kubernetes/install-loki.sh)" > loki.yml
NAMESPACE="monitoring" /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/grafana/agent/release/production/kubernetes/install-tempo.sh)" > tempo.yml
- default loki logging plugin is
docker: {}
(in its configmap), change tocri: {}
to get cri log parsing - grafana-agent-logs crashing with machine id mounting problems
Install and use the debugger pod:
k apply -f debugger.yml
kubectl exec -it deploy/debugger -- sh
apk add --no-cache bash curl
bash
curl grafana-agent-traces.monitoring.svc.cluster.local:55680
Official otel demo with exemplars (doesn't work) + my rust controller with tracing:
k apply -f otel-demo-go.yml
k apply -f foo-controller.yml