[receiver/receiver_creator] Add support for metrics' hints

[ChrsMark]

Description

This PR implements the metrics' part suggested at #34427 (comment). The current implementation only supports hints coming from Pods.

Users can use the following annotations to automatically enable receivers to start collecting metrics from the target Pods/containers.

Supported metrics annotations

1. io.opentelemetry.collector.receiver-creator.metrics/receiver (example: nginx)
2. io.opentelemetry.collector.receiver-creator.metrics/endpoint (example: "http://`endpoint`/nginx_status", if not provided it defaults to endpoint which is of form pod_ip:container_port.)
3. io.opentelemetry.collector.receiver-creator.metrics/collection_interval (example: 20s)
4. io.opentelemetry.collector.receiver-creator.metrics/timeout (example: 1m)
5. io.opentelemetry.collector.receiver-creator.metrics/username (example: admin)
6. io.opentelemetry.collector.receiver-creator.metrics/password (example: passpass)

Support multiple target containers

Users can target the annotation to a specific container by suffixing it with the name of the port that container exposes, for example io.opentelemetry.collector.receiver-creator.metrics/endpoint.webserver: "http://`endpoint`/nginx_status" where webserver is the name of the port the target container exposes.

If a Pod is annotated with both container level hints and pod level hints the container level hints have priority and the Pod level hints are used as a fallback (see detailed example bellow).

The current implementation relies on the implementation of k8sobserver extension and specifically the pod_endpoint. The hints are evaluated per container by extracting the annotations from each Port endpoint that is emitted.

TODOs:

• turn-on/off the feature using a setting
• add docs
• add tests

Link to tracking issue

Part of #34427

Testing

Added unit-tests.

Documentation

Added

How to test this locally

1. Using the follow values to install the Collector's Helm chart:

mode: daemonset

image:
  repository: otelcontribcol-dev
  tag: "latest"
  pullPolicy: IfNotPresent

command:
  name: otelcontribcol

clusterRole:
  create: true
  rules:
   - apiGroups:
     - ''
     resources:
     - 'pods'
     - 'nodes'
     verbs:
     - 'get'
     - 'list'
     - 'watch'
   - apiGroups: [ "" ]
     resources: [ "nodes/proxy"]
     verbs: [ "get" ]
   - apiGroups:
       - ""
     resources:
       - nodes/stats
     verbs:
       - get
   - nonResourceURLs:
       - "/metrics"
     verbs:
       - get

config:
  extensions:
    k8s_observer:
      auth_type: serviceAccount
      node: ${env:K8S_NODE_NAME}
      observe_nodes: true
  exporters:
    debug:
      verbosity: basic
  receivers:
    receiver_creator/metrics:
      watch_observers: [ k8s_observer ]
      hints:
        k8s:
          metrics:
            enabled: true
      receivers:

  service:
    extensions: [health_check, k8s_observer]
    telemetry:
      logs:
        level: DEBUG
    pipelines:
      metrics:
        receivers: [ receiver_creator/metrics ]
        processors: [ batch ]
        exporters: [ debug ]

2. Use the following target Pod which consists of 2 containers:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-conf
data:
  nginx.conf: |
    user  nginx;
    worker_processes  1;
    error_log  /dev/stderr warn;
    pid        /var/run/nginx.pid;
    events {
      worker_connections  1024;
    }
    http {
      include       /etc/nginx/mime.types;
      default_type  application/octet-stream;
  
      log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for"';
      access_log  /dev/stdout main;
      server {
          listen 80;
          server_name localhost;
  
          location /nginx_status {
              stub_status on;
          }
      }
      include /etc/nginx/conf.d/*;
    }
---
apiVersion: v1
kind: Pod
metadata:
  name: redis
  annotations:
    io.opentelemetry.collector.receiver-creator.metrics/receiver: redis
    io.opentelemetry.collector.receiver-creator.metrics/collection_interval: '20s'
    io.opentelemetry.collector.receiver-creator.metrics/receiver.webserver: nginx 
    io.opentelemetry.collector.receiver-creator.metrics/endpoint.webserver: "http://`endpoint`/nginx_status"
  labels:
    k8s-app: redis
    app: redis
spec:
  volumes:
  - name: nginx-conf
    configMap:
     name: nginx-conf
     items:
       - key: nginx.conf
         path: nginx.conf
  containers:
  - name: webserver
    image: nginx:latest
    ports:
    - containerPort: 80
      name: webserver
    volumeMounts:
    - mountPath: /etc/nginx/nginx.conf
      readOnly: true
      subPath: nginx.conf
      name: nginx-conf
  - image: redis
    imagePullPolicy: IfNotPresent
    name: redis
    ports:
    - name: redis
      containerPort: 6379
      protocol: TCP

3. After the target pod is deployed check the Collector logs and verify that 2 receivers are started properly and metrics are collected from both target containers:

2024-10-04T13:53:02.550Z	info	[email protected]/observerhandler.go:115	starting receiver	{"kind": "receiver", "name": "receiver_creator/metrics", "data_type": "metrics", "name": "nginx/e6d880c9-13a0-45e7-a3a1-27fa021802b4_80", "endpoint": "10.244.0.52:80", "endpoint_id": "k8s_observer/e6d880c9-13a0-45e7-a3a1-27fa021802b4/webserver(80)", "config": {"collection_interval":"20s","endpoint":"http://`endpoint`/nginx_status"}}
....
2024-10-04T13:53:02.550Z	info	[email protected]/observerhandler.go:115	starting receiver	{"kind": "receiver", "name": "receiver_creator/metrics", "data_type": "metrics", "name": "redis/e6d880c9-13a0-45e7-a3a1-27fa021802b4_6379", "endpoint": "10.244.0.52:6379", "endpoint_id": "k8s_observer/e6d880c9-13a0-45e7-a3a1-27fa021802b4/redis(6379)", "config": {"collection_interval":"20s","endpoint":"10.244.0.52:6379"}}
2024-10-04T13:53:03.646Z	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 2, "metrics": 30, "data points": 38}
2024-10-04T13:53:06.547Z	debug	memorylimiter/memorylimiter.go:181	Currently used memory.	{"kind": "processor", "name": "memory_limiter", "pipeline": "traces", "cur_mem_mib": 39}
2024-10-04T13:53:11.547Z	debug	memorylimiter/memorylimiter.go:181	Currently used memory.	{"kind": "processor", "name": "memory_limiter", "pipeline": "traces", "cur_mem_mib": 39}
2024-10-04T13:53:16.546Z	debug	memorylimiter/memorylimiter.go:181	Currently used memory.	{"kind": "processor", "name": "memory_limiter", "pipeline": "traces", "cur_mem_mib": 40}

[ChrsMark]

@dmitryax I have opened this for review, please feel free to take a look when you get the chance.

Since #35544 was merged I could also add the Logs' part here if that's easier for you to review, however that would make the PR significantly bigger. Let me know what you prefer.