Installing elastic agent on AWS EKS

[Nutties93]

apiVersion: apps/v1
kind: DaemonSet
metadata:
name: elastic-agent
namespace: kube-system
labels:
app: elastic-agent
spec:
selector:
matchLabels:
app: elastic-agent
template:
metadata:
labels:
app: elastic-agent
spec:
tolerations:
- key: node-role.kubernetes.io/control-plane
effect: NoSchedule
- key: node-role.kubernetes.io/master
effect: NoSchedule
serviceAccountName: elastic-agent
hostNetwork: true
hostPID: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: elastic-agent
image: docker.elastic.co/beats/elastic-agent:8.14.1
command: ["/usr/bin/tini", "-s", "--"]
args: ["elastic-agent"]
env:
- name: FLEET_ENROLL
value: "1"
- name: FLEET_INSECURE
value: "false"
- name: FLEET_URL
value: "https://.fleet.vpce.ap-southeast-1.aws.elastic-cloud.com:443"
- name: FLEET_ENROLLMENT_TOKEN
value: ""
- name: KIBANA_HOST
value: "http://kibana:5601"
- name: KIBANA_FLEET_USERNAME
value: "elastic"
- name: KIBANA_FLEET_PASSWORD
value: "changeme"
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: ELASTIC_NETINFO
value: "false"
securityContext:
runAsUser: 0
resources:
limits:
memory: 700Mi
requests:
cpu: 100m
memory: 400Mi
volumeMounts:
- name: proc
mountPath: /hostfs/proc
readOnly: true
- name: cgroup
mountPath: /hostfs/sys/fs/cgroup
readOnly: true
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: varlog
mountPath: /var/log
readOnly: true
- name: etc-full
mountPath: /hostfs/etc
readOnly: true
- name: var-lib
mountPath: /hostfs/var/lib
readOnly: true
- name: etc-mid
mountPath: /etc/machine-id
readOnly: true
- name: sys-kernel-debug
mountPath: /sys/kernel/debug
- name: elastic-agent-state
mountPath: /usr/share/elastic-agent/state
volumes:
- name: proc
hostPath:
path: /proc
- name: cgroup
hostPath:
path: /sys/fs/cgroup
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: varlog
hostPath:
path: /var/log
- name: etc-full
hostPath:
path: /etc
- name: var-lib
hostPath:
path: /var/lib
- name: etc-mid
hostPath:
path: /etc/machine-id
type: File
- name: sys-kernel-debug
hostPath:
path: /sys/kernel/debug
- name: elastic-agent-state
hostPath:
path: /var/lib/elastic-agent-managed/kube-system/state
type: DirectoryOrCreate

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: elastic-agent
subjects:

• kind: ServiceAccount
  name: elastic-agent
  namespace: kube-system
  roleRef:
  kind: ClusterRole
  name: elastic-agent
  apiGroup: rbac.authorization.k8s.io

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
namespace: kube-system
name: elastic-agent
subjects:

• kind: ServiceAccount
  name: elastic-agent
  namespace: kube-system
  roleRef:
  kind: Role
  name: elastic-agent
  apiGroup: rbac.authorization.k8s.io

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: elastic-agent-kubeadm-config
namespace: kube-system
subjects:

• kind: ServiceAccount
  name: elastic-agent
  namespace: kube-system
  roleRef:
  kind: Role
  name: elastic-agent-kubeadm-config
  apiGroup: rbac.authorization.k8s.io

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: elastic-agent
labels:
k8s-app: elastic-agent
rules:

• apiGroups: [""]
  resources:
  • nodes
  • namespaces
  • events
  • pods
  • services
  • configmaps
  • serviceaccounts
  • persistentvolumes
  • persistentvolumeclaims
    verbs: ["get", "list", "watch"]
• apiGroups: ["extensions"]
  resources:
  • replicasets
    verbs: ["get", "list", "watch"]
• apiGroups: ["apps"]
  resources:
  • statefulsets
  • deployments
  • replicasets
  • daemonsets
    verbs: ["get", "list", "watch"]
• apiGroups:
  • ""
    resources:
  • nodes/stats
    verbs:
  • get
• apiGroups: [ "batch" ]
  resources:
  • jobs
  • cronjobs
    verbs: [ "get", "list", "watch" ]
• nonResourceURLs:
  • "/metrics"
    verbs:
  • get
• apiGroups: ["rbac.authorization.k8s.io"]
  resources:
  • clusterrolebindings
  • clusterroles
  • rolebindings
  • roles
    verbs: ["get", "list", "watch"]
• apiGroups: ["policy"]
  resources:
  • podsecuritypolicies
    verbs: ["get", "list", "watch"]
• apiGroups: [ "storage.k8s.io" ]
  resources:
  • storageclasses
    verbs: [ "get", "list", "watch" ]

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: elastic-agent
namespace: kube-system
labels:
k8s-app: elastic-agent
rules:

• apiGroups:
  • coordination.k8s.io
    resources:
  • leases
    verbs: ["get", "create", "update"]

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: elastic-agent-kubeadm-config
namespace: kube-system
labels:
k8s-app: elastic-agent
rules:

• apiGroups: [""]
  resources:
  • configmaps
    resourceNames:
  • kubeadm-config
    verbs: ["get"]

apiVersion: v1
kind: ServiceAccount
metadata:
name: elastic-agent
namespace: kube-system
labels:
k8s-app: elastic-agent

I have tried to install the above daemon sets in my EKS cluster in AWS. the agents are up and running however, it is still not enrolled in my elastic cloud. I am see the errors below in the containers. Can someone advice on this? Are they anyone who has enrolled elastic agent on AWS EKS successfully?

{"log.level":"info","@timestamp":"2024-06-24T08:20:03.754Z","message":"Non-zero metrics in the last 30s","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.logger":"monitoring","log.origin":{"file.line":187,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot"},"service.name":"metricbeat","monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":239444180}},"memory":{"mem":{"usage":{"bytes":347721728}}}},"cpu":{"system":{"ticks":190,"time":{"ms":10}},"total":{"ticks":550,"time":{"ms":20},"value":550},"user":{"ticks":360,"time":{"ms":10}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":18},"info":{"ephemeral_id":"b2335678-40f2-4934-8f5b-593f2d2fcd1d","uptime":{"ms":120163},"version":"8.14.1"},"memstats":{"gc_next":72382608,"memory_alloc":54185456,"memory_sys":4194304,"memory_total":79991488,"rss":182398976},"runtime":{"goroutines":98}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":6}},"output":{"events":{"active":0},"write":{"latency":{"histogram":{"count":0,"max":0,"mean":0,"median":0,"min":0,"p75":0,"p95":0,"p99":0,"p999":0,"stddev":0}}}},"pipeline":{"clients":6,"events":{"active":12,"retry":6}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":1.1,"15":0.75,"5":0.82,"norm":{"1":0.275,"15":0.1875,"5":0.205}}}}},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-06-24T08:20:09.293Z","message":"Failed to connect to backoff(elasticsearch(http://elasticsearch:9200)): Get "http://elasticsearch:9200\": lookup elasticsearch on 10.100.0.10:53: no such host","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":148,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-24T08:20:09.293Z","message":"Attempting to reconnect to backoff(elasticsearch(http://elasticsearch:9200)) with 7 reconnect attempt(s)","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":139,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-06-24T08:20:09.301Z","message":"DNS lookup failure "elasticsearch": lookup elasticsearch on 10.100.0.10:53: no such host","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"transport","log.origin":{"file.line":52,"file.name":"transport/tcp.go","function":"github.com/elastic/elastic-agent-libs/transport/httpcommon.(*HTTPTransportSettings).RoundTripper.NetDialer.TestNetDialer.func3"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-06-24T08:20:09.301Z","message":"Error dialing lookup elasticsearch on 10.100.0.10:53: no such host","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.origin":{"file.line":38,"file.name":"transport/logging.go","function":"github.com/elastic/elastic-agent-libs/transport/httpcommon.(*HTTPTransportSettings).RoundTripper.LoggingDialer.func1"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"esclientleg","network":"tcp","address":"elasticsearch:9200","ecs.version":"1.6.0"}

[cmacknz]

Get "[http://elasticsearch:9200](http://elasticsearch:9200%5C/)": lookup elasticsearch on 10.100.0.10:53: no such host

http://elasticsearch:9200 is not resolving to an IP. The DNS lookup is failing.

[Nutties93]

@cmacknz yes i know but from the documentation and elastic cloud instructions, the manifest didn't provide any environment variables to configure this portion. It only shows to define the fleet_url and token.

[pierrehilbert]

You have them here: https://www.elastic.co/guide/en/fleet/current/running-on-kubernetes-standalone.html#_step_2_connect_to_the_elastic_stack

[cmacknz]

That is for a standalone agent. When enrolling in Fleet, agent should be using the output that was defined in the Fleet UI: https://www.elastic.co/guide/en/fleet/current/fleet-settings.html#output-settings

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)