Manage external connections to Kubernetes pods or nodes using AWS Elastic IPs (EIPs).
This operator manages the following:
- Creation of an EIP Custom Resource Definition in K8S.
- Creation/destruction of EIP allocations in AWS.
- These AWS EIPs will be tagged with the K8S EIP uid they will be assigned to (
eip.materialize.cloud/eip_uid
) for later identification.
- These AWS EIPs will be tagged with the K8S EIP uid they will be assigned to (
- Association/disassociation of the EIP to the Elastic Network Interface (ENI) on the private IP of the pod or node.
- Annotation of
external-dns.alpha.kubernetes.io/target
on the pod, so that theexternal-dns
operator knows to use the IP of the EIP instead of the external IP of the primary ENI on the host.
-
You must be using the AWS VPC CNI plugin. This is the default for AWS EKS, so you probably don't need to change anything here.
-
You must set
AWS_VPC_K8S_CNI_EXTERNALSNAT=true
on the aws-node daemonset.kubectl set env daemonset -n kube-system aws-node AWS_VPC_K8S_CNI_EXTERNALSNAT=true
See the AWS external-snat docs for more details.
-
For
external-dns
support, you must be using a version with headless ClusterIp support, either by waiting until this PR is merged or by using a fork with it already included.
- Create an AWS IAM role with the following policy:
{ "Version": "2012-10-17", "Statement": [ { "Action": [ "ec2:AllocateAddress", "ec2:ReleaseAddress", "ec2:DescribeAddresses", "ec2:AssociateAddress", "ec2:DisassociateAddress", "ec2:DescribeNetworkInterfaces", "ec2:CreateTags", "ec2:DeleteTags", "ec2:DescribeInstances", "ec2:ModifyNetworkInterfaceAttribute", "servicequotas:GetServiceQuota" ], "Effect": "Allow", "Resource": "*" } ] }
- Create a K8S ServiceAccount.
apiVersion: v1 kind: ServiceAccount metadata: name: eip-operator annotations: eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT-ID:role/IAM-SERVICE-ROLE-NAME
- Create a K8S ClusterRole.
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: eip-operator rules: - apiGroups: [""] resources: ["pods", "pods/status"] verbs: ["get", "watch", "list", "update", "patch"] - apiGroups: [""] resources: ["nodes", "nodes/status"] verbs: ["get"]
- Create a K8S ClusterRoleBinding.
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: eip-operator-viewer roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: eip-operator subjects: - kind: ServiceAccount name: eip-operator namespace: default
- Create a K8S Deployment.
You must specify the
CLUSTER_NAME
environment variable.NAMESPACE
andDEFAULT_TAGS
are optional. If you do not set theNAMESPACE
environment variable, the eip-operator will operate on all namespaces. Even in this global mode, the eip and pod must be in the same namespace.apiVersion: apps/v1 kind: Deployment metadata: name: eip-operator spec: strategy: type: Recreate selector: matchLabels: app: eip-operator template: metadata: labels: app: eip-operator spec: containers: - name: eip-operator image: materialize/k8s-eip-operator:latest env: - name: NAMESPACE value: default - name: CLUSTER_NAME value: my-example-cluster - name: DEFAULT_TAGS value: '{"tag1": "value1", "tag2": "value2"}'
A. If you want your EIP to survive beyond the lifetime of the pod (ie: for static reservations when updating a statefulset):
Instantiate an Eip Kubernetes object, specifying the podName
in the spec.
apiVersion: "materialize.cloud/v2"
kind: Eip
metadata:
name: my-new-eip
spec:
selector:
pod:
podName: my-pod
Add the eip.materialize.cloud/manage=true
label to the pod with name matching the podName
specified above.
No need to manually create the Eip in this case, the operator can do it for you.
Add both the eip.materialize.cloud/manage=true
and eip.materialize.cloud/autocreate_eip=true
labels to your pod.
Do NOT manually create the Eip Kubernetes object if setting the eip.materialize.cloud/autocreate_eip=true
label, or the two objects will fight over your pod.
C. If you want to attach an EIP to attach to a node directly instead of a pod, specify a node selector instead in the Eip Kubernetes resource:
apiVersion: "materialize.cloud/v2"
kind: Eip
metadata:
name: my-new-eip
spec:
selector:
node:
selector:
some-label: some-value
some-other-label: some-other-value
The node selector should contain a set of labels which should match a single node - if multiple nodes are matched, the EIP will be attached to one of them arbitrarily.
Add the eip.materialize.cloud/manage=true
label to the node whose labels match the labels in the selector.
If using Cilium in ENI mode, you can still use this operator, but you will need to disable masquerade for pods with EIPs assigned. Cilium (as of 1.12.0) does not seem to support configuring masquerade on a per-pod basis, so you will need to do one of the following:
B. Run a privileged daemonset in the host network to inject ip rules for pods managed by the eip-operator.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: cilium-eip-no-masquerade-agent
spec:
selector:
matchLabels:
app: cilium-eip-no-masquerade-agent
template:
metadata:
labels:
app: cilium-eip-no-masquerade-agent
spec:
containers:
- command:
- ./cilium-eip-no-masquerade-agent
env:
- name: RUST_LOG
value: INFO
image: materialize/k8s-eip-operator
name: eip-operator
securityContext:
privileged: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
restartPolicy: Always
serviceAccount: eip-operator
serviceAccountName: eip-operator
- Install the KUTTL testing tool
- Run
./bin/run-tests
We now have support for sending traces using the OpenTelemetry OTLP format. This is configured through environment variables:
OPENTELEMETRY_ENDPOINT
is the endpoint to send the logs to.
OPENTELEMETRY_HEADERS
is a json formatted map of key/value pairs to be included in the GRPC request headers.
OPENTELEMETRY_TOPLEVEL_FIELDS
is a json formatted map of key/value pairs to be included in all trace spans emitted from the service.
OPENTELEMETRY_LEVEL_TARGETS
is a tracing
crate level filter. Defaults to "DEBUG"
.
OPENTELEMETRY_SAMPLE_RATE
is a float value controlling the trace sample rate. Default is 0.05.