An error log occurs repeatedly in the deployment logs of amazon-cloudwatch-observability-controller-manager.

[wontae913]

eks version : 1.30
add-on for eks: amazon-cloudwatch-observability
version: v2.2.0-eksbuild.1

deployment.apps/amazon-cloudwatch-observability-controller-manager
image: cloudwatch-agent-operator:1.8.0

kubectl -namazon-cloudwatch logs deployment.apps/amazon-cloudwatch-observability-controller-manager

{"level":"error","ts":"2024-11-06T01:27:19Z","logger":"controllers.AmazonCloudWatchAgent","msg":"cannot create liveness probe.","error":"no service available as part of the configuration","stacktrace":"github.com/aws/amazon-cloudwatch-agent-operator/internal/manifests/collector.Container\n\t/workspace/internal/manifests/collector/container.go:92\ngithub.com/aws/amazon-cloudwatch-agent-operator/internal/manifests/collector.DaemonSet\n\t/workspace/internal/manifests/collector/daemonset.go:42\ngithub.com/aws/amazon-cloudwatch-agent-operator/internal/manifests/collector.Build.func5\n\t/workspace/internal/manifests/builder.go:20\ngithub.com/aws/amazon-cloudwatch-agent-operator/internal/manifests/collector.Build\n\t/workspace/internal/manifests/collector/collector.go:51\ngithub.com/aws/amazon-cloudwatch-agent-operator/controllers.BuildCollector\n\t/workspace/controllers/common.go:50\ngithub.com/aws/amazon-cloudwatch-agent-operator/controllers.(*AmazonCloudWatchAgentReconciler).Reconcile\n\t/workspace/controllers/amazoncloudwatchagent_controller.go:107\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}

Is it possible to take action by changing the settings? There appears to be no livenessprobe related information in yaml.

[mitali-salvi]

Hey @wontae913 , are you overriding the default add-on configuration ? It seems like the operator did not create a service for the AmazonCloudWatchAgent and hence failed to create the liveness probe as well

[wontae913]

hello @mitali-salvi
As you mentioned, I initially thought the issue was caused by reusing the configuration from the old version when installing the new version of the addon. However, even after creating a new eks cluster and installing the addon again, the logger is still encountering same liveness probe error logs repeatedly.
Regardless of the specific error messages, the running services seem to be functioning normally, and cloudwatch-agent and fluent-bit are also working as expected.

[ovaldi]

@mitali-salvi I'm also facing the same issue.

EKS version is v1.31
amazon-cloudwatch-observability version is v2.3.1-eksbuild

here is my add-on configuration:

{
    "containerLogs": {
        "enabled": false
    },
    "agent": {
        "config": {
            "logs": {
                "metrics_collected": {
                    "kubernetes": {
                        "enhanced_container_insights": true,
                        "accelerated_compute_metrics": false
                    }
                }
            }
        }
    }
}

[ovaldi]

Hi @movence @mitali-salvi Is there anything we can do to assist with this?

[andrewhharmon]

I'm facing the same problem as well. Is this expected? It spams the logs constantly.

[mk2134226]

I am facing same problem eks 1.31 aws-observability addon verison 3.1.0