Error while reconciling 4.17.0-okd-scos.0: ingress, machine-config has an unknown error: ClusterOperatorsDegraded

[RomanEpo]

#Query about 4.17.0-okd-scos.0 , Expect team feedback.

Does that issue arise due to the installation configuration? or any other issue?

##Installation details

Single node fresh installation on KVM

##issue details##

$ openshift-install --dir okd-install wait-for install-complete --log-level=debug

ERROR Cluster operator ingress Degraded is True with IngressDegraded: The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing. Last 1 error messages:

ERROR error sending canary HTTP request: DNS error: Get "https://canary-openshift-ingress-canary.apps.cluster-okd.sno.shoddho.com/": dial tcp: lookup canary-openshift-ingress-canary.apps.cluster-okd.sno.shoddho.com on 172.30.0.10:53: no such host (x413 over 6h54m0s))

$ oc get clusterversion

NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version False False 36m Error while reconciling 4.17.0-okd-scos.0: ingress, machine-config has an unknown error: ClusterOperatorsDegraded

$ oc get co

ingress 4.17.0-okd-scos.0 True False True 2d21h The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing. Last 1 error messages:...

machine-config True True True 2d21h Unable to apply 4.17.0-okd-scos.0: error during syncRequiredMachineConfigPools: [context deadline exceeded, MachineConfigPool master has not progressed to latest configuration: configuration status for pool master is empty: 0 (ready 0) out of 1 nodes are updating to latest configuration rendered-master-b7850cdee6237daae025120bf20bd8b8, retrying]

#Console logs ClusterOperator

CO machine-config Cannot update - One or more machine config pools are updating, please see oc get mcp for further details

CO ingress Degraded 4.17.0-okd-scos.0 The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing. Last 2 error messages: error sending canary HTTP Request: Timeout: Get "https://canary-openshift-ingress-canary.apps.cluster-okd.sno.shoddho.com/": context deadline exceeded (Client.Timeout exceeded while awaiting headers) error sending canary HTTP request: DNS error: Get "https://canary-openshift-ingress-canary.apps.cluster-okd.sno.shoddho.com/": dial tcp: lookup canary-openshift-ingress-canary.apps.cluster-okd.sno.shoddho.com on 172.30.0.10:53: no such host (x8 over 7m0s))

#What i did to resolve this issue

1. Remove machine-config , dns and ingress pods
2. created Canary https route $ oc create route edge canary --service=router-default --port=https -n openshift-ingress

##Result
There are no changes in the cluster. The issue is still there.

##Attachment

machine-config · Cluster Operator · Details · OKD.pdf

ingress · Cluster Operator · Details · OKD.pdf

[Tyrone-W]

Are you able to provide logs of the machine-config-daemon pod running on the node?

[RomanEpo]

machine-config-daemon Pod status: CrashLoopBackOff

#Logs :

I0205 11:56:38.708787 111914 start.go:68] Version: machine-config-daemon-4.6.0-202006240615.p0-2999-g911e8829-dirty (911e8829ee867544a650b61751c66f571396bd5c)
I0205 11:56:38.708849 111914 update.go:2639] Running: mount --rbind /run/secrets /rootfs/run/secrets
I0205 11:56:38.710975 111914 update.go:2639] Running: mount --rbind /usr/bin /rootfs/run/machine-config-daemon-bin
I0205 11:56:38.712523 111914 daemon.go:518] using appropriate binary for source=rhel-9 target=rhel-9
I0205 11:56:38.764731 111914 daemon.go:571] Invoking re-exec /run/bin/machine-config-daemon
I0205 11:56:38.791638 111914 start.go:68] Version: machine-config-daemon-4.6.0-202006240615.p0-2999-g911e8829-dirty (911e8829ee867544a650b61751c66f571396bd5c)
E0205 11:56:38.791800 111914 image_manager_helper.go:133] Merged secret file does not exist; defaulting to cluster pull secret
I0205 11:56:38.791834 111914 image_manager_helper.go:194] Linking rpm-ostree authfile to /var/lib/kubelet/config.json
F0205 11:56:38.884455 111914 start.go:106] Failed to initialize single run daemon: error reading osImageURL from rpm-ostree: exit status 1

[Tyrone-W]

I see you have marked my reply as the answer, is the issue fixed? I was going to suggest checking the content of the rendered machine config to find the value of osImageURL.

[RomanEpo]

Yes , Ingress status is available after restarting the nameserver local server,

$oc get machineconfig rendered-master-b7850cdee6237daae025120bf20bd8b8 -o yaml
osImageURL: 'quay.io/okd/scos-content@sha256:1ddb74181a918712054aad400c0a83b69cafc948ccbf77116782b4b82d3bcbca'