Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rename node-bootstrapper => aks-node-controller #5191

Merged
merged 5 commits into from
Nov 3, 2024

Merge branch 'dev' into r2k1/aks-node-controller

1ed7831
Select commit
Loading
Failed to load commit list.
Merged

Rename node-bootstrapper => aks-node-controller #5191

Merge branch 'dev' into r2k1/aks-node-controller
1ed7831
Select commit
Loading
Failed to load commit list.
Azure Pipelines / Agentbaker E2E failed Nov 3, 2024 in 38m 1s

Build #20241103.2 had test failures

1 errors / 1 warnings

Details

Tests

  • Failed: 4 (7.55%)
  • Passed: 49 (92.45%)
  • Other: 0 (0.00%)
  • Total: 53

Annotations

Check failure on line 3181 in Build log

See this annotation in the file changed.

@azure-pipelines azure-pipelines / Agentbaker E2E

Build log #L3181 

Bash exited with code '1'.

Check failure on line 1 in Test_azurelinuxv2gpu

See this annotation in the file changed.

@azure-pipelines azure-pipelines / Agentbaker E2E

Test_azurelinuxv2gpu 

Failed
Raw output 
    scenario_helpers_test.go:96: running scenario "Test_azurelinuxv2gpu" with vhd: "/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/AzureLinuxV2gen2/versions/1.1730621031.3821", tags {Name:Test_azurelinuxv2gpu ImageName:AzureLinuxV2gen2 OS:azurelinux Arch:amd64 Airgap:false GPU:true WASM:false ServerTLSBootstrapping:false Scriptless:false KubeletCustomConfig:false}
    scenario_helpers_test.go:102: running scenario "Test_azurelinuxv2gpu" with image "/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/AzureLinuxV2gen2/versions/1.1730621031.3821" in aks cluster "/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourcegroups/abe2e-westus3/providers/Microsoft.ContainerService/managedClusters/abe2e-kubenet-331fc"
    vmss.go:36: creating VMSS "abe2e-2024-11-03-ug8s-azurelinuxv2gpu" in resource group "MC_abe2e-westus3_abe2e-kubenet-331fc_westus3"
    scenario_helpers_test.go:110: vmss abe2e-2024-11-03-ug8s-azurelinuxv2gpu creation succeeded, proceeding with node readiness and pod checks...
    pollers.go:29: waiting for node abe2e-2024-11-03-ug8s-azurelinuxv2gpu to be ready
    pollers.go:57: node abe2e-2024-11-03-ug8s-azurelinuxv2gpu000000 is ready
    scenario_helpers_test.go:122: node abe2e-2024-11-03-ug8s-azurelinuxv2gpu is ready, proceeding with validation commands...
    validation.go:53: running live VM validator on abe2e-2024-11-03-ug8s-azurelinuxv2gpu: "assert /etc/default/kubelet should not contain dynamic config dir flag"
    validation.go:53: running live VM validator on abe2e-2024-11-03-ug8s-azurelinuxv2gpu: "assert sysctl settings"
    validation.go:53: running live VM validator on abe2e-2024-11-03-ug8s-azurelinuxv2gpu: "assert /var/log/azure/aks contents"
    validation.go:53: running live VM validator on abe2e-2024-11-03-ug8s-azurelinuxv2gpu: "check that curl to wireserver succeeds from host's network"
    validation.go:53: running live VM validator on abe2e-2024-11-03-ug8s-azurelinuxv2gpu: "check that curl to wireserver fails"
    validation.go:53: running live VM validator on abe2e-2024-11-03-ug8s-azurelinuxv2gpu: "check that curl to wireserver port 32526 fails"
    exec.go:48: dumping stderr:
        ----------------------------------- begin stderr -----------------------------------
          % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                         Dload  Upload   Total   Spent    Left  Speed
        
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
        ------------------------------------ end stderr ------------------------------------
    scenario_helpers_test.go:129: 
        	Error Trace:	/mnt/vss/_work/1/s/e2e/scenario_helpers_test.go:129
        	            				/mnt/vss/_work/1/s/e2e/scenario_helpers_test.go:60
        	            				/mnt/vss/_work/1/s/e2e/scenario_test.go:200
        	Error:      	Received unexpected error:
        	            	failed validator on node abe2e-2024-11-03-ug8s-azurelinuxv2gpu assertion: validator command terminated with exit code "0" but expected code 28 (CURL timeout)
        	Test:       	Test_azurelinuxv2gpu
    exec.go:74: executing command on remote VM at 10.224.0.40 of VMSS abe2e-2024-11-03-ug8s-azurelinuxv2gpu: "cat /var/log/azure/cluster-provision-cse-output.log"
    exec.go:74: executing command on remote VM at 10.224.0.40 of VMSS abe2e-2024-11-03-ug8s-azurelinuxv2gpu: "sysctl -a"
    exec.go:74: executing command on remote VM at 10.224.0.40 of VMSS abe2e-2024-11-03-ug8s-azurelinuxv2gpu: "cat /var/log/azure/aks-node-controller.log"
    exec.go:74: executing command on remote VM at 10.224.0.40 of VMSS abe2e-2024-11-03-ug8s-azurelinuxv2gpu: "cat /var/log/azure/cluster-provision.log"

Check failure on line 1 in Test_marinerv2gpu

See this annotation in the file changed.

@azure-pipelines azure-pipelines / Agentbaker E2E

Test_marinerv2gpu 

Failed
Raw output 
    scenario_helpers_test.go:96: running scenario "Test_marinerv2gpu" with vhd: "/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/CBLMarinerV2gen2/versions/1.1730621045.8255", tags {Name:Test_marinerv2gpu ImageName:CBLMarinerV2gen2 OS:mariner Arch:amd64 Airgap:false GPU:true WASM:false ServerTLSBootstrapping:false Scriptless:false KubeletCustomConfig:false}
    scenario_helpers_test.go:102: running scenario "Test_marinerv2gpu" with image "/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/CBLMarinerV2gen2/versions/1.1730621045.8255" in aks cluster "/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourcegroups/abe2e-westus3/providers/Microsoft.ContainerService/managedClusters/abe2e-kubenet-331fc"
    vmss.go:36: creating VMSS "abe2e-2024-11-03-f7qg-marinerv2gpu" in resource group "MC_abe2e-westus3_abe2e-kubenet-331fc_westus3"
    scenario_helpers_test.go:110: vmss abe2e-2024-11-03-f7qg-marinerv2gpu creation succeeded, proceeding with node readiness and pod checks...
    pollers.go:29: waiting for node abe2e-2024-11-03-f7qg-marinerv2gpu to be ready
    pollers.go:57: node abe2e-2024-11-03-f7qg-marinerv2gpu000000 is ready
    scenario_helpers_test.go:122: node abe2e-2024-11-03-f7qg-marinerv2gpu is ready, proceeding with validation commands...
    validation.go:53: running live VM validator on abe2e-2024-11-03-f7qg-marinerv2gpu: "assert /etc/default/kubelet should not contain dynamic config dir flag"
    validation.go:53: running live VM validator on abe2e-2024-11-03-f7qg-marinerv2gpu: "assert sysctl settings"
    validation.go:53: running live VM validator on abe2e-2024-11-03-f7qg-marinerv2gpu: "assert /var/log/azure/aks contents"
    validation.go:53: running live VM validator on abe2e-2024-11-03-f7qg-marinerv2gpu: "check that curl to wireserver succeeds from host's network"
    validation.go:53: running live VM validator on abe2e-2024-11-03-f7qg-marinerv2gpu: "check that curl to wireserver fails"
    validation.go:53: running live VM validator on abe2e-2024-11-03-f7qg-marinerv2gpu: "check that curl to wireserver port 32526 fails"
    exec.go:48: dumping stderr:
        ----------------------------------- begin stderr -----------------------------------
          % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                         Dload  Upload   Total   Spent    Left  Speed
        
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
        ------------------------------------ end stderr ------------------------------------
    scenario_helpers_test.go:129: 
        	Error Trace:	/mnt/vss/_work/1/s/e2e/scenario_helpers_test.go:129
        	            				/mnt/vss/_work/1/s/e2e/scenario_helpers_test.go:60
        	            				/mnt/vss/_work/1/s/e2e/scenario_test.go:449
        	Error:      	Received unexpected error:
        	            	failed validator on node abe2e-2024-11-03-f7qg-marinerv2gpu assertion: validator command terminated with exit code "0" but expected code 28 (CURL timeout)
        	Test:       	Test_marinerv2gpu
    exec.go:74: executing command on remote VM at 10.224.0.29 of VMSS abe2e-2024-11-03-f7qg-marinerv2gpu: "journalctl -u kubelet"
    exec.go:74: executing command on remote VM at 10.224.0.29 of VMSS abe2e-2024-11-03-f7qg-marinerv2gpu: "cat /var/log/azure/cluster-provision-cse-output.log"
    exec.go:74: executing command on remote VM at 10.224.0.29 of VMSS abe2e-2024-11-03-f7qg-marinerv2gpu: "sysctl -a"
    exec.go:74: executing command on remote VM at 10.224.0.29 of VMSS abe2e-2024-11-03-f7qg-marinerv2gpu: "cat /var/log/azure/aks-node-controller.log"
    exe

Check failure on line 1 in Test_ubuntu2204ArtifactStreaming

See this annotation in the file changed.

@azure-pipelines azure-pipelines / Agentbaker E2E

Test_ubuntu2204ArtifactStreaming 

Failed
Raw output 
    scenario_helpers_test.go:96: running scenario "Test_ubuntu2204ArtifactStreaming" with vhd: "/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/2204gen2containerd/versions/1.1730621068.17631", tags {Name:Test_ubuntu2204ArtifactStreaming ImageName:2204gen2containerd OS:ubuntu Arch:amd64 Airgap:false GPU:false WASM:false ServerTLSBootstrapping:false Scriptless:false KubeletCustomConfig:false}
    scenario_helpers_test.go:102: running scenario "Test_ubuntu2204ArtifactStreaming" with image "/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/2204gen2containerd/versions/1.1730621068.17631" in aks cluster "/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourcegroups/abe2e-westus3/providers/Microsoft.ContainerService/managedClusters/abe2e-kubenet-331fc"
    vmss.go:36: creating VMSS "abe2e-2024-11-03-cgcg-ubuntu2204artifactstreaming" in resource group "MC_abe2e-westus3_abe2e-kubenet-331fc_westus3"
    scenario_helpers_test.go:110: vmss abe2e-2024-11-03-cgcg-ubuntu2204artifactstreaming creation succeeded, proceeding with node readiness and pod checks...
    pollers.go:29: waiting for node abe2e-2024-11-03-cgcg-ubuntu2204artifactstreaming to be ready
    pollers.go:57: node abe2e-2024-11-03-cgcg-ubuntu2204artifactstreaming000000 is ready
    scenario_helpers_test.go:122: node abe2e-2024-11-03-cgcg-ubuntu2204artifactstreaming is ready, proceeding with validation commands...
    validation.go:53: running live VM validator on abe2e-2024-11-03-cgcg-ubuntu2204artifactstreaming: "assert /etc/default/kubelet should not contain dynamic config dir flag"
    validation.go:53: running live VM validator on abe2e-2024-11-03-cgcg-ubuntu2204artifactstreaming: "assert sysctl settings"
    validation.go:53: running live VM validator on abe2e-2024-11-03-cgcg-ubuntu2204artifactstreaming: "assert /var/log/azure/aks contents"
    validation.go:53: running live VM validator on abe2e-2024-11-03-cgcg-ubuntu2204artifactstreaming: "check that curl to wireserver succeeds from host's network"
    validation.go:53: running live VM validator on abe2e-2024-11-03-cgcg-ubuntu2204artifactstreaming: "check that curl to wireserver fails"
    validation.go:53: running live VM validator on abe2e-2024-11-03-cgcg-ubuntu2204artifactstreaming: "check that curl to wireserver port 32526 fails"
    validation.go:53: running live VM validator on abe2e-2024-11-03-cgcg-ubuntu2204artifactstreaming: "assert /var/log/azure/cluster-provision.log does not contain bootstrap token"
    validation.go:53: running live VM validator on abe2e-2024-11-03-cgcg-ubuntu2204artifactstreaming: "assert /var/log/azure/cluster-provision.log does not contain client private key"
    validation.go:53: running live VM validator on abe2e-2024-11-03-cgcg-ubuntu2204artifactstreaming: "assert /var/log/azure/cluster-provision.log does not contain service principal secret"
    validation.go:53: running live VM validator on abe2e-2024-11-03-cgcg-ubuntu2204artifactstreaming: "assert /var/log/azure/aks-node-controller.log does not contain bootstrap token"
    validation.go:53: running live VM validator on abe2e-2024-11-03-cgcg-ubuntu2204artifactstreaming: "assert /var/log/azure/aks-node-controller.log does not contain client private key"
    validation.go:53: running live VM validator on abe2e-2024-11-03-cgcg-ubuntu2204artifactstreaming: "assert /var/log/azure/aks-node-controller.log does not contain service principal secret"
    validation.go:53: running live VM validator on abe2e-2024-11-03-cgcg-ubuntu2204artifactstreaming: "assert /etc/default/kubelet has --node-ip flag set"
    validation.go:53: running live VM validator on abe2e-2024-11-03-cgcg-ubuntu2204artifactstreaming: "assert that there are files in /etc/overlaybd"
    scenario_helpers_test.go:129: 
        	Error Trace:	/mnt/vss/_work/1/s/e2e/scena

Check failure on line 1 in Test_ubuntu2204ScriptlessInstaller

See this annotation in the file changed.

@azure-pipelines azure-pipelines / Agentbaker E2E

Test_ubuntu2204ScriptlessInstaller 

Failed
Raw output 
    scenario_helpers_test.go:96: running scenario "Test_ubuntu2204ScriptlessInstaller" with vhd: "/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/2204gen2containerd/versions/1.1730621068.17631", tags {Name:Test_ubuntu2204ScriptlessInstaller ImageName:2204gen2containerd OS:ubuntu Arch:amd64 Airgap:false GPU:false WASM:false ServerTLSBootstrapping:false Scriptless:false KubeletCustomConfig:false}
    scenario_helpers_test.go:102: running scenario "Test_ubuntu2204ScriptlessInstaller" with image "/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/2204gen2containerd/versions/1.1730621068.17631" in aks cluster "/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourcegroups/abe2e-westus3/providers/Microsoft.ContainerService/managedClusters/abe2e-kubenet-331fc"
    vmss.go:36: creating VMSS "abe2e-2024-11-03-w0ji-ubuntu2204scriptlessinstaller" in resource group "MC_abe2e-westus3_abe2e-kubenet-331fc_westus3"
    scenario_helpers_test.go:110: vmss abe2e-2024-11-03-w0ji-ubuntu2204scriptlessinstaller creation succeeded, proceeding with node readiness and pod checks...
    pollers.go:29: waiting for node abe2e-2024-11-03-w0ji-ubuntu2204scriptlessinstaller to be ready
    pollers.go:54: node "abe2e-2024-11-03-w0ji-ubuntu2204scriptlessinstaller" isn't connected to the AKS cluster
    pollers.go:56: 
        	Error Trace:	/mnt/vss/_work/1/s/e2e/pollers.go:56
        	            				/mnt/vss/_work/1/s/e2e/validation.go:15
        	            				/mnt/vss/_work/1/s/e2e/scenario_helpers_test.go:111
        	            				/mnt/vss/_work/1/s/e2e/scenario_helpers_test.go:60
        	            				/mnt/vss/_work/1/s/e2e/scenario_test.go:574
        	Error:      	Received unexpected error:
        	            	context deadline exceeded
        	Test:       	Test_ubuntu2204ScriptlessInstaller
        	Messages:   	failed to find or wait for "abe2e-2024-11-03-w0ji-ubuntu2204scriptlessinstaller" to be ready {Capacity:map[] Allocatable:map[] Phase: Conditions:[] Addresses:[] DaemonEndpoints:{KubeletEndpoint:{Port:0}} NodeInfo:{MachineID: SystemUUID: BootID: KernelVersion: OSImage: ContainerRuntimeVersion: KubeletVersion: KubeProxyVersion: OperatingSystem: Architecture:} Images:[] VolumesInUse:[] VolumesAttached:[] Config:nil RuntimeHandlers:[] Features:nil}
    exec.go:74: executing command on remote VM at 10.224.0.5 of VMSS abe2e-2024-11-03-w0ji-ubuntu2204scriptlessinstaller: "cat /var/log/azure/cluster-provision-cse-output.log"
    exec.go:74: executing command on remote VM at 10.224.0.5 of VMSS abe2e-2024-11-03-w0ji-ubuntu2204scriptlessinstaller: "sysctl -a"
    exec.go:74: executing command on remote VM at 10.224.0.5 of VMSS abe2e-2024-11-03-w0ji-ubuntu2204scriptlessinstaller: "cat /var/log/azure/aks-node-controller.log"
    exec.go:74: executing command on remote VM at 10.224.0.5 of VMSS abe2e-2024-11-03-w0ji-ubuntu2204scriptlessinstaller: "cat /var/log/azure/cluster-provision.log"
    exec.go:74: executing command on remote VM at 10.224.0.5 of VMSS abe2e-2024-11-03-w0ji-ubuntu2204scriptlessinstaller: "journalctl -u kubelet"
    vmss.go:123: vmss "abe2e-2024-11-03-w0ji-ubuntu2204scriptlessinstaller" deleted successfully
View more details on Azure Pipelines