From 6c824146059fdcd6640ad25f5a9cdf53cc717b36 Mon Sep 17 00:00:00 2001 From: Antonin Bas Date: Thu, 18 Apr 2024 16:29:13 -0700 Subject: [PATCH 1/4] [Windows] Finish removing Docker and userspace kube-proxy support We update the Windows documentation to remove all Docker-specific instructions, and all mentions of (userspace) kube-proxy. We also update Windows scripts to remove Docker support. The Prepare-AntreaAgent.ps1 script was still defaulting to installing kube-proxy, so we update the script to change this default behavior. For all the scripts, "omitting" kube-proxy is now the default behavior. The corresponding script parameters have not been removed yet, but they are not officially deprecated, and we can remove them in a future release, such as Antrea v2.3. Fixes #5630 Signed-off-by: Antonin Bas --- docs/windows.md | 279 ++------------------------- hack/windows/Prepare-AntreaAgent.ps1 | 9 +- hack/windows/Prepare-Node.ps1 | 45 ++--- hack/windows/Start-AntreaAgent.ps1 | 1 + 4 files changed, 43 insertions(+), 291 deletions(-) diff --git a/docs/windows.md b/docs/windows.md index 1aff4ef160b..772d8eb0fd6 100644 --- a/docs/windows.md +++ b/docs/windows.md @@ -19,14 +19,7 @@ - [5. Run kubeadm to join the Node](#5-run-kubeadm-to-join-the-node) - [Verify your installation](#verify-your-installation) - [Installation as a Service](#installation-as-a-service) - - [Installation as a Pod using wins for Docker (DEPRECATED)](#installation-as-a-pod-using-wins-for-docker-deprecated) - - [Add Windows antrea-agent DaemonSet](#add-windows-antrea-agent-daemonset-1) - - [Join Windows worker Nodes](#join-windows-worker-nodes-1) - - [Add Windows kube-proxy DaemonSet (only for Kubernetes versions prior to 1.26)](#add-windows-kube-proxy-daemonset-only-for-kubernetes-versions-prior-to-126) - - [Common steps](#common-steps) - - [For containerd](#for-containerd) - - [For Docker](#for-docker) - - [Manually run kube-proxy and antrea-agent on Windows worker Nodes](#manually-run-kube-proxy-and-antrea-agent-on-windows-worker-nodes) + - [Manually run antrea-agent on Windows worker Nodes](#manually-run-antrea-agent-on-windows-worker-nodes) - [Known issues](#known-issues) @@ -49,15 +42,8 @@ The following components should be configured and run on the Windows Node. * [kubernetes components](https://kubernetes.io/docs/setup/production-environment/windows/user-guide-windows-nodes/) * OVS daemons * antrea-agent -* kube-proxy -antrea-agent and kube-proxy run as processes on host and are managed by -management Pods. It is recommended to run OVS daemons as Windows services. -We also support running OVS processes inside a container. If you don't want to -run antrea-agent and kube-proxy from the management Pods Antrea also provides -scripts which help to install and run these two components directly without Pod. -Please see [Manually run kube-proxy and antrea-agent on Windows worker Nodes](#manually-run-kube-proxy-and-antrea-agent-on-windows-worker-nodes) -section for details. +antrea-agent and the OVS daemons can either run as Pods or as Windows services. ### Antrea Windows demo @@ -76,10 +62,6 @@ prefer running the Antrea Agent as a Windows service, or if you are using containerd 1.5, you can use the [Installation as a Service](#installation-as-a-service) method. -Note that [Docker support](#installation-as-a-pod-using-wins-for-docker-deprecated) -is deprecated. We no longer test Antrea support with Docker on Windows, and the -installation method will be removed from the documentation in a later release. - ### Prerequisites * Create a Kubernetes cluster. @@ -100,12 +82,9 @@ higher (containerd 1.7 or higher is recommended). It relies on support for [Windows HostProcess Pods](https://kubernetes.io/docs/tasks/configure-pod-container/create-hostprocess-pod/), which is generally available starting with K8s 1.26. -Starting with Antrea v1.13, Antrea will take over all the responsibilities of -kube-proxy for Windows nodes by default. Since Kubernetes 1.26, kube-proxy -should not be deployed on Windows Nodes with Antrea, as kube-proxy userspace -mode is deprecated. For Kubernetes versions prior to 1.26, Antrea can work -with userspace kube-proxy on Windows Nodes. -For more information refer to section [Add Windows kube-proxy DaemonSet (only for Kubernetes versions prior to 1.26)](#add-windows-kube-proxy-daemonset-only-for-kubernetes-versions-prior-to-126) +Starting with Antrea v1.13, Antrea takes over all the responsibilities of +kube-proxy for Windows nodes by default, and kube-proxy should not be deployed +on Windows Nodes with Antrea. #### Download & Configure Antrea for Linux @@ -246,12 +225,11 @@ Run the following commands to prepare the Node environment needed by antrea-agen ```powershell mkdir c:\k\antrea cd c:\k\antrea -$TAG="v1.14.0" +$TAG="v1.15.1" curl.exe -LO https://raw.githubusercontent.com/antrea-io/antrea/${TAG}/hack/windows/Clean-AntreaNetwork.ps1 -curl.exe -LO https://raw.githubusercontent.com/antrea-io/antrea/${TAG}/hack/windows/Prepare-ServiceInterface.ps1 curl.exe -LO https://raw.githubusercontent.com/antrea-io/antrea/${TAG}/hack/windows/Prepare-AntreaAgent.ps1 # use -RunOVSServices $false for containerized OVS! -.\Prepare-AntreaAgent.ps1 -InstallKubeProxy $false [-RunOVSServices $false] +.\Prepare-AntreaAgent.ps1 [-RunOVSServices $false] ``` The script `Prepare-AntreaAgent.ps1` performs the following tasks: @@ -268,7 +246,7 @@ The script `Prepare-AntreaAgent.ps1` performs the following tasks: need to specify the parameter `RunOVSServices` as false. ```powershell - .\Prepare-AntreaAgent.ps1 -InstallKubeProxy $false -RunOVSServices $false + .\Prepare-AntreaAgent.ps1 -RunOVSServices $false ``` The script must be executed every time you restart the Node to prepare the @@ -283,7 +261,7 @@ Insert following line in kubelet service script `c:\k\StartKubelet.ps1` to invok `Prepare-AntreaAgent.ps1` when starting kubelet service: ```powershell -& C:\k\antrea\Prepare-AntreaAgent.ps1 -InstallKubeProxy $false -RunOVSServices $false +& C:\k\antrea\Prepare-AntreaAgent.ps1 -RunOVSServices $false ``` * Example 2: Create a ScheduledJob that runs at startup. @@ -291,7 +269,7 @@ Insert following line in kubelet service script `c:\k\StartKubelet.ps1` to invok ```powershell $trigger = New-JobTrigger -AtStartup -RandomDelay 00:00:30 $options = New-ScheduledJobOption -RunElevated -Register-ScheduledJob -Name PrepareAntreaAgent -Trigger $trigger -ScriptBlock { Invoke-Expression C:\k\antrea\Prepare-AntreaAgent.ps1 -InstallKubeProxy $false -RunOVSServices $false } -ScheduledJobOption $options +Register-ScheduledJob -Name PrepareAntreaAgent -Trigger $trigger -ScriptBlock { Invoke-Expression C:\k\antrea\Prepare-AntreaAgent.ps1 -RunOVSServices $false } -ScheduledJobOption $options ``` ##### 5. Run kubeadm to join the Node @@ -321,10 +299,9 @@ control-plane Ready control-plane 1h v1.29.0 10.176.2 win-5akrf2tpq91 Ready 1h v1.29.0 10.176.27.150 Windows Server 2019 Datacenter 10.0.17763.5206 containerd://1.6.6 win-5akrf2tpq92 Ready 1h v1.29.0 10.176.27.197 Windows Server 2019 Datacenter 10.0.17763.5206 containerd://1.6.6 -# Show antrea-agent and kube-proxy Pods +# Show antrea-agent Pods kubectl get pods -o wide -n kube-system | grep windows antrea-agent-windows-6hvkw 1/1 Running 0 100s -kube-proxy-windows-2d45w 1/1 Running 0 102s ``` ### Installation as a Service @@ -347,269 +324,55 @@ containerized), and you will need to run `Install-OVS.ps1` to install them. When your Nodes are ready, run the following scripts to install the antrea-agent service. NOTE: ``, `` and -`` should be set by you. `` is -an optional parameter that is specific to kube-proxy mode. For example: +`` should be set by you. For example: ```powershell -# kube-proxy mode is no longer supported starting with K8s version 1.26 -$InstallKubeProxy=$false -$KubernetesVersion="v1.23.5" +$KubernetesVersion="v1.29.0" $KubeConfig="C:/Users/Administrator/.kube/config" # admin kubeconfig $KubeletKubeconfigPath="C:/etc/kubernetes/kubelet.conf" -if ($InstallKubeProxy) { $KubeProxyKubeconfigPath="C:/Users/Administrator/kubeproxy.conf" } ``` ```powershell -$TAG="v1.14.0" +$TAG="v1.15.0" $KubernetesVersion="" $KubeConfig="" $KubeletKubeconfigPath="" -if ($InstallKubeProxy) { $KubeProxyKubeconfigPath="" } $KubernetesHome="c:/k" $AntreaHome="c:/k/antrea" -$KubeProxyLogPath="c:/var/log/kube-proxy" curl.exe -LO "https://raw.githubusercontent.com/antrea-io/antrea/${TAG}/hack/windows/Helper.psm1" Import-Module ./Helper.psm1 Install-AntreaAgent -KubernetesVersion "$KubernetesVersion" -KubernetesHome "$KubernetesHome" -KubeConfig "$KubeConfig" -AntreaVersion "$TAG" -AntreaHome "$AntreaHome" -New-KubeProxyServiceInterface New-DirectoryIfNotExist "${AntreaHome}/logs" -New-DirectoryIfNotExist "${KubeProxyLogPath}" -# Install kube-proxy service -if ($InstallKubeProxy) { nssm install kube-proxy "${KubernetesHome}/kube-proxy.exe" "--proxy-mode=userspace --kubeconfig=${KubeProxyKubeconfigPath} --log-dir=${KubeProxyLogPath} --logtostderr=false --alsologtostderr" } nssm install antrea-agent "${AntreaHome}/bin/antrea-agent.exe" "--config=${AntreaHome}/etc/antrea-agent.conf --logtostderr=false --log_dir=${AntreaHome}/logs --alsologtostderr --log_file_max_size=100 --log_file_max_num=4" nssm set antrea-agent DependOnService ovs-vswitchd -if ($InstallKubeProxy) { nssm set antrea-agent DependOnService kube-proxy ovs-vswitchd } nssm set antrea-agent Start SERVICE_DELAYED_AUTO_START -if ($InstallKubeProxy) { Start-Service kube-proxy } Start-Service antrea-agent ``` -### Installation as a Pod using wins for Docker (DEPRECATED) - -*Dockershim was deprecated in K8s 1.20, and removed in K8s version 1.24. These - steps may work with [cri-dockerd](https://github.com/Mirantis/cri-dockerd) but - this is not something we validated. Antrea is no longer tested with Docker on - Windows, and we intend to remove these steps from the documentation in Antrea - version 2.0.* - -Running Antrea with Docker on Windows uses -[wins](https://github.com/rancher/wins), which lets you run services on the -Window hosts, while managing them as if they were Pods. - -#### Add Windows antrea-agent DaemonSet - -For example, these commands will download the antrea-agent manifest, set -`kubeAPIServerOverride`, and deploy the antrea-agent DaemonSet when using the -Docker container runtime: - -```bash -KUBE_APISERVER=$(kubectl config view -o jsonpath='{.clusters[0].cluster.server}') && \ -curl -sL https://github.com/antrea-io/antrea/releases/download//antrea-windows.yml | \ -sed "s|.*kubeAPIServerOverride: \"\"| kubeAPIServerOverride: \"${KUBE_APISERVER}\"|g" | \ -kubectl apply -f - -``` - -#### Join Windows worker Nodes - -The steps to join Windows worker Nodes are similar to the ones for the -containerd runtime, with the following differences: - -1. OVS containerization is not supported, so OVS userspace processes need to be - run as Windows native services. -2. When running the `Prepare-Node.ps1` script, you will need to explicitly - specify that you are using the Docker container runtime. The script will then - take care of installing wins. For example: - - ```powershell - .\Prepare-Node.ps1 -KubernetesVersion v1.23.5 -NodeIP 192.168.1.10 -ContainerRuntime docker - ``` - -If you want to install and use userspace kube-proxy on the Node (no longer -supported since K8s version 1.26), follow instructions in [Add Windows -kube-proxy DaemonSet (only for Kubernetes versions prior to 1.26)](#add-windows-kube-proxy-daemonset-only-for-kubernetes-versions-prior-to-126). - -### Add Windows kube-proxy DaemonSet (only for Kubernetes versions prior to 1.26) - -Starting from Kubernetes 1.26, Antrea no longer supports Windows kube-proxy -because the kube-proxy userspace mode has been removed, and the kernel -implementation does not work with Antrea. Clusters using recent K8s versions -will need to follow the normal [installation guide](#deploying-antrea-on-windows-worker-nodes) -and use AntreaProxy with `proxyAll` enabled. - -For older K8s versions, you can use kube-proxy userspace mode by following the -instructions below. - -#### Common steps - -When running `Prepare-Node.ps1`, make sure that you set `InstallKubeProxy` to -true. For example: - -```powershell -.\Prepare-Node.ps1 -KubernetesVersion v1.25.0 -InstallKubeProxy:$true -NodeIP 192.168.1.10 -``` - -When running `Prepare-AntreaAgent.ps1`, make sure that you set -`InstallKubeProxy` to true. For example: - -```powershell -.\Prepare-AntreaAgent.ps1 -InstallKubeProxy $true` -``` - -This will take care of preparing the network adapter for kube-proxy. kube-proxy -needs a network adapter to configure Kubernetes Services IPs and uses the -adapter for proxying connections to Services. The adapter will be deleted -automatically by Windows after the Windows Node reboots -(`Prepare-AntreaAgent.ps1` needs to run at every startup). - -After that, you will need to deploy a Windows-compatible version of -kube-proxy. You can download `kube-proxy.yml` from the Kubernetes github -repository to deploy kube-proxy. The kube-proxy version in the YAML file must be -set to a Windows compatible version. The following command downloads -`kube-proxy.yml`: - -```bash -curl -L "https://github.com/kubernetes-sigs/sig-windows-tools/releases/download/v0.1.5/kube-proxy.yml" | sed 's/VERSION-nanoserver/v1.20.0/g' > kube-proxy.yml -``` - -Before applying the downloaded manifest, you will need to make some changes -(which depend on your container runtime). - -#### For containerd - -Replace the content of `run-script.ps1` in the `kube-proxy-windows` ConfigMap -with the following: - -```yaml -apiVersion: v1 -data: - run-script.ps1: |- - $mountPath = $env:CONTAINER_SANDBOX_MOUNT_POINT - $mountPath = ($mountPath.Replace('\', '/')).TrimEnd('/') - New-Item -Path "c:/var/lib" -Name "kube-proxy" -ItemType "directory" -Force - ((Get-Content -path $mountPath/var/lib/kube-proxy/kubeconfig.conf -Raw) -replace '/var',"$($mountPath)/var") | Set-Content -Path /var/lib/kube-proxy/kubeconfig.conf - ((Get-Content -path /var/lib/kube-proxy/kubeconfig.conf -Raw) -replace '\/',"/") | Set-Content -Path /var/lib/kube-proxy/kubeconfig.conf - sed -i 's/mode: iptables/mode: \"\"/g' $mountPath/var/lib/kube-proxy/config.conf - & "$mountPath/k/kube-proxy/kube-proxy.exe" --config=$mountPath/var/lib/kube-proxy/config.conf --v=10 --proxy-mode=userspace --hostname-override=$env:NODE_NAME -kind: ConfigMap -metadata: - labels: - app: kube-proxy - name: kube-proxy-windows - namespace: kube-system -``` - -Set the `hostNetwork` option to `true` and add the following to the -kube-proxy-windows DaemonSet spec: - -```yaml -apiVersion: apps/v1 -kind: DaemonSet -metadata: - labels: - k8s-app: kube-proxy - name: kube-proxy-windows - namespace: kube-system -spec: - selector: - matchLabels: - k8s-app: kube-proxy-windows - template: - metadata: - labels: - k8s-app: kube-proxy-windows - spec: - securityContext: - windowsOptions: - hostProcess: true - runAsUserName: "NT AUTHORITY\\SYSTEM" - hostNetwork: true - serviceAccountName: kube-proxy - containers: - - command: - - pwsh - args: - - -file - - $env:CONTAINER_SANDBOX_MOUNT_POINT/var/lib/kube-proxy-windows/run-script.ps1 -``` - -#### For Docker - -Replace the content of `run-script.ps1` in the `kube-proxy-windows` ConfigMap -with the following: - -```yaml -apiVersion: v1 -data: - run-script.ps1: |- - $ErrorActionPreference = "Stop"; - mkdir -force /host/var/lib/kube-proxy/var/run/secrets/kubernetes.io/serviceaccount - mkdir -force /host/k/kube-proxy - - cp -force /k/kube-proxy/* /host/k/kube-proxy - cp -force /var/lib/kube-proxy/* /host/var/lib/kube-proxy - cp -force /var/run/secrets/kubernetes.io/serviceaccount/* /host/var/lib/kube-proxy/var/run/secrets/kubernetes.io/serviceaccount - - wins cli process run --path /k/kube-proxy/kube-proxy.exe --args "--v=3 --config=/var/lib/kube-proxy/config.conf --proxy-mode=userspace --hostname-override=$env:NODE_NAME" - -kind: ConfigMap -metadata: - labels: - app: kube-proxy - name: kube-proxy-windows - namespace: kube-system -``` - -Set the `hostNetwork` option to `true` in the spec of kube-proxy-windows -DaemonSet spec: - -```yaml -apiVersion: apps/v1 -kind: DaemonSet -metadata: - labels: - k8s-app: kube-proxy - name: kube-proxy-windows - namespace: kube-system -spec: - selector: - matchLabels: - k8s-app: kube-proxy-windows - template: - metadata: - labels: - k8s-app: kube-proxy-windows - spec: - hostNetwork: true -``` - -### Manually run kube-proxy and antrea-agent on Windows worker Nodes +### Manually run antrea-agent on Windows worker Nodes -Aside from starting kube-proxy and antrea-agent from the management Pods, Antrea -also provides powershell scripts which help install and run these two components -directly without Pods. Please complete the steps in +Antrea also provides powershell scripts which help install and run the Antrea +Agent manually, please complete the steps in [Installation](#installation-as-a-pod) section, and skip the [Add Windows antrea-agent DaemonSet](#add-windows-antrea-agent-daemonset) step. -Then run the following commands in powershell. +Then run the following commands in powershell: ```powershell mkdir c:\k\antrea cd c:\k\antrea curl.exe -LO https://github.com/antrea-io/antrea/releases/download//Start-AntreaAgent.ps1 -# Run antrea-agent without kube-proxy +# Run antrea-agent # $KubeConfigPath is the path of kubeconfig file -./Start-AntreaAgent.ps1 -kubeconfig $KubeConfigPath -StartKubeProxy $false -# Run Antrea-Agent with kube-proxy (deprecated since Kubernetes 1.26) -# ./Start-AntreaAgent.ps1 -kubeconfig $KubeConfigPath -StartKubeProxy $true +./Start-AntreaAgent.ps1 -kubeconfig $KubeConfigPath ``` > Note: Some features such as supportbundle collection are not supported in this -> way. It's recommended to start kube-proxy and antrea-agent through management -> Pods. +> way. It's recommended to run antrea-agent as a Pod. ## Known issues diff --git a/hack/windows/Prepare-AntreaAgent.ps1 b/hack/windows/Prepare-AntreaAgent.ps1 index 78eca1ff5fc..8e7d4b27ec4 100644 --- a/hack/windows/Prepare-AntreaAgent.ps1 +++ b/hack/windows/Prepare-AntreaAgent.ps1 @@ -9,11 +9,13 @@ This script prepares environment needed by antrea-agent which includes: provide the proxy for Kubernetes Services. .PARAMETER InstallKubeProxy -Specifies whether kube-proxy interface is included in the installation. If false, this interface will not -be installed on the host. +[DEPRECATED] Specifies whether kube-proxy interface is included in the installation. + +.PARAMETER RunOVSServices +Specifies whether the OVS userspace daemons should be started as Windows services. #> Param( - [parameter(Mandatory = $false)] [bool] $InstallKubeProxy = $true, + [parameter(Mandatory = $false)] [bool] $InstallKubeProxy = $false, [parameter(Mandatory = $false)] [bool] $RunOVSServices= $true ) @@ -48,6 +50,7 @@ if ($RunOVSServices -eq $true) { } # Prepare service network interface for kube-proxy. if ($InstallKubeProxy -eq $true) { + Write-Host "Running Antrea with kube-proxy is no longer supported, this parameter will be removed soon" Write-Host "Preparing service network interface for kube-proxy..." & $PrepareServiceInterfaceScript } diff --git a/hack/windows/Prepare-Node.ps1 b/hack/windows/Prepare-Node.ps1 index e3cf93104c0..db2d55ef9ac 100644 --- a/hack/windows/Prepare-Node.ps1 +++ b/hack/windows/Prepare-Node.ps1 @@ -17,7 +17,7 @@ script from the sig-windows-tools repo release page: https://github.com/kubernet Kubernetes version to download and use .PARAMETER InstallKubeProxy -Install kube-proxy +[DEPRECATED] Install kube-proxy .PARAMETER InstallOVS Install OVS @@ -26,11 +26,12 @@ Install OVS The node ip used by kubelet .PARAMETER ContainerRuntime -Container runtime that Kubernetes will use. (docker or containerd) +[DEPRECATED] Container runtime that Kubernetes will use (docker or containerd). +Starting with Antrea v2, only containerd is supported. .PARAMETER InstallOVSUserspace -Specifies whether OVS userspace processes are included in the installation. If false, these processes will not -be installed as a Windows service on the host. +Specifies whether OVS userspace processes are included in the installation. If false, these +processes will not be installed as a Windows service on the host. .EXAMPLE PS> .\Prepare-Node.ps1 -KubernetesVersion v1.27.0 -NodeIP 192.168.1.10 -ContainerRuntime containerd @@ -43,8 +44,8 @@ Param( [parameter(Mandatory = $false)] [switch] $InstallKubeProxy = $false, [parameter(Mandatory = $false)] [switch] $InstallOVS = $false, [parameter(Mandatory = $false, HelpMessage="Kubernetes download")] [string] $KubernetesURL="dl.k8s.io", - [parameter(HelpMessage="Container runtime that Kubernets will use")] [ValidateSet("containerd", "docker")] [string] $ContainerRuntime = "containerd", - [parameter(Mandatory = $false)] [bool] $InstallOVSUserspace = $true + [parameter(Mandatory = $false)] [ValidateSet("containerd", "docker")] [string] $ContainerRuntime = "containerd", + [parameter(Mandatory = $false)] [bool] $InstallOVSUserspace = $true ) $ErrorActionPreference = 'Stop' @@ -64,15 +65,13 @@ If (Get-Service kubelet -ErrorAction SilentlyContinue) { } if ($ContainerRuntime -eq "docker") { - if (-not(Test-Path "//./pipe/docker_engine")) { - Write-Error "Docker service was not detected - please install and start Docker before calling Prepare-Node.ps1 with -ContainerRuntime docker" - exit 1 - } -} elseif ($ContainerRuntime -eq "containerd") { - if (-not(Test-Path "//./pipe/containerd-containerd")) { - Write-Error "Containerd service was not detected - please install and start Containerd before calling Prepare-Node.ps1 with -ContainerRuntime containerd" - exit 1 - } + Write-Error "Docker container runtime is no longer supported" + exit 1 +} + +if (-not(Test-Path "//./pipe/containerd-containerd")) { + Write-Error "Containerd service was not detected - please install and start Containerd before calling Prepare-Node.ps1" + exit 1 } if (!$KubernetesVersion.StartsWith("v")) { @@ -94,13 +93,6 @@ $env:Path += ";$global:KubernetesPath" DownloadFile $kubeletBinPath "https:/$KubernetesURL/$KubernetesVersion/bin/windows/amd64/kubelet.exe" DownloadFile "$global:KubernetesPath\kubeadm.exe" "https:/$KubernetesURL/$KubernetesVersion/bin/windows/amd64/kubeadm.exe" -if ($ContainerRuntime -eq "docker") { - Write-Host "Registering wins service" - DownloadFile "$global:KubernetesPath\wins.exe" https://github.com/rancher/wins/releases/download/v0.0.4/wins.exe - wins.exe srv app run --register - start-service rancher-wins -} - mkdir -force C:\var\log\kubelet mkdir -force C:\var\lib\kubelet\etc\kubernetes @@ -111,15 +103,8 @@ New-Item -path C:\var\lib\kubelet\etc\kubernetes\pki -type SymbolicLink -value C $StartKubeletFileContent = '$FileContent = Get-Content -Path "/var/lib/kubelet/kubeadm-flags.env" $global:KubeletArgs = $FileContent.Trim("KUBELET_KUBEADM_ARGS=`"")'+ [Environment]::NewLine -if ($ContainerRuntime -eq "docker") { - $StartKubeletFileContent +=[Environment]::NewLine +'$netId = docker network ls -f name=host --format "{{ .ID }}" - -if ($netId.Length -lt 1) { - docker network create -d nat host -}' + [Environment]::NewLine -} - if ($InstallKubeProxy) { + Write-Host "Running Antrea with kube-proxy is no longer supported, this parameter will be removed soon" $StartKubeletFileContent += [Environment]::NewLine + '& C:\k\Prepare-ServiceInterface.ps1 -InterfaceAlias "HNS Internal NIC"' + [Environment]::NewLine } diff --git a/hack/windows/Start-AntreaAgent.ps1 b/hack/windows/Start-AntreaAgent.ps1 index fa70d1d4f1c..877c0302c0b 100644 --- a/hack/windows/Start-AntreaAgent.ps1 +++ b/hack/windows/Start-AntreaAgent.ps1 @@ -76,6 +76,7 @@ if ($LastExitCode) { } if ($StartKubeProxy) { + Write-Host "Running Antrea with kube-proxy is no longer supported, this parameter will be removed soon" Write-Host "Starting kube-proxy..." if (!(Start-KubeProxy -KubeProxy $KubernetesHome\kube-proxy.exe -KubeConfig $KubeConfig)) { Write-Host "Failed to start kube-proxy, exit" From 411cda44c76072e938f2f9b8b71b835053c78dfa Mon Sep 17 00:00:00 2001 From: Antonin Bas Date: Tue, 23 Apr 2024 11:00:26 -0700 Subject: [PATCH 2/4] Rename Antrea Windows YAML manifests Now that containerd is the only supported container runtime, we rename antrea-windows-containerd.yml to antrea-windows.yml and antrea-windows-containerd-with-ovs.yml to antrea-windows-with-ovs.yml. Signed-off-by: Antonin Bas --- Makefile | 4 ++-- ...rd-with-ovs.yml => antrea-windows-with-ovs.yml} | 0 ...a-windows-containerd.yml => antrea-windows.yml} | 0 ci/jenkins/test.sh | 6 +++--- docs/design/windows-design.md | 10 +++++----- docs/windows.md | 14 +++++++------- hack/release/prepare-assets.sh | 4 ++-- hack/update-checksum-windows.sh | 4 ++-- 8 files changed, 21 insertions(+), 21 deletions(-) rename build/yamls/{antrea-windows-containerd-with-ovs.yml => antrea-windows-with-ovs.yml} (100%) rename build/yamls/{antrea-windows-containerd.yml => antrea-windows.yml} (100%) diff --git a/Makefile b/Makefile index cc337b1f7c6..ea691137060 100644 --- a/Makefile +++ b/Makefile @@ -417,8 +417,8 @@ build-migrator: manifest: @echo "===> Generating dev manifest for Antrea <===" $(CURDIR)/hack/generate-standard-manifests.sh --mode dev --out build/yamls - $(CURDIR)/hack/generate-manifest-windows.sh --mode dev > build/yamls/antrea-windows-containerd.yml - $(CURDIR)/hack/generate-manifest-windows.sh --mode dev --include-ovs > build/yamls/antrea-windows-containerd-with-ovs.yml + $(CURDIR)/hack/generate-manifest-windows.sh --mode dev > build/yamls/antrea-windows.yml + $(CURDIR)/hack/generate-manifest-windows.sh --mode dev --include-ovs > build/yamls/antrea-windows-with-ovs.yml $(CURDIR)/hack/update-checksum-windows.sh $(CURDIR)/hack/generate-manifest-flow-aggregator.sh --mode dev > build/yamls/flow-aggregator.yml diff --git a/build/yamls/antrea-windows-containerd-with-ovs.yml b/build/yamls/antrea-windows-with-ovs.yml similarity index 100% rename from build/yamls/antrea-windows-containerd-with-ovs.yml rename to build/yamls/antrea-windows-with-ovs.yml diff --git a/build/yamls/antrea-windows-containerd.yml b/build/yamls/antrea-windows.yml similarity index 100% rename from build/yamls/antrea-windows-containerd.yml rename to build/yamls/antrea-windows.yml diff --git a/ci/jenkins/test.sh b/ci/jenkins/test.sh index d00942f1fc7..48bd73174d2 100755 --- a/ci/jenkins/test.sh +++ b/ci/jenkins/test.sh @@ -375,7 +375,7 @@ function revert_snapshot_windows { function build_and_deliver_antrea_windows_and_linux_containerd_images { echo "====== Cleanup Antrea Installation Before Delivering Antrea Windows and Antrea Linux containerd Images =====" clean_antrea - kubectl delete -f ${WORKDIR}/antrea-windows-containerd-with-ovs.yml --ignore-not-found=true || true + kubectl delete -f ${WORKDIR}/antrea-windows-with-ovs.yml --ignore-not-found=true || true kubectl delete -f ${WORKDIR}/kube-proxy-windows-containerd.yml --ignore-not-found=true || true kubectl delete daemonset antrea-agent -n kube-system --ignore-not-found=true || true kubectl delete -f ${WORKDIR}/antrea.yml --ignore-not-found=true || true @@ -388,11 +388,11 @@ function build_and_deliver_antrea_windows_and_linux_containerd_images { ${PRINT_DOCKER_STATUS} export_govc_env_var # Enable verbose log for troubleshooting. - sed -i "s/--v=0/--v=4/g" build/yamls/antrea.yml build/yamls/antrea-windows-containerd-with-ovs.yml + sed -i "s/--v=0/--v=4/g" build/yamls/antrea.yml build/yamls/antrea-windows-with-ovs.yml echo "====== Updating yaml files to enable proxyAll ======" KUBE_API_SERVER=$(kubectl --kubeconfig=$KubeConfigFile config view -o jsonpath='{.clusters[0].cluster.server}') - sed -i "s|.*kubeAPIServerOverride: \"\"| kubeAPIServerOverride: \"${KUBE_API_SERVER}\"|g" build/yamls/antrea.yml build/yamls/antrea-windows-containerd-with-ovs.yml + sed -i "s|.*kubeAPIServerOverride: \"\"| kubeAPIServerOverride: \"${KUBE_API_SERVER}\"|g" build/yamls/antrea.yml build/yamls/antrea-windows-with-ovs.yml cp -f build/yamls/*.yml $WORKDIR set +e diff --git a/docs/design/windows-design.md b/docs/design/windows-design.md index a42b2156716..0d80658c8c4 100644 --- a/docs/design/windows-design.md +++ b/docs/design/windows-design.md @@ -190,11 +190,11 @@ these scenarios: ## Antrea and OVS Management on Windows -While we provide different installation methods for Windows, the recommended one starting with -Antrea v1.13 is to use the `antrea-windows-containerd-with-ovs.yml` manifest. With this method, the -antrea-agent process and the OVS daemons (ovsdb-server and ovs-vswitchd) run as a Pod on Windows -worker Nodes, and are managed by a DaemonSet. This installation method relies on -[Windows HostProcess Pod](https://kubernetes.io/docs/tasks/configure-pod-container/create-hostprocess-pod/) +While we provide different installation methods for Windows, the recommended one +is to use the `antrea-windows-with-ovs.yml` manifest. With this method, the +antrea-agent process and the OVS daemons (ovsdb-server and ovs-vswitchd) run as +a Pod on Windows worker Nodes, and are managed by a DaemonSet. This installation +method relies on [Windows HostProcess Pod](https://kubernetes.io/docs/tasks/configure-pod-container/create-hostprocess-pod/) support. ## Traffic walkthrough diff --git a/docs/windows.md b/docs/windows.md index 772d8eb0fd6..b1cc3c27bbc 100644 --- a/docs/windows.md +++ b/docs/windows.md @@ -119,29 +119,29 @@ For earlier versions of Antrea, you will need to enable `proxyAll` manually. Starting with Antrea 1.13, you can run both the Antrea Agent and the OVS daemons on Windows Nodes using a single DaemonSet, by applying the file -`antrea-windows-containerd-with-ovs.yml`. This is the recommended installation +`antrea-windows-with-ovs.yml`. This is the recommended installation method. The following commands download the manifest, set `kubeAPIServerOverride`, and create the DaemonSet: ```bash KUBE_APISERVER=$(kubectl config view -o jsonpath='{.clusters[0].cluster.server}') && \ -curl -sL https://github.com/antrea-io/antrea/releases/download//antrea-windows-containerd-with-ovs.yml | \ +curl -sL https://github.com/antrea-io/antrea/releases/download//antrea-windows-with-ovs.yml | \ sed "s|.*kubeAPIServerOverride: \"\"| kubeAPIServerOverride: \"${KUBE_APISERVER}\"|g" | \ kubectl apply -f - ``` Alternatively, to deploy the antrea-agent Windows DaemonSet without the OVS -daemons, apply the file `antrea-windows-containerd.yml` with the following +daemons, apply the file `antrea-windows.yml` with the following commands: ```bash KUBE_APISERVER=$(kubectl config view -o jsonpath='{.clusters[0].cluster.server}') && \ -curl -sL https://github.com/antrea-io/antrea/releases/download//antrea-windows-containerd.yml | \ +curl -sL https://github.com/antrea-io/antrea/releases/download//antrea-windows.yml | \ sed "s|.*kubeAPIServerOverride: \"\"| kubeAPIServerOverride: \"${KUBE_APISERVER}\"|g" | \ kubectl apply -f - ``` -When using `antrea-windows-containerd.yml`, you will need to install OVS +When using `antrea-windows.yml`, you will need to install OVS userspace daemons as services when you prepare your Windows worker Nodes, in the next section. @@ -161,7 +161,7 @@ you will need to invoke the `Install-OVS.ps1` script differently (or not at all) | No | Yes | `.\Install-OVS.ps1` | | No | No | `.\Install-OVS.ps1 -ImportCertificate $false -Local -LocalFile ` | -If you used `antrea-windows-containerd-with-ovs.yml` to create the antrea-agent +If you used `antrea-windows-with-ovs.yml` to create the antrea-agent Windows DaemonSet, then you are using "Containerized OVS daemons". For all other methods, you are *not* using "Containerized OVS daemons". @@ -178,7 +178,7 @@ Restart-Computer ``` As an example, if you are using containerized OVS -(`antrea-windows-containerd-with-ovs.yml`), and you want to use the test-signed +(`antrea-windows-with-ovs.yml`), and you want to use the test-signed OVS kernel driver provided by Antrea (not recommended for production), you would run the following commands: diff --git a/hack/release/prepare-assets.sh b/hack/release/prepare-assets.sh index d1d389a80c8..e2081104a57 100755 --- a/hack/release/prepare-assets.sh +++ b/hack/release/prepare-assets.sh @@ -110,8 +110,8 @@ export CONTROLLER_IMG_NAME=antrea/antrea-controller-ubuntu ./hack/generate-standard-manifests.sh --mode release --out "$OUTPUT_DIR" export IMG_NAME=antrea/antrea-windows -./hack/generate-manifest-windows.sh --mode release > "$OUTPUT_DIR"/antrea-windows-containerd.yml -./hack/generate-manifest-windows.sh --mode release --include-ovs > "$OUTPUT_DIR"/antrea-windows-containerd-with-ovs.yml +./hack/generate-manifest-windows.sh --mode release > "$OUTPUT_DIR"/antrea-windows.yml +./hack/generate-manifest-windows.sh --mode release --include-ovs > "$OUTPUT_DIR"/antrea-windows-with-ovs.yml export IMG_NAME=antrea/flow-aggregator ./hack/generate-manifest-flow-aggregator.sh --mode release > "$OUTPUT_DIR"/flow-aggregator.yml diff --git a/hack/update-checksum-windows.sh b/hack/update-checksum-windows.sh index f418b4d2c20..66f52b2d4d4 100755 --- a/hack/update-checksum-windows.sh +++ b/hack/update-checksum-windows.sh @@ -36,5 +36,5 @@ for file in ${MANIFESTS[@]}; do sed -i.bak "s/windows-config-checksum-placeholder/${checksum_windows_config}/g" ${file} done -sed -i.bak "s/agent-windows-checksum-placeholder/${checksum_containerd}/g" ${YAMLS_DIR}/antrea-windows-containerd.yml -sed -i.bak "s/agent-windows-checksum-placeholder/${checksum_containerd_with_ovs}/g" ${YAMLS_DIR}/antrea-windows-containerd-with-ovs.yml +sed -i.bak "s/agent-windows-checksum-placeholder/${checksum_containerd}/g" ${YAMLS_DIR}/antrea-windows.yml +sed -i.bak "s/agent-windows-checksum-placeholder/${checksum_containerd_with_ovs}/g" ${YAMLS_DIR}/antrea-windows-with-ovs.yml From fe6e1ac53422b7cb90acbe368dee5fb7c3a3e380 Mon Sep 17 00:00:00 2001 From: Antonin Bas Date: Tue, 23 Apr 2024 11:11:05 -0700 Subject: [PATCH 3/4] Address review comments To preserve documentation acuracy, we remove some references to older Antrea versions. Signed-off-by: Antonin Bas --- docs/windows.md | 50 ++++++++++++++++++++++------------- hack/windows/Prepare-Node.ps1 | 4 +-- 2 files changed, 33 insertions(+), 21 deletions(-) diff --git a/docs/windows.md b/docs/windows.md index b1cc3c27bbc..a6e0b14b8a4 100644 --- a/docs/windows.md +++ b/docs/windows.md @@ -35,6 +35,15 @@ Node to an existing Kubernetes cluster. For the detailed design of how antrea-agent works on Windows, please refer to the [design doc](design/windows-design.md). +**Note: Docker support on Windows Nodes was dropped completely in Antrea v2.0, + making containerd the only supported container runtime. As part of this + change, we renamed the `antrea-windows-containerd.yml` manifest to + `antrea-windows.yml`, and the `antrea-windows-containerd-with-ovs.yml` + manifest to `antrea-windows-with-ovs.yml`. Prior to the Antrea v2.0 release, + the `antrea-windows.yml` manifest was used to support Windows Nodes with + Docker. For the best experience, make sure that you refer to the version of + the documentation that matches the Antrea version you are deploying.** + ### Components that run on Windows The following components should be configured and run on the Windows Node. @@ -43,7 +52,15 @@ The following components should be configured and run on the Windows Node. * OVS daemons * antrea-agent -antrea-agent and the OVS daemons can either run as Pods or as Windows services. +antrea-agent and the OVS daemons can either run as Pods (containerized) or as +Windows services, and the following configurations are supported: + +| OVS daemons | antrea-agent | Supported | Refer to | +| ---------------- | ---------------- | ----------------- | -------- | +| Containerized | Containerized | Yes (recommended) | [Installation as a Pod](#installation-as-a-pod) | +| Containerized | Windows Service | No | N/A | +| Windows Services | Containerized | Yes | [Installation as a Pod](#installation-as-a-pod) | +| Windows Services | Windows Services | Yes | [Installation as a Service](#installation-as-a-service) | ### Antrea Windows demo @@ -77,13 +94,13 @@ Service](#installation-as-a-service) method. ### Installation as a Pod -This installation method requires Antrea 1.10 or higher, and containerd 1.6 or +This installation method requires Antrea v1.10 or higher, and containerd 1.6 or higher (containerd 1.7 or higher is recommended). It relies on support for [Windows HostProcess Pods](https://kubernetes.io/docs/tasks/configure-pod-container/create-hostprocess-pod/), which is generally available starting with K8s 1.26. Starting with Antrea v1.13, Antrea takes over all the responsibilities of -kube-proxy for Windows nodes by default, and kube-proxy should not be deployed +kube-proxy for Windows Nodes by default, and kube-proxy should not be deployed on Windows Nodes with Antrea. #### Download & Configure Antrea for Linux @@ -97,9 +114,8 @@ kubectl apply -f https://github.com/antrea-io/antrea/releases/download//ant #### Add Windows antrea-agent DaemonSet -Starting from Antrea 1.13, you need to manually set the `kubeAPIServerOverride` -field in the YAML configuration file as the Antrea Proxy `proxyAll` mode is -enabled by default. +You need to manually set the `kubeAPIServerOverride` field in the YAML +configuration file as the Antrea Proxy `proxyAll` mode is enabled by default. ```yaml # Provide the address of Kubernetes apiserver, to override any value provided in kubeconfig or InClusterConfig. @@ -115,13 +131,10 @@ enabled by default. proxyAll: true ``` -For earlier versions of Antrea, you will need to enable `proxyAll` manually. - -Starting with Antrea 1.13, you can run both the Antrea Agent and the OVS daemons -on Windows Nodes using a single DaemonSet, by applying the file -`antrea-windows-with-ovs.yml`. This is the recommended installation -method. The following commands download the manifest, set -`kubeAPIServerOverride`, and create the DaemonSet: +You can run both the Antrea Agent and the OVS daemons on Windows Nodes using a +single DaemonSet, by applying the file `antrea-windows-with-ovs.yml`. This is +the recommended installation method. The following commands download the +manifest, set `kubeAPIServerOverride`, and create the DaemonSet: ```bash KUBE_APISERVER=$(kubectl config view -o jsonpath='{.clusters[0].cluster.server}') && \ @@ -131,8 +144,7 @@ kubectl apply -f - ``` Alternatively, to deploy the antrea-agent Windows DaemonSet without the OVS -daemons, apply the file `antrea-windows.yml` with the following -commands: +daemons, apply the file `antrea-windows.yml` with the following commands: ```bash KUBE_APISERVER=$(kubectl config view -o jsonpath='{.clusters[0].cluster.server}') && \ @@ -225,7 +237,7 @@ Run the following commands to prepare the Node environment needed by antrea-agen ```powershell mkdir c:\k\antrea cd c:\k\antrea -$TAG="v1.15.1" +$TAG="v2.0.0" curl.exe -LO https://raw.githubusercontent.com/antrea-io/antrea/${TAG}/hack/windows/Clean-AntreaNetwork.ps1 curl.exe -LO https://raw.githubusercontent.com/antrea-io/antrea/${TAG}/hack/windows/Prepare-AntreaAgent.ps1 # use -RunOVSServices $false for containerized OVS! @@ -306,8 +318,8 @@ antrea-agent-windows-6hvkw 1/1 Running 0 ### Installation as a Service -Install Antrea (v0.13.0+ is required for containerd) as usual. The following -command deploys Antrea with the version specified by ``: +Install Antrea as usual. The following command deploys Antrea with the version +specified by ``: ```bash kubectl apply -f https://github.com/antrea-io/antrea/releases/download//antrea.yml @@ -333,7 +345,7 @@ $KubeletKubeconfigPath="C:/etc/kubernetes/kubelet.conf" ``` ```powershell -$TAG="v1.15.0" +$TAG="v2.0.0" $KubernetesVersion="" $KubeConfig="" $KubeletKubeconfigPath="" diff --git a/hack/windows/Prepare-Node.ps1 b/hack/windows/Prepare-Node.ps1 index db2d55ef9ac..3efff3f681c 100644 --- a/hack/windows/Prepare-Node.ps1 +++ b/hack/windows/Prepare-Node.ps1 @@ -27,7 +27,7 @@ The node ip used by kubelet .PARAMETER ContainerRuntime [DEPRECATED] Container runtime that Kubernetes will use (docker or containerd). -Starting with Antrea v2, only containerd is supported. +Starting with Antrea v2.0, only containerd is supported. .PARAMETER InstallOVSUserspace Specifies whether OVS userspace processes are included in the installation. If false, these @@ -70,7 +70,7 @@ if ($ContainerRuntime -eq "docker") { } if (-not(Test-Path "//./pipe/containerd-containerd")) { - Write-Error "Containerd service was not detected - please install and start Containerd before calling Prepare-Node.ps1" + Write-Error "Containerd service was not detected - please install and start containerd before calling Prepare-Node.ps1" exit 1 } From 9b7cc640c2e581cdc30759b60b933c5cf7fe1c8d Mon Sep 17 00:00:00 2001 From: Antonin Bas Date: Tue, 23 Apr 2024 19:45:27 -0700 Subject: [PATCH 4/4] Update WINDOWS_YAML_SUFFIX in ci/jenkins/test.sh Signed-off-by: Antonin Bas --- ci/jenkins/test.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/jenkins/test.sh b/ci/jenkins/test.sh index 48bd73174d2..4c562da6511 100755 --- a/ci/jenkins/test.sh +++ b/ci/jenkins/test.sh @@ -1000,7 +1000,7 @@ fi trap clean_antrea EXIT if [[ ${TESTCASE} =~ "windows" ]]; then if [[ ${TESTCASE} =~ "containerd" ]]; then - WINDOWS_YAML_SUFFIX="windows-containerd-with-ovs" + WINDOWS_YAML_SUFFIX="windows-with-ovs" build_and_deliver_antrea_windows_and_linux_containerd_images if [[ ${TESTCASE} =~ "e2e" ]]; then run_e2e_windows