From 4c4f3dda44970ed1f83f525a2cfe604e4cab9153 Mon Sep 17 00:00:00 2001 From: rfisher001 Date: Wed, 20 Dec 2023 15:53:31 +0000 Subject: [PATCH] =?UTF-8?q?[CI]=20Publish=20Documentation=20for=20cd450ab8?= =?UTF-8?q?20d8ea4f4f26b137f590b603a78c0c57=20-=20cd450ab820d8ea4f4f26b137?= =?UTF-8?q?f590b603a78c0c57=20=F0=9F=9A=80?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .nojekyll | 0 4.12-4.14/API-Compatibility.html | 174 ++++ 4.12-4.14/Applying-MCPs.html | 182 ++++ 4.12-4.14/CNF-Upgrade-Prep.html | 193 ++++ 4.12-4.14/OCP-upgrade-prep.html | 213 ++++ 4.12-4.14/_images/5Rack-MCP.jpg | Bin 0 -> 43808 bytes 4.12-4.14/_images/LBorHT-MCP.jpg | Bin 0 -> 6776 bytes 4.12-4.14/_images/PDB-down-2.jpg | Bin 0 -> 4558 bytes 4.12-4.14/_images/PDB-full.jpg | Bin 0 -> 4252 bytes 4.12-4.14/_images/Worker-MCP.jpg | Bin 0 -> 9947 bytes 4.12-4.14/_images/k8s-vers-skew.png | Bin 0 -> 271165 bytes 4.12-4.14/index.html | 173 ++++ 4.12-4.14/introduction.html | 218 +++++ 4.12-4.14/lcm_upgrades.html | 906 ++++++++++++++++++ 404.html | 76 ++ _/css/site.css | 3 + _/font/JetBrainsMono-Bold-Italic.woff | Bin 0 -> 81268 bytes _/font/JetBrainsMono-Bold.woff | Bin 0 -> 75240 bytes _/font/JetBrainsMono-ExtraBold-Italic.woff | Bin 0 -> 79616 bytes _/font/JetBrainsMono-ExtraBold.woff | Bin 0 -> 73536 bytes _/font/JetBrainsMono-Italic.woff | Bin 0 -> 78444 bytes _/font/JetBrainsMono-Medium-Italic.woff | Bin 0 -> 80912 bytes _/font/JetBrainsMono-Medium.woff | Bin 0 -> 75084 bytes _/font/JetBrainsMono-Regular.woff | Bin 0 -> 72552 bytes _/font/RedHatDisplay-Black.woff | Bin 0 -> 36396 bytes _/font/RedHatDisplay-BlackItalic.woff | Bin 0 -> 37464 bytes _/font/RedHatDisplay-Bold.woff | Bin 0 -> 36920 bytes _/font/RedHatDisplay-BoldItalic.woff | Bin 0 -> 37944 bytes _/font/RedHatDisplay-Italic.woff | Bin 0 -> 37172 bytes _/font/RedHatDisplay-Medium.woff | Bin 0 -> 36532 bytes _/font/RedHatDisplay-MediumItalic.woff | Bin 0 -> 37568 bytes _/font/RedHatDisplay-Regular.woff | Bin 0 -> 36432 bytes _/font/RedHatText-Bold.woff | Bin 0 -> 36396 bytes _/font/RedHatText-BoldItalic.woff | Bin 0 -> 37384 bytes _/font/RedHatText-Italic.woff | Bin 0 -> 37348 bytes _/font/RedHatText-Medium.woff | Bin 0 -> 37096 bytes _/font/RedHatText-MediumItalic.woff | Bin 0 -> 37888 bytes _/font/RedHatText-Regular.woff | Bin 0 -> 35980 bytes _/img/Logo-Red_Hat-A-Reverse-RGB.png | Bin 0 -> 7920 bytes _/img/Logo-Red_Hat-A-White-RGB.png | Bin 0 -> 6984 bytes _/img/NewRHDFullLogo_4-color.png | Bin 0 -> 14642 bytes .../NewRHDFullLogo_4-color_white-wordmark.png | Bin 0 -> 14794 bytes _/img/back.svg | 1 + _/img/caret.svg | 1 + _/img/chevron.svg | 1 + _/img/clippy.svg | 1 + _/img/close.svg | 1 + _/img/home-o.svg | 1 + _/img/home.svg | 1 + _/img/menu.svg | 1 + _/js/site.js | 6 + _/js/vendor/clipboard.js | 1 + _/js/vendor/highlight.js | 1 + sitemap.xml | 31 + 54 files changed, 2185 insertions(+) create mode 100644 .nojekyll create mode 100644 4.12-4.14/API-Compatibility.html create mode 100644 4.12-4.14/Applying-MCPs.html create mode 100644 4.12-4.14/CNF-Upgrade-Prep.html create mode 100644 4.12-4.14/OCP-upgrade-prep.html create mode 100644 4.12-4.14/_images/5Rack-MCP.jpg create mode 100644 4.12-4.14/_images/LBorHT-MCP.jpg create mode 100644 4.12-4.14/_images/PDB-down-2.jpg create mode 100644 4.12-4.14/_images/PDB-full.jpg create mode 100644 4.12-4.14/_images/Worker-MCP.jpg create mode 100644 4.12-4.14/_images/k8s-vers-skew.png create mode 100644 4.12-4.14/index.html create mode 100644 4.12-4.14/introduction.html create mode 100644 4.12-4.14/lcm_upgrades.html create mode 100644 404.html create mode 100644 _/css/site.css create mode 100644 _/font/JetBrainsMono-Bold-Italic.woff create mode 100644 _/font/JetBrainsMono-Bold.woff create mode 100644 _/font/JetBrainsMono-ExtraBold-Italic.woff create mode 100644 _/font/JetBrainsMono-ExtraBold.woff create mode 100644 _/font/JetBrainsMono-Italic.woff create mode 100644 _/font/JetBrainsMono-Medium-Italic.woff create mode 100644 _/font/JetBrainsMono-Medium.woff create mode 100644 _/font/JetBrainsMono-Regular.woff create mode 100644 _/font/RedHatDisplay-Black.woff create mode 100644 _/font/RedHatDisplay-BlackItalic.woff create mode 100644 _/font/RedHatDisplay-Bold.woff create mode 100644 _/font/RedHatDisplay-BoldItalic.woff create mode 100644 _/font/RedHatDisplay-Italic.woff create mode 100644 _/font/RedHatDisplay-Medium.woff create mode 100644 _/font/RedHatDisplay-MediumItalic.woff create mode 100644 _/font/RedHatDisplay-Regular.woff create mode 100644 _/font/RedHatText-Bold.woff create mode 100644 _/font/RedHatText-BoldItalic.woff create mode 100644 _/font/RedHatText-Italic.woff create mode 100644 _/font/RedHatText-Medium.woff create mode 100644 _/font/RedHatText-MediumItalic.woff create mode 100644 _/font/RedHatText-Regular.woff create mode 100644 _/img/Logo-Red_Hat-A-Reverse-RGB.png create mode 100644 _/img/Logo-Red_Hat-A-White-RGB.png create mode 100644 _/img/NewRHDFullLogo_4-color.png create mode 100644 _/img/NewRHDFullLogo_4-color_white-wordmark.png create mode 100644 _/img/back.svg create mode 100644 _/img/caret.svg create mode 100644 _/img/chevron.svg create mode 100644 _/img/clippy.svg create mode 100644 _/img/close.svg create mode 100644 _/img/home-o.svg create mode 100644 _/img/home.svg create mode 100644 _/img/menu.svg create mode 100644 _/js/site.js create mode 100644 _/js/vendor/clipboard.js create mode 100644 _/js/vendor/highlight.js create mode 100644 sitemap.xml diff --git a/.nojekyll b/.nojekyll new file mode 100644 index 0000000..e69de29 diff --git a/4.12-4.14/API-Compatibility.html b/4.12-4.14/API-Compatibility.html new file mode 100644 index 0000000..5b94084 --- /dev/null +++ b/4.12-4.14/API-Compatibility.html @@ -0,0 +1,174 @@ + + + + + + OCP API Compatibility Policy :: LAB - Telco OCP Upgrade Lab + + + + + + +
+ +
+
+
+ +
+
+ +
+
+

OCP API Compatibility Policy

+
+

CNF API Compatibility

+
+
+

OpenShift and Kubernetes are strong because they use APIs for so many functions. This also means that the APIs will change over time as components are updated. Therefore, it is important to verify that an API call is still compatible with the cluster so that you will receive the desired response. Please review the documented section called “Understanding API Tiers”.

+
+
+

The most important thing to understand when considering which Z-release to upgrade to inside of a new Y-release is what patches need to be in the new Z-release. In other words if you are currently at OCP 4.11.28 you will need to make sure to upgrade to a Z-release of 4.12 that has all of the patches in it that were applied to 4.11.28, otherwise you will break the built-in compatibility of Kubernetes.

+
+
+
+
+

Kubernetes Version Skew

+
+
+

Support of specific API versions needs to be maintained by each cluster operator. With new releases of operators come new APIs. Therefore, the changes or skews in APIs need to be maintained. To a certain extent the APIs can be compatible across several releases of an operator. This list of operators and the releases that are compatible are at: https://kubernetes.io/releases/version-skew-policy

+
+
+

The easiest way verify your application functionality will still work, is to make sure that you follow

+
+
+
+
+

OpenShift Upgrade Path

+
+
+

Please also note that not all releases of OCP can be upgraded to any arbitrary Z-release even if they contain all of the required patches. +OpenShift upgrade process mandates that: +If fix “A” is present in a specific X.Y.Z release of OCP +Then fix “A” MUST be present in the X.Y+1.Z release that OCP is upgraded TO

+
+
+

Consequence of the chosen destination version of 4.12.z defines which is the maximum version of OCP4.11.z, OCP4.10.z and OCP4.9.z +not all 4.9.z version will permit to upgrade to a given version of OCP4.12.z +A given version of OCP4.12.z will have requirements to a maximum version of OCP4.9z +This is due to how fixes are backported into older releases of OCP.

+
+
+

You can use the upgrade graph tool to determine if the path is valid for your z-release. You should also always verify with your Sales Engineer or Technical Account Manager at Red Hat to make sure the upgrade path is valid for Telco implementations.

+
+
+
+k8s vers skew +
+
Figure 1. K8s Version Skey
+
+
+
+ +
+ +
+
+
+
+
+
+
+ +
+
+ +
+
+

Applying MCPs

+
+

First you can run “oc get mcp” to show your current list of MCPs:

+
+
+
+
# oc get mcp
+
+
+
+

List out all of your nodes:

+
+
+
+
# oc get no
+
+
+
+

Determine, from the above suggestions, how you would like to separate out your worker nodes into machine config pools +(MCP).
+In this example we will just use 1 node in each MCP.
+We first need to label the nodes so that they can be put into MCPs. We will do this with the following commands:

+
+
+
+
oc label node euschannel-worker-0.test.corp node-role.kubernetes.io/*mcp-1*=
+
+
+
+

This will show up when you run the “oc get node” command:

+
+
+
+
# oc get no
+
+
+
+

Now you need to create yaml files that will apply the labels as MCPs. Here is one example:

+
+
+
+
apiVersion: machineconfiguration.openshift.io/v1
+
+
+
+

For each of these, just run “oc apply -f {filename.yaml}”:

+
+
+
+
# oc apply -f test-mcp-2.yaml
+
+
+
+

Now you can run “oc get mcp” again and your new MCPs will show. Please note that you will still see the original worker +and master MCPs that are part of the cluster.

+
+
+
+
# oc get mcp
+
+
+
+ +
+
+
+
+
+
+
+ +
+
+ +
+
+

CNF Upgrade Preparation

+
+
+
+

The life of a POD is an important topic to understand. This section will describe several topics that are important to +keeping your CNF PODs healthy and allow the cluster to properly schedule them during an upgrade.

+
+
+
+
+

CNF Requirements Document

+
+
+

Before you go any further, please read through the CNF requirements document. +In this section a few of the most important points will be discussed but the CNF Requirements Document has additional +detail and other important topics.

+
+
+
+
+

POD Disruption Budget

+
+
+

Each set of PODs in a deployment can be given a specific minimum number of PODs that should be running in order to keep +from disrupting the functionality of the CNF, thus called the POD disruption budget (PDB). However, this budget can be +improperly configured.
+For example, if you have 4 PODs in a deployment and your PDB is set to 4, this means that you are telling the scheduler +that you NEED 4 PODs running at all times. Therefore, in this scenario ZERO PODs can come down.

+
+
+
+PDB full +
+
Figure 1. Deployment with no PDB
+
+
+

To fix this, the PDB can be set to 2, letting 2 of the 4 pods to be scheduled as down and this would then let the worker +nodes where those PODs are located be rebooted.

+
+
+
+PDB down 2 +
+
Figure 2. Deployment with PDB
+
+
+
+
+

POD Anti-affinity

+
+
+

True high availability requires a duplication of a process to be running on separate hardware, thus making sure that an +application will continue to run if one piece of hardware goes down. OpenShift can easily make that happen since +processes are automatically duplicated in separate PODs within a deployment. However, those PODs need to have +anti-affinity set on them so that they are NOT running on the same hardware. It so happens that anti-affinity also +helps during upgrades because it makes sure that PODs are on different worker nodes, therefore allowing enough PODs to +come down even after considering their PDB.

+
+
+
+
+

Liveness / Readiness Probes

+
+
+

OpenShift and Kubernetes have some built in features that not everyone takes advantage of called +liveness and readiness probes. +These are very important when POD deployments are dependent upon keeping state for their application. This document +won’t go into detail regarding these probes but please review the OpenShift documentation +on how to implement their use.

+
+
+
+
+ +
+
+
+
+
+
+
+ +
+
+ +
+
+

OCP Upgrade Preparation

+
+

Firmware compatibility

+
+
+

All hardware vendors will advise that it is always best to be on their latest certified version of firmware for their +hardware. In the telco world this comes with a trust but verify approach due to the high throughput nature of telco +CNFs. Therefore, it is important to have a regimented group who can test the current and latest firmware from any vendor +to make sure that all components will work with both. It is not always recommended to upgrade firmware in conjunction +with an OCP upgrade however if it is possible to test the latest release of firmware that will improve the odds that +you won’t run into issues down the road.
+Upgrading firmware is a very important debate because the process can be very intrusive and has a potential for causing +a node to require manual interventions before the node will come back online. On the other hand it may be imperative to +upgrade the firmware due to security fixes, new required functionality or compatibility with the new release of OCP +components. Therefore, it is up to everyone to verify with their hardware vendors, verify compatibility with OCP +components and perform tests in their lab before moving forward.

+
+
+
+
+

Layer product compatibility

+
+
+

It is important to make sure all layered products will run on the new version of OCP that you will be moving to. This, +very much, includes all Operators.

+
+
+

Verify the current installed list of Operators installed on your cluster. For example:

+
+
+
+
# oc get csv -A
+NAMESPACE                              NAME                                 DISPLAY          VERSION   REPLACES                             PHASE
+chapter2                               gitlab-operator-kubernetes.v0.17.2   GitLab           0.17.2    gitlab-operator-kubernetes.v0.17.1   Succeeded
+openshift-operator-lifecycle-manager   packageserver                        Package Server   0.19.0                                         Succeeded
+
+
+
+
+
+

Prepare MCPs

+
+
+

Prepare your Machine Config Pool (MCP) labels by grouping your nodes, depending on the number of nodes in your cluster. +MCPs should be split up into 8 to 10 nodes per group. However, there is no hard fast rule as to how many nodes need to +be in each MCP. The purpose of these MCPs is to group nodes together so that a group of nodes can be controlled +independently of the rest. Additional information and examples can be found here, under the canary rollout documentation.
+These MCPs will be used to un-pause a set of nodes during the upgrade process, thus allowing them to be upgraded and +rebooted at a determined time instead of at the pleasure of the scheduler. Please review the upgrade process flow +section, below, for more details on the pause/un-pause process.

+
+
+
+5Rack MCP +
+
Figure 1. Worker node MCPs in a 5 rack cluster
+
+
+

The division and size of these MCPs can vary depending on many factors. In general the standard division is between 8 +and 10 nodes per MCP to allow the operations team to control how many nodes are taken down at a time.

+
+
+
+LBorHT MCP +
+
Figure 2. Separate MCPs inside of a group of Load Balancer or purpose built nodes
+
+
+

In larger clusters there is quite often a need to separate out several nodes for purposes like Load Balancing or other +high throughput purposes, which usually have different machine sets to configure SR-IOV. In these cases we do not want +to upgrade all of these nodes without getting a chance to test during the upgrade. Therefore, we need to separate them +out into at least 3 different MCPs and unpause them individually.

+
+
+
+Worker MCP +
+
Figure 3. Small cluster worker MCPs
+
+
+

Smaller cluster example with 1 rack

+
+
+

The process and pace at which you un-pause the MCPs is determined by your CNFs and their configuration. Please review +the sections on PDB and anti-affinity for CNFs. If your CNF can properly handle scheduling within an OpenShift cluster +you can un-pause several MCPs at a time and set the MaxUnavailable to as high as 50%. This will allow as many as half +of the nodes in your MCPs to restart and upgrade. This will reduce the amount of time that is needed for a specific +maintenance window and allow your cluster to upgrade quickly. Hopefully you can see how keeping your PDB and +anti-affinity correctly configured will help in the long run.

+
+
+
+
+ +
+
+
+
+