+ User workloads should not be colocated with control plane workloads. To ensure that the scheduler won't + schedule workloads on the master nodes, the taint "node-role.kubernetes.io/master" with the "NoSchedule" + effect is set by default in most cluster configurations (excluding SNO and Compact Clusters). +
++ The scheduling of the master nodes is centrally configurable without reboot via +
oc edit schedulers.config.openshift.io clusterfor details see the Red Hat Solution + {{{ weblink(link="https://access.redhat.com/solutions/4564851") }}} + +
+ If you run a setup, which requires the colocation of control plane and user workload you need to + exclude this rule. +
+ +rationale: -| + By separating user workloads and the control plane workloads we can better ensure that there is + no ill effects from workload boosts to each other. Furthermore we ensure that an adversary who gets + control over a badly secured workload container is not colocated to critical components of the control plane. + In some setups it might be necessary to make the control plane schedulable for workloads i.e. + Single Node Openshift (SNO) or Compact Cluster (Three Node Cluster) setups. + +{{% set jqfilter = '.items[] | select(.metadata.labels."node-role.kubernetes.io/master" == "" or .metadata.labels."node-role.kubernetes.io/control-plane" == "" ) | .spec.taints[] | select(.key == "node-role.kubernetes.io/master" and .effect == "NoSchedule")' %}} + +identifiers: + cce@ocp4: CCE-88731-5 + +severity: medium + +ocil_clause: 'Control Plane is schedulable' + +ocil: |- + Run the following command to see if control planes are schedulable +$oc get --raw /api/v1/nodes | jq '.items[] | select(.metadata.labels."node-role.kubernetes.io/master" == "" or .metadata.labels."node-role.kubernetes.io/control-plane" == "" ) | .spec.taints[] | select(.key == "node-role.kubernetes.io/master" and .effect == "NoSchedule" )'+ for each master node, there should be an output of a key with the NoSchedule effect. + + By editing the cluster scheduler you can centrally configure the masters as schedulable or not + by setting .spec.mastersSchedulable to true. + Use
$oc edit schedulers.config.openshift.io clusterto configure the scheduling. + +warnings: + - general: |- + {{{ openshift_filtered_cluster_setting({'/api/v1/nodes': jqfilter}) | indent(8) }}} + +template: + name: yamlfile_value + vars: + ocp_data: "true" + filepath: |- + {{{ openshift_filtered_path('/api/v1/nodes', jqfilter) }}} + yamlpath: ".effect" + check_existence: "at_least_one_exists" + entity_check: "at least one" + values: + - value: "NoSchedule" + operation: "pattern match" diff --git a/applications/openshift/master/master_taint_noschedule/tests/ocp4/e2e.yml b/applications/openshift/master/master_taint_noschedule/tests/ocp4/e2e.yml new file mode 100644 index 00000000000..b49fd368b98 --- /dev/null +++ b/applications/openshift/master/master_taint_noschedule/tests/ocp4/e2e.yml @@ -0,0 +1,2 @@ +--- +default_result: PASS diff --git a/controls/bsi_app_4_4.yml b/controls/bsi_app_4_4.yml index 087fb1892ed..f88dd594e09 100644 --- a/controls/bsi_app_4_4.yml +++ b/controls/bsi_app_4_4.yml @@ -377,18 +377,37 @@ controls: levels: - elevated description: >- - In a Kubernetes cluster, nodes SHOULD be assigned dedicated tasks and only run pods that are + (1) In a Kubernetes cluster, nodes SHOULD be assigned dedicated tasks and only run pods that are assigned to each task. - Bastion nodes SHOULD handle all incoming and outgoing data connections of between + (2) Bastion nodes SHOULD handle all incoming and outgoing data connections of between applications and other networks. - Management nodes SHOULD operate control plane pods and only handle control plane data + (3) Management nodes SHOULD operate control plane pods and only handle control plane data connections. - If deployed, storage nodes SHOULD only operate the fixed persistent volume services pods in + (4) If deployed, storage nodes SHOULD only operate the fixed persistent volume services pods in a cluster. notes: >- - TBD - status: pending - rules: [] + Section 1: + This requirement must be solved organizationally. OpenShift can bind applications to specific + nodes or node groups (via labels and node selectors). ACM can take over the labeling of nodes + and ensure that the nodes are labeled accordingly. + Section 2: + OpenShift uses the concept of infra-nodes. The incoming connections can be bound to these and, + by using Egress-IP, the incoming connections can also be bound. + Section 3: + OpenShift uses control plane nodes for management, on which no applications are running. + Data connections between applications to the outside world and to one another are not routed + via the control plane as standard. The necessary requirements must be taken into account as + part of the planning. + Section 4: + OpenShift Data Foundation (ODF) can be linked to its own infra nodes using the OpenShift + mechanisms, which only run storage services. This can be implemented equivalently with other + storage solutions. + status: partial + rules: + # Section 1,2,3,4 + - general_node_separation + # Section 3 + - master_taint_noschedule - id: APP.4.4.A15 title: Separation of Applications at Node and Cluster Level From 9bb13cc1b82bb8307ccf8dc472b1fa4210fb4c85 Mon Sep 17 00:00:00 2001 From: sluetze <13255307+sluetze@users.noreply.github.com> Date: Tue, 16 Jul 2024 09:01:31 +0200 Subject: [PATCH 03/10] add missing identifier and fix indentation --- .../general/general_node_separation/rule.yml | 18 +++++++++++++----- shared/references/cce-redhat-avail.txt | 1 - 2 files changed, 13 insertions(+), 6 deletions(-) diff --git a/applications/openshift/general/general_node_separation/rule.yml b/applications/openshift/general/general_node_separation/rule.yml index 1e2e49bd723..bb74983a1df 100644 --- a/applications/openshift/general/general_node_separation/rule.yml +++ b/applications/openshift/general/general_node_separation/rule.yml @@ -3,17 +3,25 @@ documentation_complete: true title: 'Create Boundaries between Resources using Nodes or Clusters' description: |- - Use Nodes or Clusters to isolate Workloads with high protection requirements. + Use Nodes or Clusters to isolate Workloads with high protection requirements. - Run the following command and review the pods and how they are deployed on Nodes.
$ oc get pod -o=custom-columns=NAME:.metadata.name,NAMESPACE:.metadata.namespace,APP:.metadata.labels.app\.kubernetes\.io/name,NODE:.spec.nodeName --all-namespaces | grep -v "openshift-"- You can use labels or other data as custom field which helps you to identify parts of an application. - Ensure that Applications with high protection requirements are not colocated on Nodes or in Clusters with workloads of lower protection requirements. + Run the following command and review the pods and how they are deployed on Nodes. +
$ oc get pod -o=custom-columns=NAME:.metadata.name,NAMESPACE:.metadata.namespace,APP:.metadata.labels.app\.kubernetes\.io/name,NODE:.spec.nodeName --all-namespaces | grep -v "openshift-"+ You can use labels or other data as custom field which helps you to identify parts of an application. + Ensure that Applications with high protection requirements are not colocated on Nodes or in Clusters + with workloads of lower protection requirements. rationale: |- - Assigning workloads with high protection requirements to specific nodes creates and additional boundary (the node) between workloads of high protection requirements and workloads which might follow less strict requirements. An adversary which attacked a lighter protected workload now has additional obstacles for their movement towards the higher protected workloads. + Assigning workloads with high protection requirements to specific nodes creates and additional + boundary (the node) between workloads of high protection requirements and workloads which might + follow less strict requirements. An adversary which attacked a lighter protected workload now has + additional obstacles for their movement towards the higher protected workloads. severity: medium +identifiers: + cce@ocp4: CCE-90279-1 + ocil_clause: 'Application placement on Nodes and Clusters needs review' ocil: |- diff --git a/shared/references/cce-redhat-avail.txt b/shared/references/cce-redhat-avail.txt index 33debb230d4..597f1c4b495 100644 --- a/shared/references/cce-redhat-avail.txt +++ b/shared/references/cce-redhat-avail.txt @@ -1347,7 +1347,6 @@ CCE-88725-7 CCE-88727-3 CCE-88728-1 CCE-88729-9 -CCE-88731-5 CCE-88734-9 CCE-88735-6 CCE-88736-4 From 10ff1410441176165d3a5fd2ed27129b1ffaf001 Mon Sep 17 00:00:00 2001 From: sluetze <13255307+sluetze@users.noreply.github.com> Date: Tue, 16 Jul 2024 12:45:59 +0200 Subject: [PATCH 04/10] fix duplicate reference --- applications/openshift/general/general_node_separation/rule.yml | 2 +- shared/references/cce-redhat-avail.txt | 1 - 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/applications/openshift/general/general_node_separation/rule.yml b/applications/openshift/general/general_node_separation/rule.yml index bb74983a1df..1e012d86eb7 100644 --- a/applications/openshift/general/general_node_separation/rule.yml +++ b/applications/openshift/general/general_node_separation/rule.yml @@ -20,7 +20,7 @@ rationale: |- severity: medium identifiers: - cce@ocp4: CCE-90279-1 + cce@ocp4: CCE-88903-0 ocil_clause: 'Application placement on Nodes and Clusters needs review' diff --git a/shared/references/cce-redhat-avail.txt b/shared/references/cce-redhat-avail.txt index 597f1c4b495..d3f18735b4b 100644 --- a/shared/references/cce-redhat-avail.txt +++ b/shared/references/cce-redhat-avail.txt @@ -1442,7 +1442,6 @@ CCE-88898-2 CCE-88899-0 CCE-88900-6 CCE-88902-2 -CCE-88903-0 CCE-88904-8 CCE-88905-5 CCE-88906-3 From 8657b6ffe8cc0a0d44be20870696cebb39a346b5 Mon Sep 17 00:00:00 2001 From: sluetze <13255307+sluetze@users.noreply.github.com> Date: Fri, 4 Oct 2024 14:28:59 +0000 Subject: [PATCH 05/10] add section numbering --- controls/bsi_app_4_4.yml | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/controls/bsi_app_4_4.yml b/controls/bsi_app_4_4.yml index f88dd594e09..b4d51c20d4b 100644 --- a/controls/bsi_app_4_4.yml +++ b/controls/bsi_app_4_4.yml @@ -392,7 +392,7 @@ controls: and ensure that the nodes are labeled accordingly. Section 2: OpenShift uses the concept of infra-nodes. The incoming connections can be bound to these and, - by using Egress-IP, the incoming connections can also be bound. + by using Egress-IP, the outgoing connections can also be bound. Section 3: OpenShift uses control plane nodes for management, on which no applications are running. Data connections between applications to the outside world and to one another are not routed @@ -414,7 +414,7 @@ controls: levels: - elevated description: >- - Applications with very high protection needs SHOULD each use their own Kubernetes clusters + (1) Applications with very high protection needs SHOULD each use their own Kubernetes clusters or dedicated nodes that are not available for other applications notes: '' status: manual @@ -426,8 +426,7 @@ controls: levels: - elevated description: >- - The automation of operational tasks in operators SHOULD be used for particularly critical - applications and control plane programs. + (1) The automation of operational tasks in operators SHOULD be used for particularly critical notes: >- OpenShift relies consistently on the application of the concept of operators. The platform itself is operated and managed 100% by operators, meaning that all internal components of From fae96d7d667ad3f25c75e668f85e391649ff21c8 Mon Sep 17 00:00:00 2001 From: sluetze <13255307+sluetze@users.noreply.github.com> Date: Tue, 8 Oct 2024 11:52:18 +0200 Subject: [PATCH 06/10] add rule to check for egress ip assignable nodes --- .../ocp4/e2e.yml | 2 + .../rule.yml | 57 +++++++++++++++++++ shared/references/cce-redhat-avail.txt | 1 - 3 files changed, 59 insertions(+), 1 deletion(-) create mode 100644 applications/openshift/networking/configure_egress_ip_node_assignable/ocp4/e2e.yml create mode 100644 applications/openshift/networking/configure_egress_ip_node_assignable/rule.yml diff --git a/applications/openshift/networking/configure_egress_ip_node_assignable/ocp4/e2e.yml b/applications/openshift/networking/configure_egress_ip_node_assignable/ocp4/e2e.yml new file mode 100644 index 00000000000..f426dc3d7ea --- /dev/null +++ b/applications/openshift/networking/configure_egress_ip_node_assignable/ocp4/e2e.yml @@ -0,0 +1,2 @@ +--- +default_result: FAIL diff --git a/applications/openshift/networking/configure_egress_ip_node_assignable/rule.yml b/applications/openshift/networking/configure_egress_ip_node_assignable/rule.yml new file mode 100644 index 00000000000..6249f48548b --- /dev/null +++ b/applications/openshift/networking/configure_egress_ip_node_assignable/rule.yml @@ -0,0 +1,57 @@ +documentation_complete: true + +title: Check Egress IPs Assignable to Nodes + +description: -| +
+ The OpenShift Container Platform egress IP address functionality allows you to ensure that the + traffic from one or more pods in one or more namespaces has a consistent source IP address for + services outside the cluster network. +
++ The necessary labeling on the designated nodes is configurable without reboot via +
$ oc label nodes $NODENAME k8s.ovn.org/egress-assignable=""for details see the + Red Hat Documentation + {{{ weblink(link="https://docs.redhat.com/en/documentation/openshift_container_platform/4.17/html/networking/ovn-kubernetes-network-plugin#nw-egress-ips-about_configuring-egress-ips-ovn") }}} + + +rationale: -| + By using egress IPs you can provide a consistent IP to external services and configure special + firewall rules which precisely select this IP. This allows for more control on external systems. + Furthermore you can bind the IPs to specific nodes, which handle all the network connections to + achieve a better separation of duties between the different nodes. + +identifiers: + cce@ocp4: CCE-86787-9 + +severity: medium + +ocil_clause: 'Check Egress IPs Assignable to Nodes' + +ocil: |- + Run the following command to see if nodes are assignable for egress IPs +
$ oc get --raw /api/v1/nodes | jq '.items[] | select(.metadata.labels."k8s.ovn.org/egress-assignable" != null) | .metadata.name'+ This commands prints the name of each node which is configured to get egress IPs assigned. If + the output is empty, there are no nodes available. + +{{% set old_jqfilter = 'if any(.items[]?; .metadata.labels."k8s.ovn.org/egress-assignable" != null) then true else false end' %}} +{{% set jqfilter = '[ .items[] | .metadata.labels["k8s.ovn.org/egress-assignable"] != null ]' %}} + + +warnings: + - general: |- + {{{ openshift_filtered_cluster_setting({'/api/v1/nodes': jqfilter}) | indent(8) }}} + +template: + name: yamlfile_value + vars: + ocp_data: "true" + filepath: |- + {{{ openshift_filtered_path('/api/v1/nodes', jqfilter) }}} + yamlpath: '[:]' + check_existence: at_least_one_exists + entity_check: "at least one" + values: + - value: 'true' + type: "string" + entity_check: "at least one" diff --git a/shared/references/cce-redhat-avail.txt b/shared/references/cce-redhat-avail.txt index d3f18735b4b..52a376cd244 100644 --- a/shared/references/cce-redhat-avail.txt +++ b/shared/references/cce-redhat-avail.txt @@ -173,7 +173,6 @@ CCE-86780-4 CCE-86781-2 CCE-86784-6 CCE-86785-3 -CCE-86787-9 CCE-86788-7 CCE-86789-5 CCE-86790-3 From 48d4f52bd6b76df17fa5358a6bf8b2f7ce2e72c3 Mon Sep 17 00:00:00 2001 From: sluetze <13255307+sluetze@users.noreply.github.com> Date: Tue, 8 Oct 2024 12:05:33 +0200 Subject: [PATCH 07/10] add testcase for egress ip --- .../configure_egress_ip_node_assignable/ocp4/e2e.yml | 2 -- .../tests/ocp4/e2e-remediation.sh | 9 +++++++++ .../tests/ocp4/e2e.yml | 3 +++ 3 files changed, 12 insertions(+), 2 deletions(-) delete mode 100644 applications/openshift/networking/configure_egress_ip_node_assignable/ocp4/e2e.yml create mode 100755 applications/openshift/networking/configure_egress_ip_node_assignable/tests/ocp4/e2e-remediation.sh create mode 100644 applications/openshift/networking/configure_egress_ip_node_assignable/tests/ocp4/e2e.yml diff --git a/applications/openshift/networking/configure_egress_ip_node_assignable/ocp4/e2e.yml b/applications/openshift/networking/configure_egress_ip_node_assignable/ocp4/e2e.yml deleted file mode 100644 index f426dc3d7ea..00000000000 --- a/applications/openshift/networking/configure_egress_ip_node_assignable/ocp4/e2e.yml +++ /dev/null @@ -1,2 +0,0 @@ ---- -default_result: FAIL diff --git a/applications/openshift/networking/configure_egress_ip_node_assignable/tests/ocp4/e2e-remediation.sh b/applications/openshift/networking/configure_egress_ip_node_assignable/tests/ocp4/e2e-remediation.sh new file mode 100755 index 00000000000..90a19e0dca9 --- /dev/null +++ b/applications/openshift/networking/configure_egress_ip_node_assignable/tests/ocp4/e2e-remediation.sh @@ -0,0 +1,9 @@ +#!/bin/bash +set -xe + +echo "Labeling Node for egress IP" + +NODENAME=`oc get node | tail -1 | cut -d" " -f1` +oc label node $NODENAME k8s.ovn.org/egress-assignable="" + +sleep 5 diff --git a/applications/openshift/networking/configure_egress_ip_node_assignable/tests/ocp4/e2e.yml b/applications/openshift/networking/configure_egress_ip_node_assignable/tests/ocp4/e2e.yml new file mode 100644 index 00000000000..fd9b313e87b --- /dev/null +++ b/applications/openshift/networking/configure_egress_ip_node_assignable/tests/ocp4/e2e.yml @@ -0,0 +1,3 @@ +--- +default_result: FAIL +result_after_remediation: PASS From 5c5aaa6245a63e5da9d33706dfc37d0f0f932ddc Mon Sep 17 00:00:00 2001 From: sluetze <13255307+sluetze@users.noreply.github.com> Date: Tue, 8 Oct 2024 13:01:22 +0200 Subject: [PATCH 08/10] add Control for section 2 --- controls/bsi_app_4_4.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/controls/bsi_app_4_4.yml b/controls/bsi_app_4_4.yml index b4d51c20d4b..4e430afe676 100644 --- a/controls/bsi_app_4_4.yml +++ b/controls/bsi_app_4_4.yml @@ -406,6 +406,8 @@ controls: rules: # Section 1,2,3,4 - general_node_separation + # Section 2 + - configure_egress_ip_node_assignable # Section 3 - master_taint_noschedule From 024b9427df9219dd882bd4c59292fc87502061ce Mon Sep 17 00:00:00 2001 From: sluetze <13255307+sluetze@users.noreply.github.com> Date: Tue, 8 Oct 2024 14:39:36 +0200 Subject: [PATCH 09/10] add network separation rule to A14 --- .../openshift/general/general_network_separation/rule.yml | 2 ++ controls/bsi_app_4_4.yml | 1 + 2 files changed, 3 insertions(+) diff --git a/applications/openshift/general/general_network_separation/rule.yml b/applications/openshift/general/general_network_separation/rule.yml index 8144cfc3ffa..87ccb3c8b76 100644 --- a/applications/openshift/general/general_network_separation/rule.yml +++ b/applications/openshift/general/general_network_separation/rule.yml @@ -18,3 +18,5 @@ ocil_clause: 'Network separation needs review' ocil: |- Create separate Ingress Controllers for the API and your Applications. Also setup your environment in a way, that Control Plane Nodes are in another network than your worker nodes. If you implement multiple Nodes for different purposes evaluate if these should be in different network segments (i.e. Infra-Nodes, Storage-Nodes, ...). + Also evaluate how you handle outgoing connections and if they have to be pinned to + specific nodes or IPs. diff --git a/controls/bsi_app_4_4.yml b/controls/bsi_app_4_4.yml index 4e430afe676..e617e078381 100644 --- a/controls/bsi_app_4_4.yml +++ b/controls/bsi_app_4_4.yml @@ -406,6 +406,7 @@ controls: rules: # Section 1,2,3,4 - general_node_separation + - general_network_separation # Section 2 - configure_egress_ip_node_assignable # Section 3 From 33a1da5fbdfcf64177b005f2461977340f31da5a Mon Sep 17 00:00:00 2001 From: sluetze <13255307+sluetze@users.noreply.github.com> Date: Mon, 14 Oct 2024 12:21:30 +0000 Subject: [PATCH 10/10] fix removed sentence --- controls/bsi_app_4_4.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/controls/bsi_app_4_4.yml b/controls/bsi_app_4_4.yml index e617e078381..d8ebd323833 100644 --- a/controls/bsi_app_4_4.yml +++ b/controls/bsi_app_4_4.yml @@ -430,6 +430,7 @@ controls: - elevated description: >- (1) The automation of operational tasks in operators SHOULD be used for particularly critical + applications and control plane programs. notes: >- OpenShift relies consistently on the application of the concept of operators. The platform itself is operated and managed 100% by operators, meaning that all internal components of