To fix CMP-2548 #12008

rutvik23 · 2024-05-20T11:43:27Z

Description:

This rule controller_rotate_kubelet_server_certs is no longer valid as it still checking for feature-gate parameter which does not exist in any supported versions of OpenShift 4.

$ ./oc get ccr | grep cert | grep -i fail

ocp4-high-controller-rotate-kubelet-server-certs FAIL medium

Reproducible in Compliance Operator v1.4.0 & OpenShift v4.14.z

Rationale:

Fixes #CMP-2548
The existing rules like kubelet_enable_cert_rotation, kubelet_enable_server_cert_rotation already satisfy this check.

Review Hints:

Another rule that rely on feature-gate parameter which does not exist, was disabled recently through CMP-2331

openshift-ci · 2024-05-20T11:43:38Z

Hi @rutvik23. Thanks for your PR.

I'm waiting for a ComplianceAsCode member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

github-actions · 2024-05-20T11:44:44Z

Start a new ephemeral environment with changes proposed in this pull request:

Fedora Environment

Oracle Linux 8 Environment

github-actions · 2024-05-20T11:47:47Z

🤖 A k8s content image for this PR is available at:
ghcr.io/complianceascode/k8scontent:12008
This image was built from commit: 3beb2f6

Click here to see how to deploy it

If you alread have Compliance Operator deployed:
utils/build_ds_container.py -i ghcr.io/complianceascode/k8scontent:12008

Otherwise deploy the content and operator together by checking out ComplianceAsCode/compliance-operator and:
CONTENT_IMAGE=ghcr.io/complianceascode/k8scontent:12008 make deploy-local

yuumasato · 2024-05-22T11:54:04Z

/ok-to-test

yuumasato · 2024-05-22T11:56:08Z

/test 4.13-e2e-aws-ocp4-moderate
/test 4.14-e2e-aws-ocp4-moderate
/test 4.15-e2e-aws-ocp4-moderate
/test 4.16-e2e-aws-ocp4-moderate
/test e2e-aws-ocp4-moderate

applications/openshift/controller/controller_rotate_kubelet_server_certs/rule.yml

xiaojiey · 2024-05-27T05:56:26Z

/hold for test

BhargaviGudi · 2024-05-27T14:00:10Z

Verification passed with 4.16.0-0.nightly-2024-05-23-173505 + https://github.com/ComplianceAsCode/compliance-operator code + PR #12008 code

$ oc get pb
NAME              CONTENTIMAGE                                 CONTENTFILE         STATUS
ocp4              ghcr.io/complianceascode/k8scontent:latest   ssg-ocp4-ds.xml     VALID
rhcos4            ghcr.io/complianceascode/k8scontent:latest   ssg-rhcos4-ds.xml   VALID
upstream-ocp4     ghcr.io/complianceascode/k8scontent:12008    ssg-ocp4-ds.xml     VALID
upstream-rhcos4   ghcr.io/complianceascode/k8scontent:12008    ssg-rhcos4-ds.xml   VALID

$ oc get rule upstream-ocp4-controller-rotate-kubelet-server-certs -ojsonpath={.instructions}
RotateKubeletServerCertificate is no longer a valid check, instead an user should run below commands and confirm the rotation settings are enabled as default.
$ for node in $(oc get nodes -ojsonpath='{.items[*].metadata.name}'); do oc get --raw /api/v1/nodes/$node/proxy/configz | jq '.kubeletconfig.serverTLSBootstrap'; done
$ for NODE_NAME in $(oc get nodes -ojsonpath='{.items[*].metadata.name}'); do oc get --raw /api/v1/nodes/$NODE_NAME/proxy/configz | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"' | grep rotateCertificates; done
The output should return true
      Is it the case that <tt>serverTLSBootstrap</tt> argument is set to <tt>false</tt> in the

$ for node in $(oc get nodes -ojsonpath='{.items[*].metadata.name}'); do oc get --raw /api/v1/nodes/$node/proxy/configz | jq '.kubeletconfig.serverTLSBootstrap'; done
true
true
true
true
true
true
$ for NODE_NAME in $(oc get nodes -ojsonpath='{.items[*].metadata.name}'); do oc get --raw /api/v1/nodes/$NODE_NAME/proxy/configz | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"' | grep rotateCertificates; done
  "rotateCertificates": true,
  "rotateCertificates": true,
  "rotateCertificates": true,
  "rotateCertificates": true,
  "rotateCertificates": true,
  "rotateCertificates": true,

BhargaviGudi · 2024-05-27T14:00:29Z

/unhold

xiaojiey · 2024-05-29T06:14:28Z

/hold for test

yuumasato

Let's unselect the rule so that no profile uses this rule anymore.

grep -r controller_rotate_kubelet_server_certs controls/ products/
controls/srg_ctr/SRG-APP-000516-CTR-001325.yml:  - controller_rotate_kubelet_server_certs
controls/nist_ocp4.yml:  - controller_rotate_kubelet_server_certs
controls/nist_ocp4.yml:  - controller_rotate_kubelet_server_certs
controls/nist_ocp4.yml:  - controller_rotate_kubelet_server_certs
controls/nist_ocp4.yml:  - controller_rotate_kubelet_server_certs
products/ocp4/profiles/stig-v1r1.profile:    - controller_rotate_kubelet_server_certs

And let's add a general warning noting that this rule is deprecated.
Example warning in the rule:https://github.com/ComplianceAsCode/content/blob/master/linux_os/guide/auditing/auditd_configure_rules/audit_rules_etc_passwd_openat/rule.yml#L37
Inspiration for warning text: https://github.com/ComplianceAsCode/content/blob/master/shared/macros/10-warning.jinja#L30

applications/openshift/controller/controller_rotate_kubelet_server_certs/rule.yml

BhargaviGudi · 2024-06-18T07:45:36Z

Verification passed with 4.16.0-0.nightly-2024-06-14-130320 + https://github.com/ComplianceAsCode/compliance-operator code + PR #12008 code

Install CO

$ oc get pb
NAME              CONTENTIMAGE                                 CONTENTFILE         STATUS
ocp4              ghcr.io/complianceascode/k8scontent:latest   ssg-ocp4-ds.xml     VALID
rhcos4            ghcr.io/complianceascode/k8scontent:latest   ssg-rhcos4-ds.xml   VALID
upstream-ocp4     ghcr.io/complianceascode/k8scontent:12008    ssg-ocp4-ds.xml     VALID
upstream-rhcos4   ghcr.io/complianceascode/k8scontent:12008    ssg-rhcos4-ds.xml   VALID

Verify rule controller-rotate-kubelet-server-certs does not exists

$ oc get rule | grep controller-rotate-kubelet-server-certs | grep upstream
$ 
$ oc compliance bind -N test -S default-auto-apply profile/upstream-ocp4-high
Creating ScanSettingBinding test
$ oc get scan
NAME                 PHASE   RESULT
upstream-ocp4-high   DONE    NON-COMPLIANT
$ oc get ccr | grep controller-rotate-kubelet-server-certs
$

BhargaviGudi · 2024-06-18T07:45:42Z

/unhold

yuumasato

From what I understand, what makes this rule fails is that feature-gates doesn't list RotateKubeletServerCertificate=true any more. At some point OCP stopped listing it and this rule became not applicable.

And for the rule to result in NOT APPLICABLE the platform needs to be updated.
For example:
platform: not ocp4-on-hypershift-hosted and (ocp4.10 or ocp4.11)

I tried checking which versions have RotateKubeletServerCertificate listed in feature-gates but my clusters installs are failing... I can try getting the list of versions tomorrow.

applications/openshift/controller/controller_rotate_kubelet_server_certs/rule.yml

applications/openshift/controller/controller_rotate_kubelet_server_certs/tests/ocp4/e2e.yml

applications/openshift/controller/controller_rotate_kubelet_server_certs/rule.yml

yuumasato

Hi, @rutvik23 sorry for the back and forth.

The RotateKubeletServerCertificate=true featture gate is available on 4.12, 4.13:

$ oc version
Client Version: 4.14.1
Kustomize Version: v5.0.1
Server Version: 4.12.0-0.nightly-2024-06-20-082732
Kubernetes Version: v1.25.16+306a47e

$ oc get configmaps config -n openshift-kube-controller-manager -ojson | jq -r '.data["config.yaml"]' | jq -r '.extendedArguments["feature-gates"]'                                                                                                                                        
[
  "APIPriorityAndFairness=true",
  "RotateKubeletServerCertificate=true",
  "DownwardAPIHugePages=true",
  "CSIMigrationAzureFile=false",
  "CSIMigrationvSphere=false"
]

So I think the rule should be available and working for these versions.
keep the rule selected in the profiles. The following changes should be undone:

controls/nist_ocp4.yml
controls/srg_ctr/SRG-APP-000516-CTR-001325.yml
products/ocp4/profiles/stig-v1r1.profile

To make the rules not applicable on 4.13 and 4.14:
platform: not ocp4-on-hypershift-hosted and (ocp4.12 or ocp4.13)

Then override the e2e test results for 4.12 and 4.13. Copy the e2e.yml and name it 4.12.yml and 4.13.yml.
For inspiration: https://github.com/ComplianceAsCode/content/blob/master/applications/openshift/worker/file_permissions_proxy_kubeconfig/tests/ocp4/4.12.yml

applications/openshift/controller/controller_rotate_kubelet_server_certs/rule.yml

xiaojiey · 2024-06-21T02:30:42Z

/hold for test

yuumasato · 2024-06-24T10:41:36Z

/test 4.12-e2e-aws-ocp4-high
/test 4.13-e2e-aws-ocp4-high
/test 4.14-e2e-aws-ocp4-high
/test 4.15-e2e-aws-ocp4-high
/test 4.16-e2e-aws-ocp4-high
/test e2e-aws-ocp4-high

yuumasato

@rutvik23 Sorry, looks like the e2e assertions need to be changed here too:

default_result: NOT-APPLICABLE

codeclimate · 2024-06-24T16:51:49Z

Code Climate has analyzed commit 3beb2f6 and detected 0 issues on this pull request.

The test coverage on the diff in this pull request is 100.0% (50% is the threshold).

This pull request will bring the total coverage in the repository to 59.4% (0.0% change).

View more on Code Climate.

yuumasato · 2024-06-24T17:34:30Z

/retest

yuumasato · 2024-06-24T17:56:16Z

/test 4.12-e2e-aws-ocp4-high
/test 4.13-e2e-aws-ocp4-high
/test 4.14-e2e-aws-ocp4-high
/test 4.15-e2e-aws-ocp4-high
/test 4.16-e2e-aws-ocp4-high
/test e2e-aws-ocp4-high

yuumasato

Thank you @rutvik23

yuumasato · 2024-06-25T10:49:05Z

Skipping the ansible hardening tests.

yuumasato · 2024-06-25T10:52:37Z

@xiaojiey Apologies. I just noticed the hold for merge label.
Let me know if you run into any issues with this PR..

To fix CMP-2548

a582345

openshift-ci bot added the needs-ok-to-test Used by openshift-ci bot. label May 20, 2024

marcusburghardt added the OpenShift OpenShift product related. label May 21, 2024

marcusburghardt requested review from yuumasato, Vincent056 and rhmdnd May 21, 2024 09:53

openshift-ci bot added ok-to-test Used by openshift-ci bot. and removed needs-ok-to-test Used by openshift-ci bot. labels May 22, 2024

yuumasato reviewed May 22, 2024

View reviewed changes

applications/openshift/controller/controller_rotate_kubelet_server_certs/rule.yml Outdated Show resolved Hide resolved

applications/openshift/controller/controller_rotate_kubelet_server_certs/rule.yml Outdated Show resolved Hide resolved

openshift-ci bot added the do-not-merge/hold Used by openshift-ci-robot bot. label May 27, 2024

openshift-ci bot removed the do-not-merge/hold Used by openshift-ci-robot bot. label May 27, 2024

openshift-ci bot added the do-not-merge/hold Used by openshift-ci-robot bot. label May 29, 2024

yuumasato requested changes Jun 11, 2024

View reviewed changes

applications/openshift/controller/controller_rotate_kubelet_server_certs/rule.yml Outdated Show resolved Hide resolved

applications/openshift/controller/controller_rotate_kubelet_server_certs/rule.yml Outdated Show resolved Hide resolved

rutvik23 added 3 commits June 14, 2024 20:33

modifying rule content and selection criteria

651cfa9

adding warning section

35f5869

fixing whitespaces in warning section

028049f

yuumasato self-assigned this Jun 17, 2024

BhargaviGudi closed this Jun 18, 2024

openshift-ci bot removed the do-not-merge/hold Used by openshift-ci-robot bot. label Jun 18, 2024

BhargaviGudi reopened this Jun 18, 2024

rutvik23 requested a review from yuumasato June 18, 2024 11:45

yuumasato reviewed Jun 19, 2024

View reviewed changes

yuumasato requested changes Jun 20, 2024

View reviewed changes

applications/openshift/controller/controller_rotate_kubelet_server_certs/rule.yml Show resolved Hide resolved

openshift-ci bot added the do-not-merge/hold Used by openshift-ci-robot bot. label Jun 21, 2024

updating platform check and tests

c378c08

rutvik23 requested a review from yuumasato June 24, 2024 10:11

yuumasato requested changes Jun 24, 2024

View reviewed changes

fixing e2e assertions

3beb2f6

yuumasato approved these changes Jun 25, 2024

View reviewed changes

yuumasato merged commit 9758517 into ComplianceAsCode:master Jun 25, 2024
72 of 119 checks passed

yuumasato added this to the 0.1.74 milestone Jun 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

To fix CMP-2548 #12008

To fix CMP-2548 #12008

rutvik23 commented May 20, 2024

openshift-ci bot commented May 20, 2024

github-actions bot commented May 20, 2024

github-actions bot commented May 20, 2024 •

edited

Loading

yuumasato commented May 22, 2024

yuumasato commented May 22, 2024 •

edited

Loading

xiaojiey commented May 27, 2024

BhargaviGudi commented May 27, 2024

BhargaviGudi commented May 27, 2024

xiaojiey commented May 29, 2024

yuumasato left a comment

BhargaviGudi commented Jun 18, 2024

BhargaviGudi commented Jun 18, 2024

yuumasato left a comment

yuumasato left a comment

xiaojiey commented Jun 21, 2024

yuumasato commented Jun 24, 2024

yuumasato left a comment

codeclimate bot commented Jun 24, 2024

yuumasato commented Jun 24, 2024

yuumasato commented Jun 24, 2024

yuumasato left a comment

yuumasato commented Jun 25, 2024

yuumasato commented Jun 25, 2024

To fix CMP-2548 #12008

To fix CMP-2548 #12008

Conversation

rutvik23 commented May 20, 2024

Description:

Rationale:

Review Hints:

openshift-ci bot commented May 20, 2024

github-actions bot commented May 20, 2024

github-actions bot commented May 20, 2024 • edited Loading

yuumasato commented May 22, 2024

yuumasato commented May 22, 2024 • edited Loading

xiaojiey commented May 27, 2024

BhargaviGudi commented May 27, 2024

BhargaviGudi commented May 27, 2024

xiaojiey commented May 29, 2024

yuumasato left a comment

Choose a reason for hiding this comment

BhargaviGudi commented Jun 18, 2024

BhargaviGudi commented Jun 18, 2024

yuumasato left a comment

Choose a reason for hiding this comment

yuumasato left a comment

Choose a reason for hiding this comment

xiaojiey commented Jun 21, 2024

yuumasato commented Jun 24, 2024

yuumasato left a comment

Choose a reason for hiding this comment

codeclimate bot commented Jun 24, 2024

yuumasato commented Jun 24, 2024

yuumasato commented Jun 24, 2024

yuumasato left a comment

Choose a reason for hiding this comment

yuumasato commented Jun 25, 2024

yuumasato commented Jun 25, 2024

github-actions bot commented May 20, 2024 •

edited

Loading

yuumasato commented May 22, 2024 •

edited

Loading