Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HIVE-2777: Implement hive Nutanix provisioning #2573

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

eliorerz
Copy link

@eliorerz eliorerz commented Feb 18, 2025

This is a partial implementation of #2550 PR, the final code will (probably) extend ClusterDeployment CR and will add support with MachinePools.

This PR adds support for provisioning OpenShift clusters on Nutanix using the OpenShift Installer's IPI installation method within Hive.

Key changes include:

  • Integration with the OpenShift Installer IPI workflow for Nutanix.
  • Implementation of necessary controllers, validations, and configuration options.
  • Automatically set install-config nutanix platform credentials from secret nutanix-creds (see pasteInProviderCredentials method)

Implementation:

  • ClusterDeployment
apiVersion: hive.openshift.io/v1
kind: ClusterDeployment
metadata:
  name: cluster-nutanix
  namespace: hive
spec:
  clusterName: cluster-nutanix
  baseDomain: example.com
  platform:
    nutanix:
      credentialsSecretRef:
        name: nutanix-creds
      prismCentral:
        address: cluster-nutanix.prism-central.nutanix.com
        port: 9440
  provisioning:
    installConfigSecretRef:
      name: cluster-nutanix-install-config
    imageSetRef:
      name: cluster-nutanix-image-set
  pullSecretRef:
    name: pull-secret
****

Secrets

  • nutanix-creds.yaml
apiVersion: v1
data:
  password: <password>
  username: <username>
kind: Secret
metadata:
  name: nutanix-creds
  namespace: hive
  • install-config.yaml
apiVersion: v1
baseDomain: example.com
compute:
- name: worker
controlPlane:
  name: master
metadata:
  name: cluster-nutanix
platform:
  nutanix:
    apiVIPs:
      - 10.0.0.123
    ingressVIPs:
      - 10.0.0.124
    prismCentral:
      endpoint:
        address: cluster-nutanix.prism-central.nutanix.com
        port: 9440
    prismElements: 
      - endpoint:
          address: cluster-nutanix.prism-element.nutanix.com
          port: 9440
        uuid: 0005de05-75a3-dacb-ba00-2c5da2ac4c1a
        name: "NAME"
    subnetUUIDs:
      -  0005de05-75a3-dacb-ba00-123456789012
    failureDomains:
      - name: "LD Name"
        subnetUUIDs:
          -  0005de05-75a3-dacb-ba00-123456789012
        prismElements: 
          - endpoint:
              address: cluster-nutanix.prism-element.nutanix.com
              port: 9440
            uuid: 0005de05-75a3-dacb-ba00-2c5da2ac4c1a
            name: "NAME"

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Feb 18, 2025
@openshift-ci-robot
Copy link

openshift-ci-robot commented Feb 18, 2025

@eliorerz: This pull request references HIVE-2777 which is a valid jira issue.

In response to this:

/cc @eliorerz

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 18, 2025
Copy link
Contributor

openshift-ci bot commented Feb 18, 2025

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Copy link
Contributor

openshift-ci bot commented Feb 18, 2025

@eliorerz: GitHub didn't allow me to request PR reviews from the following users: eliorerz.

Note that only openshift members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @eliorerz

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@eliorerz eliorerz force-pushed the HIVE-2777-Implement-Hive-Nutanix-Provisioning branch 3 times, most recently from dc89df3 to c68f669 Compare February 19, 2025 12:22
@openshift-ci-robot
Copy link

openshift-ci-robot commented Feb 19, 2025

@eliorerz: This pull request references HIVE-2777 which is a valid jira issue.

In response to this:

This PR adds support for provisioning OpenShift clusters on Nutanix using the OpenShift Installer's IPI installation method within Hive.

Key changes include:

  • Integration with the OpenShift Installer IPI workflow for Nutanix.
  • Implementation of necessary controllers, validations, and configuration options.

Implementation:

  • ClusterDeployment
apiVersion: hive.openshift.io/v1
kind: ClusterDeployment
metadata:
 name: cluster-nutanix
 namespace: hive
spec:
 clusterName: cluster-nutanix
 baseDomain: example.com
 platform:
   nutanix:
     credentialsSecretRef:
       name: nutanix-creds
     prismCentral:
       address: cluster-nutanix.prism-central.nutanix.com
       port: 9440
 provisioning:
   installConfigSecretRef:
     name: cluster-nutanix-install-config
   imageSetRef:
     name: cluster-nutanix-image-set
 pullSecretRef:
   name: pull-secret
****

Secrets

  • nutanix-creds.yaml
apiVersion: v1
data:
 password: <password>
 username: <username>
kind: Secret
metadata:
 name: nutanix-creds
 namespace: hive
  • install-config.yaml
apiVersion: v1
baseDomain: example.com
compute:
- name: worker
controlPlane:
 name: master
metadata:
 name: cluster-nutanix
platform:
 nutanix:
   apiVIPs:
     - 10.0.0.123
   ingressVIPs:
     - 10.0.0.124
   prismCentral:
     endpoint:
       address: cluster-nutanix.prism-central.nutanix.com
       port: 9440
   prismElements: 
     - endpoint:
         address: cluster-nutanix.prism-element.nutanix.com
         port: 9440
       uuid: 0005de05-75a3-dacb-ba00-2c5da2ac4c1a
       name: "NAME"
   subnetUUIDs:
     -  0005de05-75a3-dacb-ba00-123456789012
   failureDomains:
     - name: "LD Name"
       subnetUUIDs:
         -  0005de05-75a3-dacb-ba00-123456789012
       prismElements: 
         - endpoint:
             address: cluster-nutanix.prism-element.nutanix.com
             port: 9440
           uuid: 0005de05-75a3-dacb-ba00-2c5da2ac4c1a
           name: "NAME"

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@eliorerz eliorerz marked this pull request as ready for review February 19, 2025 12:47
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 19, 2025
@eliorerz
Copy link
Author

/cc @2uasimojo

@openshift-ci-robot
Copy link

openshift-ci-robot commented Feb 19, 2025

@eliorerz: This pull request references HIVE-2777 which is a valid jira issue.

In response to this:

This PR adds support for provisioning OpenShift clusters on Nutanix using the OpenShift Installer's IPI installation method within Hive.

Key changes include:

  • Integration with the OpenShift Installer IPI workflow for Nutanix.
  • Implementation of necessary controllers, validations, and configuration options.
  • Automatically set install-config nutanix platform credentials from secret nutanix-creds (see pasteInProviderCredentials method)

Implementation:

  • ClusterDeployment
apiVersion: hive.openshift.io/v1
kind: ClusterDeployment
metadata:
 name: cluster-nutanix
 namespace: hive
spec:
 clusterName: cluster-nutanix
 baseDomain: example.com
 platform:
   nutanix:
     credentialsSecretRef:
       name: nutanix-creds
     prismCentral:
       address: cluster-nutanix.prism-central.nutanix.com
       port: 9440
 provisioning:
   installConfigSecretRef:
     name: cluster-nutanix-install-config
   imageSetRef:
     name: cluster-nutanix-image-set
 pullSecretRef:
   name: pull-secret
****

Secrets

  • nutanix-creds.yaml
apiVersion: v1
data:
 password: <password>
 username: <username>
kind: Secret
metadata:
 name: nutanix-creds
 namespace: hive
  • install-config.yaml
apiVersion: v1
baseDomain: example.com
compute:
- name: worker
controlPlane:
 name: master
metadata:
 name: cluster-nutanix
platform:
 nutanix:
   apiVIPs:
     - 10.0.0.123
   ingressVIPs:
     - 10.0.0.124
   prismCentral:
     endpoint:
       address: cluster-nutanix.prism-central.nutanix.com
       port: 9440
   prismElements: 
     - endpoint:
         address: cluster-nutanix.prism-element.nutanix.com
         port: 9440
       uuid: 0005de05-75a3-dacb-ba00-2c5da2ac4c1a
       name: "NAME"
   subnetUUIDs:
     -  0005de05-75a3-dacb-ba00-123456789012
   failureDomains:
     - name: "LD Name"
       subnetUUIDs:
         -  0005de05-75a3-dacb-ba00-123456789012
       prismElements: 
         - endpoint:
             address: cluster-nutanix.prism-element.nutanix.com
             port: 9440
           uuid: 0005de05-75a3-dacb-ba00-2c5da2ac4c1a
           name: "NAME"

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Feb 19, 2025

@eliorerz: This pull request references HIVE-2777 which is a valid jira issue.

In response to this:

This is a partial implementation of #2550 PR, the final code will (probably) extend ClusterDeployment CR and will add support with MachinePools.

This PR adds support for provisioning OpenShift clusters on Nutanix using the OpenShift Installer's IPI installation method within Hive.

Key changes include:

  • Integration with the OpenShift Installer IPI workflow for Nutanix.
  • Implementation of necessary controllers, validations, and configuration options.
  • Automatically set install-config nutanix platform credentials from secret nutanix-creds (see pasteInProviderCredentials method)

Implementation:

  • ClusterDeployment
apiVersion: hive.openshift.io/v1
kind: ClusterDeployment
metadata:
 name: cluster-nutanix
 namespace: hive
spec:
 clusterName: cluster-nutanix
 baseDomain: example.com
 platform:
   nutanix:
     credentialsSecretRef:
       name: nutanix-creds
     prismCentral:
       address: cluster-nutanix.prism-central.nutanix.com
       port: 9440
 provisioning:
   installConfigSecretRef:
     name: cluster-nutanix-install-config
   imageSetRef:
     name: cluster-nutanix-image-set
 pullSecretRef:
   name: pull-secret
****

Secrets

  • nutanix-creds.yaml
apiVersion: v1
data:
 password: <password>
 username: <username>
kind: Secret
metadata:
 name: nutanix-creds
 namespace: hive
  • install-config.yaml
apiVersion: v1
baseDomain: example.com
compute:
- name: worker
controlPlane:
 name: master
metadata:
 name: cluster-nutanix
platform:
 nutanix:
   apiVIPs:
     - 10.0.0.123
   ingressVIPs:
     - 10.0.0.124
   prismCentral:
     endpoint:
       address: cluster-nutanix.prism-central.nutanix.com
       port: 9440
   prismElements: 
     - endpoint:
         address: cluster-nutanix.prism-element.nutanix.com
         port: 9440
       uuid: 0005de05-75a3-dacb-ba00-2c5da2ac4c1a
       name: "NAME"
   subnetUUIDs:
     -  0005de05-75a3-dacb-ba00-123456789012
   failureDomains:
     - name: "LD Name"
       subnetUUIDs:
         -  0005de05-75a3-dacb-ba00-123456789012
       prismElements: 
         - endpoint:
             address: cluster-nutanix.prism-element.nutanix.com
             port: 9440
           uuid: 0005de05-75a3-dacb-ba00-2c5da2ac4c1a
           name: "NAME"

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Copy link

codecov bot commented Feb 19, 2025

Codecov Report

Attention: Patch coverage is 46.40625% with 343 lines in your changes missing coverage. Please review.

Project coverage is 49.79%. Comparing base (66218d8) to head (62a98fc).

Files with missing lines Patch % Lines
contrib/pkg/createcluster/nutanix.go 0.00% 77 Missing ⚠️
contrib/pkg/deprovision/nutanix.go 0.00% 60 Missing ⚠️
pkg/controller/machinepool/nutanixactuator.go 57.81% 49 Missing and 5 partials ⚠️
pkg/installmanager/installmanager.go 28.88% 28 Missing and 4 partials ⚠️
pkg/install/generate.go 0.00% 23 Missing ⚠️
...s/hive/v1/machinepool_validating_admission_hook.go 0.00% 15 Missing and 1 partial ⚠️
...g/controller/clusterpool/clusterpool_controller.go 0.00% 15 Missing ⚠️
...g/controller/machinepool/machinepool_controller.go 0.00% 14 Missing ⚠️
pkg/controller/utils/nutainx.go 90.47% 13 Missing and 1 partial ⚠️
contrib/pkg/createcluster/create.go 0.00% 13 Missing ⚠️
... and 5 more
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2573      +/-   ##
==========================================
- Coverage   49.84%   49.79%   -0.05%     
==========================================
  Files         281      287       +6     
  Lines       32981    33607     +626     
==========================================
+ Hits        16439    16735     +296     
- Misses      15209    15528     +319     
- Partials     1333     1344      +11     
Files with missing lines Coverage Δ
pkg/constants/constants.go 100.00% <ø> (ø)
...oller/clusterdeployment/installconfigvalidation.go 100.00% <100.00%> (ø)
.../v1/clusterdeployment_validating_admission_hook.go 85.86% <100.00%> (+0.25%) ⬆️
...shift/hive/apis/hive/v1/clusterdeployment_types.go 0.00% <ø> (ø)
...hift/hive/apis/hive/v1/clusterdeprovision_types.go 0.00% <ø> (ø)
...m/openshift/hive/apis/hive/v1/machinepool_types.go 0.00% <ø> (ø)
contrib/pkg/deprovision/deprovision.go 0.00% <0.00%> (ø)
pkg/controller/utils/credentials.go 0.00% <0.00%> (ø)
contrib/pkg/utils/nutanix/nutanix.go 0.00% <0.00%> (ø)
.../clusterdeployment/clusterdeployment_controller.go 66.80% <25.00%> (-0.23%) ⬇️
... and 11 more

@2uasimojo
Copy link
Member

Going to get started on this.

One question about the install-config: I noticed the prismElements and subnetUUIDs are duplicated at the top level and under failureDomains. We talked yesterday about the ClusterDeployment.Spec.Platform.Nutanix schema not containing this redundancy. I assume it's still supported via install-config, but it's not necessary, right? IOW your PoC still works if you omit the top-level copy?

@eliorerz
Copy link
Author

eliorerz commented Feb 19, 2025

Going to get started on this.

One question about the install-config: I noticed the prismElements and subnetUUIDs are duplicated at the top level and under failureDomains. We talked yesterday about the ClusterDeployment.Spec.Platform.Nutanix schema not containing this redundancy. I assume it's still supported via install-config, but it's not necessary, right? IOW your PoC still works if you omit the top-level copy?

Thanks, Regarding your question, the problem is that PrismElements seems to be mandatory in the install install-config while the failureDomains is optional. In any case you want to define failureDomains you have to also set at least PrismElements[0].UUID.

@2uasimojo
Copy link
Member

/test e2e e2e-pool

Weird flakery probably due to some upstream bug that seems to be fixed now.

Copy link
Member

@2uasimojo 2uasimojo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great! With a couple of minor tweaks, I think it's ready to go. (Except that, as we discussed, we're going to need to clone the appropriate install-config fields into cd.Spec.Platform.Nutanix in anticipation of MachinePools before we can actually "release" :( )

Reminders:

  • We're going to need doc updates.
  • Let's not forget to look into ClusterPools (will probably need to use Inventory). Note that the work required for ClusterPools overlaps quite a bit with that needed for hiveutil create-cluster, although the latter is optional as we've discussed.

@eliorerz eliorerz force-pushed the HIVE-2777-Implement-Hive-Nutanix-Provisioning branch from c68f669 to c092981 Compare February 20, 2025 16:47
@eliorerz eliorerz force-pushed the HIVE-2777-Implement-Hive-Nutanix-Provisioning branch from c092981 to c17e725 Compare February 24, 2025 13:04
@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 25, 2025
@eliorerz eliorerz force-pushed the HIVE-2777-Implement-Hive-Nutanix-Provisioning branch from bd300f9 to 3908c64 Compare February 25, 2025 21:24
@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 25, 2025
@eliorerz eliorerz closed this Feb 27, 2025
@eliorerz eliorerz force-pushed the HIVE-2777-Implement-Hive-Nutanix-Provisioning branch from b9b80e8 to 66218d8 Compare February 27, 2025 21:00
@eliorerz eliorerz reopened this Feb 27, 2025
@openshift-ci-robot
Copy link

openshift-ci-robot commented Feb 27, 2025

@eliorerz: This pull request references HIVE-2777 which is a valid jira issue.

In response to this:

This is a partial implementation of #2550 PR, the final code will (probably) extend ClusterDeployment CR and will add support with MachinePools.

This PR adds support for provisioning OpenShift clusters on Nutanix using the OpenShift Installer's IPI installation method within Hive.

Key changes include:

  • Integration with the OpenShift Installer IPI workflow for Nutanix.
  • Implementation of necessary controllers, validations, and configuration options.
  • Automatically set install-config nutanix platform credentials from secret nutanix-creds (see pasteInProviderCredentials method)

Implementation:

  • ClusterDeployment
apiVersion: hive.openshift.io/v1
kind: ClusterDeployment
metadata:
 name: cluster-nutanix
 namespace: hive
spec:
 clusterName: cluster-nutanix
 baseDomain: example.com
 platform:
   nutanix:
     credentialsSecretRef:
       name: nutanix-creds
     prismCentral:
       address: cluster-nutanix.prism-central.nutanix.com
       port: 9440
 provisioning:
   installConfigSecretRef:
     name: cluster-nutanix-install-config
   imageSetRef:
     name: cluster-nutanix-image-set
 pullSecretRef:
   name: pull-secret
****

Secrets

  • nutanix-creds.yaml
apiVersion: v1
data:
 password: <password>
 username: <username>
kind: Secret
metadata:
 name: nutanix-creds
 namespace: hive
  • install-config.yaml
apiVersion: v1
baseDomain: example.com
compute:
- name: worker
controlPlane:
 name: master
metadata:
 name: cluster-nutanix
platform:
 nutanix:
   apiVIPs:
     - 10.0.0.123
   ingressVIPs:
     - 10.0.0.124
   prismCentral:
     endpoint:
       address: cluster-nutanix.prism-central.nutanix.com
       port: 9440
   prismElements: 
     - endpoint:
         address: cluster-nutanix.prism-element.nutanix.com
         port: 9440
       uuid: 0005de05-75a3-dacb-ba00-2c5da2ac4c1a
       name: "NAME"
   subnetUUIDs:
     -  0005de05-75a3-dacb-ba00-123456789012
   failureDomains:
     - name: "LD Name"
       subnetUUIDs:
         -  0005de05-75a3-dacb-ba00-123456789012
       prismElements: 
         - endpoint:
             address: cluster-nutanix.prism-element.nutanix.com
             port: 9440
           uuid: 0005de05-75a3-dacb-ba00-2c5da2ac4c1a
           name: "NAME"

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Copy link
Contributor

openshift-ci bot commented Feb 27, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: eliorerz
Once this PR has been reviewed and has the lgtm label, please assign suhanime for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

1 similar comment
Copy link
Contributor

openshift-ci bot commented Feb 27, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: eliorerz
Once this PR has been reviewed and has the lgtm label, please assign suhanime for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@eliorerz eliorerz force-pushed the HIVE-2777-Implement-Hive-Nutanix-Provisioning branch from 815fb16 to 62a98fc Compare February 28, 2025 11:05
Copy link
Contributor

openshift-ci bot commented Feb 28, 2025

@eliorerz: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-vsphere 62a98fc link true /test e2e-vsphere
ci/prow/security 62a98fc link true /test security
ci/prow/verify 62a98fc link true /test verify

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants