Add deleting cells automation #846

mrkisaolamb · 2024-08-19T13:23:12Z

mrkisaolamb · 2024-08-19T13:24:30Z

Still require more testing and adding more func/kuttle tests

templates/nova-manage/bin/delete_cell.sh

gibizer

I tried to use this locally but it fails with:

 ❯ oc logs nova-cell1-cell-delete-tw8ts
+ sudo -E kolla_set_configs
sudo: unable to send audit message: Operation not permitted
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Copying service configuration files
INFO:__main__:Deleting /etc/nova/nova.conf
INFO:__main__:Copying /var/lib/openstack/config/nova-blank.conf to /etc/nova/nova.conf
INFO:__main__:Setting permission for /etc/nova/nova.conf
INFO:__main__:Copying /var/lib/openstack/config/01-nova.conf to /etc/nova/nova.conf.d/01-nova.conf
INFO:__main__:Setting permission for /etc/nova/nova.conf.d/01-nova.conf
INFO:__main__:Copying /var/lib/openstack/bin/delete_cell.sh to /bin/delete_cell.sh
INFO:__main__:Setting permission for /bin/delete_cell.sh
INFO:__main__:Copying /var/lib/openstack/config/my.cnf to /etc/my.cnf
INFO:__main__:Setting permission for /etc/my.cnf
INFO:__main__:Writing out command to execute
++ cat /run_command
+ CMD=/bin/ensure_cell_mapping.sh
+ ARGS=
+ sudo kolla_copy_cacerts
sudo: unable to send audit message: Operation not permitted
+ [[ ! -n '' ]]
+ . kolla_extend_start
+ echo 'Running command: '\''/bin/ensure_cell_mapping.sh'\'''
Running command: '/bin/ensure_cell_mapping.sh'
+ umask 0022
+ exec /bin/ensure_cell_mapping.sh
/usr/local/bin/kolla_start: line 22: /bin/ensure_cell_mapping.sh: No such file or directory

templates/nova-manage/config/cell-delete-config.json

templates/nova-manage/bin/delete_cell.sh

softwarefactory-project-zuul · 2024-09-02T14:50:00Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/4f7309eceb0b40788c94630721b7dbec

❌ openstack-meta-content-provider NODE_FAILURE Node request 100-0007561214 failed in 0s
⚠️ nova-operator-kuttl SKIPPED Skipped due to failed job openstack-meta-content-provider
⚠️ nova-operator-tempest-multinode SKIPPED Skipped due to failed job openstack-meta-content-provider
⚠️ nova-operator-tempest-multinode-ceph SKIPPED Skipped due to failed job openstack-meta-content-provider

mrkisaolamb · 2024-09-03T06:24:15Z

recheck

gibizer · 2024-09-03T09:33:08Z

Tried this again locally by deleting cell1.

I see two issues:

the cell_mapping is not deleted

❯ oc logs nova-cell1-cell-delete-t8ddb
+ sudo -E kolla_set_configs
sudo: unable to send audit message: Operation not permitted
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Copying service configuration files
INFO:__main__:Deleting /etc/nova/nova.conf
INFO:__main__:Copying /var/lib/openstack/config/nova-blank.conf to /etc/nova/nova.conf
INFO:__main__:Setting permission for /etc/nova/nova.conf
INFO:__main__:Copying /var/lib/openstack/config/01-nova.conf to /etc/nova/nova.conf.d/01-nova.conf
INFO:__main__:Setting permission for /etc/nova/nova.conf.d/01-nova.conf
INFO:__main__:Copying /var/lib/openstack/bin/delete_cell.sh to /bin/delete_cell.sh
INFO:__main__:Setting permission for /bin/delete_cell.sh
INFO:__main__:Copying /var/lib/openstack/config/my.cnf to /etc/my.cnf
INFO:__main__:Setting permission for /etc/my.cnf
INFO:__main__:Writing out command to execute
++ cat /run_command
+ CMD=/bin/delete_cell.sh
+ ARGS=
+ sudo kolla_copy_cacerts
sudo: unable to send audit message: Operation not permitted
+ [[ ! -n '' ]]
+ . kolla_extend_start
+ echo 'Running command: '\''/bin/delete_cell.sh'\'''
Running command: '/bin/delete_cell.sh'
+ umask 0022
+ exec /bin/delete_cell.sh
+ export CELL_NAME=cell1
+ CELL_NAME=cell1
++ nova-manage cell_v2 list_cells
++ tr ' ' '|'
++ tr --squeeze-repeats '|'
++ cut -d '|' -f 3
++ grep -e '^|cell1|'
Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code.
+ cell_uuid=30e83114-9261-4476-bfa4-e1301953e85f
+ '[' -z 30e83114-9261-4476-bfa4-e1301953e85f ']'

❯ oc rsh nova-cell0-conductor-0 nova-manage cell_v2 list_cells
Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code.
+-------+--------------------------------------+-----------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+----------+
|  Name |                 UUID                 |                                      Transport URL                                      |                                             Database Connection                                             | Disabled |
+-------+--------------------------------------+-----------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+----------+
| cell0 | 00000000-0000-0000-0000-000000000000 |                                         rabbit:                                         |    mysql+pymysql://****:****@openstack.openstack.svc/nova_cell0?read_default_file=/etc/my.cnf    |  False   |
| cell1 | 30e83114-9261-4476-bfa4-e1301953e85f | rabbit://****:****@rabbitmq-cell1.openstack.svc:5671/?ssl=1 | mysql+pymysql://****:****@openstack-cell1.openstack.svc/nova_cell1?read_default_file=/etc/my.cnf |  False   |
+-------+--------------------------------------+-----------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+----------+

MariaDBAccount and MariaDBDatabase is not deleted

❯ oc get MariaDBAccount | grep nova
nova-api     18h
nova-cell0   18h
nova-cell1   18h
❯ oc get MariaDBDatabase | grep nova
nova-api     18h
nova-cell0   18h
nova-cell1   18h
 
❯ oc get MariaDBAccount/nova-cell1 -o yaml
apiVersion: mariadb.openstack.org/v1beta1
kind: MariaDBAccount
metadata:
  creationTimestamp: "2024-09-02T15:04:44Z"
  finalizers:
  - openstack.org/nova
  - openstack.org/mariadbaccount
  generation: 1
  labels:
    mariaDBDatabaseName: nova-cell1
  name: nova-cell1
  namespace: openstack
 
 
❯ oc get MariaDBDatabase/nova-cell1 -o yaml
apiVersion: mariadb.openstack.org/v1beta1
kind: MariaDBDatabase
metadata:
  creationTimestamp: "2024-09-02T15:04:44Z"
  finalizers:
  - openstack.org/nova
  - openstack.org/mariadbdatabase
  - openstack.org/mariadbaccount-nova-cell1
  generation: 1
  labels:
    dbName: openstack-cell1
  name: nova-cell1
  namespace: openstack

templates/nova-manage/bin/delete_cell.sh

gibizer

I found more leftovers. See inline.

controllers/nova_controller.go

gibizer · 2024-09-03T12:03:48Z

Also we need to call cleanServiceFromNovaDb in the reconcileDelete of the NovaConductor to ensure that the nova service is also deleted form the nova api DB.

softwarefactory-project-zuul · 2024-09-03T13:53:35Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/2009990370ba428c97acb4d073f5198b

✔️ openstack-meta-content-provider SUCCESS in 2h 31m 38s
❌ nova-operator-kuttl FAILURE in 49m 15s
✔️ nova-operator-tempest-multinode SUCCESS in 2h 14m 02s
❌ nova-operator-tempest-multinode-ceph FAILURE in 1h 21m 37s

softwarefactory-project-zuul · 2024-09-05T20:14:58Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/c15b74bd5d604f8885fe61832ad48869

✔️ openstack-meta-content-provider SUCCESS in 3h 39m 35s
❌ nova-operator-kuttl FAILURE in 47m 51s
✔️ nova-operator-tempest-multinode SUCCESS in 2h 16m 15s
❌ nova-operator-tempest-multinode-ceph FAILURE in 2h 29m 23s

softwarefactory-project-zuul · 2024-09-06T10:59:10Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/9b2f094508b3481bb79b652b9a624e3b

✔️ openstack-meta-content-provider SUCCESS in 2h 47m 22s
❌ nova-operator-kuttl FAILURE in 50m 14s
✔️ nova-operator-tempest-multinode SUCCESS in 1h 55m 46s
✔️ nova-operator-tempest-multinode-ceph SUCCESS in 2h 31m 27s

controllers/nova_controller.go

softwarefactory-project-zuul · 2024-10-04T15:55:08Z

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/openstack-k8s-operators/nova-operator for 846,70ff3a21897276fed1cf76b23467c226c6f86575

We need to delete only nova finalizer

gibizer · 2024-10-04T15:57:53Z

rebased on main
removed the vendored change and calling add DeleteDatabaseAndAccountdFinalizers mariadb-operator#268 directly
fixed potential bugs when cleanup needs to be retried
refactored a bit.

gibizer

I consider this ready now (except the mariadb-operator dependency). I cannot approve it as I wrote a sizeable chunk of fixes for it.

gibizer

Opened trackers for the remaining items

gibizer · 2024-10-07T12:49:31Z

test/kuttl/test-suites/default/cell-tests/02-assert.yaml

+  - script: |
+      oc get -n nova-kuttl-default Secret/nova-cell1-manage-config-data || exit 0
+      exit 1
+  # TODO(gibi): this needs an openstack-operator change to clean up the cell1


This is tracked in https://issues.redhat.com/browse/OSPRH-10549 now

controllers/nova_controller.go

Both CR is created by the nova controller for the given cell, so when that cell is deleted the CRs also needs to be deleted. We also need to delete the secret connected to the MariaDBAccount as mariadb-operator does not delete it today. It also added graceful handling of NotFound during resource deletion and refactored the cleanup logic to avoid bugs if the delete needs to be retried. This now depends on the openstack-k8s-operators/mariadb-operator#268 instead of copying that change.

gibizer · 2024-10-08T11:28:18Z

the mariadb-operator dependency now landed, so for me this is ready to go.

zzzeek · 2024-10-08T13:16:59Z

I'm assuming the nova cells use case is unusual enough that the routines here are still needing to deal with MariaDBAccount and MariaDBDatabase explicitly, rather than using the new DeleteDatabaseAndAccountFinalizers routine, is that correct?

gibizer · 2024-10-08T13:55:46Z

I'm assuming the nova cells use case is unusual enough that the routines here are still needing to deal with MariaDBAccount and MariaDBDatabase explicitly, rather than using the new DeleteDatabaseAndAccountFinalizers routine, is that correct?

Yes. We do call DeleteDatabaseAndAccountFinalizers but we also need to explicitly delete the MariaDBDatabase CR afterwards as it is owned by the Nova CR but the Nova CR is not going away, just the NovaCell CR, so we cannot rely on the k8s garbage collector to delete the MariaDBDatabase.

gibis concerns have either been adress, are tracked by a follow up jira issue or we have delegated part of the db handeling to the maraidb operator so clearing this stale review

openshift-ci · 2024-10-10T00:03:06Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mrkisaolamb, SeanMooney

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [SeanMooney,mrkisaolamb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci bot requested review from olliewalsh and SeanMooney August 19, 2024 13:23

openshift-ci bot added the approved label Aug 19, 2024

mrkisaolamb requested review from gibizer and removed request for olliewalsh August 19, 2024 13:23

mrkisaolamb added the do-not-merge/work-in-progress label Aug 19, 2024

mrkisaolamb changed the title ~~Add deleting cells automation~~ WIP: Add deleting cells automation Aug 19, 2024

mrkisaolamb force-pushed the cell_delete branch 3 times, most recently from f67ed5e to 8fc5a10 Compare August 22, 2024 13:16

mrkisaolamb changed the title ~~WIP: Add deleting cells automation~~ Add deleting cells automation Aug 22, 2024

openshift-ci bot removed the do-not-merge/work-in-progress label Aug 22, 2024

SeanMooney reviewed Aug 22, 2024

View reviewed changes

templates/nova-manage/bin/delete_cell.sh Outdated Show resolved Hide resolved

gibizer requested changes Sep 2, 2024

View reviewed changes

templates/nova-manage/config/cell-delete-config.json Outdated Show resolved Hide resolved

templates/nova-manage/bin/delete_cell.sh Outdated Show resolved Hide resolved

openshift-ci bot assigned gibizer Sep 2, 2024

mrkisaolamb force-pushed the cell_delete branch from 6c7718b to c7e80ca Compare September 2, 2024 14:32

mrkisaolamb force-pushed the cell_delete branch from c7e80ca to 8d09b32 Compare September 2, 2024 14:53

mrkisaolamb requested a review from gibizer September 3, 2024 06:21

gibizer reviewed Sep 3, 2024

View reviewed changes

templates/nova-manage/bin/delete_cell.sh Outdated Show resolved Hide resolved

gibizer requested changes Sep 3, 2024

View reviewed changes

controllers/nova_controller.go Show resolved Hide resolved

mrkisaolamb force-pushed the cell_delete branch from 8d09b32 to eb8a8b0 Compare September 5, 2024 15:25

mrkisaolamb force-pushed the cell_delete branch from eb8a8b0 to b316362 Compare September 6, 2024 08:10

gibizer reviewed Oct 4, 2024

View reviewed changes

controllers/nova_controller.go Outdated Show resolved Hide resolved

controllers/nova_controller.go Outdated Show resolved Hide resolved

gibizer reviewed Oct 4, 2024

View reviewed changes

controllers/nova_controller.go Outdated Show resolved Hide resolved

gibizer force-pushed the cell_delete branch from 48f7fb3 to 70ff3a2 Compare October 4, 2024 15:55

openshift-merge-robot added the needs-rebase label Oct 4, 2024

mrkisaolamb added 7 commits October 4, 2024 17:56

Add remove cell logic

c3082cd

Add job to clear cell from db

d829bf9

Add kuttle test for cell deletion

b896753

Correct cell deletion logic

f4765e4

Move cell-delete test

1f6c1f6

Delete additional cell delete job

2b92841

We need to delete only nova finalizer

Reorganize kuttl tests for nova-cell-deletion

83c08b0

gibizer force-pushed the cell_delete branch from 70ff3a2 to 539669a Compare October 4, 2024 15:56

openshift-merge-robot removed the needs-rebase label Oct 4, 2024

gibizer self-requested a review October 4, 2024 15:59

gibizer reviewed Oct 7, 2024

View reviewed changes

gibizer added 2 commits October 8, 2024 13:27

[kuttl]Assert DB and Secrets are deleted for deleted cell

37f25bd

gibizer force-pushed the cell_delete branch from e6aefd9 to 37f25bd Compare October 8, 2024 11:27

SeanMooney approved these changes Oct 10, 2024

View reviewed changes

openshift-ci bot assigned SeanMooney Oct 10, 2024

openshift-ci bot added the lgtm label Oct 10, 2024

openshift-merge-bot bot merged commit 0721f8b into openstack-k8s-operators:main Oct 10, 2024
7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add deleting cells automation #846

Add deleting cells automation #846

mrkisaolamb commented Aug 19, 2024 •

edited by openshift-ci bot

Loading

mrkisaolamb commented Aug 19, 2024

gibizer left a comment

softwarefactory-project-zuul bot commented Sep 2, 2024

mrkisaolamb commented Sep 3, 2024

gibizer commented Sep 3, 2024

gibizer left a comment

gibizer commented Sep 3, 2024

softwarefactory-project-zuul bot commented Sep 3, 2024

softwarefactory-project-zuul bot commented Sep 5, 2024

softwarefactory-project-zuul bot commented Sep 6, 2024

softwarefactory-project-zuul bot commented Oct 4, 2024

gibizer commented Oct 4, 2024

gibizer left a comment

gibizer left a comment

gibizer Oct 7, 2024

gibizer commented Oct 8, 2024

zzzeek commented Oct 8, 2024

gibizer commented Oct 8, 2024

openshift-ci bot commented Oct 10, 2024

Add deleting cells automation #846

Add deleting cells automation #846

Conversation

mrkisaolamb commented Aug 19, 2024 • edited by openshift-ci bot Loading

mrkisaolamb commented Aug 19, 2024

gibizer left a comment

Choose a reason for hiding this comment

softwarefactory-project-zuul bot commented Sep 2, 2024

mrkisaolamb commented Sep 3, 2024

gibizer commented Sep 3, 2024

gibizer left a comment

Choose a reason for hiding this comment

gibizer commented Sep 3, 2024

softwarefactory-project-zuul bot commented Sep 3, 2024

softwarefactory-project-zuul bot commented Sep 5, 2024

softwarefactory-project-zuul bot commented Sep 6, 2024

softwarefactory-project-zuul bot commented Oct 4, 2024

gibizer commented Oct 4, 2024

gibizer left a comment

Choose a reason for hiding this comment

gibizer left a comment

Choose a reason for hiding this comment

gibizer Oct 7, 2024

Choose a reason for hiding this comment

gibizer commented Oct 8, 2024

zzzeek commented Oct 8, 2024

gibizer commented Oct 8, 2024

openshift-ci bot commented Oct 10, 2024

mrkisaolamb commented Aug 19, 2024 •

edited by openshift-ci bot

Loading