Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement image builder database maintenance (HMS-4244) #1402

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

schuellerf
Copy link
Contributor

@schuellerf schuellerf commented Nov 14, 2024

This implements the retention policy requested in https://issues.redhat.com/browse/HMS-4244

image-builder-maintenance for now runs vacuum
and cleans all composes older than 2 years (by default)
to remove customer related data according to the
retention policy

As this is not really urgent, it's scheduled once
a week on Tuesdays to avoid problems on weekends

…should be merged after #1398
Note: this implementation is almost a copy of https://github.com/osbuild/osbuild-composer/tree/main/cmd/osbuild-service-maintenance to stick to the same mechanism

@schuellerf schuellerf force-pushed the HMS-4244-implement-image-builder-database-maintenance branch from 8e097b5 to 12c937d Compare November 14, 2024 12:58
Copy link

codecov bot commented Nov 14, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

@schuellerf schuellerf force-pushed the HMS-4244-implement-image-builder-database-maintenance branch from 12c937d to 0e26d6e Compare November 14, 2024 13:06
@schuellerf schuellerf force-pushed the HMS-4244-implement-image-builder-database-maintenance branch from 6af80e6 to dc5cafe Compare November 15, 2024 10:50
@schuellerf
Copy link
Contributor Author

Note: failed code/snyk check just indicates that in our testcases we have a password hardcoded …

lzap
lzap previously approved these changes Nov 15, 2024
Copy link
Collaborator

@lzap lzap left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great. The only concern I have is why this is a separate image, but this could be app-sre requirement, no issues.

cmd/image-builder-maintenance/config.go Outdated Show resolved Hide resolved
cmd/image-builder-maintenance/db.go Outdated Show resolved Hide resolved
Makefile Show resolved Hide resolved
build_deploy.sh Show resolved Hide resolved
cmd/image-builder-maintenance/main.go Show resolved Hide resolved
@schuellerf
Copy link
Contributor Author

schuellerf commented Nov 15, 2024

@lzap
I forgot to add this to the commit message that this, for now, is almost a copy of
https://github.com/osbuild/osbuild-composer/tree/main/cmd/osbuild-service-maintenance
to stick to the same approach.

@schuellerf
Copy link
Contributor Author

Refactoring this into a separate repo or somehow sharing code with composer was considered but discarded for less effort.
Still this discussion is valid and ongoing.

will be used in more database tests like the
image-builder-maintenance
@schuellerf schuellerf force-pushed the HMS-4244-implement-image-builder-database-maintenance branch 3 times, most recently from 918f709 to 255d1fd Compare November 19, 2024 09:16
image-builder-maintenance for now runs vacuum
and cleans all composes older than 2 years (by default)
to remove customer related data according to the
retention policy

As this is not really urgent, it's scheduled once
a week on Tuesdays to avoid problems on weekends
the behavior of flushing seems to have changed between
postgres 12 and the current (17) so we need a "CHECKPOINT"
to assure for vacuum to be tested correctly
for now we'll skip this test as it's not really
mission critical and should not stop us from delivering
the maintenance job.
instead of implementing our selfs, this should be a
more clean and feature rich solution
the test executable was called like the target program
@schuellerf schuellerf force-pushed the HMS-4244-implement-image-builder-database-maintenance branch from 255d1fd to 808d4ec Compare November 19, 2024 09:40
also introduces a github action test for the container
@schuellerf schuellerf force-pushed the HMS-4244-implement-image-builder-database-maintenance branch from 808d4ec to e5de212 Compare November 19, 2024 09:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants