Feature/productionize image #24

leiicamundi · 2024-03-05T17:08:49Z

Pull Request Description:
This pull request addresses the need for a production-ready Camunda-flavored Keycloak Docker image that supports AWS IAM roles for Service Accounts (IRSA) out of the box. Currently, customers are required to create their own Keycloak Docker image for AWS with IRSA support, and this enhancement aims to simplify this process.

Problem Description:
Customers running Keycloak in AWS with IRSA face challenges in creating custom Docker images. This PR introduces a Camunda-flavored Keycloak image for both ARM and AMD architectures, facilitating easier deployment in AWS environments.

Proposed Solution:

Renames the existing repository from https://github.com/camunda/keycloak-aws to camunda/keycloak to accommodate multi-cloud support in the future (already implemented, repository is now public).
Adds Open Container Initiative (OCI) labels to the Docker image for enhanced metadata.
Establishes a release pipeline and strategy that supports all currently supported Keycloak versions for ARM and AMD architectures.
Publishes the Docker image to DockerHub, aligning with the Infrastructure team's guidelines.
Creates an integration test within https://github.com/camunda/keycloak-aws to validate AWS usage via self-hosted runners, specifically testing IRSA against existing shared Aurora databases.

Desired Outcome:

Repository renamed to camunda/keycloak.
Docker image includes OCI labels.
Release pipeline supports all currently supported Keycloak versions for ARM and AMD architectures.
Docker image is published to DockerHub.
Integration test validates AWS usage with IRSA against shared Aurora databases.

Implementation Details:

Multi-cloud support implemented.
Integration test added for AWS usage.
Multi-architecture image CI build.
Added static image analysis in CI using Snyk.
DockerHub project set up for Keycloak images, with token for image publishing => currently waiting for the official repo (tested w/ https://hub.docker.com/r/leiicamundi/keycloak-camunda-test).
Nightly builds scheduled by cron.
Pre-commit tests integrated with image linting (hadolint and YAML check).
Harbor retention policy for CI implemented.
Renovate enabled for automated dependency updates.

Implementation Details:

In addition to the previously mentioned implementation details, a dynamic test matrix is now generated for all Keycloak versions. Each version undergoes a suite of tests on GCP, AWS, and Docker Compose environments.

The testing process follows these steps:

Dynamic Test Matrix:
- A matrix is dynamically generated to cover various Keycloak versions.
- Each version undergoes tests on GCP, AWS, and Docker Compose environments to ensure cross-cloud compatibility.
Semantic Versioning and Image Tagging:
- Images are tagged with semantic versioning for easy identification and version tracking.
- The following tags are used when publishing images on docker.io:
  - keycloak-<version>-latest: Represents the latest version.
  - keycloak-<version>-<yyyy-mm-dd-xxx>: Timestamped version for historical reference and traceability.
  - latest: Used if the version is the most recent.
Publishing Process:
- After passing all tests and completing the security scan, the images are automatically published on docker.io with the appropriate semantic tags.
Pending Tasks:
- Efforts are ongoing to enhance documentation.
- Further refinement of the publication process is needed.

PS: This PR will be squashed merged due to the number of commits

add pre-commit in the project add a pre-commit gh action add multi arch steps change names disable load init labels re-order steps of workflows use dockerhub push image and fix wkflw fix ci update ci update ci remove redundant OCI labels as it's set by builder update README

.github/workflows/build-images.yml

Langleu

I had another look over the action.
The remaining files looked good / Maxim already left some comments.

gsed not present in the runner

leiicamundi · 2024-03-13T15:32:44Z

Thanks @maxdanilov and @Langleu for the review.
The last blocking point is regarding usage of snyk action.

Currently, it is not a blocking action (so even if snyk reports CVEs, we will still be able to merge/release). I think it could help but as discussed with Lars, it could be a little overkilled.

@maxdanilov we'd like to hear your opinion on this, then the PR should be ready to be merged

maxdanilov · 2024-03-13T16:30:11Z

@leiicamundi I don't have a strong opinion, but more in favor of keeping it light for now and drop it.
You could move the Snyk logic addition to a separate PR (that will be closed) so we can have a reference implementation for the future, should we rethink the scanning approach later, wdyt?

maxdanilov

Looks good in general, thank you 🚀
Just leaving small final styling suggestions + discovered hiccup with the pre-commit hook when running it locally.

README.md

.pre-commit-config.yaml

.github/scripts/utils/find_latest_keycloak.sh

.pre-commit-hooks/update_readme_version.sh

unified scripts naming with snake case fix pre-commit runs update README

leiicamundi · 2024-03-14T07:25:13Z

Hello @maxdanilov, thank you for the review.

I have implemented the suggestions and also extracted the Snyk scan into its own issue (#25).

I am unable to reproduce the pre-commit bug. Could you please retry with the latest version of the repository? Otherwise, I will likely switch back to the Python script, as it seems to work without the need to consider the underlying platform.

Langleu

looks good from my side.

maxdanilov

All good, just last suggestion for fixing the colored output of ls in one of the scripts so the pre-commit hook works consistently for all users.

.github/scripts/utils/find_latest_keycloak.sh

leiicamundi · 2024-03-15T08:43:11Z

Hey @Langleu, @maxdanilov, thank you for the reviews!

Now that everything is aligned, I'm merging the PR.

We'll continue to keep track of the remaining items to process at https://github.com/orgs/camunda/projects/87/views/1?pane=issue&itemId=47441722.

leiicamundi added 30 commits February 26, 2024 14:37

update labels

e0576a8

fix ci

72746da

add multiarch action

bed4f4e

replace workflow_run by workflow call due to main branch limitation

56b4f68

fix path

03f4371

fix call

c1e7543

fix yaml

aca2004

merge workflows

ae71701

update image name

5479971

fix ci

7d8ed02

fix ci

6e48afe

fix tr

7c97627

fix ci

9806540

fix quotes

0a4dd26

fix outputs

d6a5178

fix ci

3d753de

add matrix runner desc for different envs

af55bf2

fix version

584e53f

init complex runners

f4f7adb

add harbor integ

c3d0388

fix templating

ab34a1a

fix templating

9b3b815

fix password

ea28a7e

only take first

e11ceac

update db host templating

ef2258b

fix tear down aws

58d700c

integrate aurora

dae95fd

fix wrong aurora script name

c5bc884

fix template

e08d702

Langleu reviewed Mar 13, 2024

View reviewed changes

.github/workflows/build-images.yml Outdated Show resolved Hide resolved

Langleu reviewed Mar 13, 2024

View reviewed changes

.github/workflows/build-images.yml Outdated Show resolved Hide resolved

Langleu requested changes Mar 13, 2024

View reviewed changes

leiicamundi added 12 commits March 13, 2024 11:49

applied PR suggestions

ad606a6

use sed instead of gsed

21acf0c

gsed not present in the runner

prevent workflows to run twice in prs

09ff8be

debug

f5da318

apply separation of steps

d0ac189

fix sed linux

d0f7990

debug find latest

3d062d5

continue to debug

215755b

finish debug

056b342

fix external test

d9c9281

fix password

4432f55

fix secret

67b0f1c

leiicamundi requested review from Langleu and maxdanilov March 13, 2024 15:32

maxdanilov requested changes Mar 13, 2024

View reviewed changes

leiicamundi added 2 commits March 14, 2024 08:10

apply pr suggestions

8b33b6c

unified scripts naming with snake case fix pre-commit runs update README

remove snyk integration from this pr

d79a658

use [[ instead of [

eaef286

Langleu approved these changes Mar 14, 2024

View reviewed changes

maxdanilov approved these changes Mar 14, 2024

View reviewed changes

.github/scripts/utils/find_latest_keycloak.sh Outdated Show resolved Hide resolved

fix(pre-commit): ensure ls no color

4f18a09

leiicamundi merged commit 6a15db2 into main Mar 15, 2024
17 checks passed

leiicamundi deleted the feature/productionize-image branch March 15, 2024 08:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/productionize image #24

Feature/productionize image #24

leiicamundi commented Mar 5, 2024 •

edited

Loading

Langleu left a comment

leiicamundi commented Mar 13, 2024

maxdanilov commented Mar 13, 2024 •

edited

Loading

maxdanilov left a comment •

edited

Loading

leiicamundi commented Mar 14, 2024

Langleu left a comment

maxdanilov left a comment

leiicamundi commented Mar 15, 2024 •

edited

Loading

Feature/productionize image #24

Feature/productionize image #24

Conversation

leiicamundi commented Mar 5, 2024 • edited Loading

Langleu left a comment

Choose a reason for hiding this comment

leiicamundi commented Mar 13, 2024

maxdanilov commented Mar 13, 2024 • edited Loading

maxdanilov left a comment • edited Loading

Choose a reason for hiding this comment

leiicamundi commented Mar 14, 2024

Langleu left a comment

Choose a reason for hiding this comment

maxdanilov left a comment

Choose a reason for hiding this comment

leiicamundi commented Mar 15, 2024 • edited Loading

leiicamundi commented Mar 5, 2024 •

edited

Loading

maxdanilov commented Mar 13, 2024 •

edited

Loading

maxdanilov left a comment •

edited

Loading

leiicamundi commented Mar 15, 2024 •

edited

Loading