Feature: replace aissemble-kafka 1.7.0 references with community docker image

[d-ryan-ashcraft]

Description

The aissemble-kafka docker image had minimal impact and was removed as part of feature #221. We purposely left helm references to the aissemble 1.7.0 container until this ticket migrates this reference to an appropriate community chart and validates that it also still works.

DOD

• Replace aissemble-kafka docker image references with the chosen community replacement (apache/kafka).
  • Ensure the replacement does NOT break the build
• Develop migration script for existing values.yaml that is using the old listeners configuration for Kafka Helm charts.
• Update release notes indicating kafka docker image has been migrated over to the community docker image.

Stretch Goal:

• Upgrade Kafka Helm chart to the latest stable version.
  • This was removed out of this ticket.

Test Strategy/Script

OTS Test:

1. Generate a downstream project using the following maven command: mvn archetype:generate -DarchetypeGroupId=com.boozallen.aissemble -DarchetypeArtifactId=foundation-archetype -DarchetypeVersion=1.9.0-SNAPSHOT -DgroupId=com.kafkatest -DartifactId=kafkatest -DprojectGitUrl=url -DprojectName=kafkatest && cd kafkatest.
2. Place the Spark pipeline metamodel in the following path: kafkatest/kafkatest-pipeline-models/src/main/resources/pipelines
3. Run mvn clean install until all the manual actions have been resolved.
4. Navigate to kafkatest/kafkatest-deploy/src/main/resources/apps/kafka-cluster/Chart.yaml and replace the image repository with the following: repository: file:///_path to aissemble_/aissemble/extensions/extensions-helm/aissemble-kafka-chart.
5. Navigate to aissemble and change the version and appVersion 1.9.0-SNAPSHOT in /_path to aissemble_/aissemble/extensions/extensions-helm/aissemble-kafka-chart/Chart.yaml.
6. Run the following in the aiSSEMBLE repo: mvn clean install -pl :aissemble-kafka-chart.
7. Go back to the downstream project and re-run mvn clean install.
8. Ensure there are no more manual actions, then run tilt up; tilt down.
9. Navigate to the tilt dashboard to the kafka-cluster-controller pod and verify the docker image is being pulled from: http://docker.io/bitnami/kafka:3.5.1-debian-11-r1.

Test 1: Ensure downstream project that is using kafka builds the community docker image.

1. Generate a downstream project using the following maven command: mvn archetype:generate -DarchetypeGroupId=com.boozallen.aissemble -DarchetypeArtifactId=foundation-archetype -DarchetypeVersion=1.9.0-SNAPSHOT -DgroupId=com.kafkatest -DartifactId=kafkatest -DprojectGitUrl=url -DprojectName=kafkatest && cd kafkatest
2. Place the attached Spark pipeline metamodel in the following path: kafkatest/kafkatest-pipeline-models/src/main/resources/pipelines
3. Run mvn clean install until all the manual actions have been resolved.
4. Ensure there are no more manual actions, then run tilt up; tilt down.
5. Navigate to the tilt dashboard to the kafka-cluster-controller pod and verify the docker image is being pulled from: http://docker.io/bitnami/kafka:3.5.1-debian-11-r1.

Test 2: Upgrade downstream project versions.

1. Download the attached downstream project (kafka2.zip).
2. Run mvn clean install until all the manual actions have been resolved.
3. Once all the manual actions are completed (there shouldn't be any), navigate to the root pom.xml and change the version from 1.8.2 to 1.9.0-SNAPSHOT.
4. Run mvn clean install until all the manual actions are satisfied.
5. Ensure there are no more manual actions, then run tilt up; tilt down.
6. Navigate to the tilt dashboard to the kafka-cluster-controller pod and verify the docker image is being pulled from: http://docker.io/bitnami/kafka:3.5.1-debian-11-r1.

References/Additional Context

• Docker Image for Apache Kafka
• Kafka Helm Chart
• Kafka Helm Chart upgrade breaking changes

[habibimoiz]

DoD reviewed and approved by @meliz19.

[habibimoiz]

Testing files:

kafkatest2.zip
SparkPipeline.json

[habibimoiz]

• Updated the Kafka Helm chart to pull community docker image.
• Upgraded the Kafka Helm chart 23.0.7 to 30.1.1.

[habibimoiz]

OTS completed w/ @meliz19 & @jaebchoi.

[habibimoiz]

Based on findings, the stretch goal of this ticket will be remove as part of the PR #348 and the changes will continue to only consist of migrating the Kafka Docker image to the community maintained Docker image due to major changes requires by the Kafka Helm chart upgrade (per bitnami/charts#17507).

[cpointe-ibllanos]

Fully tested. 6 day lead time.