Create index image from pre-built bundles (#597)

* Create index image from pre-built bundles

Deploy from bundles can now be used along with
deploy from index enabled.

This way, an index image can be created with
specific pre built bundles and deploy STF with
those.

* Fix set_fact syntax for SGO/SGO bundle info dicts

* Add ImageStreams for STO and SGO

* Fix minor typo in STO/SGO bundle info dict

* Set correct STO and SGO bundles path

We need to use the internal registry address
to build the index image that will serve
as base for the catalog

* Drop build number from SGO and STO bundles

We only need the tag when creating the index image
Keeping this value will generate errors

* Get STO and SGO tags from the bundle path

* Fix typo on SGO and STO images path

* Update README with bundle + index instructions

Add more details on how to deploy from index with
pre built bundles

* Use stf_channel on the deploy_stf step

Define this value to stable-1.5 when deploying
from bundles and index, unstable (the default)
if bundles have been built locally.

Also, honor the sgo and sto bundles tags. Ideally,
we would update the sgo and sto image path to be
only the path (without the tag) and define the tag
separately. In this case, it is responsibility
from the user to make sure that the tag in the
image path and the tag provided in sto and sgo
image path match when deploying from externally
built bundles and index.

* Update missing reference to SGO bundles channel

* Update some logic paths to simplify the execution

* Honor the stf_channel variable

Nightly bundles use the "unstable" channel while
RH catalog bundles use "stable-1.5" channel

We should use nightly bundles "unstable" by default
and pass "stable-1.5" when deploying from RH catalog

* Get opm for getting bundles information

We need opm to get the bundle version when deploying
from index with pre-built bundles

Get opm following the same logic we use for operator-sdk

* Get the operator bundle version for STO and SGO

Inspect the provided bundles with opm render and
obtain the operator bundle version for each of them.

* Set a default version for opm

Let's stick to latest-4.14 for now, with the option
of updating to a different one if needed

* Fix typo when executing opm render

* Revert "Fix typo when executing opm render"

This reverts commit 1050227.

* Revert "Set a default version for opm"

This reverts commit d6a51df.

* Revert "Get the operator bundle version for STO and SGO"

This reverts commit c1d9db3.

* Revert "Get opm for getting bundles information"

This reverts commit 642df50.

* Get STO and SGO operator bundle version using oc

oc image info command provides operator bundle
version information. Use this to get the proper
version numbers when building the index image

* Get STO and SGO bundle versions from info

Parse the oc image info output to get
the STO and SGO bundle versions

* Add extra logic to avoid variables overwrite

sto_bundle_info and sgo_bundle_info is overriding
even though the step is skipped

Add an extra logic check to avoid variables
override

build/stf-run-ci/README.md

@@ -20,7 +20,7 @@ choose to override:
 | `__deploy_stf`                                                       | {true,false}                                                | true                                                                          | Whether to deploy an instance of STF                                                                                                                                                                                    |
 | `__local_build_enabled`                                              | {true,false}                                                | true                                                                          | Whether to deploy STF from local built artifacts. Also see `working_branch`, `sg_branch`, `sgo_branch`                                                                                                                  |
 | `__deploy_from_bundles_enabled`                                      | {true,false}                                                | false                                                                         | Whether to deploy STF from OLM bundles (TODO: compat with `__local_build_enabled`)                                                                                                                                      |
-| `__deploy_from_index_enabled`                                        | {true,false}                                                | false                                                                         | Whether to deploy STF from locally built bundles and index image.                                                                                                                                                       |
+| `__deploy_from_index_enabled`                                        | {true,false}                                                | false                                                                         | Whether to deploy STF from locally built bundles/OLM bundles and index image.                                                                                                                                                       |
 | `__disconnected_deploy`                                              | {true,false}                                                | false                                                                         | Whether to deploy on a disconnected cluster                                                                                                                                                                               |
 | `__service_telemetry_bundle_image_path`                              | <image_path>                                                | `quay.io/infrawatch-operators/service-telemetry-operator-bundle:nightly-head` | Image path to Service Telemetry Operator bundle                                                                                                                                                                         |
 | `__smart_gateway_bundle_image_path`                                  | <image_path>                                                | `quay.io/infrawatch-operators/smart-gateway-operator-bundle:nightly-head`     | Image path to Smart Gateway Operator bundle                                                                                                                                                                             |
@@ -83,9 +83,10 @@ choose to override:
 You can deploy Service Telemetry Framework using this role in a few
 configuration methods:
 
-* local build artifacts from Git repository cloned locally
-* local build artifacts, local bundle artifacts, and Subscription via OLM using locally built index image
-* standard deployment using Subscription and OLM
+* local build artifacts from Git repository cloned locally (local build)
+* local build artifacts, local bundle artifacts, and Subscription via OLM using locally built index image (local build + deploy from index)
+* externally build bundle artifacts and Subscription via OLM using locally built index image (deploy from bundles + deploy from index)
+* standard deployment using Subscription and OLM (deploy from bundles)
 * supporting components but no instance of Service Telemetry Operator
 
 ## Basic deployment
@@ -134,6 +135,27 @@ You can perform a deployment using OLM and a Subscription from locally built art
 ansible-playbook -e __local_build_enabled=true -e __deploy_from_index_enabled=true run-ci.yaml
 ```
 
+## Deployment with pre-build bundles and index
+
+Instead of relying on the operator-sdk to deploy from selected bundles using the "operator-sdk run bundle" utility,
+you can perform a deployment using OLM and a Subscription to a locally created index image like this:
+
+```sh
+ansible-playbook -e __local_build_enabled=false -e __deploy_from_bundles_enabled=true \
+  -e __deploy_from_index_enabled=true \
+  -e __service_telemetry_bundle_image_path=<registry>/<namespace>/stf-service-telemetry-operator-bundle:<tag> \
+  -e __smart_gateway_bundle_image_path=<registry>/<namespace>/stf-smart-gateway-operator-bundle:<tag> \
+  -e pull_secret_registry=<registry> \
+  -e pull_secret_user=<username> \
+  -e pull_secret_pass=<password>
+  run-ci.yaml
+```
+
+Since you will fetch the selected images from a bundle registry, it is required that you have all the required
+access credentials for the desired registry correctly configured. Check the "Deployment with pre-build bundles"
+docs above to get more information about this.
+
+
 # License
 
 Apache v2.0

build/stf-run-ci/tasks/create_catalog.yml

@@ -7,11 +7,65 @@
 
 # Updating to use stdout_lines[-1] so that any additional info that gets added to generate_bundles (e.g. for debug) doesn't break this task
 # Adding from_json so that the JSON output is parsed into a dictionary
-- name: Create info variables from bundle generation output
+- name: Create info variables from bundle generation output (local build)
+  when: __local_build_enabled | bool and not __deploy_from_bundles_enabled | bool
   ansible.builtin.set_fact:
     sto_bundle_info: "{{ generate_bundle_sto.stdout_lines[-1] | from_json }}"
     sgo_bundle_info: "{{ generate_bundle_sgo.stdout_lines[-1] | from_json }}"
 
+- name: Create info variables from provided pre-built bundles (deploy from bundles)
+  when: __deploy_from_bundles_enabled | bool and not __local_build_enabled | bool
+  block:
+    - name: Get STO operator bundle info
+      ansible.builtin.command: oc image info {{ __service_telemetry_bundle_image_path }}
+      register: sto_prebuilt_image_info
+
+    - name: Get SGO operator bundle info
+      ansible.builtin.command: oc image info {{ __smart_gateway_bundle_image_path }}
+      register: sgo_prebuilt_image_info
+
+    - name: Get STO and SGO bundle versions
+      ansible.builtin.set_fact:
+        sto_prebuilt_bundle_version: "{{ sto_prebuilt_image_info.stdout_lines[-1] | split('=') | last }}"
+        sgo_prebuilt_bundle_version: "{{ sgo_prebuilt_image_info.stdout_lines[-1] | split('=') | last }}"
+
+    - name: Set info variables from provided pre-built bundles
+      ansible.builtin.set_fact:
+        sto_bundle_info:
+          'bundle_default_channel': "{{ stf_channel }}"
+          'bundle_channels': "{{ stf_channel }}"
+          'operator_bundle_version': "{{ sto_prebuilt_bundle_version }}"
+        sgo_bundle_info:
+          'bundle_default_channel': "{{ stf_channel }}"
+          'bundle_channels': "{{ stf_channel }}"
+          'operator_bundle_version': "{{ sgo_prebuilt_bundle_version }}"
+
+- name: Show STO and SGO bundle info that will used in the index image
+  ansible.builtin.debug:
+    msg:
+    - "{{ sto_bundle_info }}"
+    - "{{ sgo_bundle_info }}"
+
+- name: Create ImageStream for STO and SGO (deploying from bundles)
+  when: __deploy_from_bundles_enabled | bool
+  block:
+    - name: Set correct STO and SGO bundle paths when deploying from index with pre-built bundles
+      ansible.builtin.set_fact:
+        sto_bundle_image_path: "{{ __internal_registry_path }}/{{ namespace }}/service-telemetry-operator-bundle:{{ sto_bundle_image_tag }}"
+        sgo_bundle_image_path: "{{ __internal_registry_path }}/{{ namespace }}/smart-gateway-operator-bundle:{{ sgo_bundle_image_tag }}"
+
+    - name: Create ImageStream for STO
+      ansible.builtin.command:
+        cmd: |
+          oc import-image -n {{ namespace }} service-telemetry-operator-bundle:{{ sto_bundle_image_tag }} --from={{ __service_telemetry_bundle_image_path }} --confirm --insecure
+      register: sto_is
+
+    - name: Create ImageStream for SGO
+      ansible.builtin.command:
+        cmd: |
+          oc import-image -n {{ namespace }} smart-gateway-operator-bundle:{{ sgo_bundle_image_tag }} --from={{ __smart_gateway_bundle_image_path }} --confirm --insecure
+      register: sgo_is
+
 - name: Get the builder-dockercfg Secret name
   ansible.builtin.command: oc get secret -n {{ namespace }} --field-selector='type==kubernetes.io/dockercfg' -ojsonpath='{.items[?(@.metadata.annotations.kubernetes\.io/service-account\.name=="builder")].metadata.name}'
   register: secret_builder_dockercfg_name

build/stf-run-ci/tasks/main.yml

@@ -28,27 +28,6 @@
     - __local_build_enabled | bool
     - __deploy_from_bundles_enabled | bool
 
-- name: Fail when deploying from index image and local build disabled
-  ansible.builtin.fail:
-    msg: __deploy_from_index_enabled must also have __local_build_enabled
-  when:
-   - __deploy_from_index_enabled | bool
-   - not __local_build_enabled | bool
-
-- name: Fail when deploying from index images and deployment from bundles also requested (mutually exclusive methods)
-  ansible.builtin.fail:
-    msg: __deploy_from_index_enabled can not be used with __deploy_from_bundles_enabled
-  when:
-   - __deploy_from_index_enabled | bool
-   - __deploy_from_bundles_enabled | bool
-
-- name: Fail when disconnected deploy and other deployment options also requested
-  ansible.builtin.fail:
-    msg: __disconnected_deploy cannot be used if __deploy_from_bundles_enabled, __deploy_from_index_enabled or __local_build_enabled
-  when:
-   - __disconnected_deploy | bool
-   - __deploy_from_bundles_enabled | bool or __deploy_from_index_enabled | bool or __local_build_enabled | bool 
-
 - name: Get the list of nodes
   kubernetes.core.k8s_info:
     kind: Node
@@ -159,7 +138,7 @@
           pod-security.kubernetes.io/audit: restricted
           pod-security.kubernetes.io/warn: restricted
 
-- when: __deploy_from_index_enabled | bool
+- when: __deploy_from_index_enabled | bool and __local_build_enabled | bool
   tags:
     - create_bundles
   block:
@@ -180,11 +159,13 @@
       tags:
         - build
 
+- when: __deploy_from_index_enabled | bool
+  block:
     - name: Create file-based catalog
       ansible.builtin.include_tasks: create_catalog.yml
 
 # -- deploy
-- when: not __local_build_enabled | bool
+- when: not __local_build_enabled | bool and not __deploy_from_index_enabled | bool
   block:
   - name: Setup Service Telemetry Framework from supplied bundle URLs
     ansible.builtin.include_tasks: setup_stf_from_bundles.yml
@@ -206,7 +187,7 @@
         name: service-telemetry-operator
         namespace: "{{ namespace }}"
       spec:
-        channel: unstable
+        channel: "{{ stf_channel }}"
         installPlanApproval: Automatic
         name: service-telemetry-operator
         source: service-telemetry-framework-operators