Rename test workflow and test job to benchmark workflow and job

* Clean up empty sections to be ready to merge

Co-authored-by: AntonioDiTuri <[email protected]>
Signed-off-by: nikimanoledaki <[email protected]>

docs/proposals/proposal-002-run.md

@@ -1,18 +1,19 @@
 # Proposal 002 - Run the CNCF project benchmark tests as part of the automated pipeline
 
-This is step two from the automated pipeline which evaluates the carbon emissions of a CNCF project: run benchmarking tests of the project (probably created for this purpose).
-See also step 1: [Trigger and Deploy](https://github.com/cncf-tags/green-reviews-tooling/pull/88).
+This is step 2 from the automated pipeline to evaluate the carbon emissions of a CNCF project: Run benchmarking tests of the project (probably created for this purpose). See also step 1: [Trigger and Deploy](./proposal-001-trigger-and-deploy.md).
+
 - Tracking issue: [#83](https://github.com/cncf-tags/green-reviews-tooling/issues/83)
 - Implementation issue: [#86](https://github.com/cncf-tags/green-reviews-tooling/issues/86)
 
 ## Authors
 
 - @locomundo
 - @nikimanoledaki
+- @AntonioDiTuri
 
 ## Status
 
-Draft
+Approved
 
 ## Table of Contents
 <!-- toc -->
@@ -24,17 +25,15 @@ Draft
   - [Motivation](#motivation)
     - [Goals](#goals)
     - [Non-Goals](#non-goals)
-    - [Linked Docs](#linked-docs)
   - [Proposal](#proposal)
     - [User Stories](#user-stories)
-    - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional)
     - [Risks and Mitigations](#risks-and-mitigations)
   - [Design Details](#design-details)
     - [Definitions](#definitions)
-    - [Calling the test workflow](#calling-the-test-workflow)
+    - [Calling the benchmark workflow](#calling-the-benchmark-workflow)
       - [Use Case 1: A GitHub Action job using a workflow defined in the _same_ repository (preferred)](#use-case-1-a-github-action-job-using-a-workflow-defined-in-the-same-repository-preferred)
       - [Use Case 2: A GitHub Action job using a workflow defined in a _different_ repository](#use-case-2-a-github-action-job-using-a-workflow-defined-in-a-different-repository)
-    - [Test instructions](#test-instructions)
+    - [Benchmark jobs](#benchmark-jobs)
     - [Authentication](#authentication)
     - [Versioning](#versioning)
   - [Drawbacks (Optional)](#drawbacks-optional)
@@ -101,15 +100,6 @@ within the scope of this work. This helps make sure everyone is crystal clear on
 
 * Defining or designing the content of benchmark tests themselves, or assigning responsibility for who should write them.
 
-### Linked Docs
-
-<!--
-Provide links to previous discussions, Slack threads, motivation issues or any other document
-with context. It is really helpful to provide a "source of truth" for the work
-so that people aren't searching all over the place for lost context.
--->
-
-
 ## Proposal
 
 <!--
@@ -138,7 +128,6 @@ Since different CNCF projects need different benchmarks to reproduce the right m
 
 If the available benchmarks are not enough to set a realistic context, I would like to create and run my own benchmark
 
-
 **Green Reviews maintainer helps to create a new benchmark test for a specific CNCF project**
 
 As a Green Reviews maintainer, I can help a CNCF project maintainers to define the Functional Unit of a project so that the project maintainers can create a benchmark test.
@@ -147,21 +136,6 @@ As a Green Reviews maintainer, I can help a CNCF project maintainers to define t
 
 As a project maintainer, I can edit or remove a benchmark test if it is in a repository owned by the CNCF project itself, or otherwise if it’s in the Green Reviews repository by making a pull request with the changes.
 
-
-<!--
-As a Green Review WG reviewer,
-
-As a Green Review WG cluster maintainer, -->
-
-### Notes/Constraints/Caveats (Optional)
-
-<!--
-What are the caveats to the proposal?
-What are some important details that didn't come across above?
-Go in to as much detail as necessary here.
-This might be a good place to talk about core concepts and how they relate.
--->
-
 ### Risks and Mitigations
 
 <!--
@@ -190,7 +164,10 @@ change are understandable. This may include manifests or workflow examples
 about HOW your proposal will be implemented, this is the place to discuss them.
 -->
 
-The Green Reviews automated pipeline relies on putting together different reusable GitHub Action workflows to modularise the different moving parts. A workflow runs one or more jobs, and each job runs one or more actions. It may be helpful to familiarise oneself with the documentation on [GitHub Action workflows](https://docs.github.com/en/actions/using-workflows/about-workflows) and especially [Reusing workflows](https://docs.github.com/en/actions/using-workflows/reusing-workflows) first to better understand the rest of the proposal as it explains some of these concepts well.  The section on [Calling reusable workflows](https://docs.github.com/en/actions/using-workflows/reusing-workflows#calling-a-reusable-workflow) describes an important concept that will be referenced later in this proposal.
+The Green Reviews automated pipeline relies on putting together different reusable GitHub Action workflows to modularise the different moving parts. It would be helpful to familiarise oneself with the following documentation:
+- [GitHub Action workflows](https://docs.github.com/en/actions/using-workflows/about-workflows) - in summary, a workflow runs one or more jobs, and each job runs one or more actions.
+- [Reusing workflows](https://docs.github.com/en/actions/using-workflows/reusing-workflows)
+- [Calling reusable workflows](https://docs.github.com/en/actions/using-workflows/reusing-workflows#calling-a-reusable-workflow)
 
 ### Definitions
 
@@ -201,68 +178,65 @@ There are different components defined here and shown in the following diagram.
 Let's recap some of the components defined in [Proposal 1](proposal-001-trigger-and-deploy.md):
 1. **Green Reviews pipeline**: the Continuous Integration pipeline which deploys a CNCF project to a test cluster, runs a set of benchmarks while measuring carbon emissions and stores the results. It is implemented by the workflows listed below.
 2. **Cron workflow**: This refers to the initial GitHub Action workflow (described in proposal 1) and which dispatches a project workflow (see next definition), as well as a delete workflow to clean up the resources created by the project workflow.
-3. **Project workflow**: The project workflow is dispatched by the Cron workflow. A project workflow can be, for example, a Falco workflow. A project workflow deploys the project and calls the test workflow (see below). A project workflow can be dispatched more than once if there are multiple project variants/setups. In addition, a Project workflow, which is also just another GitHub Action workflow, contains a list of GitHub Action Jobs. These are a list of test jobs - more info below.
+3. **Project workflow**: The project workflow is dispatched by the Cron workflow. A project workflow can be, for example, a Falco workflow. A project workflow deploys the project and calls the benchmark workflow (see below). A project workflow can be dispatched more than once if there are multiple project variants/setups. In addition, a Project workflow, which is also just another GitHub Action workflow, contains a list of GitHub Action Jobs.
 4. **Delete/cleanup workflow**: This is the one to make sure that the resources created by the project workflow are deleted so the environments go back to the initial state.
 
 This proposal adds the following components:
-1. **[new] Test job**: A test job is an instance of a GitHub Action Job. It is called right after deploying the CNCF project from the test workflow. The test job runs the test workflow and instructions for a CNCF project. Which benchmark test to run is defined by inputs in the calling workflow: a CNCF project and a variant.
-2. **[new] Test workflow**: A test workflow is a separate manifest containing the test instructions.
-3. **[new] Test instructions**: Test instructions define what should run tests on the cluster. These are usually related to the tool's Functional Unit as defined by the SCI. It is described further in the sections below.
+5. **[new] Benchmark workflow**: A list of benchmark jobs that needs to be run in parallel. A benchmark workflow has a `1:many` relationship with benchmark jobs.
+6. **[new] Benchmark job**: A list of benchmark instructions that are executed on the cluster. A benchmark job is an instance of a GitHub Action Job. Which benchmark test to run is defined by inputs in the calling workflow: a CNCF project and a variant.
 
-### Calling the test workflow
+### Calling the benchmark workflow
 
-When the project workflow starts, it deploys the project on the test environment and then runs the test job. For modularity and/or clarity, the test instructions could be defined in two different ways:
+When the project workflow starts, it deploys the project on the test environment and then runs the test job. For modularity and/or clarity, the benchmark job could be defined in two different ways:
 
-As a Job that calls another GitHub Action workflow (yes, yet another workflow 🙂) that contains the instructions. The workflow can be either:
-  1. In the Green Reviews WG repository (**preferred**)
-  2. In a separate repository, such as an upstream CNCF project repository
+As a job that calls another GitHub Action workflow (yes, yet another workflow 🙂) that contains the instructions. The workflow can be either:
+  1. Internal: In the Green Reviews WG repository (**preferred**)
+  2. External: In a separate repository, such as an upstream CNCF project repository
+The two use cases for defining a benchmark workflow are illustrated below.
 
-The two use cases for defining a test workflow are illustrated below.
+![Calling the benchmark job](diagrams/calling-benchmark-job.png "Calling the benchmark job")
 
-![Calling the test job](diagrams/calling-test-job.png "Calling the test job")
+This section defines _benchmark workflow_ and _benchmark job_. It describes how to run them from the _project workflow_. It dives deeper into the following:
 
-This section defines _test workflow_ and _test instructions_. It describes how to run them from the _project workflow_. It dives deeper into the following:
+* How a benchmark workflow should be called from the project workflow
+* What a benchmark workflow must contain in order to run on the cluster
 
-* How a test workflow should be called from the project workflow
-* What a test workflow must contain in order to run on the cluster
-* How a test workflow is related to test instructions
+At a bare minimum, the benchmark workflow must contain a benchmark job with steps of what should run in the Kubernetes cluster. For example, the Falco project maintainers have identified that one way to test the Falco project is through a test that runs `stress-ng` for a given period of time. The steps are contained in a Deployment manifest which can be directly applied to the community cluster using `kubectl`
 
-At a bare minimum, the test workflow must contain test instructions of what should run in the Kubernetes cluster. For example, the Falco project maintainers have identified that one way to test the Falco project is through a test that runs `stress-ng` for a given period of time. The test instructions are contained in a Deployment manifest which can be directly applied to the community cluster using `kubectl`
-
-The test workflows will be stored in the same JSON file as the other parameters for CNCF projects as defined in [Proposal 1](./proposal-001-trigger-and-deploy.md). It can be added as an additional input.
+The benchmark workflows will be stored in the same JSON file as the other parameters for CNCF projects as defined in [Proposal 1](./proposal-001-trigger-and-deploy.md). It can be added as an additional input.
 
 ```yaml
 # .github/workflows/benchmark-pipeline.yaml
 jobs:
   # first, must authenticate to the Kubernetes cluster
-  # this is a Job
-  test-job:
-    # test job calls on test workflow
-    uses: ${{ inputs.test_path }} # refers to test workflow path
+  # this is a benchmark job
+  benchmark-job:
+    # benchmark job calls on benchmark workflow
+    uses: ${{ inputs.benchmark_path }} # refers to benchmark workflow path
 ```
 
 This will fetch the workflow using the `jobs.<job_id>.uses` syntax defined [here](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_iduses).
 
-Below are two use cases: the test workflow may be defined in the Green Reviews repository or in a separate repository.
+Below are two use cases: the benchmark workflow may be defined in the Green Reviews repository or in a separate repository.
 
 #### Use Case 1: A GitHub Action job using a workflow defined in the _same_ repository (preferred)
 
-If the test workflow is located in the Green Reviews repository, `test_path` would refer to, for example, `cncf-tags/green-reviews-tooling/.github/workflows/falco-test-workflow.yml@v1`.
+If the benchmark workflow is located in the Green Reviews repository, `benchmark_path` would refer to, for example, `cncf-tags/green-reviews-tooling/.github/workflows/falco-benchmark-workflow.yml@v1`.
 
-In terms of the directory structure, in the `green-reviews-tooling` repository, we could create a subfolder such as `./github/workflows/tests` to contain the test workflows.
+In terms of the directory structure, in the `green-reviews-tooling` repository, we could create a subfolder such as `./github/workflows/benchmark-workflows` to contain the benchmark workflows.
 
 #### Use Case 2: A GitHub Action job using a workflow defined in a _different_ repository
 
-We want to accommodate different methods of setting up the tests depending on the CNCF project. Given this, the test workflow containing the test instructions could be defined in a different repository. In this case, the `test_path` would be, for example, `falcosecurity/cncf-green-review-testing/.github/workflows/workflow.yml@v1`.
+We want to accommodate different methods of setting up the tests depending on the CNCF project. Given this, the benchmark workflow containing the benchmark job could be defined in a different repository. In this case, the `benchmark_path` would be, for example, `falcosecurity/cncf-green-review-testing/.github/workflows/workflow.yml@v1`.
 
 ![Pipeline run](diagrams/pipeline-run.png "An example pipeline run")
 
-### Test instructions
+### Benchmark jobs
 
-The test workflow which contains the test instructions may look like the following:
+The benchmark workflow which contains the benchmark jobs and their test steps may look like the following:
 
 ```yaml
-# .github/workflows/tests/falco-test-workflow.yaml
+# .github/workflows/tests/falco-benchmark-workflow.yaml
 jobs:
   stress-ng-test:
     runs-on: ubuntu-latest
@@ -275,18 +249,17 @@ jobs:
          kubectl delete -f https://raw.githubusercontent.com/falcosecurity/cncf-green-review-testing/main/kustomize/falco-driver/ebpf/stress-ng.yaml  # other Falco tests:
   # - redis-test e.g. https://github.com/falcosecurity/cncf-green-review-testing/blob/main/kustomize/falco-driver/ebpf/redis.yaml
   # - event-generator-test e.g. https://github.com/falcosecurity/cncf-green-review-testing/blob/main/kustomize/falco-driver/ebpf/falco-event-generator.yaml
-  # TODO: should each test be a workflow or a job in a single workflow? as in, one test workflow per cncf project or multiple workflows per cncf project? TBD
 ```
 
-The job has test instructions to apply the upstream Kubernetes manifest which contains a `while` loop that runs `stress-ng`. The manifest already defines where the test should run in the cluster i.e. in which namespace. The functional unit test is time-bound in this case and scoped to 15 minutes. Therefore, we deploy this test, wait for 15 minutes, then delete the manifest to end the loop. The test instructions depend on the functional unit of each CNCF project.
+The benchmark job has some test instructions/steps. In this case, it applies an upstream Kubernetes manifest. This manifest contains a `while` loop that runs `stress-ng`. The manifest already defines where the test should run in the cluster i.e. in which namespace. The functional unit test is time-bound in this case and scoped to 15 minutes. Therefore, we deploy this test, wait for 15 minutes, then delete the manifest to end the loop. The test steps depend on the functional unit of each CNCF project.
 
 In the example above, the Kubernetes manifest that is applied to the cluster is located in a different repository: this is the case of an externally defined benchmark
 
-Each workflow should ensure that any artefacts that were deployed as part of the test instructions should be deleted at the end of the test run.
+Each workflow should ensure that any artefacts that were deployed as part of the benchmark job should be deleted at the end of the test run.
 
 ### Authentication
 
-Before the test workflow is called on, we assume that the workflow already contains a secret with a kubeconfig to authenticate with the test cluster and Falco has already been deployed to it. It is required that the pipeline authenticates with the Kubernetes cluster before running the job with the test.
+Before the benchmark workflow is called on, we assume that the workflow already contains a secret with a kubeconfig to authenticate with the test cluster and Falco has already been deployed to it. It is required that the pipeline authenticates with the Kubernetes cluster before running the job with the test.
 
 ### Versioning
 
@@ -302,18 +275,12 @@ information to express the idea and why it was not acceptable.
 
 ## Alternatives
 
-Here a list of the alternatives we considered: 
+Here a list of the alternatives we considered:
 
 - **mapping between workflows and CNCF projects**: we have decided for a 1:1 relationship, every project will only have one workflow, again for simplicity. We could add support for 1:many in the future
 
 - **mapping between workflows and jobs**: we have decided a 1:Many relationship, 1 workflow and many jobs, but a different option we evaluated was a 1:1 relationship. We choose for the first option cause it is simpler and gives a clear overview about what jobs are needed for a project workflow
 
-What other approaches did you consider, and why did you rule them out? These do
-not need to be as detailed as the proposal (pros and cons are fine),
-but should include enough information to express the idea and why it was not acceptable
-as well as illustrate why the final solution was selected.
--->
-
 ## Infrastructure Needed (Optional)
 
 <!--