Add new integration tests for Azure OIDC for git repositories

[dipti-pai]

Changes include enhancements to tests/integration to add git tests

• testapp is modified to accept category git and test cloning azure devops git repository using workload identity.
  Azure terraform files are separated into oci and git. Azure terraform files for git require an organization and PAT and create ADO project and repository
• suite_test.go is modified to add new git category and setup provider specific test configuration and permissions on git repository
• azure_test.go has new functions to return test configuration based on the git terraform file output and API to give the workload identity access to the ADO repository.
• repo_list_test.go is deleted with new files for oci_test.go with existing oci tests and git_test.go with tests to validate Azure OIDC for ADO repository.
• util_test.go is a new utility with functions to get the gogit client to clone, commit and push changes to a git repository.
  In a subsequent PR, we can consider moving oci//tests/integration/ out of oci package to a root directory tests/integration/

Depends on PR in test-infra #44 and PR in pkg #789

Sample test run outputs -

$ make test-azure GO_TEST_ARGS="-retain -existing"
make test PROVIDER_ARG="-provider azure"
make[1]: Entering directory '/home/dipti/code/pkg/oci/tests/integration'
docker image inspect fluxcd/testapp:test >/dev/null
TEST_IMG=fluxcd/testapp:test go test -timeout 30m -v ./ -retain -existing -provider azure --tags=integration
2024/08/22 12:19:26 Terraform binary:  /snap/bin/terraform
2024/08/22 12:19:26 Init Terraform
2024/08/22 12:19:28 Applying Terraform
2024/08/22 12:22:07 pushing flux test image fluxtesthipomule.azurecr.io/app:test
2024/08/22 12:22:08 Git is enabled, get test config
URL without username: https://dev.azure.com/diptipai-flux/fluxProjhipomule/_git/fluxRepohipomule
2024/08/22 12:22:08 Giving permissions to workload identity to access repository
2024/08/22 12:22:10 Added managed identity to organization:
2024/08/22 12:22:10 Workload identity is enabled, initializing service account with annotations
=== RUN   TestGitCloneUsingProvider
=== RUN   TestGitCloneUsingProvider/Git_oidc_credential_test
--- PASS: TestGitCloneUsingProvider (6.70s)
    --- PASS: TestGitCloneUsingProvider/Git_oidc_credential_test (5.94s)
=== RUN   TestOciImageRepositoryListTags
    oci_test.go:30: Skipping test as oci is not enabled in env
--- SKIP: TestOciImageRepositoryListTags (0.00s)
=== RUN   TestOciRepositoryRootLoginListTags
    oci_test.go:46: Skipping test as oci is not enabled in env
--- SKIP: TestOciRepositoryRootLoginListTags (0.00s)
=== RUN   TestOciOIDCLoginListTags
    oci_test.go:64: Skipping test as oci is not enabled in env
--- SKIP: TestOciOIDCLoginListTags (0.00s)
PASS
ok      github.com/fluxcd/pkg/oci/tests/integration     171.350s
make[1]: Leaving directory '/home/dipti/code/pkg/oci/tests/integration'

source .env (After enabling oci in env)
$ make test-azure GO_TEST_ARGS="-retain -existing"
make test PROVIDER_ARG="-provider azure"
make[1]: Entering directory '/home/dipti/code/pkg/oci/tests/integration'
docker image inspect fluxcd/testapp:test >/dev/null
TEST_IMG=fluxcd/testapp:test go test -timeout 30m -v ./ -retain -existing -provider azure --tags=integration
2024/08/22 12:23:28 Terraform binary:  /snap/bin/terraform
2024/08/22 12:23:28 Init Terraform
2024/08/22 12:23:31 Applying Terraform
2024/08/22 12:26:11 pushing flux test image fluxtesthipomule.azurecr.io/app:test
2024/08/22 12:26:12 OCI is enabled, push oci test images
2024/08/22 12:26:15 pushing test image fluxtesthipomule.azurecr.io/dytqs:v0.1.0
2024/08/22 12:26:17 pushing test image fluxtesthipomule.azurecr.io/dytqs:v0.1.2
2024/08/22 12:26:18 pushing test image fluxtesthipomule.azurecr.io/dytqs:v0.1.3
2024/08/22 12:26:19 pushing test image fluxtesthipomule.azurecr.io/dytqs:v0.1.4
2024/08/22 12:26:20 Git is enabled, get test config
URL without username: https://dev.azure.com/diptipai-flux/fluxProjhipomule/_git/fluxRepohipomule
2024/08/22 12:26:20 Giving permissions to workload identity to access repository
2024/08/22 12:26:21 Added managed identity to organization:
2024/08/22 12:26:21 Workload identity is enabled, initializing service account with annotations
=== RUN   TestGitCloneUsingProvider
=== RUN   TestGitCloneUsingProvider/Git_oidc_credential_test
--- PASS: TestGitCloneUsingProvider (6.38s)
    --- PASS: TestGitCloneUsingProvider/Git_oidc_credential_test (5.74s)
=== RUN   TestOciImageRepositoryListTags
--- PASS: TestOciImageRepositoryListTags (0.00s)
=== RUN   TestOciRepositoryRootLoginListTags
--- PASS: TestOciRepositoryRootLoginListTags (0.00s)
=== RUN   TestOciOIDCLoginListTags
--- PASS: TestOciOIDCLoginListTags (0.00s)
PASS
ok      github.com/fluxcd/pkg/oci/tests/integration     179.322s
make[1]: Leaving directory '/home/dipti/code/pkg/oci/tests/integration'

$ make test-azure-git GO_TEST_ARGS="-retain -existing"
make test PROVIDER_ARG="-provider azure" GO_TEST_PREFIX="-run TestGit"
make[1]: Entering directory '/home/dipti/code/pkg/oci/tests/integration'
docker image inspect fluxcd/testapp:test >/dev/null
TEST_IMG=fluxcd/testapp:test go test -timeout 30m -v ./ -run TestGit -retain -existing -provider azure --tags=integration
2024/08/22 12:43:05 Terraform binary:  /snap/bin/terraform
2024/08/22 12:43:05 Init Terraform
2024/08/22 12:43:07 Applying Terraform
2024/08/22 12:45:50 pushing flux test image fluxtesthipomule.azurecr.io/app:test
2024/08/22 12:45:51 OCI is enabled, push oci test images
2024/08/22 12:45:55 pushing test image fluxtesthipomule.azurecr.io/6vxpb:v0.1.0
2024/08/22 12:45:56 pushing test image fluxtesthipomule.azurecr.io/6vxpb:v0.1.2
2024/08/22 12:45:57 pushing test image fluxtesthipomule.azurecr.io/6vxpb:v0.1.3
2024/08/22 12:45:58 pushing test image fluxtesthipomule.azurecr.io/6vxpb:v0.1.4
2024/08/22 12:45:59 Git is enabled, get test config
URL without username: https://dev.azure.com/diptipai-flux/fluxProjhipomule/_git/fluxRepohipomule
2024/08/22 12:45:59 Giving permissions to workload identity to access repository
2024/08/22 12:46:01 Added managed identity to organization:
2024/08/22 12:46:01 Workload identity is enabled, initializing service account with annotations
=== RUN   TestGitCloneUsingProvider
=== RUN   TestGitCloneUsingProvider/Git_oidc_credential_test
--- PASS: TestGitCloneUsingProvider (6.97s)
    --- PASS: TestGitCloneUsingProvider/Git_oidc_credential_test (5.95s)
PASS
ok      github.com/fluxcd/pkg/oci/tests/integration     183.375s
make[1]: Leaving directory '/home/dipti/code/pkg/oci/tests/integration'

$ make test-azure-oci GO_TEST_ARGS="-retain -existing"
make test PROVIDER_ARG="-provider azure" GO_TEST_PREFIX="-run TestOci"
make[1]: Entering directory '/home/dipti/code/pkg/oci/tests/integration'
docker image inspect fluxcd/testapp:test >/dev/null
TEST_IMG=fluxcd/testapp:test go test -timeout 30m -v ./ -run TestOci -retain -existing -provider azure --tags=integration
2024/08/22 12:46:32 Terraform binary:  /snap/bin/terraform
2024/08/22 12:46:32 Init Terraform
2024/08/22 12:46:34 Applying Terraform
2024/08/22 12:49:23 pushing flux test image fluxtesthipomule.azurecr.io/app:test
2024/08/22 12:49:24 OCI is enabled, push oci test images
2024/08/22 12:49:28 pushing test image fluxtesthipomule.azurecr.io/1uavx:v0.1.0
2024/08/22 12:49:29 pushing test image fluxtesthipomule.azurecr.io/1uavx:v0.1.2
2024/08/22 12:49:30 pushing test image fluxtesthipomule.azurecr.io/1uavx:v0.1.3
2024/08/22 12:49:31 pushing test image fluxtesthipomule.azurecr.io/1uavx:v0.1.4
2024/08/22 12:49:32 Git is enabled, get test config
URL without username: https://dev.azure.com/diptipai-flux/fluxProjhipomule/_git/fluxRepohipomule
2024/08/22 12:49:32 Giving permissions to workload identity to access repository
2024/08/22 12:49:33 Added managed identity to organization:
2024/08/22 12:49:33 Workload identity is enabled, initializing service account with annotations
=== RUN   TestOciImageRepositoryListTags
--- PASS: TestOciImageRepositoryListTags (0.00s)
=== RUN   TestOciRepositoryRootLoginListTags
--- PASS: TestOciRepositoryRootLoginListTags (0.00s)
=== RUN   TestOciOIDCLoginListTags
--- PASS: TestOciOIDCLoginListTags (0.00s)
PASS
ok      github.com/fluxcd/pkg/oci/tests/integration     180.909s
make[1]: Leaving directory '/home/dipti/code/pkg/oci/tests/integration'

[darkowlzz]

I finally ran this in my account. Good to see it working 🙂
Left more comments based on my observations after running it.

[darkowlzz]

LGTM!

The git and auth package dependencies need to be updated when those packages are available before merging this.

Thanks for all the work in adding git tests.

[stefanprodan]

This can now be rebased on top of main with the following packages updates:

• auth/v0.0.1
• cache/v0.0.4
• git/v0.21.0
• git/gogit/v0.21.0
• oci/v0.41.1

[dipti-pai]

This can now be rebased on top of main with the following packages updates:

• auth/v0.0.1
• cache/v0.0.4
• git/v0.21.0
• git/gogit/v0.21.0
• oci/v0.41.1

@stefanprodan I rebased on top of upstream main and see the auth pkg is now updated to v0.0.1. However, git package is at v0.20.0 and git/gogit is at v0.19.0 which is lower than the versions you shared above. Is this ok? The new git integration tests are green.

EDIT: In source controller, the updated packages are used correctly. In pkg/oci/tests/integration, the replace directives are for local paths so I understand this should be ok.

[stefanprodan]

LGTM

Thanks @dipti-pai