Feature: Add support for GCR buckets with uniform_bucket_level_access…

… = true (#32)

The new default setting for buckets is to use bucket-level policies (instead of an ACL), this also applies to buckets created for the container-registry. These buckets don't support setting google_storage_bucket_access_control policies, so the ACL grant fails.

This PR uses the new (as of v3.88 of the google terraform provider) google_storage_bucket datasource to determine whether or not uniform_bucket_level_access = true is set, and if so uses the google_storage_bucket_iam_member resource to grant access.

Special thanks to @andyroyle and @patricklubach

README.md

@@ -116,7 +116,9 @@ No modules.
 | [google_service_account.cleaner](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/service_account) | resource |
 | [google_service_account.invoker](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/service_account) | resource |
 | [google_storage_bucket_access_control.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket_access_control) | resource |
+| [google_storage_bucket_iam_member.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket_iam_member) | resource |
 | [google_project.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/project) | data source |
+| [google_storage_bucket.bucket](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/storage_bucket) | data source |
 
 ## Inputs
 
@@ -139,7 +141,7 @@ No modules.
 | <a name="input_disable_dependent_services"></a> [disable\_dependent\_services](#input\_disable\_dependent\_services) | If `true`, services that are enabled and which depend on this service should also be disabled when this service is destroyed. If `false` or unset, an error will be generated if any enabled services depend on this service when destroying it. | `bool` | `false` | no |
 | <a name="input_disable_on_destroy"></a> [disable\_on\_destroy](#input\_disable\_on\_destroy) | If `true`, disable the service when the terraform resource is destroyed. May be useful in the event that a project is long-lived but the infrastructure running in that project changes frequently. | `bool` | `false` | no |
 | <a name="input_gcr_cleaner_image"></a> [gcr\_cleaner\_image](#input\_gcr\_cleaner\_image) | The docker image of the gcr cleaner to deploy to Cloud Run. | `string` | `"gcr.io/gcr-cleaner/gcr-cleaner:latest"` | no |
-| <a name="input_gcr_repositories"></a> [gcr\_repositories](#input\_gcr\_repositories) | List of Google Container Registries objects to create:<pre>list(object({<br>    project_id     = Value of the Google project id, if ommited, it will be assigned `google_project_id` variable value (optional(string))<br>    storage_region = Location of the storage bucket (optional(string))<br>    repositories = Docker image repositories to clean (optional(list(object({<br>      name           = Name of the repository (string)<br>      grace          = Relative duration in which to ignore references. This value is specified as a time duration value like "5s" or "3h". If set, refs newer than the duration will not be deleted. If unspecified, the default is no grace period (all untagged image refs are deleted) (optional(string))<br>      keep           = If an integer is provided, it will always keep that minimum number of images. Note that it will not consider images inside the `grace` duration (optional(string))<br>      tag_filter     = (Deprecated) If specified, any image where the first tag matches this given regular expression will be deleted. The image will not be deleted if other tags match the regular expression (optional(string))<br>      tag_filter_any = If specified, any image with at least one tag that matches this given regular expression will be deleted. The image will be deleted even if it has other tags that do not match the given regular expression (optional(string))<br>      tag_filter_all = If specified, any image where all tags match this given regular expression will be deleted. The image will not be delete if it has other tags that do not match the given regular expression (optional(string))<br>      recursive      = If set to true, will recursively search all child repositories (optional(bool))<br>    }))))<br>    clean_all  = Set to `true` to clean all project's repositories (optional(bool))<br>    parameters = Map of parameters to apply to all repositories when `clean_all` is set to `true` (optional(object({<br>      grace          = Relative duration in which to ignore references. This value is specified as a time duration value like "5s" or "3h". If set, refs newer than the duration will not be deleted. If unspecified, the default is no grace period (all untagged image refs are deleted) (optional(string))<br>      keep           = If an integer is provided, it will always keep that minimum number of images. Note that it will not consider images inside the `grace` duration (optional(string))<br>      tag_filter     = (Deprecated) If specified, any image where the first tag matches this given regular expression will be deleted. The image will not be deleted if other tags match the regular expression (optional(string))<br>      tag_filter_any = If specified, any image with at least one tag that matches this given regular expression will be deleted. The image will be deleted even if it has other tags that do not match the given regular expression (optional(string))<br>      tag_filter_all = If specified, any image where all tags match this given regular expression will be deleted. The image will not be delete if it has other tags that do not match the given regular expression (optional(string))<br>    })))<br>}))</pre> | <pre>list(object({<br>    project_id     = optional(string)<br>    storage_region = optional(string)<br>    repositories = optional(list(object({<br>      name           = string<br>      grace          = optional(string)<br>      keep           = optional(string)<br>      tag_filter     = optional(string)<br>      tag_filter_any = optional(string)<br>      tag_filter_all = optional(string)<br>      recursive      = optional(bool)<br>    })))<br>    clean_all = optional(bool)<br>    parameters = optional(object({<br>      grace          = optional(string)<br>      keep           = optional(string)<br>      tag_filter     = optional(string)<br>      tag_filter_any = optional(string)<br>      tag_filter_all = optional(string)<br>    }))<br>  }))</pre> | `[]` | no |
+| <a name="input_gcr_repositories"></a> [gcr\_repositories](#input\_gcr\_repositories) | List of Google Container Registries objects to create:<pre>list(object({<br>    project_id     = Value of the Google project id, if ommited, it will be assigned `google_project_id` local value, which is the provider's project_id (optional(string))<br>    storage_region = Location of the storage bucket (optional(string))<br>    repositories = Docker image repositories to clean (optional(list(object({<br>      name           = Name of the repository (string)<br>      grace          = Relative duration in which to ignore references. This value is specified as a time duration value like "5s" or "3h". If set, refs newer than the duration will not be deleted. If unspecified, the default is no grace period (all untagged image refs are deleted) (optional(string))<br>      keep           = If an integer is provided, it will always keep that minimum number of images. Note that it will not consider images inside the `grace` duration (optional(string))<br>      tag_filter     = (Deprecated) If specified, any image where the first tag matches this given regular expression will be deleted. The image will not be deleted if other tags match the regular expression (optional(string))<br>      tag_filter_any = If specified, any image with at least one tag that matches this given regular expression will be deleted. The image will be deleted even if it has other tags that do not match the given regular expression (optional(string))<br>      tag_filter_all = If specified, any image where all tags match this given regular expression will be deleted. The image will not be delete if it has other tags that do not match the given regular expression (optional(string))<br>      recursive      = If set to true, will recursively search all child repositories (optional(bool))<br>    }))))<br>    clean_all  = Set to `true` to clean all project's repositories (optional(bool))<br>    parameters = Map of parameters to apply to all repositories when `clean_all` is set to `true` (optional(object({<br>      grace          = Relative duration in which to ignore references. This value is specified as a time duration value like "5s" or "3h". If set, refs newer than the duration will not be deleted. If unspecified, the default is no grace period (all untagged image refs are deleted) (optional(string))<br>      keep           = If an integer is provided, it will always keep that minimum number of images. Note that it will not consider images inside the `grace` duration (optional(string))<br>      tag_filter     = (Deprecated) If specified, any image where the first tag matches this given regular expression will be deleted. The image will not be deleted if other tags match the regular expression (optional(string))<br>      tag_filter_any = If specified, any image with at least one tag that matches this given regular expression will be deleted. The image will be deleted even if it has other tags that do not match the given regular expression (optional(string))<br>      tag_filter_all = If specified, any image where all tags match this given regular expression will be deleted. The image will not be delete if it has other tags that do not match the given regular expression (optional(string))<br>    })))<br>}))</pre> | <pre>list(object({<br>    project_id     = optional(string)<br>    storage_region = optional(string)<br>    repositories = optional(list(object({<br>      name           = string<br>      grace          = optional(string)<br>      keep           = optional(string)<br>      tag_filter     = optional(string)<br>      tag_filter_any = optional(string)<br>      tag_filter_all = optional(string)<br>      recursive      = optional(bool)<br>    })))<br>    clean_all = optional(bool)<br>    parameters = optional(object({<br>      grace          = optional(string)<br>      keep           = optional(string)<br>      tag_filter     = optional(string)<br>      tag_filter_any = optional(string)<br>      tag_filter_all = optional(string)<br>    }))<br>  }))</pre> | `[]` | no |
 
 ## Outputs
 

data.tf

@@ -1,2 +1,8 @@
 # get project details
 data "google_project" "this" {}
+
+data "google_storage_bucket" "bucket" {
+  for_each = toset(local.buckets)
+
+  name = each.value
+}

iam.tf

@@ -1,14 +1,23 @@
 # Grant cleaner service account access to delete references in Google Container Registry
+# for buckets with uniform_bucket_level_access = false
 resource "google_storage_bucket_access_control" "this" {
-  for_each = {
-    for item in toset(local.project_storage_region) : "${item.storage_region}.${item.project_id}" => item
-  }
+  for_each = toset(local.google_storage_bucket_access_control)
 
-  bucket = each.value.storage_region != "" ? "${each.value.storage_region}.artifacts.${each.value.project_id}.appspot.com" : "artifacts.${each.value.project_id}.appspot.com"
+  bucket = each.value
   role   = "WRITER"
   entity = "user-${google_service_account.cleaner.email}"
 }
 
+# Grant cleaner service account access to delete references in Google Container Registry
+# for buckets with uniform_bucket_level_access = true
+resource "google_storage_bucket_iam_member" "this" {
+  for_each = toset(local.google_storage_bucket_iam_member)
+
+  bucket = each.value
+  role   = "roles/storage.legacyBucketWriter"
+  member = "serviceAccount:${google_service_account.cleaner.email}"
+}
+
 # Add IAM policy binding to the Cloud Run service
 resource "google_cloud_run_service_iam_binding" "this" {
   location = google_cloud_run_service.this.location

locals.tf

@@ -14,6 +14,21 @@ locals {
   # More on https://cloud.google.com/appengine/docs/locations
   cloud_scheduler_job_location = contains(["europe-west", "us-central"], var.app_engine_application_location) == true ? "${var.app_engine_application_location}1" : var.app_engine_application_location
 
+  # create a list of buckets from the gcr_repositories input
+  buckets = [
+    for repo in var.gcr_repositories : repo.storage_region != null ? "${repo.storage_region}.artifacts.${repo.project_id != null ? repo.project_id : local.google_project_id}.appspot.com" : "artifacts.${repo.project_id != null ? repo.project_id : local.google_project_id}.appspot.com"
+  ]
+
+  # Buckets having uniform_bucket_level_access = true
+  google_storage_bucket_iam_member = [
+    for bucket in local.buckets : bucket if data.google_storage_bucket.bucket[bucket].uniform_bucket_level_access
+  ]
+
+  # Buckets having uniform_bucket_level_access = false
+  google_storage_bucket_access_control = [
+    for bucket in local.buckets : bucket if !data.google_storage_bucket.bucket[bucket].uniform_bucket_level_access
+  ]
+
   # create project_all_repositories list, appending storage_region and project_id when ommited
   # for projects having clean_all = true.
   # Set default values for optional fields.
@@ -24,7 +39,7 @@ locals {
       keep           = repo.parameters != null ? (repo.parameters.keep != null ? repo.parameters.keep : "0") : "0"
       tag_filter     = repo.parameters != null ? (repo.parameters.tag_filter != null ? repo.parameters.tag_filter : "") : ""
       tag_filter_any = repo.parameters != null ? (repo.parameters.tag_filter_any != null ? repo.parameters.tag_filter_any : "") : ""
-      tag_filter_all = repo.parameters != null ? (repo.parameters.ttag_filter_allag_filter != null ? repo.parameters.tag_filter_all : "") : ""
+      tag_filter_all = repo.parameters != null ? (repo.parameters.tag_filter_all != null ? repo.parameters.tag_filter_all : "") : ""
       recursive      = true
       filter         = repo.parameters != null ? "grace-${repo.parameters.grace != null ? repo.parameters.grace : "0"}-keep-${repo.parameters.keep != null ? repo.parameters.keep : "0"}-tag_filter-${repo.parameters.tag_filter != null ? repo.parameters.tag_filter : "no"}-tag_filter_any-${repo.parameters.tag_filter_any != null ? repo.parameters.tag_filter_any : "no"}-tag_filter_any-${repo.parameters.tag_filter_any != null ? repo.parameters.tag_filter_any : "no"}" : "delete-all-untagged-images-recursive"
     } if repo.clean_all == true
@@ -51,20 +66,4 @@ locals {
 
   # create final repositories list
   fetched_repositories = concat(local.project_all_repositories, local.repositories)
-
-  # group repositories by project
-  repositories_by_project = {
-    for repo in var.gcr_repositories : repo.project_id != null ? repo.project_id : local.google_project_id => repo...
-  }
-
-  # create a list of tuple { project_id, storage_region } from repositories_by_project
-  # that will be used in `iam.tf`
-  project_storage_region = flatten([
-    for key, repos in local.repositories_by_project : [
-      for repo in repos : {
-        project_id     = key
-        storage_region = repo.storage_region != null ? repo.storage_region : ""
-      }
-    ]
-  ])
 }

variables.tf

@@ -62,7 +62,7 @@ variable "gcr_repositories" {
 List of Google Container Registries objects to create:
 ```
 list(object({
-    project_id     = Value of the Google project id, if ommited, it will be assigned `google_project_id` variable value (optional(string))
+    project_id     = Value of the Google project id, if ommited, it will be assigned `google_project_id` local value, which is the provider's project_id (optional(string))
     storage_region = Location of the storage bucket (optional(string))
     repositories = Docker image repositories to clean (optional(list(object({
       name           = Name of the repository (string)