Add bootstrap IAM/services instructions on GCP bootstrap project (the one that is used to create the 2 prj-b-cicd and prj-b-seed projects)

[obriensystems]

TL;DR

add to https://github.com/CloudLandingZone/terraform-example-foundation?tab=readme-ov-file#0-bootstrap
and the roles section https://github.com/terraform-google-modules/terraform-example-foundation/tree/master/0-bootstrap#prerequisites

add bootstrap project to https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/docs/GLOSSARY.md

See service enablements on the project and IAM roles for the super admin or terraform service account being impersonated
details in the list on
#1133 (comment)

Terraform Resources

No response

Detailed design

No response

Additional information

No response

[obriensystems]

Testing IAM Role and services enablement on the bootstrap project
see
#1133 (comment)

Continue olxyz org local and cloud shell terraform 1.3 run

• pull in 1-org changes 2 days ago in https://github.com/terraform-google-modules/terraform-example-foundation/pull/1110/files
• run with TEF groups creation instead of cloud-setup (foundations)

Target bootstrap PR for the following

see scripting example in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/gh766-script/solutions/setup.sh

create folder then use script

wait 2 min for propagation

folder id in this case is 736660879367

IAM SA roles

Billing Account Administrator
Compute Shared VPC Admin
Folder Admin
Folder Creator
Organization Administrator
Organization Policy Administrator
Project Billing Manager
Project Creator
Project Deleter
Project IAM Admin
Security Center Admin
Security Center Notification Configurations Editor
Service Account Token Creator
Service Usage Admin


michael@cloudshell:~$ ROOT_FOLDER_ID=736660879367
michael@cloudshell:~$ BOOT_PROJECT_ID=tef-olxyz
#BILLING_ID=$(gcloud billing projects describe $BOOT_PROJECT_ID $BILLING_FORMAT | sed 's/.*\///')
michael@cloudshell:~$ gcloud projects create "$BOOT_PROJECT_ID" --name="${BOOT_PROJECT_ID}" --set-as-default --folder="$ROOT_FOLDER_ID"
Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/tef-olxyz].
Waiting for [operations/cp.9134984346595650639] to finish...done.                                                                                                                         
Enabling service [cloudapis.googleapis.com] on project [tef-olxyz]...
Operation "operations/acat.p2-438381210056-0f410b88-37c8-4953-baf3-3af50f7d4db1" finished successfully.
Updated property [core/project] to [tef-olxyz].
michael@cloudshell:~ (tef-olxyz)$ BILLING_ID=01BCCE-4EC0EE-DC58C8


michael@cloudshell:~ (tef-olxyz)$ SUPER_ADMIN_EMAIL=$(gcloud config list --format json|jq .core.account | sed 's/"//g')
michael@cloudshell:~ (tef-olxyz)$ echo $SUPER_ADMIN_EMAIL
mich...bs.xyz
michael@cloudshell:~ (tef-olxyz)$ ORG_ID=$(gcloud projects get-ancestors $BOOT_PROJECT_ID --format='get(id)' | tail -1)
michael@cloudshell:~ (tef-olxyz)$ gcloud beta billing projects link "${BOOT_PROJECT_ID}" --billing-account "${BILLING_ID}"

gcloud organizations add-iam-policy-binding ${ORG_ID}  --member=user:$SUPER_ADMIN_EMAIL --role=roles/billing.admin --quiet > /dev/null 1>&1
gcloud organizations add-iam-policy-binding ${ORG_ID}  --member=user:$SUPER_ADMIN_EMAIL --role=roles/accesscontextmanager.policyAdmin --quiet > /dev/null 1>&1
gcloud organizations add-iam-policy-binding ${ORG_ID}  --member=user:$SUPER_ADMIN_EMAIL --role=roles/billing.user --quiet > /dev/null 1>&1
gcloud organizations add-iam-policy-binding ${ORG_ID}  --member=user:$SUPER_ADMIN_EMAIL --role=roles/resourcemanager.folderAdmin --quiet > /dev/null 1>&1
gcloud organizations add-iam-policy-binding ${ORG_ID}  --member=user:$SUPER_ADMIN_EMAIL --role=roles/resourcemanager.organizationAdmin --quiet > /dev/null 1>&1
gcloud organizations add-iam-policy-binding ${ORG_ID}  --member=user:$SUPER_ADMIN_EMAIL --role=roles/resourcemanager.projectCreator --quiet > /dev/null 1>&1
gcloud organizations add-iam-policy-binding ${ORG_ID}  --member=user:$SUPER_ADMIN_EMAIL --role=roles/resourcemanager.projectDeleter --quiet > /dev/null 1>&1
gcloud organizations add-iam-policy-binding ${ORG_ID}  --member=user:$SUPER_ADMIN_EMAIL --role=roles/orgpolicy.policyAdmin --quiet > /dev/null 1>&1
gcloud organizations add-iam-policy-binding ${ORG_ID}  --member=user:$SUPER_ADMIN_EMAIL --role=roles/resourcemanager.projectIamAdmin --quiet > /dev/null 1>&1
gcloud organizations add-iam-policy-binding ${ORG_ID}  --member=user:$SUPER_ADMIN_EMAIL --role=roles/serviceusage.serviceUsageAdmin --quiet > /dev/null 1>&1
gcloud organizations add-iam-policy-binding ${ORG_ID}  --member=user:$SUPER_ADMIN_EMAIL --role=roles/storage.admin --quiet > /dev/null 1>&1
gcloud organizations add-iam-policy-binding ${ORG_ID}  --member=user:$SUPER_ADMIN_EMAIL --role=roles/iam.serviceAccountTokenCreator --quiet > /dev/null 1>&1

gcloud organizations add-iam-policy-binding ${ORG_ID}  --member=user:$SUPER_ADMIN_EMAIL --role=roles/securitycenter.admin --quiet > /dev/null 1>&1

bootstrap project

#1139
#1140
#1142
#1143

# check services
michael@cloudshell:~ (tef-olxyz)$ gcloud services list | grep NAME
NAME: analyticshub.googleapis.com
NAME: bigquery.googleapis.com
NAME: bigqueryconnection.googleapis.com
NAME: bigquerydatapolicy.googleapis.com
NAME: bigquerymigration.googleapis.com
NAME: bigqueryreservation.googleapis.com
NAME: bigquerystorage.googleapis.com
NAME: cloudapis.googleapis.com
NAME: cloudtrace.googleapis.com
NAME: dataform.googleapis.com
NAME: dataplex.googleapis.com
NAME: datastore.googleapis.com
NAME: logging.googleapis.com
NAME: monitoring.googleapis.com
NAME: servicemanagement.googleapis.com
NAME: serviceusage.googleapis.com
NAME: sql-component.googleapis.com
NAME: storage-api.googleapis.com
NAME: storage-component.googleapis.com
NAME: storage.googleapis.com

michael@cloudshell:~ (tef-olxyz)$ gcloud services enable cloudresourcemanager.googleapis.com
Operation "operations/acat.p2-438381210056-f7e0fb71-9240-434b-918d-6e4bfe1fadfb" finished successfully.
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable cloudbilling.googleapis.com
Operation "operations/acat.p2-438381210056-9316b2a5-637a-40ad-bd64-981dc1af28f3" finished successfully.
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable iam.googleapis.com
Operation "operations/acat.p2-438381210056-2344a0fc-dc11-4432-b9aa-53a89f0fbb4d" finished successfully.
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable cloudkms.googleapis.com
Operation "operations/acat.p2-438381210056-4379d575-1d69-40b6-9104-113ff4f48704" finished successfully.
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable servicenetworking.googleapis.com
Operation "operations/acat.p2-438381210056-b21bf432-2188-49c1-a9b9-44ec824f8d43" finished successfully.

# existing
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable  accesscontextmanager.googleapis.com
Operation "operations/acat.p2-438381210056-66c8f99c-fe16-4481-8d64-e9d467f8fe8d" finished successfully.
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable appengine.googleapis.com
Operation "operations/acat.p2-438381210056-4426e470-344b-4b1e-9d47-52a0f2befe21" finished successfully.
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable artifactregistry.googleapis.com
Operation "operations/acat.p2-438381210056-0d5d0ccd-3845-4edf-a062-e3ff3448ff03" finished successfully.
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable assuredworkloads.googleapis.com
Operation "operations/acat.p2-438381210056-f189974a-ab71-49e1-a81b-48bd896df8a0" finished successfully.
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable billingbudgets.googleapis.com
Operation "operations/acat.p2-438381210056-7b38125e-23ec-4564-bbfc-277fc3297802" finished successfully.
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable cloudasset.googleapis.com
gcloud services enable cloudbuild.googleapis.comOperation "operations/acat.p2-438381210056-bafe06bf-3786-4608-b994-3c3069f6042e" finished successfully.
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable cloudbuild.googleapis.com
Operation "operations/acf.p2-438381210056-039eab93-9f6a-420b-943f-01297e27b339" finished successfully.
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable cloudidentity.googleapis.com
Operation "operations/acat.p2-438381210056-7989271a-604a-4e7a-9c4e-4acd5e8ab8a5" finished successfully.
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable containerregistry.googleapis.com
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable essentialcontacts.googleapis.com
Operation "operations/acat.p2-438381210056-8230082d-b801-4096-a74c-4977faf1a85c" finished successfully.
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable securitycenter.googleapis.com
Operation "operations/acat.p2-438381210056-e1a24825-5697-465d-ad46-1496b47f1b58" finished successfully.

# post check
michael@cloudshell:~ (tef-olxyz)$ gcloud services list | grep NAME 
NAME: accesscontextmanager.googleapis.com
NAME: analyticshub.googleapis.com
NAME: appengine.googleapis.com
NAME: artifactregistry.googleapis.com
NAME: assuredworkloads.googleapis.com
NAME: bigquery.googleapis.com
NAME: bigqueryconnection.googleapis.com
NAME: bigquerydatapolicy.googleapis.com
NAME: bigquerymigration.googleapis.com
NAME: bigqueryreservation.googleapis.com
NAME: bigquerystorage.googleapis.com
NAME: billingbudgets.googleapis.com
NAME: cloudapis.googleapis.com
NAME: cloudasset.googleapis.com
NAME: cloudbilling.googleapis.com
NAME: cloudbuild.googleapis.com
NAME: cloudidentity.googleapis.com
NAME: cloudkms.googleapis.com
NAME: cloudresourcemanager.googleapis.com
NAME: cloudtrace.googleapis.com
NAME: containerregistry.googleapis.com
NAME: dataform.googleapis.com
NAME: dataplex.googleapis.com
NAME: datastore.googleapis.com
NAME: essentialcontacts.googleapis.com
NAME: iam.googleapis.com
NAME: iamcredentials.googleapis.com
NAME: logging.googleapis.com
NAME: monitoring.googleapis.com
NAME: pubsub.googleapis.com
NAME: securitycenter.googleapis.com
NAME: securitycentermanagement.googleapis.com
NAME: servicemanagement.googleapis.com
NAME: servicenetworking.googleapis.com
NAME: serviceusage.googleapis.com
NAME: sql-component.googleapis.com
NAME: storage-api.googleapis.com
NAME: storage-component.googleapis.com
NAME: storage.googleapis.com

michael@cloudshell:~ (tef-olxyz)$ gcloud services list | grep NAME | wc -l
39

[fmichaelobrien]

stale bot timer restart - https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/.github/workflows/stale.yml#L21

[fmichaelobrien]

see related
#1189

Note: with the service enablements and role additions (owner on the sa-terraform-bootstrap@seed account we are ok and get past 2-environments

michael@cloudshell:~/tef-olapp/github/gcp-org (tef-olapp)$ gcloud scc notifications describe "scc-notify" --organization=${ORGANIZATION_ID}
ERROR: (gcloud.scc.notifications.describe) INVALID_ARGUMENT: Security Command Center Legacy has been permanently disabled as of June 7, 2021. Migrate to Security Command Center's Standard tier or Premium tier to maintain access to Security Command Center. See https://cloud.google.com/security-command-center/docs/quickstart-security-command-center for more info.

#1145

on a 2nd run after enabling roles/securitycenterAdmin on the super admin running the deployment
see PR chang https://github.com/terraform-google-modules/terraform-example-foundation/pull/1175/files#diff-510bd7bb34f359f116d7029a41db09abb222834e57c39feb56e2acf6f74be8eeR86

although this sa is only used for 0-bootstrap, as sa-terraform-bootstrap@seed... is used for steps 1+

check SCC

michael@cloudshell:~/tef-olxyz/github/gcp-org (tef-olxyz)$ mv ./envs/shared/terraform.example.tfvars ./envs/shared/terraform.tfvars
michael@cloudshell:~/tef-olxyz/github/gcp-org (tef-olxyz)$ export ORGANIZATION_ID=$(terraform -chdir="../pbmm-on-gcp-onboarding/0-bootstrap/" output -json common_config | jq '.org_id' --raw-output)
michael@cloudshell:~/tef-olxyz/github/gcp-org (tef-olxyz)$ gcloud scc notifications describe "scc-notify" --organization=${ORGANIZATION_ID}
ERROR: (gcloud.scc.notifications.describe) NOT_FOUND: Requested entity was not found.


Deployment of 1-org was ok

Step #2 - "tf plan": Plan: 288 to add, 0 to change, 0 to destroy.

Step #4 - "tf apply": Apply complete! Resources: 288 added, 0 changed, 0 destroyed.
Step #4 - "tf apply": 
Step #4 - "tf apply": Outputs:
Step #4 - "tf apply": 
Step #4 - "tf apply": base_net_hub_project_id = "prj-c-base-net-hub-5y8h"
Step #4 - "tf apply": billing_sink_names = {
Step #4 - "tf apply":   "prj" = "sk-c-logging-prj-billing-wh58"
Step #4 - "tf apply":   "pub" = "sk-c-logging-pub-billing-wh58"
Step #4 - "tf apply":   "sto" = "sk-c-logging-bkt-billing-wh58"
Step #4 - "tf apply": }
Step #4 - "tf apply": cai_monitoring_artifact_registry = "ar-cai-monitoring-4241"
Step #4 - "tf apply": cai_monitoring_asset_feed = "organizations/1064386348915/feeds/fd-cai-monitoring-4241"
Step #4 - "tf apply": cai_monitoring_bucket = "bkt-cai-monitoring-4241-sources-726972909649-us-central1"
Step #4 - "tf apply": cai_monitoring_topic = "top-cai-monitoring-4241-event"
Step #4 - "tf apply": common_folder_name = "folders/96486704059"
Step #4 - "tf apply": dns_hub_project_id = "prj-c-dns-hub-6f4b"
Step #4 - "tf apply": domains_to_allow = tolist([
Step #4 - "tf apply":   "obrienlabs.xyz",
Step #4 - "tf apply": ])
Step #4 - "tf apply": interconnect_project_id = "prj-c-interconnect-s2zg"
Step #4 - "tf apply": interconnect_project_number = "6710...54"
Step #4 - "tf apply": logs_export_project_linked_dataset_name = "projects/prj-c-logging-fn0h/locations/us-central1/buckets/AggregatedLogs/links/ds_c_prj_aggregated_logs_analytics"
Step #4 - "tf apply": logs_export_project_logbucket_name = "AggregatedLogs"
Step #4 - "tf apply": logs_export_pubsub_topic = "tp-org-logs-o9q2"
Step #4 - "tf apply": logs_export_storage_bucket_name = "bkt-prj-c-logging-fn0h-org-logs-o9q2"
Step #4 - "tf apply": network_folder_name = "folders/65...643"
Step #4 - "tf apply": org_audit_logs_project_id = "prj-c-logging-fn0h"
Step #4 - "tf apply": org_billing_logs_project_id = "prj-c-billing-logs-ve1w"
Step #4 - "tf apply": org_id = "10...5"
Step #4 - "tf apply": org_kms_project_id = "prj-c-kms-eeg3"
Step #4 - "tf apply": org_secrets_project_id = "prj-c-secrets-2lxo"
Step #4 - "tf apply": parent_resource_id = "73...67"
Step #4 - "tf apply": parent_resource_type = "folder"
Step #4 - "tf apply": restricted_net_hub_project_id = "prj-c-restricted-net-hub-a8d5"
Step #4 - "tf apply": restricted_net_hub_project_number = "12...62"
Step #4 - "tf apply": scc_notification_name = "scc-notify"
Step #4 - "tf apply": scc_notifications_project_id = "prj-c-scc-8zsj"
Step #4 - "tf apply": shared_vpc_projects = {
Step #4 - "tf apply":   "development" = {
Step #4 - "tf apply":     "base_shared_vpc_project_id" = "prj-d-shared-base-nlqs"
Step #4 - "tf apply":     "base_shared_vpc_project_number" = "10...505"
Step #4 - "tf apply":     "restricted_shared_vpc_project_id" = "prj-d-shared-restricted-j004"
Step #4 - "tf apply":     "restricted_shared_vpc_project_number" = "10..921"
Step #4 - "tf apply":   }
Step #4 - "tf apply":   "non-production" = {
Step #4 - "tf apply":     "base_shared_vpc_project_id" = "prj-n-shared-base-b12y"
Step #4 - "tf apply":     "base_shared_vpc_project_number" = "60..5"
Step #4 - "tf apply":     "restricted_shared_vpc_project_id" = "prj-n-shared-restricted-qnv6"
Step #4 - "tf apply":     "restricted_shared_vpc_project_number" = "85...26"
Step #4 - "tf apply":   }
Step #4 - "tf apply":   "production" = {
Step #4 - "tf apply":     "base_shared_vpc_project_id" = "prj-p-shared-base-oae0"
Step #4 - "tf apply":     "base_shared_vpc_project_number" = "17..1"
Step #4 - "tf apply":     "restricted_shared_vpc_project_id" = "prj-p-shared-restricted-2pqc"
Step #4 - "tf apply":     "restricted_shared_vpc_project_number" = "98..8"
Step #4 - "tf apply":   }
Step #4 - "tf apply": }
Step #4 - "tf apply": tags = {
Step #4 - "tf apply":   "environment_bootstrap" = "tagValues/281484537587812"
Step #4 - "tf apply":   "environment_development" = "tagValues/281483791828482"
Step #4 - "tf apply":   "environment_non-production" = "tagValues/281484388371311"
Step #4 - "tf apply":   "environment_production" = "tagValues/281483304603502"
Step #4 - "tf apply": }
Step #4 - "tf apply": policy-library/policies  doesn't match production; skipping
Step #4 - "tf apply": policy-library/lib  doesn't match production; skipping
Step #4 - "tf apply": policy-library/.git  doesn't match production; skipping
Finished Step #4 - "tf apply"

[obriensystems]

I'll raise in a separate PR but 3-networks-hub-and-spoke also needs the compute.orgSecurityPolicyAdmin or User or more recently compute.orgFirewallPolicyAdmin IAM role on the super admin to be able to view Hierarchical Firewall Policies created under the common branch

https://cloud.google.com/firewall/docs/firewall-policies#iam
Compute Organization Firewall Policy Admin
https://cloud.google.com/compute/docs/access/iam#compute.orgFirewallPolicyAdmin

As well as compute api on bootstrap project in step 11 of 3-networks-hub-and-spoke

michael@cloudshell:~/tef-olxyz/github/gcp-networks (tef-olxyz)$ gcloud services enable compute.googleapis.com
Operation "operations/acf.p2-438381210056-2415ed08-fad8-4333-8c8e-1017881efb60" finished successfully.

tested entire 3-networks-hub-and-spoke with these changes including the hierarchical firewall policy retry rename procedure

[fmichaelobrien]

fixed by

PRs merged to the terraform-example-foundation upstream repo

• 20240426 fix: Add roles and service enablements to bootstrap project #1175