From d35171c72f2c82066fde7241f97bb36b442351ac Mon Sep 17 00:00:00 2001 From: Merlijn Sebrechts Date: Tue, 5 Dec 2023 14:49:24 +0100 Subject: [PATCH 1/5] unwrap md --- README.md | 26 ++++++++------------------ 1 file changed, 8 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 738e83b..fe23dd0 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,10 @@ # Snapcrafters CI -This repository contains common actions and tools used throughout the Snapcrafters -[organisation](https://github.com/snapcrafters) for the testing and delivery of our snaps. +This repository contains common actions and tools used throughout the Snapcrafters [organisation](https://github.com/snapcrafters) for the testing and delivery of our snaps. ## Snapcrafters Actions -The actions in this repo are all used during the build, test and release of our snaps. Each of them -listed below has it's own README +The actions in this repo are all used during the build, test and release of our snaps. Each of them listed below has it's own README - [snapcrafters/ci/call-for-testing](call-for-testing/README.md) - [snapcrafters/ci/get-architectures](get-architectures/README.md) @@ -30,18 +28,10 @@ If you'd like to contribute to this repository, please feel free to fork and cre There are a few style guidelines to keep in mind: -- Code should be linted using [Prettier](https://prettier.io/). You can achieve this with `make -lint` and `make format`. The only requirements are [`npx`](https://www.npmjs.com/package/npx) and - [`shellcheck`](https://github.com/koalaman/shellcheck). -- When defining inputs/outputs in `action.yaml`, or listing them in the tables within `README.md`, - they should be listed in alphabetical order for easy reading and updating. -- Github Action inputs/outputs should be named all lowercase, separated by `-` where needed. The - applies to inputs/outputs to actions themselves, and for individual steps within the actions. For - example: `snap-name` or `token`. -- Environment variables referring to repository level secrets and variables should be named all - uppercase, and separated by `_`. For example: `SNAPCRAFTERS_BOT_COMMIT`. -- Step/job level environment variables should be named all lowercase, and separated by `_`. For - example: `snap_name` or `yaml_path`. +- Code should be linted using [Prettier](https://prettier.io/). You can achieve this with `make lint` and `make format`. The only requirements are [`npx`](https://www.npmjs.com/package/npx) and [`shellcheck`](https://github.com/koalaman/shellcheck). +- When defining inputs/outputs in `action.yaml`, or listing them in the tables within `README.md`, they should be listed in alphabetical order for easy reading and updating. +- Github Action inputs/outputs should be named all lowercase, separated by `-` where needed. The applies to inputs/outputs to actions themselves, and for individual steps within the actions. For example: `snap-name` or `token`. +- Environment variables referring to repository level secrets and variables should be named all uppercase, and separated by `_`. For example: `SNAPCRAFTERS_BOT_COMMIT`. +- Step/job level environment variables should be named all lowercase, and separated by `_`. For example: `snap_name` or `yaml_path`. - All `bash` variables should be quoted. -- Scripts of all kinds, including those within actions `run:|` directives should follow the [Google - styleguide](https://google.github.io/styleguide/shellguide.html) +- Scripts of all kinds, including those within actions `run:|` directives should follow the [Google styleguide](https://google.github.io/styleguide/shellguide.html) From a6869c67413b94945c1b989018ce0cc7a10b5fff Mon Sep 17 00:00:00 2001 From: Merlijn Sebrechts Date: Tue, 5 Dec 2023 17:35:38 +0100 Subject: [PATCH 2/5] add terraform files --- terraform/.gitignore | 39 +++++++++++++++++++++++++++ terraform/.terraform.lock.hcl | 24 +++++++++++++++++ terraform/README.md | 37 +++++++++++++++++++++++++ terraform/branch-protection-rules.tf | 19 +++++++++++++ terraform/collaborators.tf | 23 ++++++++++++++++ terraform/import_branch_protection.sh | 6 +++++ terraform/main.tf | 13 +++++++++ 7 files changed, 161 insertions(+) create mode 100644 terraform/.gitignore create mode 100644 terraform/.terraform.lock.hcl create mode 100644 terraform/README.md create mode 100644 terraform/branch-protection-rules.tf create mode 100644 terraform/collaborators.tf create mode 100755 terraform/import_branch_protection.sh create mode 100644 terraform/main.tf diff --git a/terraform/.gitignore b/terraform/.gitignore new file mode 100644 index 0000000..923b242 --- /dev/null +++ b/terraform/.gitignore @@ -0,0 +1,39 @@ +# Snapcrafters-specific files to ignore +repos.csv + + +# Local .terraform directories +**/.terraform/* + +# .tfstate files +*.tfstate +*.tfstate.* + +# Crash log files +crash.log +crash.*.log + +# Exclude all .tfvars files, which are likely to contain sensitive data, such as +# password, private keys, and other secrets. These should not be part of version +# control as they are data points which are potentially sensitive and subject +# to change depending on the environment. +*.tfvars +*.tfvars.json + +# Ignore override files as they are usually used to override resources locally and so +# are not checked in +override.tf +override.tf.json +*_override.tf +*_override.tf.json + +# Include override files you do wish to add to version control using negated pattern +# !example_override.tf + +# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan +# example: *tfplan* +*tfplan* + +# Ignore CLI configuration files +.terraformrc +terraform.rc diff --git a/terraform/.terraform.lock.hcl b/terraform/.terraform.lock.hcl new file mode 100644 index 0000000..38830a9 --- /dev/null +++ b/terraform/.terraform.lock.hcl @@ -0,0 +1,24 @@ +# This file is maintained automatically by "terraform init". +# Manual edits may be lost in future updates. + +provider "registry.terraform.io/integrations/github" { + version = "5.42.0" + constraints = "~> 5.0" + hashes = [ + "h1:vHTdYL6eXJfUzz4bs0ICyg2f8ct/K2EnGAjwLrAmL3U=", + "zh:0f97039c6b70295c4a82347bc8a0bcea700b3fb3df0e0be53585da025584bb7c", + "zh:12e78898580cc2a72b5f2a77e191b158f88e974b0500489b691f34842288745c", + "zh:23660933e4f00293c0d4d6cd6b4d72e382c0df46b70cecf22b5c4c090d3b61e3", + "zh:74119174b46d8d197dd209a246bf8b5db113c66467e02c831e68a8ceea312d3e", + "zh:829c4c0c202fc646eb0e1759eb9c8f0757df5295be2d3344b8fd6ca8ce9ef33b", + "zh:92043e667f520aee4e08a10a183ad5abe5487f3e9c8ad5a55ea1358b14b17b1a", + "zh:998909806b4ff42cf480fcd359ec1f12b868846f89284b991987f55de24876b7", + "zh:9f758447db3bf386516562abd6da1e54d22ddc207bda25961d2b5b049f32da0f", + "zh:a6259215612d4d6a281c671b2d5aa3a0a0b0a3ae92ed60b633998bb692e922d3", + "zh:ad7d78056beb44191911db9443bf5eec41a3d60e7b01def2a9e608d1c4288d27", + "zh:b697e7b0abef3000e1db482c897b82cd455621b488bb6c4cd3d270763d7b08ac", + "zh:db8e849eded8aebff780f89ab7e1339053d2f15c1c8f94103d70266a090527ad", + "zh:e5bdbb85fb148dd75877a7b94b595d4e8680e495c241db02c4b12b91e9d08953", + "zh:ee812c5fd77d3817fb688f720e5eb42d7ff04db67a125de48b05458c9f657483", + ] +} diff --git a/terraform/README.md b/terraform/README.md new file mode 100644 index 0000000..a0217b1 --- /dev/null +++ b/terraform/README.md @@ -0,0 +1,37 @@ +# Terraform files managing our organization + +## Setup + +1. Install terraform: `snap install terraform` +1. [Install the GitHub CLI](https://github.com/cli/cli/blob/trunk/docs/install_linux.md) +1. Login to the GitHub CLI and initialize terraform. + + ```shell + + cd tfcrafters + terraform init + gh auth cli + ``` + +## Usage + +From the directory `tfcrafters`, execute the following commands. + +```shell +# Generate the list of repositories +echo "name" > repos.csv +gh repo list snapcrafters --no-archived --limit 1000 --json name | jq .[].name -r >> repos.csv + +# Apply the terraform config +terraform apply +``` + +## Viewing the plan before executing + +When changing many repositories at once, it might be difficult to review the plan in the terminal. This is how you can easily investigate a plan using `less`. + +```shell +terraform plan -out=tfplan +terraform show | less -R +terraform apply tfplan +``` diff --git a/terraform/branch-protection-rules.tf b/terraform/branch-protection-rules.tf new file mode 100644 index 0000000..ae1d5fa --- /dev/null +++ b/terraform/branch-protection-rules.tf @@ -0,0 +1,19 @@ +resource "github_branch_protection" "list" { + for_each = { + for repo in csvdecode(file("repos.csv")) : + repo.name => repo + } + + repository_id = each.value.name + + + pattern = "*" + + required_pull_request_reviews { + required_approving_review_count = 1 + dismiss_stale_reviews = true + pull_request_bypassers = [ + "/snapcrafters-bot" + ] + } +} diff --git a/terraform/collaborators.tf b/terraform/collaborators.tf new file mode 100644 index 0000000..c397276 --- /dev/null +++ b/terraform/collaborators.tf @@ -0,0 +1,23 @@ +resource "github_repository_collaborators" "repo_collaborators" { + for_each = { + for repo in csvdecode(file("repos.csv")) : + repo.name => repo + } + + repository = each.value.name + + user { + permission = "push" + username = "snapcrafters-bot" + } + + team { + permission = "push" + team_id = "reviewers" + } + + team { + permission = "admin" + team_id = "administrators" + } +} diff --git a/terraform/import_branch_protection.sh b/terraform/import_branch_protection.sh new file mode 100755 index 0000000..3865b7a --- /dev/null +++ b/terraform/import_branch_protection.sh @@ -0,0 +1,6 @@ +#!/usr/bin/bash +repos=("discord" "ffmpeg-sdk" "ci-screenshots" ".github" "mattermost-desktop" "signal-desktop" "ci" "ghvmctl" "sublime-text" "webkitgtk-sdk" "jenkins" "eclipse" "sentry" "obs-studio" "tmnationsforever" "scummvm" "xonotic" "atom" "offlineimap" "sommelier-core" "android-studio" "ffmpeg" "opentyrian" "spelunky" "flightgear" "sdlpop" "mumble" "corsixth" "magic-wormhole" "alacritty" "gimp" "helm" "sublime-merge" "minetest" "opentoonz" "fork-and-rename-me" "axel" "dosbox-x" "ddgr" "term2048" "arduino" "photoscape" "sommelier" "gradle" "s4a" "cumulonimbus" "android-studio-canary" "cassandra" "cryptool" "thelounge" "irssi" "marktext" "fkill" "pyradio" "mdl" "mosaic" "pogo" "unifi" "nano" "opentoonz-morevna" "kompozer" "get-iplayer" "simplenote" "mutt" "inadyn" "duckmarines" "brackets" "wordpress-desktop" "mrrescue" "wire" "tcpie" "slack-term" "links" "wethr" "qtctl" "bridge-designer" "climate-trail" "hello-snap-travis-ci") + +for repo in ${repos[@]}; do + terraform import github_branch_protection.list['"'$repo'"'] $repo:main +done diff --git a/terraform/main.tf b/terraform/main.tf new file mode 100644 index 0000000..b54f68f --- /dev/null +++ b/terraform/main.tf @@ -0,0 +1,13 @@ +terraform { + required_providers { + github = { + source = "integrations/github" + version = "~> 5.0" + } + } +} + +# Configure the GitHub Provider +provider "github" { + owner = "snapcrafters" +} From c2cb6fb9d24baa2f4a71f00289c25fc0a1cc1c7b Mon Sep 17 00:00:00 2001 From: Merlijn Sebrechts Date: Tue, 5 Dec 2023 17:36:18 +0100 Subject: [PATCH 3/5] add scripts --- scripts/README.md | 67 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+) create mode 100644 scripts/README.md diff --git a/scripts/README.md b/scripts/README.md new file mode 100644 index 0000000..9d92245 --- /dev/null +++ b/scripts/README.md @@ -0,0 +1,67 @@ +# Useful scripts + +This folder contains a bunch of useful links, scripts and commands for automating and managing Snapcrafters stuff. + +## Handy links + +* Show all open pull requests in our org: https://github.com/pulls?q=is%3Aopen+is%3Apr+org%3Asnapcrafters +* Show all open calls for testing: https://github.com/issues?q=is%3Aopen+is%3Aissue+org%3Asnapcrafters+label%3Atesting + +## Get all distros that have the Discord snap installed + +```bash +snapcraft metrics discord --name installed_base_by_operating_system --format table | cat > installed-base-raw.txt +tail -n +2 installed-base-raw.txt | cut -d/ -f1 | uniq > unique-oses.txt +``` + +## Get credentials + +For more information on how to generate the credentials we use, take a look at [Permissions and Secrets](https://github.com/snapcrafters/.github/wiki/Permissions-and-Secrets) + +## Show contributor activity + +Admins can download an audit log of all activity in a repository. The URL takes the time period as GET parameter. For example, to get all activity in 2023, go to https://github.com/organizations/snapcrafters/settings/audit-log?q=created%3A2023 and download the CSV. + +The following oneliners are useful to get some interesting metrics. + +```shell +# Useful columns +# * actor: 11 +# * timestamp: 1 + +# Get all unique contributors +cat export-snapcrafters-1701710994.csv | cut -d "," -f 11 | sort | uniq > contributors-2023.txt + +# Get all active members +comm -13 contributors-2023.txt members.txt + +# Get all members of our org that are not actually snapcrafters +comm -13 members.txt snapcrafters-github-org-members.txt > non-member-org-members.txt +``` + +## Getting collaborators of snaps + +The snap store doesn't have an API to get or change the collaborators. The underlying commands scrape the HTML pages and simulate button clicks. The `` stuf needs to be extracted from your browser by clicking on a button and looking at the cookies with the developer console. + +```shell +COOKIE="csrftoken=; sessionid=" + +readarray -t array < merlijn-snapcrafters-snaps.txt + +# Get all collaborators of all snaps +for SNAP in "${array[@]}" +do + echo "# Collaborators of $SNAP" + curl -s https://dashboard.snapcraft.io/snaps/$SNAP/collaboration/ --cookie "${COOKIE}" | sed '/data-test="shares-actions"/q' | grep @ | tr -d " " | cut -c 5- | rev | cut -c 6- | rev + echo +done + + +# Add a collaborator to all snaps +for SNAP in "${array[@]}" +do + echo "Adding bot to $SNAP" + + curl -s https://dashboard.snapcraft.io/snaps/$SNAP/collaboration/ --cookie "${COOKIE}" -X POST -d "csrfmiddlewaretoken=&emails=merlijn.sebrechts%2Bsnapcrafters-bot%40gmail.com&send-invites=" -H "Referer: https://dashboard.snapcraft.io/snaps/$SNAP/collaboration/" +done +``` From 42afd76bf81bef0616715aecb3c424949a5c4c1e Mon Sep 17 00:00:00 2001 From: Merlijn Sebrechts Date: Tue, 5 Dec 2023 17:36:52 +0100 Subject: [PATCH 4/5] add overview --- README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/README.md b/README.md index fe23dd0..4ff38ac 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,11 @@ The actions in this repo are all used during the build, test and release of our - [snapcrafters/ci/sync-version](sync-version/README.md) - [snapcrafters/ci/test-snap-build](test-snap-build/README.md) +## Other CI infrastructure + +- [terraform plans](terraform/) +- [general automation scripts](scripts/) + ### Usage You can see examples of these actions in use in the following repos: From c8871ba5085bbbc84edb17fc67ba2fe5e0cc44e7 Mon Sep 17 00:00:00 2001 From: Merlijn Sebrechts Date: Tue, 5 Dec 2023 20:54:26 +0100 Subject: [PATCH 5/5] Update terraform/README.md Co-authored-by: Jon Seager --- terraform/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/terraform/README.md b/terraform/README.md index a0217b1..be26fa4 100644 --- a/terraform/README.md +++ b/terraform/README.md @@ -15,7 +15,7 @@ ## Usage -From the directory `tfcrafters`, execute the following commands. +From the directory `terraform`, execute the following commands. ```shell # Generate the list of repositories