Add semi-automated CI stuff

README.md

@@ -14,6 +14,11 @@ The actions in this repo are all used during the build, test and release of our
 - [snapcrafters/ci/sync-version](sync-version/README.md)
 - [snapcrafters/ci/test-snap-build](test-snap-build/README.md)
 
+## Other CI infrastructure
+
+- [terraform plans](terraform/)
+- [general automation scripts](scripts/)
+
 ### Usage
 
 You can see examples of these actions in use in the following repos:

scripts/README.md

@@ -0,0 +1,67 @@
+# Useful scripts
+
+This folder contains a bunch of useful links, scripts and commands for automating and managing Snapcrafters stuff.
+
+## Handy links
+
+* Show all open pull requests in our org: https://github.com/pulls?q=is%3Aopen+is%3Apr+org%3Asnapcrafters
+* Show all open calls for testing: https://github.com/issues?q=is%3Aopen+is%3Aissue+org%3Asnapcrafters+label%3Atesting
+
+## Get all distros that have the Discord snap installed
+
+```bash
+snapcraft metrics discord --name installed_base_by_operating_system --format table | cat > installed-base-raw.txt
+tail -n +2 installed-base-raw.txt | cut -d/ -f1 | uniq > unique-oses.txt
+```
+
+## Get credentials
+
+For more information on how to generate the credentials we use, take a look at [Permissions and Secrets](https://github.com/snapcrafters/.github/wiki/Permissions-and-Secrets)
+
+## Show contributor activity
+
+Admins can download an audit log of all activity in a repository. The URL takes the time period as GET parameter. For example, to get all activity in 2023, go to https://github.com/organizations/snapcrafters/settings/audit-log?q=created%3A2023 and download the CSV.
+
+The following oneliners are useful to get some interesting metrics.
+
+```shell
+# Useful columns
+# * actor: 11
+# * timestamp: 1
+
+# Get all unique contributors
+cat export-snapcrafters-1701710994.csv | cut -d "," -f 11 | sort | uniq > contributors-2023.txt
+
+# Get all active members
+comm -13 contributors-2023.txt members.txt
+
+# Get all members of our org that are not actually snapcrafters
+comm -13 members.txt snapcrafters-github-org-members.txt > non-member-org-members.txt
+```
+
+## Getting collaborators of snaps
+
+The snap store doesn't have an API to get or change the collaborators. The underlying commands scrape the HTML pages and simulate button clicks. The `<FILL>` stuf needs to be extracted from your browser by clicking on a button and looking at the cookies with the developer console.
+
+```shell
+COOKIE="csrftoken=<FILL>; sessionid=<FILL>"
+
+readarray -t array < merlijn-snapcrafters-snaps.txt
+
+# Get all collaborators of all snaps
+for SNAP in "${array[@]}"
+do
+   echo "# Collaborators of $SNAP"
+   curl -s https://dashboard.snapcraft.io/snaps/$SNAP/collaboration/ --cookie "${COOKIE}"  | sed '/data-test="shares-actions"/q' | grep @ | tr -d " " | cut -c 5- | rev | cut -c 6- | rev
+   echo
+done
+
+
+# Add a collaborator to all snaps
+for SNAP in "${array[@]}"
+do
+   echo "Adding bot to $SNAP"
+
+   curl -s https://dashboard.snapcraft.io/snaps/$SNAP/collaboration/ --cookie "${COOKIE}" -X POST -d "csrfmiddlewaretoken=<FILL>&emails=merlijn.sebrechts%2Bsnapcrafters-bot%40gmail.com&send-invites=" -H "Referer: https://dashboard.snapcraft.io/snaps/$SNAP/collaboration/"
+done
+```

terraform/.gitignore

@@ -0,0 +1,39 @@
+# Snapcrafters-specific files to ignore
+repos.csv
+
+
+# Local .terraform directories
+**/.terraform/*
+
+# .tfstate files
+*.tfstate
+*.tfstate.*
+
+# Crash log files
+crash.log
+crash.*.log
+
+# Exclude all .tfvars files, which are likely to contain sensitive data, such as
+# password, private keys, and other secrets. These should not be part of version 
+# control as they are data points which are potentially sensitive and subject 
+# to change depending on the environment.
+*.tfvars
+*.tfvars.json
+
+# Ignore override files as they are usually used to override resources locally and so
+# are not checked in
+override.tf
+override.tf.json
+*_override.tf
+*_override.tf.json
+
+# Include override files you do wish to add to version control using negated pattern
+# !example_override.tf
+
+# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
+# example: *tfplan*
+*tfplan*
+
+# Ignore CLI configuration files
+.terraformrc
+terraform.rc

terraform/README.md

@@ -0,0 +1,37 @@
+# Terraform files managing our organization
+
+## Setup
+
+1. Install terraform: `snap install terraform`
+1. [Install the GitHub CLI](https://github.com/cli/cli/blob/trunk/docs/install_linux.md)
+1. Login to the GitHub CLI and initialize terraform.
+
+   ```shell
+
+   cd tfcrafters
+   terraform init
+   gh auth cli
+   ```
+
+## Usage
+
+From the directory `tfcrafters`, execute the following commands.
+
+```shell
+# Generate the list of repositories
+echo "name" > repos.csv
+gh repo list snapcrafters --no-archived --limit 1000 --json name |  jq .[].name -r >> repos.csv
+
+# Apply the terraform config
+terraform apply
+```
+
+## Viewing the plan before executing
+
+When changing many repositories at once, it might be difficult to review the plan in the terminal. This is how you can easily investigate a plan using `less`.
+
+```shell
+terraform plan -out=tfplan
+terraform show | less -R
+terraform apply tfplan
+```

terraform/branch-protection-rules.tf

@@ -0,0 +1,19 @@
+resource "github_branch_protection" "list" {
+  for_each = {
+    for repo in csvdecode(file("repos.csv")) :
+    repo.name => repo
+  }
+
+  repository_id = each.value.name
+
+
+  pattern          = "*"
+
+  required_pull_request_reviews {
+    required_approving_review_count = 1
+    dismiss_stale_reviews  = true
+    pull_request_bypassers = [
+        "/snapcrafters-bot"
+    ]
+  }
+}

terraform/collaborators.tf

@@ -0,0 +1,23 @@
+resource "github_repository_collaborators" "repo_collaborators" {
+  for_each = {
+    for repo in csvdecode(file("repos.csv")) :
+    repo.name => repo
+  }
+
+  repository = each.value.name
+
+  user {
+    permission = "push"
+    username  = "snapcrafters-bot"
+  }
+
+  team {
+    permission = "push"
+    team_id = "reviewers"
+  }
+
+  team {
+    permission = "admin"
+    team_id = "administrators"
+  }
+}

terraform/import_branch_protection.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/bash
+repos=("discord" "ffmpeg-sdk" "ci-screenshots" ".github" "mattermost-desktop" "signal-desktop" "ci" "ghvmctl" "sublime-text" "webkitgtk-sdk" "jenkins" "eclipse" "sentry" "obs-studio" "tmnationsforever" "scummvm" "xonotic" "atom" "offlineimap" "sommelier-core" "android-studio" "ffmpeg" "opentyrian" "spelunky" "flightgear" "sdlpop" "mumble" "corsixth" "magic-wormhole" "alacritty" "gimp" "helm" "sublime-merge" "minetest" "opentoonz" "fork-and-rename-me" "axel" "dosbox-x" "ddgr" "term2048" "arduino" "photoscape" "sommelier" "gradle" "s4a" "cumulonimbus" "android-studio-canary" "cassandra" "cryptool" "thelounge" "irssi" "marktext" "fkill" "pyradio" "mdl" "mosaic" "pogo" "unifi" "nano" "opentoonz-morevna" "kompozer" "get-iplayer" "simplenote" "mutt" "inadyn" "duckmarines" "brackets" "wordpress-desktop" "mrrescue" "wire" "tcpie" "slack-term" "links" "wethr" "qtctl" "bridge-designer" "climate-trail" "hello-snap-travis-ci")
+
+for repo in ${repos[@]}; do
+  terraform import github_branch_protection.list['"'$repo'"'] $repo:main
+done

terraform/main.tf

@@ -0,0 +1,13 @@
+terraform {
+  required_providers {
+    github = {
+      source  = "integrations/github"
+      version = "~> 5.0"
+    }
+  }
+}
+
+# Configure the GitHub Provider
+provider "github" {
+  owner = "snapcrafters"
+}