Add helper scripts for voter rolls (#2324)

* Add helper scripts for voter rolls

* Fix cspell config

* Fix cspell config

* Filter out empty lines

.cspell.yaml

@@ -1,5 +1,5 @@
 version: 0.2
-ignorePaths: ["elections/**/*", "tools/github/*"]
+ignorePaths: ["elections/**/*", "tools/github/*", "scripts/**/*.py", "scripts/**/*.sh"]
 # Ignore certain patterns in the sigs.yml file
 patterns:
   - name: Slack Channel ID
@@ -21,6 +21,7 @@ words:
     - Docu
     - datadog
     - devex
+    - devstats
     - dynatrace
     - easycla
     - eiffel

scripts/gc-elections/README.md

@@ -0,0 +1,42 @@
+# GC Election Voter Roll Generation
+These helper scripts are used to generate the initial voter roll for the OpenTelemetry Governance Committee elections, before any exemptions to the automatic process are considered.
+
+Two environment variables are used across these scripts to configure output files for voters rolls:
+
+* `VOTERS_ROLL_PATH`: the path to the output CSV file with the voters roll and number of contributions (defaults to `./voters-roll.csv`).
+* `VOTERS_ROLL_HELIOS_PATH`: the path to the output CSV file with the voters roll in a format accepted by Helios Voting (defaults to `./voters-roll-helios.csv`).
+
+## 1. Generate the voters roll
+The `generate-voters-roll.py` script generates a CSV file with the GitHub logins and contributions of the eligible voters for the upcoming elections. It queries the PostgreSQL Data Source backing `opentelemetry.devstats.cncf.io` to get the list of contributors eligible for voting. It then uses the GitHub REST API to get the GitHub login (capitalized) for each user.
+
+Although not required, it is recommended to set a `GITHUB_TOKEN` environment variable with a token with minimal permissions. This avoids GitHub's limits for unauthenticated requests.
+
+Example usage:
+```bash
+export GITHUB_TOKEN=your_token_here
+python generate-voters-roll.py
+```
+
+## 2. Create GitHub comments
+Using the generated voters roll file indicated by `$VOTERS_ROLL_PATH`, the `create-github-comments.sh` script tags eligible voters on a given GitHub issue, inviting them to vote in the next election.
+
+To workaround limits of mentions on a single comment, the script creates multiple comments in batches of 50 voters.
+
+This script takes two arguments:
+* `-i issue_url`: the GitHub issue URL to add comments to (e.g. `https://github.com/open-telemetry/community/issues/1173`)
+* `-d bool`: dry-run mode, which only prints the comments that would be created without actually creating them (defaults to `true`).
+
+For this script to work, you must have the GitHub CLI (`gh`) installed and authenticated.
+
+Example usage:
+```bash
+./create-github-comments.sh -i https://github.com/open-telemetry/community/issues/1173 -d true
+```
+
+## 3. Convert voters roll to Helios format
+The `convert-voters-roll-to-helios.py` script converts the voters roll file indicated by `$VOTERS_ROLL_PATH` to a format accepted by Helios Voting. It generates a new CSV file in `$VOTERS_ROLL_HELIOS_PATH`  with the GitHub logins of the eligible voters for the upcoming elections.
+
+Example usage:
+```bash
+python convert-voters-roll-to-helios.py
+```

scripts/gc-elections/convert-voter-roll-to-helios.py

@@ -0,0 +1,17 @@
+import csv
+import os
+
+VOTERS_ROLL_PATH = os.getenv('VOTERS_ROLL_PATH', './voters-roll.csv')
+VOTERS_ROLL_HELIOS_PATH = os.getenv('VOTERS_ROLL_HELIOS_PATH', './voters-roll-helios.csv')
+
+# Open the input and output CSV files
+with open(VOTERS_ROLL_PATH, mode='r') as infile, open(VOTERS_ROLL_HELIOS_PATH, mode='w', newline='') as outfile:
+    reader = csv.reader(infile)
+    writer = csv.writer(outfile)
+
+    # Process each row in the input file
+    for row in reader:
+        if row is not None and len(row) > 0:
+            writer.writerow(['github', row[0]])
+
+print(f"Voters file for Helios Voting generated as {VOTERS_ROLL_HELIOS_PATH}")

scripts/gc-elections/create-gihub-comments.sh

@@ -0,0 +1,69 @@
+#!/bin/bash
+
+file=${VOTERS_ROLL_PATH:-"./voters-roll.csv"}
+dryrun="true"
+
+command -v gh 1>/dev/null 2>&1
+if [ $? != 0 ]; then
+    echo "The command 'gh' is expected to exist and be configured in order to make comments to GitHub issues."
+    exit 1
+fi
+
+while getopts f:i:d: flag
+do
+    case "${flag}" in
+        i) issue=${OPTARG};;
+        d) dryrun=${OPTARG};;
+    esac
+done
+
+if [[ -z $issue ]]; then
+    echo "Please, specify the issue URL. Ex.: $0 -i https://github.com/open-telemetry/community/issues/1173"
+    exit 0
+fi
+
+if [ ! -f $file ]; then
+    echo "Please, specify an existing input file as a VOTERS_ROLL_PATH env variable"
+    exit 0
+fi
+
+header="We invite the people mentioned in this comment to vote in the OpenTelemetry Governance Committee, as they have provided more than 20 contributions to the project over the last year via GitHub. Your contributions were comments, code reviews, pull requests, among others. Thank you!\n"
+msg=${header}
+counter=0
+
+echo -e "Reading the file ${file} and creating comments for the issue ${issue}\n"
+while read -r line;
+do
+    re="^(.+)\,(.+)$"
+    [[ $line =~ $re ]]
+    handle="${BASH_REMATCH[1]}"
+    contributions="${BASH_REMATCH[2]//[$'\r\n']}"
+
+    if [[ -z $handle ]]; then
+        continue
+    fi
+
+    msg="${msg}\\n* @${handle}"
+    ((counter++))
+
+    if (( counter >= 50 )); then
+        # reached 50 mentions, create the comment
+        if [ "$dryrun" = true ]; then
+            echo -e $msg
+        else
+            echo -e $msg | gh issue comment "${issue}" -F -
+        fi
+
+        # reset
+        msg=${header}
+        counter=0
+    fi
+done < ${file}
+
+if (( counter > 0 )); then
+    if [ "$dryrun" = true ]; then
+        echo -e $msg
+    else
+        echo -e $msg | gh issue comment "${issue}" -F -
+    fi
+fi

scripts/gc-elections/generate-voters-roll.py

@@ -0,0 +1,92 @@
+import csv
+import os
+import requests
+import time
+
+VOTERS_ROLL_PATH = os.getenv('VOTERS_ROLL_PATH', './voters-roll.csv')
+GH_TOKEN = os.getenv('GITHUB_TOKEN')
+
+
+# Get GitHub login from lowercase username
+def get_github_login(username):
+    print(f"Getting GitHub login for {username}")
+    time.sleep(0.1)  # Sleep for 100ms to avoid rate limiting
+    url = f'https://api.github.com/users/{username}'
+    headers = {}
+    if GH_TOKEN:
+        headers = {
+            'Authorization': f'token {GH_TOKEN}'
+        }
+    response = requests.get(url, headers=headers)
+    if response.status_code == 200:
+        data = response.json()
+        return data['login']
+    else:
+        print(f"Failed to retrieve login for {username}: {response.status_code}")
+        return None
+
+
+# Use devstats to get users and contributions with more than 20 contributions in the last year
+def get_users_and_contributions():
+    print("Getting contributions data from opentelemetry.devstats.cncf.io")
+    # Define the URL and headers
+    url = 'https://opentelemetry.devstats.cncf.io/api/ds/query'
+    headers = {
+        'Accept': 'application/json',
+        'content-type': 'application/json'
+    }
+
+    # Define the JSON payload
+    payload = {
+        "queries": [
+            {
+                "datasource": {
+                    "uid": "P172949F98CB31475",
+                    "type": "postgres"
+                },
+                "rawSql": "select sub.name as name, sub.value as contributions from (select split_part(name, '$$$', 1) as name, sum(value) as value from shdev where series = 'hdev_contributionsallall' and period = 'y' group by split_part(name, '$$$', 1) ) sub where sub.value >= 20 order by name",
+                "format": "table"
+            }
+        ],
+        "range": {
+            "from": "now",
+            "to": "now"
+        }
+    }
+
+    # Make the HTTP POST request
+    response = requests.post(url, headers=headers, json=payload)
+
+    # Check if the request was successful
+    if response.status_code == 200:
+        return response.json()
+    else:
+        print(f"Failed to retrieve contributions data: {response.status_code}")
+        return None
+
+
+# Create a CSV file with the voters rolls
+def create_voters_rolls(data):
+    frames = data['results']['A']['frames']
+    rows = []
+
+    for frame in frames:
+        values = frame['data']['values']
+        names = values[0]
+        contributions = values[1]
+        for i in range(len(names)):
+            login = get_github_login(names[i])
+            if login:
+                rows.append([login, contributions[i]])
+
+    # Write the data to a CSV file
+    print(f"Writing data to {VOTERS_ROLL_PATH}")
+    with open(VOTERS_ROLL_PATH, mode='w', newline='') as file:
+        writer = csv.writer(file)
+        writer.writerows(rows)
+        file.write('\n')
+
+
+# Call the function
+devstas_data = get_users_and_contributions()
+create_voters_rolls(devstas_data)