Create simple tag generator for ADO pipelines (#222)

Creates a script and pipeline that allows for automated tagging based on
release version and commit messages. This template should be called for
an Azure Devops hosted repository. The consumer of this template is
expected to ensure the proper permissions for the build agent to be able
to create the tag and commit tag notes.

The TagGenerator Script provides the following functionality:
- Scans git history for the most recent matching tag, e.g. 202302.5.10
- Generates the new tag version, incrementing the "major" version based
on the presence of breaking changes.
- Generates release notes including commits by type, links to Azure
Devops PRs, and contributors.

Jobs/GenerateTag.yml

@@ -0,0 +1,88 @@
+## @file
+# Template file used to generate tags on ADO. This template requires that the
+# consumer specifies this repository as a resource named mu_devops.
+#
+# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+
+parameters:
+  - name: major_version
+    displayName: The major version.
+    type: string
+    default: ""
+  - name: git_name
+    displayName: Name to use for creating tag.
+    type: string
+    default: ""
+  - name: git_email
+    displayName: Email to use for creating tag.
+    type: string
+    default: ""
+  - name: notes_file
+    displayName: Path to the notes file to generate.
+    type: string
+    default: "ReleaseNotes.md"
+  - name: extra_prepare_steps
+    displayName: Extra Prepare Steps
+    type: stepList
+    default:
+      - script: echo No extra prepare steps provided
+
+jobs:
+  - job: Create_Release_Tag
+    steps:
+      - checkout: self
+        clean: true
+        fetchTags: true
+        persistCredentials: true
+        path: "target"
+        fetchDepth: 0
+
+      - checkout: mu_devops
+        path: "mu_devops"
+        fetchDepth: 1
+
+      - template: ../Steps/SetupPythonPreReqs.yml
+        parameters:
+          install_pip_modules: false
+
+      - script: |
+          python -m pip install --upgrade pip
+          pip install GitPython
+        displayName: "Install Dependencies"
+
+      - ${{ parameters.extra_prepare_steps }}
+
+      # Checking the parameters should occur after extra_prepare_steps in case
+      # the caller is using those steps to initialize a consumed variable.
+      - script: |
+          if [ -z "${{ parameters.major_version }}"] || \
+             [ -z "${{ parameters.git_name }}"] || \
+             [ -z "${{ parameters.git_email }}"]
+          then
+            echo "##vso[task.complete result=Failed;]"
+          fi
+        displayName: "Check Parameters"
+
+      - script: |
+          git config --global user.name "${{ parameters.git_name }}"
+          git config --global user.email "${{ parameters.git_email }}"
+        displayName: "Setup Git"
+
+      - script: |
+          python mu_devops/Scripts/TagGenerator/TagGenerator.py -r target/ --major ${{ parameters.major_version }} -v --printadovar tag_name --notes target/${{ parameters.notes_file }} --url $(Build.Repository.Uri)
+        displayName: "Run Tag Generator"
+        workingDirectory: $(Agent.BuildDirectory)
+
+      - script: |
+          set -e
+          git branch
+          git add ${{ parameters.notes_file }}
+          git commit -m "Release notes for $(tag_name)"
+          git tag $(tag_name)
+          git push origin HEAD:$(Build.SourceBranchName)
+          git push origin $(tag_name)
+        continueOnError: false
+        displayName: "Create Tag"
+        workingDirectory: $(Agent.BuildDirectory)/target

Scripts/TagGenerator/Readme.md

@@ -0,0 +1,36 @@
+# Tag Generator Script
+
+[TagGenerator.py](./TagGenerator.py) will automatically generate the next version tag
+and add notes to a release notes file for the current git HEAD. The Tag Generator
+script is primarily intended for use by the [Generate Tag Pipeline](../../Jobs/GenerateTag.yml)
+but can be used locally as well. This script is intended to be used for ADO repositories,
+but may be used for GitHub, though certain features may not work in their current
+form such as PR links in tag notes.
+
+## Versioning Scheme
+
+This script uses the `major.minor.patch` versioning scheme, but diverges from semantic
+versioning in some significant ways.
+
+- `major version` - Indicates the EDKII release tag that the repo is compiled against, e.g. `202302`.
+- `minor version` - Indicates the breaking change number since the last major version change.
+- `patch version` - Indicates the number of non-breaking changes since the last minor version.
+
+## Repro Requirements
+
+For this script to work properly it makes assumptions about the repository and
+project structure for tag history and generating notes.
+
+### Pull Request Template
+
+To determine what kind of change each commit is, this script expects certain strings
+exists in the commit message. It is recommended consumers include these in the PR
+templates for the repository. The script expects `[x] Breaking Change` for breaking
+changes, `[x] Security Fix` for security changes, and `[x] New Feature` for new
+features. The template forms of these are provided below.
+
+```md
+- [ ] Breaking Change
+- [ ] Security Fix
+- [ ] New Feature
+```

Scripts/TagGenerator/TagGenerator.py

@@ -0,0 +1,247 @@
+#
+# Module for automatically tagging a commit with a release version.
+#
+# Copyright (c) Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+
+import argparse
+import re
+import time
+import logging
+from git import Repo
+
+
+def main():
+    """Main entry point for the TagGenerator script"""
+
+    args = get_cli_options()
+    repo = Repo(args.repo)
+    log_level = logging.DEBUG if args.verbose else logging.INFO
+    logging.basicConfig(format="%(levelname)s - %(message)s", level=log_level)
+
+    logging.debug(f"Generating tag name for: {repo.head.commit}")
+
+    # Get the previous tag and increment values as needed.
+    prev_tag, breaking, commits = get_last_tag(repo, args.first)
+
+    # Generate the new tag name
+    minor = 0
+    patch = 0
+    if prev_tag is not None:
+        if prev_tag.commit == repo.head.commit:
+            logging.info("No changes since last tag")
+            return
+
+        version_split = prev_tag.name.split('.')
+        if version_split[0] == args.major:
+            minor = int(version_split[1])
+            patch = int(version_split[2])
+            if breaking:
+                minor += 1
+                patch = 0
+            else:
+                patch += 1
+        else:
+            logging.critical(
+                f"Different major version. {version_split[0]} -> {args.major}")
+            if int(version_split[0]) > int(args.major):
+                raise Exception("Major version has decreased!")
+
+    version = f"{args.major}.{minor}.{patch}"
+    logging.info(f"New tag: {version}")
+
+    # Before going further, ensure this is not a duplicate. This can happen if
+    # there are tags detached from their intended branch.
+    for tag in repo.tags:
+        if tag.name == version:
+            raise Exception(
+                "The new tag name already exists! Check tags already present in the repo.")
+
+    if args.create:
+        repo.create_tag(version, message=f"Release Tag {version}")
+
+    if args.notes is not None:
+        generate_notes(version, commits, args.notes, args.url)
+
+    if args.printadovar is not None:
+        print(f"##vso[task.setvariable variable={args.printadovar};]{version}")
+
+
+def get_cli_options():
+    parser = argparse.ArgumentParser()
+
+    parser.add_argument("-r", "--repo", default=".",
+                        help="Path to the repo directory.")
+    parser.add_argument("-m", "--major", type=str, required=True,
+                        help="The major release version. This must be provided")
+    parser.add_argument("-n", "--notes", type=str,
+                        help="Provides path to the release notes markdown file.")
+    parser.add_argument("--printadovar", type=str,
+                        help="An ADO variable to set to the tag name")
+    parser.add_argument("--url", type=str, default="",
+                        help="The URL to the repo, used for tag notes.")
+    parser.add_argument("--create", action="store_true",
+                        help="Create the new tag")
+    parser.add_argument("--first", action="store_true",
+                        help="Indicates this is expected to be the first tag.")
+    parser.add_argument("-v", "--verbose", action="store_true",
+                        help="Enabled verbose script prints.")
+
+    args = parser.parse_args()
+    return args
+
+
+def get_last_tag(repo, first):
+    """Retrieves the last tag name in the given HEAD history. This will
+    exclude any tag that does not match the #.#.# format.
+
+    repo - Provides the Git Repo object which will be searched.
+    first - Indicates this may be the first tag generation run.
+    """
+
+    breaking = False
+    included_commits = []
+    commits = repo.iter_commits(repo.head.commit)
+
+    # Find all the eligible tags first.
+    tags = []
+    pattern = re.compile("^[0-9]+\.[0-9]+\.[0-9]+$")
+    for tag in repo.tags:
+        if pattern.match(tag.name) is None:
+            logging.debug(f"Skipping unrecognized tag format. Tag: {tag}")
+            continue
+
+        tags.append(tag)
+
+    # Find the most recent commit with a tag.
+    for commit in commits:
+        if is_breaking_change(commit.message):
+            breaking = True
+
+        logging.debug(f"Checking commit {commit.hexsha}")
+        for tag in tags:
+            if tag.commit == commit:
+                logging.info(f"Previous tag: {tag} Breaking: {breaking}")
+                return tag, breaking, included_commits
+
+        included_commits.append(commit)
+
+    if not first:
+        raise Exception("No previous tag found!")
+
+    # No tag found, return all commits and non-breaking.
+    logging.info("No previous tag found.")
+    return None, False, commits
+
+
+def generate_notes(version, commits, filepath, url):
+    """Generates notes for the provided tag version including the provided commit
+    list. These notes will include the list of Breaking, Security, and other
+    commits. These notes will be prepended to the file specify by filepath
+
+    version - The tag version string.
+    commits - The list of commits since the last tag.
+    filepath - The path to the file to prepend the notes to.
+    url - The URL of the repository.
+    """
+
+    notes_file = open(filepath, 'r+')
+    old_lines = notes_file.readlines()
+
+    # Collect all the notable changes
+    breaking_changes = []
+    security_changes = []
+    features = []
+    other_changes = []
+    contributors = []
+
+    for commit in commits:
+        if commit.author not in contributors:
+            contributors.append(commit.author)
+
+        if is_breaking_change(commit.message):
+            breaking_changes.append(commit)
+        elif is_security_change(commit.message):
+            security_changes.append(commit)
+        elif is_new_feature(commit.message):
+            features.append(commit)
+        else:
+            other_changes.append(commit)
+
+    timestamp = time.strftime("%a, %D %T", time.gmtime())
+    notes = f"\n# Release {version}\n\n"
+    notes += f"Created {timestamp} GMT\n\n"
+    notes += f"{len(commits)} commits. {len(contributors)} contributors.\n"
+
+    if len(breaking_changes) > 0:
+        notes += f"\n## Breaking Changes\n\n"
+        notes += get_change_list(breaking_changes, url)
+
+    if len(security_changes) > 0:
+        notes += f"\n## Security Changes\n\n"
+        notes += get_change_list(security_changes, url)
+
+    if len(features) > 0:
+        notes += f"\n## New Features\n\n"
+        notes += get_change_list(features, url)
+
+    if len(other_changes) > 0:
+        notes += f"\n## Changes\n\n"
+        notes += get_change_list(other_changes, url)
+
+    notes += "\n## Contributors\n\n"
+    for contributor in contributors:
+        notes += f"- {contributor.name} <<{contributor.email}>>"
+
+    notes += "\n"
+
+    # Add new notes at the top and write out existing content.
+    notes_file.seek(0)
+    notes_file.write(notes)
+    for line in old_lines:
+        notes_file.write(line)
+    notes_file.close()
+
+
+def get_change_list(commits, url):
+    """Generates a list of changes for the given commits. The routine will attempt
+    to create links to the appropriate ADO pages from the URL script argument where
+    applicable.
+
+    commits - The list of commits which to create the list for
+    url - The URL of the repository.
+    """
+
+    changes = ""
+
+    for commit in commits:
+        pr = None
+        msg = commit.message.split('\n', 1)[0]
+        match = re.match('Merged PR [0-9]+:', msg, flags=re.IGNORECASE)
+        if match:
+            pr = msg[len("Merged PR "):match.end() - 1]
+            msg = f"[{msg[0:match.end()]}]({url}/pullrequest/{pr}){msg[match.end():]}"
+
+        changes += f"- {msg} ~ _{commit.author}_\n"
+
+    return changes
+
+
+def is_breaking_change(message):
+    """Checks if the given commit message contains the breaking change tag"""
+    return re.search('\[x\] breaking change', message, flags=re.IGNORECASE) is not None
+
+
+def is_security_change(message):
+    """Checks if the given commit message contains the security change tag"""
+    return re.search('\[x\] security fix', message, flags=re.IGNORECASE) is not None
+
+
+def is_new_feature(message):
+    """Checks if the given commit message contains the new feature tag"""
+    return re.search('\[x\] new feature', message, flags=re.IGNORECASE) is not None
+
+
+if __name__ == '__main__':
+    main()