🔔 Notify Slack on CI Failures

[jasonBirchall]

User Need

As a developer on an app team
I want CI failures to trigger notifications to a specified Slack channel
so that my team is promptly informed and can address issues quickly.

Value

• Reduces time to detect and address CI pipeline failures.
• Increases visibility of CI pipeline status across teams.
• Minimises the risk of ignoring CI failures, improving overall software quality and reliability.

Functional Requirements (What):

• Create a custom GitHub Action that triggers on CI pipeline failure.
• Add functionality to specify a Slack channel for notifications.
• Please make sure the message includes relevant details about the failure, such as the repository name, branch, and error logs.

Non-Functional Requirements (How):

• Notifications should be sent within 30 seconds of a CI failure.
• Use secure methods for storing and handling Slack Webhook URLs.
• Ensure the custom GitHub Action is reusable and documented for other projects.
• Notifications should follow a consistent and readable format.

Acceptance Criteria:

• When a CI failure occurs, a message is sent to the specified Slack channel.
• The message contains the repository name, branch, and failure summary.
• The solution is tested and works for multiple Slack channels in different configurations.
• Documentation is provided for setting up the custom GitHub Action in other repositories.

Assumptions (Optional):

• The team has access to the necessary permissions to set up and use Slack Webhook URLs.
• The CI pipelines are managed using GitHub Actions.
• Users have existing Slack channels dedicated to CI/CD alerts or can create new ones.

Notes:

• Currently, only deployment alerts are routed to #govuk-deploy-alerts, which covers the CD portion. This feature will extend similar visibility to the CI portion of pipelines to specific user channels.
• Here is the conversation that started this request.

[AgaDufrat]

Should we scope this to the main branch, as originally requested?

[jasonBirchall]

Should we scope this to the main branch, as originally requested?

I think so, Aga. It's far too noisy if "work in progress" starts screaming.

[nicholsj]

Feels like this should make use of the alerts_team which can already be specified for each repo https://github.com/alphagov/govuk-developer-docs/blob/3f4eb261e365cda88294604f4cc0cf5814f5b297/data/repos.yml#L19

[kentsanggds]

I've asked for permissions to install a new slack app on GDS slack, govuk-ci-failures, might change the name in the future but for now I just need something to test with.

[kentsanggds]

Reading these docs - https://docs.github.com/en/actions/sharing-automations/creating-actions/about-custom-actions

[kentsanggds]

we can capture failed jobs after the CI step with this step -

      - name: Catch failure
        if: ${{ failure() }}

We should then be able to trigger a slack call using this github action - https://github.com/slackapi/slack-github-action?tab=readme-ov-file#technique-2-slack-api-method

[AgaDufrat]

There's prior art by Chris Ashton alphagov/govuk-developer-docs#4885

[kentsanggds]

Moving to blocked as doing some volunteering and taking annual leave next week.

So far I have managed to capture when it fails and send a slack message but am now thinking about how to make it a more generic github custom action.

[jasonBirchall]

@kentsanggds, please include your PR's updates in this ticket

[kentsanggds]

PR created - #1729

[AgaDufrat]

I approved the PR as I’m on leave until 11 March. The only thing outstanding before we can merge is to add the secret to all govuk repos (decided in today's standup)

Re: documentation
I think we could improve the README (instead of creating a separate doc in Dev docs) or add a doc in https://github.com/alphagov/govuk-infrastructure/tree/main/docs, which gets pulled into Dev docs.
A section with a link should be added to Slack integrations page and a link should be added to Monitoring and alerting