Design: split build/deploy and test topics

[ace-n]

Context

Emblem has several CI processes that run on a nightly basis. These are triggered by Cloud Scheduler, which sends a message to Pub/Sub that in turn triggers Cloud Build:

flowchart LR
    A["Cloud Scheduler"]-- messages -->B["Cloud Pub/Sub"]-- triggers -->C["Cloud Build"]

Loading

Problem Statement

Currently, we use one Pub/Sub topic (nightly) to trigger both builds/deploys and tests.

This means that artifact builds/deploys and tests are triggered [by Cloud Scheduler] simultaneously. Simultaneous triggering can lead to race conditions, wherein tests on day n will test day n-1's deployment artifacts.

Proposed Fix

To mitigate this, we should create two Pub/Sub topics:

• nightly-builds: build artifacts
• nightly-tests: run tests

The build topic (nightly-builds) will run slightly (e.g. 1 hour) before the testing topic (nightly-tests). This should give the builds enough time to update the deployed artifacts.

Note: if any deployment steps fail, then the tests will automatically "fall back" to the previous day's artifacts.

Implementation

To implement this change, we'll need to do the following:

1. Create two new Pub/Sub topics in Terraform (nightly-builds and nightly-tests)
2. Migrate build/test processes over to their appropriate topics
3. Remove the old (nightly) Pub/Sub topic and any references to it.

Tip: each of these should be one PR, for three PRs total.

[grayside]

Not sure we're aligned on what needs to happen here yet, so removing the help wanted and good first issue labels for now. Thanks for filing!

[ace-n]

Closed in favor of #536.