Design: figure out nightly testing plans

[ace-n]

Problem Statement

To execute on Emblem's testing plans, we will need to create nightly tests of some kind. There are two key questions we need to answer:

1. What needs to be tested?
2. How should the tests be implemented?

Proposal: Design

What to test

We will need to figure out how to test the Website, Content API, and Delivery components.

How to test

We propose a three-step process for our nightly testing:

1. Build containers, and push them to Artifact Registry
2. Deploy a fresh version of the Emblem application (website and content API components) as a Delivery test.
3. Run the website's "end-to-end" (E2E) tests against the deployed application

Proposal: Implementation

We should implement this using three scheduled builds on three (separate) Cloud Pub/Sub topics:

• nightly-builds will trigger container builds + pushes
• nightly-deploys will trigger deployments to Cloud Run
• nightly-tests will trigger end-to-end tests of the deployed Emblem application

Avoiding race conditions

To avoid race conditions (and ensure we always test the latest working version of Emblem), we should create a 1-hour delay between the topics being triggered:

flowchart LR
    A["Build"]-- 1 hour -->B["Deploy"]-- 1 hour -->C["Test"]

Loading

Deployments as tests

We plan to use the website/content API's deployment process as a (smoke) test of our delivery component.

✅ We'll run our deployment 'tests' automatically
✅ Testing our 'actual' deployment workflow (as opposed to one with mocked dependencies) gives us high-fidelity deployment tests
❌ Can be slow to return results
❌ May not always test the latest versions of dependencies. (For example, failures in docker push may result in the Website or Content API tests using an old container version.)

Other options

Testing components in isolation

One alternative would be to test each component in question (website, content API, and delivery) independently from each other.

Tradeoffs relative to our proposal:
✅ Reduces test flakiness
✅ Potentially faster test runs
❌ Doesn't test cross-component integration

Sequential testing ("one big test")

Another alternative would be to test each component sequentially (for example, as part of a single Cloud Build process).

Tradeoffs relative to our proposal:
✅ Faster overall build times
❌ Upstream failures would prevent us from testing downstream dependencies

Use unit tests only

Our existing unit tests currently run on a presubmit basis. Theoretically, we could run them on a nightly basis too.

Tradeoffs relative to our proposal:
✅ Faster test runs
✅ Reduces test flakiness
❌ Not very informative (vs. presubmit unit tests)
❌ Doesn't test cross-component integration

Next steps

1. Discuss these options as a team (in this thread!).
2. Decide on a testing option.
3. Record that decision as a decision record.
4. Implement the required changes.

[ace-n]

We discussed this during a meeting today; @grayside seems in favor of the sequential testing approach.

I'm fine with that, so long as we're fine with the (major) tradeoff:

If a build failure occurs upstream, anything downstream of it won't be tested.

@grayside WDYT? (I'm curious as to @engelke's opinion as well, but he's OOO I believe).

[ace-n]

This is 🚨 semi-blocked 🚨 on #414 (to minimize churn / merge conflicts).

Update:

I spoke with @pattishin, and we thought this is worth doing even without notifications.

@grayside @rogerthatdev any objections?