Incident Report: Outage on Nov 1, 2022

[dyc3]

Timeline

• At 4:03pm EST, the official site for OpenTogetherTube crashed.
• At 5:10pm EST, the cause was identified.
• At 5:13pm EST, service was restored, continuing to monitor.

Cause

The environment variables REDIS_URL and REDIS_TLS_URL were wrong. Heroku is supposed to manage these values, so I've opened a support ticket with them.

This issue took longer than it should have to diagnose because the logs produced an unhelpful error message:

2022-11-01T20:23:54.580040+00:00 app[web.1]: Error: connect ETIMEDOUT 3.227.52.155:23810
2022-11-01T20:23:54.580054+00:00 app[web.1]: at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1278:16)
2022-11-01T20:23:54.602586+00:00 app[web.1]: error Command failed with exit code 1.
2022-11-01T20:23:54.602718+00:00 app[web.1]: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
2022-11-01T20:23:54.621736+00:00 app[web.1]: error Command failed.
2022-11-01T20:23:54.621749+00:00 app[web.1]: Exit code: 1
2022-11-01T20:23:54.621750+00:00 app[web.1]: Command: /app/.heroku/node/bin/node
2022-11-01T20:23:54.621750+00:00 app[web.1]: Arguments: /app/.heroku/yarn/lib/cli.js start
2022-11-01T20:23:54.621751+00:00 app[web.1]: Directory: /app/server
2022-11-01T20:23:54.621751+00:00 app[web.1]: Output:
2022-11-01T20:23:54.621751+00:00 app[web.1]: 
2022-11-01T20:23:54.621862+00:00 app[web.1]: info Visit https://yarnpkg.com/en/docs/cli/workspace for documentation about this command.
2022-11-01T20:23:54.643732+00:00 app[web.1]: error Command failed with exit code 1.
2022-11-01T20:23:54.643878+00:00 app[web.1]: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Nowhere in this error does it mention redis, which would have sped up diagnosing this.

Fix

1. Attached the redis addon to the app again to get a new set of environment vars. In this specific case, the new environment vars were prefixed with HEROKU_REDIS_PUCE
2. manually updated REDIS_URL and REDIS_TLS_URL to match the new set of URLs

Follow up items

• Improve the error message when failing to connect to redis fails like this.
• Get heroku support to help get the redis addon to manage the relevant environment vars.