Incident Report: Feb. 7, 2021

[dyc3]

Timeline

• Outage appeared to begin at when bb012dad was deployed at 04:00 UTC.
  • Contained a fix for a bug in the youtube info extractor: youtube: skip videos in playlist that are private or deleted, fixes #367 #368
• At 13:00 UTC, the outage was discovered. A rollback to the previous deploy restored service.
• At 17:26 UTC, the source of the problem was found to be postgres.

Relevant info

• Diff: 542e27c...bb012da
• It seems like all requests that touch the database are timing out.

Theories

Denial of service

Why would this be the cause?

• The logs contain a lot of requests to POST /api/room/generate.

Disproved

• Endpoint is rate limited.
• Endpoint actually doesn't work in production environment. It times out.

Database not responding

Why would this be the cause?

• POST /api/room/generate, POST /api/room/create, GET /api/data/previewAdd all time out, and they all touch the database

Confirmed

Troubleshooting

• Attempt to connect to postgres manually. Success
• Are tables present? No
  • select * from information_schema.tables where table_schema='public';
• Do migrations have any effect? No
• Attempting to upgrade pg to latest version Success
  • might be related to Sequelize misbehaving with Node v14 sequelize/sequelize#12158

New problem: getting no pg_hba.conf entry for host ..., SSL off

• forcing ssl and disabling self-signed cert protection in sequelize config/config.js seems to fix it