Move cache from golang/notebooks to data services

[olevski]

The kr8s python client has "Watch" which is an async generator. This means we can have an async watch on a k8s resource.

See https://docs.kr8s.org/en/stable/autoapi/kr8s/asyncio/index.html#kr8s.asyncio.Api.watch

Rough plan can be to:

• Create a background process (or background job) that watches k8s events - we need to have only 1 of these. Note that it can be problematic if we have multiple of these (as the server scales up) because they will not be perfectly sync and may overwrite each other. But this may not be that big of a deal. But if we have multiple of these (with each server instance) we will have to handle cases like (deleting but another instance has already deleted the resource from the case or creation but the creation has already occured in another instance or update with a potential conflict).
• The background process populates a table (or tables) in the DB with the resources to be cached
• The k8s cache client reads stuff from this table

So the background job(s) reads and writes to the DB tables. The cached client can only read from these.

[olevski]

Important: Every time this k8s watcher starts it will have to list all resources first and then watch. The watch does not tell you what resources are already present - just gives you updates.