Creating a session in renku v2 in a resource pool with a quota that is full will not result in any error

[olevski]

The error message about the quota getting exceeded is on the statefulset (or the statefulset events). And amalthea does not check these for errors. This should be fixed in amalthea.

Amalthea should get this error and surface it.

Currently it just stays in "NotReady" / "Starting" status forever.

[olevski]

This is an example of events that show up on the statefulset:

Events:
  Type     Reason        Age                 From                    Message
  ----     ------        ----                ----                    -------
  Warning  FailedCreate  17m (x17 over 23m)  statefulset-controller  create Pod elisabet-cap-18c263bb3dbf-0 in StatefulSet elisabet-cap-18c263bb3dbf fa
iled error: pods "elisabet-cap-18c263bb3dbf-0" is forbidden: exceeded quota: 8586da3c-af5a-4e2c-9e82-f09b622f2f78, requested: requests.memory=30752Mi,r
equests.nvidia.com/gpu=1, used: requests.memory=748288Mi,requests.nvidia.com/gpu=24, limited: requests.memory=800G,requests.nvidia.com/gpu=24
  Warning  FailedCreate  95s (x2 over 12m)   statefulset-controller  create Pod elisabet-cap-18c263bb3dbf-0 in StatefulSet elisabet-cap-18c263bb3dbf fa
iled error: pods "elisabet-cap-18c263bb3dbf-0" is forbidden: exceeded quota: 8586da3c-af5a-4e2c-9e82-f09b622f2f78, requested: requests.memory=30752Mi,r
equests.nvidia.com/gpu=1, used: requests.memory=738048Mi,requests.nvidia.com/gpu=24, limited: requests.memory=800G,requests.nvidia.com/gpu=24

[Panaetius]

Tried some things, mainly if Events could be reconciled on their own.

SwissDataScienceCenter/amalthea#895 has a working solution listing events for a stateful set but this is tabled for now due to the perfomande implications being scary.