[Bug]: Deleting an org with apps in it is slow

[georgethebeatle]

Background

When deleting an org that contains one or more running apps we observed that the kpack build pods are coming up and down for a while before they eventually disappear. This makes org deletion too slow.

[danail-branekov]

A a build workload is being finalized, its finalizer deletes related kpack builds

On the other hand, the buildworkload controller reconciles build workloads into kpack images

That is weird, a controller should finalize the resources it creates rather than clean up resources created out of them.

TL;DR: we probably need to delete kpack images rather than kpack builds. Before doing that, we need to understand why we delete builds on finalize instead of images in the first place.

Another thing - I skimmed over kpack code - I did not see the image controller skip reconciling images that have their deletion timestamp set. When we delete the org, those timestamp are set by k8s and it waits for the objects to be finalized. Could it be that kpack need to fix their reconcilers to skip reconciling objects that are being deleted?

[danail-branekov]

#3742 made the app workload reconciler not reconcile app workloads that have their deletion timestamp set (i.e. are being deleted). That improved org deletion time (as the app pod is not restarted anymore), but we do see the kpack build pod being restarted.

Maybe we need to make the kpack image reconciler not reconcile images that are being deleted, i.e. add a similar deletion timestamp check here

[danail-branekov]

Kpack PR: buildpacks-community/kpack#1820

This PR has a small effect on overall deletion time. However, it prevents build pods being rescheduled which is a good.

[danail-branekov]

For the record, here is how to build and install a local kpack version:

1. deploy-on-kind

2. Docker login on the container registry installed by deploy-on-kind:

docker login -u user -p password https://localhost:30050

3. Build kpack by running the following script in the kpack repo (note: requires ~25G disk space):

./hack/local.sh --build-type pack --registry localhost:30050/kpack-ef --output kpack.yaml

That would build kpack images, push them to the local registry and produce a kpack.yaml file that can be applied (see below)

4. The image registry address visible within the cluster is not localhost, therefore update the images in kpack.yaml:

sed -i "s/localhost:30050/localregistry-docker-registry.default.svc.cluster.local:30050/" kpack.yaml

5. Create a docker secret in the kpack namespace (used to pull our custom images):

kubectl create secret -n kpack docker-registry image-registry-credentials \
  --docker-server="localregistry-docker-registry.default.svc.cluster.local:30050" \
  --docker-username="user" \
  --docker-password="password"

6. Add that secret to both kpack service accounts:

kubectl patch \
  --namespace kpack \
  --type=merge \
  --patch '{"imagePullSecrets":[{"name":"image-registry-credentials"}]}' \
  serviceaccounts controller

kubectl patch \
  --namespace kpack \
  --type=merge \
  --patch '{"imagePullSecrets":[{"name":"image-registry-credentials"}]}' \
  serviceaccounts webhook

7. Install kpack

kubectl apply -f kpack.yaml

[danail-branekov]

blocking on the kpack pr being merged