Pulsar scaler: If no subscription exists the default number of pods is zero

[frankjkelly]

Report

We are using Keda 2.8.2 with Pulsar successfully for some time.
We often will suspend namespaces and delete pulsar PVCs.
Unfortunately when the environment is brought back to life the Keda autoscaler is unable to launch our pods because the subscription does not exist. And although the MinReplicas: 1 the error results in no pods being launched (or so it appears)
But the subscription does not exist because the pods are not launched.

  Warning  FailedGetExternalMetric       19m (x276 over 13d)  horizontal-pod-autoscaler  unable to get external metric cogito-load/s0-pulsar-persistent---healthking-event-signal-signaloffloader-event-handler/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: signal-offloader,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: no matching metrics found for s0-pulsar-persistent---healthking-event-signal-signaloffloader-event-handler
  Warning  FailedGetExternalMetric       19m (x275 over 13d)  horizontal-pod-autoscaler  unable to get external metric cogito-load/s1-pulsar-persistent---healthking-event-signal_offload_reconciliation-signaloffloader-event-handler/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: signal-offloader,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: no matching metrics found for s1-pulsar-persistent---healthking-event-signal_offload_reconciliation-signaloffloader-event-handler

At least that's my understanding of the problem.

Expected Behavior

I would expect that despite the failure - since MinReplicas: 1 that Keda would launch at least one pod
otherwise we end up with a chicken-and-egg problem - we can't launch the pods without a subcription, we can't create a subscription without a pod.

Actual Behavior

Zero pods launched.
We had to edit the deployment.yaml manually and set the replica count to 1 and once that happened Keda took over from there.

Steps to Reproduce the Problem

1. Uninstall Pulsar Cluster
2. Delete PVCs
3. Uninstall Pods relying on Keda Pulsar scaler
4. Recreate Pulsar Cluster
5. Redeploy pods relying on Keda Pulsar scaler

Logs from KEDA operator

keda-operator-64fbb8d4c9-rbjwt keda-operator 2024-01-19T17:13:03Z	ERROR	scalehandler	Error getting scale decision	{"scaledobject.Name": "signal-offloader", "scaledObject.Namespace": "cogito-load", "scaleTarget.Name": "signal-offloader", "error": "error requesting stats from url: %!s(<nil>)"}
keda-operator-64fbb8d4c9-rbjwt keda-operator github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
keda-operator-64fbb8d4c9-rbjwt keda-operator 	/workspace/pkg/scaling/scale_handler.go:278
keda-operator-64fbb8d4c9-rbjwt keda-operator github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop

KEDA Version

2.8.2

Kubernetes Version

1.25

Platform

Amazon Web Services

Scaler Details

Pulsar

Anything else?

No response

[JorTurFer]

Hello,
I don't get the point. I mean, if you set minReplicaCount:1 KEDA won't scale to 0 your workload. How is it scaled to 0? If you set minReplicaCount:0 KEDA can scale to 0 the workload, but you are totally right, if the subscription doesn't exist and KEDA can't query the value, you won't scale from 0.
Is KEDA scaling the workload to 0 if the subscription is missing, although you set minReplicaCount > 0?

You could set a fallback to scale to at least X if there is an error. In this case, if the subscription doesn't exist, I guess that KEDA will apply the fallback. If you set at least 1 instance, your pod will wake up creating the subscription and KEDA will start to work again.

[frankjkelly]

Is KEDA scaling the workload to 0 if the subscription is missing, although you set minReplicaCount > 0?
Yes that is correct.
A fallback is exactly what I was hoping for.
Probably that's all I need - thanks so much for the advise and quick response.

[JorTurFer]

Is KEDA scaling the workload to 0 if the subscription is missing, although you set minReplicaCount > 0?

It shouldn't happen, as KEDA always check the minReplicaCount before scaling to 0. If it's KEDA who scales your workload to 0, you will see a clear entry in the log saying that the scalerExecutor has scaled from X to 0. Do you see any error like that?

BTW, v2.8 is quite old (today we have released 2.13 so v2.8 has almost a year and a half). I'd suggest upgrading to a more updated version if it's possible to improve the performance and security (A lot if fixed CVEs during these months)

[frankjkelly]

Thanks FYI the fallback worked a treat - thanks so much!