KNative Serving Activator Throttler try error

[msgurikar]

Hi,
We are using KNaive serving 1.11.1 for auto scaling our grpc service running in AKS. Sometimes we get below error when we send too many requests at once(80) in the activator logs

{"severity":"ERROR","timestamp":"2023-10-10T20:31:54.809790427Z","logger":"activator","caller":"handler/handler.go:105","message":"Throttler try error","commit":"f60eb32","knative.dev/controller":"activator","knative.dev/pod":"activator-df6b87646-dz2g4","knative.dev/key":"testsvc/test-service-latest","error":"context canceled","stacktrace":"knative.dev/serving/pkg/activator/handler.(*activationHandler).ServeHTTP\n\tknative.dev/serving/pkg/activator/handler/handler.go:105\nknative.dev/serving/pkg/http/handler.(*timeoutHandler).ServeHTTP.func4\n\tknative.dev/serving/pkg/http/handler/timeout.go:118"}

And the client recieves
'Failed to connect all address" error.

Here is our service yaml:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
name: test-service
namespace: testsvc
spec:
template:
metadata:
name: test-service-latest
annotations:

    queue.sidecar.serving.knative.dev/cpu-resource-request: "100m"
    queue.sidecar.serving.knative.dev/cpu-resource-limit: "150m"
    queue.sidecar.serving.knative.dev/memory-resource-request: "900Mi"
    queue.sidecar.serving.knative.dev/memory-resource-limit: "1000Mi"
    
    autoscaling.knative.dev/class: kpa.autoscaling.knative.dev       
    # Target 10 in-flight-requests per pod.      
    autoscaling.knative.dev/minScale: "1"
    autoscaling.knative.dev/maxScale: "80"        
    autoscaling.knative.dev/scaleDownDelay: "60m"
    
spec:
  containerConcurrency: 1
  timeoutSeconds: 1000
  tolerations:
    - key: "app"
      operator: "Equal"
      value: "qi"
      effect: "NoSchedule"
  nodeSelector:
    kubernetes.io/os: linux              

  containers:          
  - image: xxxacr.azurecr.io/bis/testservice_res:latest
    imagePullPolicy: Always
    resources:
      requests:
        cpu: 650m
        memory: 1950Mi
      limits:
        cpu: 800m
        memory: 2000Mi