including istio-injection: enabled in namespace results in unreachable knative services in that namespace due to istio sidecar injection

[alita-moore]

What version of Knative?

Istio version 1.24.1
Knative version 1.16.0

resources:
  - https://github.com/knative/serving/releases/download/knative-v1.16.0/serving-core.yaml
  - https://github.com/knative/serving/releases/download/knative-v1.16.0/serving-hpa.yaml
  # NOTE: if you fail to include dns handling the kourier service will return a 404 for all requests
  - https://github.com/knative/serving/releases/download/knative-v1.16.0/serving-default-domain.yaml
  # NOTE: if you fail to include the net-istio service the service will contanstly reset the connection
  - https://github.com/knative/net-istio/releases/download/knative-v1.16.0/net-istio.yaml

Expected Behavior

I expect to be able to make a request to my service

Actual Behavior

The request hangs infinitely.

Steps to Reproduce the Problem

install the following services:

apiVersion: v1
kind: Namespace
metadata:
  name: default
  labels:
    istio-injection: enabled
---
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: debug-service
  namespace: default
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/min-scale: "1"
        sidecar.istio.io/inject: "false"
    spec:
      containers:
        - image: gcr.io/knative-samples/helloworld-go:latest
          env:
            - name: TARGET
              value: "Hello Knative!"
---
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: debug-service-sidecar
  namespace: default
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/min-scale: "1"
    spec:
      containers:
        - image: gcr.io/knative-samples/helloworld-go:latest
          env:
            - name: TARGET
              value: "Hello Knative!"

after these have started you'll notice something like the following

NAME                                                          READY   STATUS    RESTARTS   AGE
pod/debug-service-00001-deployment-78888f5b44-qd69j           2/2     Running   0          3m21s
pod/debug-service-sidecar-00001-deployment-56b7486745-bxjtn   3/3     Running   0          3m21s

NAME                                          TYPE           CLUSTER-IP       EXTERNAL-IP                                            PORT(S)                                              AGE
service/debug-service                         ExternalName   <none>           knative-local-gateway.istio-system.svc.cluster.local   80/TCP                                               3m19s
service/debug-service-00001                   ClusterIP      10.104.24.168    <none>                                                 80/TCP,443/TCP                                       3m21s
service/debug-service-00001-private           ClusterIP      10.97.193.154    <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   3m21s
service/debug-service-sidecar                 ExternalName   <none>           knative-local-gateway.istio-system.svc.cluster.local   80/TCP                                               3m18s
service/debug-service-sidecar-00001           ClusterIP      10.109.8.104     <none>                                                 80/TCP,443/TCP                                       3m21s
service/debug-service-sidecar-00001-private   ClusterIP      10.105.129.172   <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   3m21s
service/kubernetes                            ClusterIP      10.96.0.1        <none>                                                 443/TCP                                              5h17m

NAME                                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/debug-service-00001-deployment           1/1     1            1           3m21s
deployment.apps/debug-service-sidecar-00001-deployment   1/1     1            1           3m21s

NAME                                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/debug-service-00001-deployment-78888f5b44           1         1         1       3m21s
replicaset.apps/debug-service-sidecar-00001-deployment-56b7486745   1         1         1       3m21s

NAME                                                      LATESTCREATED                 LATESTREADY                   READY   REASON
configuration.serving.knative.dev/debug-service           debug-service-00001           debug-service-00001           True    
configuration.serving.knative.dev/debug-service-sidecar   debug-service-sidecar-00001   debug-service-sidecar-00001   True    

NAME                                                       CONFIG NAME             GENERATION   READY   REASON   ACTUAL REPLICAS   DESIRED REPLICAS
revision.serving.knative.dev/debug-service-00001           debug-service           1            True             1                 1
revision.serving.knative.dev/debug-service-sidecar-00001   debug-service-sidecar   1            True             1                 1

NAME                                              URL                                                READY   REASON
route.serving.knative.dev/debug-service           http://debug-service.default.example.com           True    
route.serving.knative.dev/debug-service-sidecar   http://debug-service-sidecar.default.example.com   True    

NAME                                                URL                                                LATESTCREATED                 LATESTREADY                   READY   REASON
service.serving.knative.dev/debug-service           http://debug-service.default.example.com           debug-service-00001           debug-service-00001           True    
service.serving.knative.dev/debug-service-sidecar   http://debug-service-sidecar.default.example.com   debug-service-sidecar-00001   debug-service-sidecar-00001   True

And when you make a request the following happens

➜  monorepo git:(main) curl -H "Host: debug-service.default.example.com" http://127.0.0.1
Hello Hello Knative!!
➜  monorepo git:(main) curl -H "Host: debug-service-sidecar.default.example.com" http://127.0.0.1
### runs infinitely ###