Charms aren't able to connect to other charms ingress URL after adding model to the mesh

[IbraAoad]

Bug Description

Charms are no longer able to connect to other charms external URL regardless of the ingress provider (Seen on both traefik and istio-ingress-k8s) when model is on the mesh and the beacon's waypoint is deployed.

It's also important to note that adding the model on the mesh without the waypoint resolves the issue.

See canonical/istio-k8s-operator#14 for more context

To Reproduce

1. juju deploy istio-beacon-k8s
2. juju deploy traefik-k8s & grafana-k8s & alertmanager-k8s
3. juju relate traefik-k8s grafana-k8s & alertmanager-k8s
4. juju relate grafana-k8s alertmanager-k8s
5. juju config istio-beacon-k8s model-on-mesh=true

Grafana is no longer able to connect to alertmanager's data source

Environment

latest/edge

Relevant log output

Ztunnel Logs

2024-09-02T18:59:08.773803Z     info    access  connection complete     src.addr=10.1.20.7:49858 src.workload=grafana-k8s-0 src.namespace=k8s src.identity="spiffe://cluster.local/ns/k8s/sa/grafana-k8s" dst.addr=10.1.20.42:15008 dst.hbone_addr=172.232.192.178:80 dst.service=istio-ingress-k8s-istio.k8s.svc.cluster.local dst.workload=istio-beacon-k8s-k8s-waypoint-cc46d4bd4-j7592 dst.namespace=k8s dst.identity="spiffe://cluster.local/ns/k8s/sa/istio-beacon-k8s-k8s-waypoint" direction="outbound" bytes_sent=636 bytes_recv=0 duration="1ms"
2024-09-02T18:59:08.917470Z     info    access  connection complete     src.addr=10.1.20.7:49870 src.workload=grafana-k8s-0 src.namespace=k8s src.identity="spiffe://cluster.local/ns/k8s/sa/grafana-k8s" dst.addr=10.1.20.42:15008 dst.hbone_addr=172.232.192.178:80 dst.service=istio-ingress-k8s-istio.k8s.svc.cluster.local dst.workload=istio-beacon-k8s-k8s-waypoint-cc46d4bd4-j7592 dst.namespace=k8s dst.identity="spiffe://cluster.local/ns/k8s/sa/istio-beacon-k8s-k8s-waypoint" direction="outbound" bytes_sent=639 bytes_recv=0 duration="1ms"
2024-09-02T18:59:09.045119Z     info    access  connection complete     src.addr=10.1.20.7:49872 src.workload=grafana-k8s-0 src.namespace=k8s src.identity="spiffe://cluster.local/ns/k8s/sa/grafana-k8s" dst.addr=10.1.20.42:15008 dst.hbone_addr=172.232.192.178:80 dst.service=istio-ingress-k8s-istio.k8s.svc.cluster.local dst.workload=istio-beacon-k8s-k8s-waypoint-cc46d4bd4-j7592 dst.namespace=k8s dst.identity="spiffe://cluster.local/ns/k8s/sa/istio-beacon-k8s-k8s-waypoint" direction="outbound" bytes_sent=626 bytes_recv=0 duration="1ms"



Grafana Logs
2024-09-02T18:59:08.773Z [grafana] logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/uid/P6B7DA9DF455CD317/api/v1/status/buildinfo remote_addr=172.232.192.178 referer=http://localhost:3000/connections/your-connections/datasources/edit/P6B7DA9DF455CD317 t=2024-09-02T18:59:08.773799017Z level=error msg="Proxy request failed" err=EOF
2024-09-02T18:59:08.773Z [grafana] logger=context userId=1 orgId=1 uname=admin t=2024-09-02T18:59:08.773918688Z level=error msg="Request Completed" method=GET path=/api/datasources/proxy/uid/P6B7DA9DF455CD317/api/v1/status/buildinfo status=502 remote_addr=172.232.192.178 time_ms=3 duration=3.273419ms size=0 referer=http://localhost:3000/connections/your-connections/datasources/edit/P6B7DA9DF455CD317 handler=/api/datasources/proxy/uid/:uid/*
2024-09-02T18:59:08.917Z [grafana] logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/uid/P6B7DA9DF455CD317/alertmanager/api/v2/status remote_addr=172.232.192.178 referer=http://localhost:3000/connections/your-connections/datasources/edit/P6B7DA9DF455CD317 t=2024-09-02T18:59:08.917492951Z level=error msg="Proxy request failed" err=EOF
2024-09-02T18:59:08.917Z [grafana] logger=context userId=1 orgId=1 uname=admin t=2024-09-02T18:59:08.917601632Z level=error msg="Request Completed" method=GET path=/api/datasources/proxy/uid/P6B7DA9DF455CD317/alertmanager/api/v2/status status=502 remote_addr=172.232.192.178 time_ms=2 duration=2.948266ms size=0 referer=http://localhost:3000/connections/your-connections/datasources/edit/P6B7DA9DF455CD317 handler=/api/datasources/proxy/uid/:uid/*
2024-09-02T18:59:09.045Z [grafana] logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/uid/P6B7DA9DF455CD317/api/v2/status remote_addr=172.232.192.178 referer=http://localhost:3000/connections/your-connections/datasources/edit/P6B7DA9DF455CD317 t=2024-09-02T18:59:09.045112514Z level=error msg="Proxy request failed" err=EOF
2024-09-02T18:59:09.045Z [grafana] logger=context userId=1 orgId=1 uname=admin t=2024-09-02T18:59:09.045213895Z level=error msg="Request Completed" method=GET path=/api/datasources/proxy/uid/P6B7DA9DF455CD317/api/v2/status status=502 remote_addr=172.232.192.178 time_ms=2 duration=2.562673ms size=0 referer=http://localhost:3000/connections/your-connections/datasources/edit/P6B7DA9DF455CD317 handler=/api/datasources/proxy/uid/:uid/*

Additional context

No response

[dstathis]

Are we blocking egress?

[IbraAoad]

@dstathis not sure tbh, we haven't touched any egress configuration on all 3 istio charms.

[IbraAoad]

Might be related/helpful istio/istio#52532

[IbraAoad]

Support for Cross-namespace waypoint, was only added in 1.23, upgrading to this version fixes the issue.

Fixed by canonical/istio-k8s-operator#16

Verification Test:

1. juju deploy ./istio-k8s_ubuntu-22.04-amd64.charm --trust -m istio-system (istio 1.23.1 charm)
2. juju deploy cos-lite --channel edge --trust -m test
3. juju deploy istio-beacon-k8s --trust --channel edge --config model-on-mesh=true -m test
4. From the Grafana connections page, test that all connections are routed through traefik and are healthy