Subchannel Connectivity change to TRANSIENT_FAILURE","system":"grpc"

[mikeumus]

I check the docker logs of my jaeger/all-in-one and I see this repeating:

{"level":"warn","ts":1614514430.3461173,"caller":"[email protected]/clientconn.go:1275","msg":"grpc: addrConn.createTransport failed to connect to {:443  <nil> 0 <nil>}. Err: connection error: desc = \"transport: Error while dialing dial tcp :443: connect: connection refused\". Reconnecting...","system":"grpc","grpc_log":true}
{"level":"info","ts":1614514430.3461516,"caller":"[email protected]/clientconn.go:1056","msg":"Subchannel Connectivity change to TRANSIENT_FAILURE","system":"grpc","grpc_log":true}
{"level":"info","ts":1614514441.9279923,"caller":"[email protected]/clientconn.go:1056","msg":"Subchannel Connectivity change to CONNECTING","system":"grpc","grpc_log":true}
{"level":"info","ts":1614514441.928637,"caller":"[email protected]/clientconn.go:1193","msg":"Subchannel picks a new address \":443\" to connect","system":"grpc","grpc_log":true}
{"level":"warn","ts":1614514441.930152,"caller":"[email protected]/clientconn.go:1275","msg":"grpc: addrConn.createTransport failed to connect to {:443  <nil> 0 <nil>}. Err: connection error: desc = \"transport: Error while dialing dial tcp :443: connect: connection refused\". Reconnecting...","system":"grpc","grpc_log":true}

docker run command

  docker run -d \
    --dns=1.1.1.1 \
    -p 6831:6831/udp \
    -p 6832:6832/udp \
    -p 2052:2052 \
    -p 2053:2053 \
    -p 8443:8443 \
    -p 2087:2087 \
    -p 443:443 \
    jaegertracing/all-in-one:latest \
    --collector.http-server.host-port=:2053 \
    --query.http-server.host-port=:2052 \
    --collector.grpc-server.host-port=:8443 \
    --admin.http.host-port=:2087 \
    --metrics-backend=prometheus \
    --metrics-http-route=/metrics \
    --collector.http.tls.enabled=true \
    --collector.grpc.tls.enabled=true \
    --query.grpc.tls.enabled=true \
    --reporter.grpc.host-port=:443 \
    --reporter.grpc.tls.enabled=true \
    --reporter.type=grpc

I got up from my desk, came back, tried sending some spans and nothing was showing up. So I didn't make any changes to my containers. I read the DNS on Vultr could be shotty so I changed it to 1.1.1.1, no luck.

I tried including my hostname this way: docker run -h=my.docker.container.example.com, no luck.

I saw @jpkrohling's recommendation to specify the host and namespace, but what are these for a docker container?

I tried docker0, 127.0.0.1, those didn't alleviate the issue.

I've tried so many variations and ports in the docker run command.

Any help or ideas are greatly appreciated.

[yurishkuro]

I would start with avoiding the standard SSL port 443

[mikeumus]

@yurishkuro, I'll try with jager/all-in-one:1.21, thank you.

In etc/hostname I see fv_server_4 so do I set this gRPC Reporter host:port CLI_FLAG like this?

--reporter.grpc.host-port=fv_server_4:8443

becasue I'm getting this in the jaeger/all-in-one contains logs:

grpc: addrConn.createTransport failed to connect to {fv_server_4:8443 ...

---

This is my full docker run jaeger/all-in-one command:

  docker run -d \
    --dns=1.1.1.1 \
    -p 6831:6832/udp \
    -p 6832:6832/udp \
    -p 5778:5778 \
    -p 2052:2052 \
    -p 2053:2053 \
    -p 2096:2096 \
    -p 2087:2087 \
    -p 8443:8443 \
    -p 2095:2095 \
    jaegertracing/all-in-one:latest \
    --metrics-backend=prometheus \
    --metrics-http-route=/metrics \
    --reporter.type=grpc \
    --reporter.grpc.tls.skip-host-verify=true \
    --query.http-server.host-port=:2052 \
    --collector.http-server.host-port=:2053 \
    --collector.grpc-server.host-port=:2095 \
    --collector.grpc.tls.enabled=true \
    --collector.http.tls.enabled=true \
    --query.grpc-server.host-port=:2096 \
    --query.grpc.tls.enabled=true \
    --admin.http.host-port=:2087 \
    --reporter.grpc.host-port=fv_server_4:8443 \
    --reporter.grpc.tls.enabled=true \
    --reporter.grpc.tls.cert=${envVar:CERT_PATH} \
    --reporter.grpc.tls.key=${envVar:KEY_PATH}
    ```
    
I see in the Jaeger docs they have a couple more dots in their hostname, what're those?

`--reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250`

https://www.jaegertracing.io/docs/1.22/deployment/#discovery-system-integration

[yurishkuro]

I'll try with jager/all-in-one:1.21, thank you.

There is no TLS between agent and collector in 1.21

In etc/hostname I see fv_server_4

all-in-one is a single process that you run inside the container's network namespace, you don't need to specify the host name, local host is correct. But I don't know how it will affect TLS.

I see in the Jaeger docs they have a couple more dots in their hostname, what're those?

just an example, and probably for Kubernetes.

[jpkrohling]

In your command above, you are specifying the TLS certs for the Jaeger Agent process (--reporter.grpc.tls.enabled and related), but not for the server part (--collector.grpc.tls.enabled and related).

[mikeumus]

@jpkrohling silly question, can they be the same cert?

What happens if you use the --collector.grpc.tls.enabled flag but not the cert key/path flags?
If so, this might be a good little tweak to the docs.

[jpkrohling]

In the real world, they would be different certs issued by the same CA, so that mTLS happens between the client (agent) and server (collector). The all-in-one image is just an assembly of components in the same binary, meaning that the options for the agent won't influence the values for the collector, and vice-versa. In other words: you should set the cert key/path to each individual component, even if they point to the same files.

[mikeumus]

@yurishkuro It's working without the TLS flags:

    --collector.grpc.tls.enabled=true \
    --collector.http.tls.enabled=true \
    --query.grpc.tls.enabled=true \
    --reporter.grpc.tls.enabled=true

and when I include this flag (may happen for other flags as well), I get the following error:
--collector.http.tls.enabled=true

Could not start HTTP collector","error":"open : no such file or directory","stacktrace":
"github.com/jaegertracing/jaeger/cmd/collector/app/server.serveHTTP.func1\n\tgithub.com/
jaegertracing/jaeger/cmd/collector/app/server/http.go:95

Marking this as Resolved for now. Thanks again.