Failed to connect to all addresses; Endpoint is neither UDS or TCP loopback address

[marlenekoh]

Hi, I am receiving this error when connecting to the spicedb client. The code only works when I am using localhost:50051 as the client address. Any help will be appreciated please 🙂

Traceback (most recent call last):
  File "/examples/write_schemas.py", line 20, in <module>
    resp = client.WriteSchema(WriteSchemaRequest(schema=SCHEMA))
  File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 1030, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 910, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
       status = StatusCode.UNAVAILABLE
       details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.21.0.3:50051: Endpoint is neither UDS or TCP loopback address."
       debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.21.0.3:50051: Endpoint is neither UDS or TCP loopback address. {grpc_status:14, created_time:"2023-06-23T09:18:19.7666023+00:00"}"

write_schemas.py - https://github.com/authzed/authzed-py/tree/main/examples/v1

from authzed.api.v1 import Client
from authzed.api.v1 import WriteSchemaRequest
from grpcutil import insecure_bearer_token_credentials

SCHEMA = """definition blog/user {}

definition blog/post {
    relation reader: blog/user
    relation writer: blog/user

    permission read = reader + writer
    permission write = writer
}"""

client = Client(
    "spicedb:50051", # this does not work, change to localhost:50051 to work
    insecure_bearer_token_credentials("presharedkey"),
)

resp = client.WriteSchema(WriteSchemaRequest(schema=SCHEMA))

docker-compose.yml

version: "3"

x-spicedb-env: &spicedb-env
  environment:
    SPICEDB_DATASTORE_ENGINE: postgres
    SPICEDB_DATASTORE_CONN_URI: postgres://postgres:password@database:5432/spicedb
    SPICEDB_GRPC_PRESHARED_KEY: presharedkey

services:
  api:
    image: python:3.8
    volumes:
      - ./write_schemas.py:/examples/write_schemas.py
    command: bash -c "pip install authzed==0.9.0 && python3 /examples/write_schemas.py"
    networks:
      - spicedb

  database:
    image: postgres:14.3
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
      POSTGRES_DB: spicedb
    healthcheck:
      test: ["CMD-SHELL", "pg_isready"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - spicedb

  spicedb:
    image: authzed/spicedb:v1.20.0
    command: serve
    <<: *spicedb-env
    ports:
      - 50051:50051
    networks:
      - spicedb

  migrate:
    image: authzed/spicedb:v1.20.0
    command: migrate head
    <<: *spicedb-env
    networks:
      - spicedb

networks:
  spicedb:
    driver: bridge

Steps to reproduce

1. Create write_schemas.py and docker-compose.yml
2. Run the following commands

docker-compose up -d database
docker-compose up migrate
docker-compose up -d spicedb
docker-compose up api

[jzelinskie]

This is a bug for sure.

insecure_bearer_token_credentials is restricting the usage to only local addresses by adding the following to the channel credentials: grpc.local_channel_credentials(grpc.LocalConnectionType.LOCAL_TCP).

I think this behavior was well-intentioned, but there are definitely cases where folks are going to need plaintext outside of localhost: docker compose is an excellent example.

[jurecuhalev]

There's a workaround you can do while we wait for fix:

x-spicedb-env: &spicedb-env
services:
  api:
    image: python:3.8
    volumes:
      - ./write_schemas.py:/examples/write_schemas.py
    command: bash -c "pip install authzed==0.9.0 && python3 /examples/write_schemas.py"
    network_mode: 'container:api'

  spicedb:
    image: authzed/spicedb:v1.20.0
    command: serve
    <<: *spicedb-env
    networks:
      - spicedb
networks:
  spicedb:
    driver: bridge

I've removed extra services information that stays the same but what you essentially do is move SpiceDB ports to api and map spicedb so that it runs on the same interface as api and that way tou can connect to localhost:50051 from within api.

[ghost]

adding some more context to this thread, it seems like regardless of @jzelinskie 's thread over on the grpc GH page, the experimental code that they reference would not improve the situation here without additional changes to this codebase or spicedb's codebase (though I could have missed a cli arg to spice).

Some things I observed while hacking on the problem include,

1. an insecure channel cannot be passed credentials. (etcd3 and python: UNAUTHENTICATED: Established channel does not have a sufficient security level to transfer call credential grpc/grpc#29643)

<AioRpcError of RPC that terminated with:
 status = StatusCode.UNAUTHENTICATED
 details = "Established channel does not have a sufficient security level to transfer call credential."

2. redef'ing the bearer_token_credentials function to only use an insecure channel, raises the following from the authzed-py library

def bearer_token_credentials(token: str, certChain: Optional[bytes] = None):
    """
    gRPC credentials for a service that requires a Bearer Token.
    """
    return grpc.experimental.insecure_channel_credentials()


<AioRpcError of RPC that terminated with:
 status = StatusCode.UNAUTHENTICATED
 details = "invalid preshared key: rpc error: code = Unauthenticated desc = Request unauthenticated with bearer"

which, the above, appears to be a consequence of this library requiring a sercure channel; regardless.

• https://github.com/authzed/authzed-py/blob/main/authzed/api/v1/__init__.py#L53

3. redef'ing the client to remove the credential options doesn't work because authzed requires the preshared key be provided apparently.

client side
<_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAUTHENTICATED
        details = "invalid preshared key: rpc error: code = Unauthenticated desc = Request unauthenticated with bearer"

and on the server-side
{"level":"info","requestID":"5d7c6b923354476e7e27dc392c4e5ca1","protocol":"grpc","grpc.component":"server",
"grpc.service":"authzed.api.v1.PermissionsService","grpc.method":"WriteRelationships","grpc.method_type":"unary",
"peer.address":"127.0.0.1:33106","grpc.start_time":"2023-09-14T03:19:03Z","grpc.code":"Unauthenticated",
"grpc.error":"rpc error: code = Unauthenticated desc = invalid preshared key: rpc error: code = Unauthenticated 
desc = Request unauthenticated with bearer","grpc.time_ms":"0.076","time":"2023-09-14T03:19:03Z",
"message":"finished call"}

so except for the workaround mentioned by @jurecuhalev (which I haven't tried, but I assume works) you're either out-of-luck, or you can try using the HTTP API which (per my understanding) is just a call to spicedb's loopback address and shouldn't be affected by this?

[FernandoArteaga]

I think this is also what I'm facing here 👉🏼 https://github.com/orgs/authzed/discussions/1644

[mateenkasim]

Any progress on this? I'm testing out spicedb for the first time, trying to make calls from within a k8s cluster, and I'd prefer using this client over the HTTP API if I can help it

[nediamond]

+1 that this is a serious issue for local docker compose setups

another workaround is to use socat to proxy the connection from localhost:50051 to the spicedb instance

did this by adding the following line to an entrypoint script:

# Proxy localhost:50051 to host.docker.internal:50051 (could do spicedb:50051 for same goal)
socat TCP-LISTEN:50051,fork TCP:host.docker.internal:50051 & 

[jtfidje]

+1 Also run into this with our local docker compose dev environment

[andrii-harbour]

+1 still happening

[josephschorr]

https://gist.github.com/jakedt/100d75048c7ebefdf2fb666de58cda9b has a workaround for the insecure token to a non-localhost address

[tstirrat15]

I've added an implementation of an InsecureClient in the referenced PR. I'll probably get it reviewed and in tomorrow, but I'd love some additional eyes on it if any of y'all have some time.

[tstirrat15]

The InsecureClient is now available in v0.18.0. Let me know what you think!

[tstirrat15]

Apologies, v0.18.0 never made it to pypi. Try v0.18.1.

[mateenkasim]

InsecureClient works like a charm – thank you! The new init_stubs method made it easy to add the HealthStub, too.

from authzed.api.v1 import InsecureClient
from grpc_health.v1.health_pb2_grpc import HealthStub
from grpc_health.v1.health_pb2 import HealthCheckRequest

class SpiceClient(InsecureClient, HealthStub):
    def __init__(self, options=None, compression=None):
        target = 'my-custom-target'
        token = 'my-personal-token'
        super().__init__(target, token, options, compression)

    def init_stubs(self, channel):
        super().init_stubs(channel)
        HealthStub.__init__(self, channel)

client = SpiceClient()
check = client.Check(HealthCheckRequest())
print(check) # status: SERVING

[alexshanabrook]

Thank you for adding this support! Just one small comment—on our CI/CD pipelines, mypy raises a complaint about the following code:

with the failing error:

python -m mypy . --strict --ignore-missing-imports --exclude clients/python --exclude alembic_app_db --exclude alembic_ts_db --exclude tests --implicit-reexport --explicit-package-bases
config/config.py:143: error: Incompatible return value type (got "SyncClient", expected "Client")  [return-value]
Found 1 error in 1 file (checked 164 source files)

In the __init__.py file, the inheritance that should cause this check to pass shows up as expected, with SyncClient inheriting from Client:

but in the __init__.pyi file, the inheritance isn't shown, which I'm thinking might be what's causing the error:

Is there any chance you could release a patch for this? Thank you!

[tstirrat15]

@alexshanabrook tracking in #207

[alexshanabrook]

Great, thank you!