GCP missing creds slows down significantly

[dmpetrov]

Description

See quick start.

This command take 20 seconds (sometimes 30) without any progressbar if I don't have any GCP creds setup. If I put anon=True - it works instantly like it suppose to.

meta = DataChain.from_json("gs://datachain-demo/dogs-and-cats/*json", object_name="meta")

This affects all users since we use GCP in most of our examples and, I'm sure, majority of user do not have GCP creds set up.

What's needed: we need to avoid the delay when GCP creds are not set up. Ideally, without specifying anon=True which adds more code in examples and mental overhead for users.

ToDo:

• Fix the issue
• Remove anon=True from Readme and Get Started

[shcheklein]

This is happening in SOMA co-working for me as well. The reason is the same issue as here https://github.com/iterative/studio/issues/10235 . Google lib has an exponential backoff on trying to reach out to the metadata service. So, when we don't set credentials explicitly to anon it is trying everything it could and thus hitting that exponential backoff. We need to wait for that PR that @0x2b3bfa0 made to land. And even after that users will have to specify an env variable.

I wonder if we can fix fsspec to disable metadata service if we are not on google (GC fsspec has a way to determine that AFAIU) - wdyt @0x2b3bfa0 ?

[dmpetrov]

This is happening in SOMA co-working for me as well

I'm experiencing the same at my home wifi

[skshetry]

Related: fsspec/filesystem_spec#1768

[shcheklein]

@skshetry thanks, I put a comment with the details there. It's a good / relevant discussion indeed.

[0x2b3bfa0]

I wonder if we can fix fsspec to disable metadata service if we are not on google (GC fsspec has a way to determine that AFAIU) - wdyt @0x2b3bfa0 ?

I think exactly https://github.com/iterative/studio/issues/10235#issuecomment-2420957866; I've posted a comment at fsspec/filesystem_spec#1768 (comment).