Could not read served logs: Invalid URL 'http://:8793....': Not host supplied

[stefanikjabuti]

Official Helm Chart version

1.15.0 (latest released)

Apache Airflow version

2.9.3

Kubernetes Version

Client Version: v1.30.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.30.6-eks-7f9249a

Helm Chart configuration

I deployed Airflow using the Helm chart on the AWS Kubernetes cluster, made some changes like using an existing Redis and synchronizing with GitHub. However, my DAG always gives this log error. I saw that it could be the base_url; mine is set to http://airflow-webserver.airflow.svc.cluster.local:8080/. I tried updating my Airflow version to a more recent one in the values.yaml file, but my webserver pod throws an error when I do that.

Docker Image customizations

No response

What happened

No response

What you think should happen instead

No response

How to reproduce

Using the helm chart at the link: https://airflow.apache.org/docs/helm-chart/stable/index.html

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

This is a configuration discussion not airlfow issue - converting it to a discussion.

You should likely follow the IP route described in https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#hostname-callable - for some reason the way how DNS works, default hostname callable does not provide name of the worker.

[tzuwei93]

Firstly, the base_url configuration is intended for accessing the Airflow webserver, not for retrieving logs from the worker server. The worker server might have a different hostname, e.g., airflow-worker.airflow.svc.cluster.local. The worker server is responsible for fetching logs from the worker's filesystem in airflow version of , as mentioned in a recent discussion about failing to retrieve logs after multiple task retries, and also referenced in the code snippet.

I have just encountered a similar issue, and the exception I got was “[Errno -3] Temporary failure in name resolution.” , which indicates that the hostname recorded in PostgreSQL is temporarily unresolvable. In my case, the problem was caused by the Airflow worker service is mounting an incorrect destination path for existing log files, which led to retrieving logs from the worker server instead.

However, in your situation, it seems the issue might stem from the hostname not being recorded correctly to PostgreSQL backend. As the exception stating “no host supplied” suggests that there seems to be no hostname was recorded for this specific task instance. One possible cause is that the task instances crashed unexpectedly without recording the worker's hostname.

In your case, I would first verifying whether logs for successfully executed DAGs are displayed correctly in the webserver UI. This might help determine if the issue is isolated to this specific task instances or some other issue, which might be a broader problem with hostname recording. Additionally, you could check if the log is present at the corresponding path you expected.

If any part of my reasoning or logic is incorrect, please feel free to point it out. :)

[stefanikjabuti]

Hello, tzuwei93,
First of all, thank you for the suggestions. None of the DAGs have succeeded because of this error, so I believe, as you mentioned, it might be a broader issue.
There’s also nothing in the path where the logs should be generated inside the worker pod (/opt/airflow/logs).
Do you have any idea how I should proceed now?

[insomnes]

What executor do you use, and how does the code of the task look?

I've met the same error due to issue #45529 and with empty operators.

Also, usage of the mentioned hostname callable by IP helped with tasks running on the scheduler.

[potiuk]

Yes. the default "hostname callable" might simply - depending on configuration of your DNS and such can give you empty hostname and you might want to use the IP address instead.

[stefanikjabuti]

I added the variable - name: AIRFLOW__CORE__HOSTNAME_CALLABLE
value: airflow.utils.net.get_host_ip_address to the environment, but it didn't change the error logs. I saw in some places that a PVC might be needed, but I would like the data to be stored locally instead of using storage. Do you know if this could be the issue?

[insomnes]

Have you restarted the worker? Can you check the setting with:

airflow config get-value core hostname_callable

[stefanikjabuti]

Yes, I restarted the worker, and the response to the command airflow config get-value core hostname_callable was airflow.utils.net.get_host_ip_address.

[insomnes]

Then, something may be wrong with your worker configuration. You can check that celery inspect works and DB connection.