Live log streaming not working

[elcajon]

@brenner-tobias
I'm not sure if it's a problem on my end, but since the introduction of the live log with the last core/supervisor release, I no longer get the current logs displayed. When I access HA locally (using my internal IP) it works. Is this a problem with my (remotely managed) tunnel or are you seeing this on your end as well? I am not sure as there are no reported issues in this repository so far. I've experimented a bit with tunnel run parameters and cloudflare caching options, but can't seem to solve the problem.

For reference:

latest supervisor logs displayed when viewing with Cloudflare domain:

and checking with local ip (xxx:8123):

I guess it is related to this connection, but not sure....

[brenner-tobias]

Thanks for raising, I did not realise it until now, but I do have the same issue on my end on all my add-ons.
I can scroll-up to load older entries, but no new entries arrive.

[brenner-tobias]

@AlecRust, @elcajon, @felipecrs any ideas, what might be causing this?

[felipecrs]

I once had a somewhat similar issue: esphome/issues#4989

It was fixed by: home-assistant/addons#3356

This is everything I thought of, so far.

[elcajon]

Unfortunately I have no idea, I suspect it is due to the Cloduflare proxy filtering the live logs.
It would be interesting to test:

• The new camera WebRTC feature works through the tunnel, or if there are problems there too. I suspect that the protocol is also used for the logs, but I'm not sure. We would have to check the changes in the supervisor/frontend.
• Live logging via a Cloudflare proxy domain without a tunnel works (port sharing and nginx proxy for example)

[elcajon]

The new camera WebRTC feature works through the tunnel, or if there are problems there too. I suspect that the protocol is also used for the logs, but I'm not sure. We would have to check the changes in the supervisor/frontend.

I am wrong with this, as I learned here: home-assistant/supervisor#5509 (comment)

Live logging via a Cloudflare proxy domain without a tunnel works (port sharing and nginx proxy for example)

I did a test and realized that live logs indeed do work when using Nginx Proxy add-on with Cloudflare DNS and activated Cloudflare DNS proxy --> So the problem is directly related to the cloudflared (tunnel)

[felipecrs]

EDIT: Never mind. You meant using the Nginx add-on standalone, without cloudflared.

What do you mean exactly by Nginx Proxy add-on with Cloudflare DNS? Do you mean enabling this option?

[othorp]

I described a potential workaround to still use cloudflared tunnel by using nginx to rewrite response headers on the thread at home-assistant/supervisor#5509

[elcajon]

As of v5.2.8 (not released at the time of writing) you can change your remote managed tunnel to get log streaming working again by changing your service uri from homeassistant:8123 to localhost:8321. Feel free to share your feedback after the version has been released.

[brenner-tobias]

you can also switch to the edge repository, where it is already included

[felipecrs]

An option to override the default Home Assistant address would be nice too, for users that are not using the Cloudflare Dashboard.

For example, I already use Nginx Proxy Manager. I could just override homeassistant:8123 to a0d7b954-nginxproxymanager instead.

[brenner-tobias]

I am running this since yesterday and it seems like it is not working 100% @elcajon.
Right now, I am missing logs in my view via the tunnel since yesterday 22:52, while I do have around 19 new lines when checking locally.
How does it look for you?

[felipecrs]

I totally agree.

[elcajon]

Have any of you tested the new Supervisor version (recently released in beta)? For me, nothing seems to have changed for the better.

[othorp]

The supervisor update only adds the X-Accel-Buffering header which should improve nginx behaviour but unlikely to fix cloudflared.

Full-fix likely to be more than trivial header rewrite and might require rewriting home-assistant live logging to use Server Side Events.
While the misuse of the mime-type here seems to resolve the problem, the behaviour is not fully specified and may break depending on the presence of colons and newlines in the log data. In addition, without structuring the data correctly the home assistant server won't be sending keep-alives to keep the connection open.

[elcajon]

Do you have any recommendations on how we should deal with this from an add-on perspective?
Unfortunately, I'm not deep enough into this proxy topic to judge whether there's anything we can do from an add-on perspective or not.
I'm also wondering if this means we should use the recently introduced nginx workaround as the new default for the HomeAssistant-Cloudflare tunnel, or if there's a better alternative.
I'm also not sure whether our workaround can be improved in any way.

[othorp]

From my point of view, the workaround as-is improves things (given that without this log streaming doesn't work at all). Therefore it is worth consideration for inclusion as default. API interference is only to logging API endpoints so shouldn't have wider impact. I think it's possible to restrict this further to pick up just the "follow" API endpoints with:

location ~ /api/hassio/.*/logs.*/follow

Remaining Issues to consider:

1. The nginx "follow" API calls timeout after 60s (nginx proxy_read_timeout). This issue is also there with the main nginx addon and results in the home-assistant frontend repeating log lines (TODO: Raise home assistant issue - lack of keep-alive)
2. Mapping content-type to text/event-stream does change browser behaviour as it expects SSE EventStream. Most browsers limit simultaneous SSE connections to 6. At least in Edge / Chrome I can see that that home-assistant live logs stop once they timeout more than 6 times - the SSE connections don't seem to be tidied up properly in the browser post timeout. I don't think this can be improved directly in the addon, although I think it's possible to avoid this by rewriting content-type back to text/plain via CloudFlare Dashboard Page Rules.