Suggestion to work around failed updates

[jcjveraa]

Hi @golles, perhaps this could be an idea? Ideally the API is 'fixed' to return non-corrupt json, but there might be other reasons too why the API would not be available - i.e. we can't rely on the API always being there. To allow for that, we could implement something like this to ensure a single failed update does not turn the entities 'unavailable'.

I have this running with my own API key on a 'devcontainer-hass' now, I'll leave it running for a few hours to see if this has actually worked as expected. I have zero experience with HASS development, but my assumption is that the state of the integration's entities will only become 'unavailable' if an error is thrown.

Ideally for me (and perhaps other users):

• I prefer 'stale' data (old data) over 'no data' ('unavailable), provided I can control how old the data can be;
• I would like to see how old the data is so that for "time critical" data I can perhaps deviate from the default (already provided by the integration via knmi_latest_update, so no change required).

Note: treat this as a 'stub', should be expanded with tests etc, but unfortunately I'm not too familiar with HASS python coding and my VSCode IDE PyLance doesn't seem to pick up the installed requirements_dev packages, so no guarantee I will do that for now

It seems to work so far.
As is, a 20 minute (2 failed updates) gap in the data:

With this PR (I started up HASS around 09:06):

[jcjveraa]

The logs seem to be working fine

2024-04-16 07:56:33.873 WARNING (MainThread) [custom_components.knmi] Update failed 2 times! - unexpected character: line 1 column 261 (char 260)
2024-04-16 07:56:33.874 DEBUG (MainThread) [custom_components.knmi] Finished fetching knmi data in 0.056 seconds (success: True)

But I am getting this lot below too a lot in this version. Sometimes with the coordinator.get_value(["liveweer", 0, "image"], sometimes with the alarm.

If I examine the DEBUG logs more closely I notice that after every full update HASS or this integration seems to want to load quite a few items with the alarm value first, and every 30 seonds it tries to update about 5 items with the image being the first. Long story short: it fails every now and then due to the dictionary not being populated. This could of course very well be related to this change.

TypeError: 'NoneType' object is not subscriptable
2024-04-16 07:55:10.119 ERROR (Recorder) [homeassistant] Error doing job: Task exception was never retrieved:   File "/home/vscode/.local/bin/hass", line 8, in <module>
    sys.exit(main())
  File "/home/vscode/.local/lib/python3.12/site-packages/homeassistant/__main__.py", line 210, in main
    exit_code = runner.run(runtime_conf)
  File "/home/vscode/.local/lib/python3.12/site-packages/homeassistant/runner.py", line 189, in run
    return loop.run_until_complete(setup_and_run_hass(runtime_config))
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 672, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 639, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1977, in _run_once
    handle._run()
  File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/home/vscode/.local/lib/python3.12/site-packages/homeassistant/helpers/event.py", line 1508, in _run_async_call_action
    hass.async_run_hass_job(job, time_tracker_utcnow())
  File "/home/vscode/.local/lib/python3.12/site-packages/homeassistant/core.py", line 837, in async_run_hass_job
    hassjob.target(*args)
  File "/home/vscode/.local/lib/python3.12/site-packages/homeassistant/helpers/event.py", line 1602, in _interval_listener
    hass.async_run_hass_job(self._run_job, now, background=True)
  File "/home/vscode/.local/lib/python3.12/site-packages/homeassistant/core.py", line 837, in async_run_hass_job
    hassjob.target(*args)
  File "/home/vscode/.local/lib/python3.12/site-packages/homeassistant/helpers/entity_platform.py", line 658, in _async_handle_interval_callback
    self.config_entry.async_create_background_task(
  File "/home/vscode/.local/lib/python3.12/site-packages/homeassistant/config_entries.py", line 1059, in async_create_background_task
    task = hass.async_create_background_task(target, name, eager_start)
  File "/home/vscode/.local/lib/python3.12/site-packages/homeassistant/core.py", line 761, in async_create_background_task
    task = create_eager_task(target, name=name, loop=self.loop)
  File "/home/vscode/.local/lib/python3.12/site-packages/homeassistant/util/async_.py", line 33, in create_eager_task
    return Task(
Traceback (most recent call last):
  File "/home/vscode/.local/lib/python3.12/site-packages/homeassistant/helpers/entity_platform.py", line 1027, in _update_entity_states
    await entity.async_update_ha_state(True)
  File "/home/vscode/.local/lib/python3.12/site-packages/homeassistant/helpers/entity.py", line 969, in async_update_ha_state
    self._async_write_ha_state()
  File "/home/vscode/.local/lib/python3.12/site-packages/homeassistant/helpers/entity.py", line 1119, in _async_write_ha_state
    state, attr, capabilities, shadowed_attr = self.__async_calculate_state()
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.12/site-packages/homeassistant/helpers/entity.py", line 1056, in __async_calculate_state
    state = self._stringify_state(available)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.12/site-packages/homeassistant/helpers/entity.py", line 1004, in _stringify_state
    if (state := self.state) is None:
                 ^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.12/site-packages/homeassistant/components/weather/__init__.py", line 900, in state
    return self.condition
           ^^^^^^^^^^^^^^
  File "/hass/custom_components/knmi/weather.py", line 122, in condition
    return self.map_condition(self.coordinator.get_value(["liveweer", 0, "image"]))
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/hass/custom_components/knmi/coordinator.py", line 76, in get_value
    value = value[key]
            ~~~~~^^^^^
TypeError: 'NoneType' object is not subscriptable

[jcjveraa]

Seems to be fixed by the previous commit - apparently it must return data. New logs on a failed retrieval:

2024-04-16 09:43:41.555 DEBUG (MainThread) [custom_components.knmi] Path ['liveweer', 0, 'dauwp'] returns a float (value = 5.2)
2024-04-16 09:43:41.555 DEBUG (MainThread) [custom_components.knmi] Path ['liveweer', 0, 'lv'] returns a int (value = 76)
2024-04-16 09:43:41.555 DEBUG (MainThread) [custom_components.knmi] Path ['liveweer', 0, 'luchtd'] returns a float (value = 1000.33)
2024-04-16 09:43:41.555 DEBUG (MainThread) [custom_components.knmi] Path ['liveweer', 0, 'windrgr'] returns a float (value = 343.9)
2024-04-16 09:43:41.555 DEBUG (MainThread) [custom_components.knmi] Path ['liveweer', 0, 'windkmh'] returns a float (value = 29.4)
2024-04-16 09:43:41.555 DEBUG (MainThread) [custom_components.knmi] Path ['liveweer', 0, 'zicht'] returns a int (value = 100000)
2024-04-16 09:43:41.670 WARNING (MainThread) [custom_components.knmi] Update failed 1 times! - unexpected character: line 1 column 261 (char 260)
2024-04-16 09:43:41.670 DEBUG (MainThread) [custom_components.knmi] Update failed, returning existing data
2024-04-16 09:43:41.670 DEBUG (MainThread) [custom_components.knmi] Finished fetching knmi data in 0.075 seconds (success: True)
2024-04-16 09:43:41.671 DEBUG (MainThread) [custom_components.knmi] Path ['liveweer', 0, 'alarm'] returns a int (value = 0)
2024-04-16 09:43:41.671 DEBUG (MainThread) [custom_components.knmi] Path ['liveweer', 0, 'lkop'] returns a str (value = Er zijn geen waarschuwingen)
2024-04-16 09:43:41.671 DEBUG (MainThread) [custom_components.knmi] Path ['liveweer', 0, 'ltekst'] returns a str (value =  Er zijn momenteel geen waarschuwingen van kracht.)
2024-04-16 09:43:41.671 DEBUG (MainThread) [custom_components.knmi] Path ['liveweer', 0, 'wrschklr'] returns a str (value = groen

[golles]

Hi @jcjveraa thank you for your contribution!

I find it an interesting concept to cache the data in case responses are incorrect. I never considered this as I assumed the server wouldn't have any issues.
I guess you were facing issue #130 and for that, I made a workaround to deal with that in #141

One downside of serving cached data is that it can become stale (for example the weather this week has been changing a lot; sunny, rainy, hail, lightning). With the default interval of 5 minutes and allowing 6 failures we can end in a situation with showing stale data for 30 minutes. Since we don't know why there is a server-side issue, I think having no data isn't that bad, it shouldn't trigger anything on the user side (eg. when sunny for 20 minutes open awning).

[HiDiHo01]

To keep track of api availability maybe you clould create a sensor "API beschikbaar"

[golles]

To keep track of api availability maybe you clould create a sensor "API beschikbaar"

I wouldn't suggest doing that, that will make each sensor by itself useless, you would always need to check such a status to know if you can use the value. If you really want to do such a check you can check the timestamp of sensor.knmi_laatste_update

[HiDiHo01]

To keep track of api availability maybe you clould create a sensor "API beschikbaar"

I wouldn't suggest doing that, that will make each sensor by itself useless, you would always need to check such a status to know if you can use the value. If you really want to do such a check you can check the timestamp of sensor.knmi_laatste_update

Can you see a API outtage in this?

[golles]

To keep track of api availability maybe you clould create a sensor "API beschikbaar"

I wouldn't suggest doing that, that will make each sensor by itself useless, you would always need to check such a status to know if you can use the value. If you really want to do such a check you can check the timestamp of sensor.knmi_laatste_update

Can you see a API outtage in this?

Not like this, but if you want to, you need to check that the timestamp is older than your configured scan interval