Z2M addon hangs and needs restart randomly few times a day in HA

[shaikedem]

Hello there
im trying to figure how to debug this issue

hi all,

I had 2 Z2M instances in HA working with 2 zigbee Tasmota Sonoff ethernet based coordinators for a while
this is the coordinator

https://a.aliexpress.com/_ooK11yi

in last few weeks, one of the Z2M instances (I call it Z2M2) started to hang randomly
My other Z2M instance (i call it Z2M1) is stable and its same device exactly same f/w etc.

This Z2M2 instance has like total of 19 devices, mostly Tuya - temp sensors, motion/presence sensors, few smart plugs, contact / light sensors.
Z2M1 has fewer devices (5) Also mostly Tuya

I am updated to latest HA, and both Z2M addon instances was recently updated as well (1.40.1-1) though im not sure its related to when the problem started

I didnt touch them physically nor did I add any new zigbee sensor recently.

My Coordinators have the this f/w, which is latest AFAIK.
xsp1989/zigbeeFirmware#49 (comment)

The addon for Z2M2 instance would just stop randomly. this happens now like 3-6 times a day.

i have an automation which detects it and starts it, so my HA Z2M starts working again, but its annoying and does seem right operation mode.

this is the log message of the Z2M2 addon i have when it stops

[2024-09-18 00:01:23] error: 	zh:ember:uart:ash: Received ERROR from adapter, with code=ERROR_EXCEEDED_MAXIMUM_ACK_TIMEOUT_COUNT.
[2024-09-18 00:01:23] error: 	zh:ember:uart:ash: ASH disconnected | Adapter status: ASH_NCP_FATAL_ERROR
[2024-09-18 00:01:23] error: 	zh:ember:uart:ash: Error while parsing received frame, status=ASH_NCP_FATAL_ERROR.
[2024-09-18 00:01:23] error: 	zh:ember: Adapter fatal error: HOST_FATAL_ERROR
[2024-09-18 00:01:23] debug: 	zh:controller: Adapter disconnected
[2024-09-18 00:01:23] info: 	zh:ember:uart:ash: ASH COUNTERS since last clear:
[2024-09-18 00:01:23] info: 	zh:ember:uart:ash:   Total frames: RX=293, TX=286
[2024-09-18 00:01:23] info: 	zh:ember:uart:ash:   Cancelled   : RX=0, TX=0
[2024-09-18 00:01:23] info: 	zh:ember:uart:ash:   DATA frames : RX=275, TX=103
[2024-09-18 00:01:23] info: 	zh:ember:uart:ash:   DATA bytes  : RX=6322, TX=2090
[2024-09-18 00:01:23] info: 	zh:ember:uart:ash:   Retry frames: RX=16, TX=0
[2024-09-18 00:01:23] info: 	zh:ember:uart:ash:   ACK frames  : RX=0, TX=182
[2024-09-18 00:01:23] info: 	zh:ember:uart:ash:   NAK frames  : RX=0, TX=0
[2024-09-18 00:01:23] info: 	zh:ember:uart:ash:   nRdy frames : RX=0, TX=0
[2024-09-18 00:01:23] info: 	zh:ember:uart:ash:   CRC errors      : RX=0
[2024-09-18 00:01:23] info: 	zh:ember:uart:ash:   Comm errors     : RX=0
[2024-09-18 00:01:23] info: 	zh:ember:uart:ash:   Length < minimum: RX=0
[2024-09-18 00:01:23] info: 	zh:ember:uart:ash:   Length > maximum: RX=0
[2024-09-18 00:01:23] info: 	zh:ember:uart:ash:   Bad controls    : RX=0
[2024-09-18 00:01:23] info: 	zh:ember:uart:ash:   Bad lengths     : RX=0
[2024-09-18 00:01:23] info: 	zh:ember:uart:ash:   Bad ACK numbers : RX=0
[2024-09-18 00:01:23] info: 	zh:ember:uart:ash:   Out of buffers  : RX=0
[2024-09-18 00:01:23] info: 	zh:ember:uart:ash:   Retry dupes     : RX=16
[2024-09-18 00:01:23] info: 	zh:ember:uart:ash:   Out of sequence : RX=0
[2024-09-18 00:01:23] info: 	zh:ember:uart:ash:   ACK timeouts    : RX=0
[2024-09-18 00:01:24] error: 	zh:ember:uart:ash: Error while parsing received frame, status=ASH_NCP_FATAL_ERROR.
[2024-09-18 00:01:24] error: 	zh:ember:uart:ash: Error while parsing received frame, status=ASH_NCP_FATAL_ERROR.
[2024-09-18 00:01:24] info: 	zh:ember:uart:ash: ======== ASH stopped ========
[2024-09-18 00:01:24] info: 	zh:ember:ezsp: ======== EZSP stopped ========
[2024-09-18 00:01:24] info: 	zh:ember: ======== Ember Adapter Stopped ========
[2024-09-18 00:01:24] error: 	z2m: Adapter disconnected, stopping
[2024-09-18 00:01:24] debug: 	z2m: Saving state to file /config/zigbee2mqtt2/state.json
[2024-09-18 00:01:24] info: 	z2m:mqtt: MQTT publish: topic 'zigbee2mqtt2/bridge/state', payload '{"state":"offline"}'
[2024-09-18 00:01:24] info: 	z2m: Disconnecting from MQTT server
[2024-09-18 00:01:24] info: 	z2m: Stopping zigbee-herdsman...
[2024-09-18 00:01:24] debug: 	zh:controller:database: Writing database to '/config/zigbee2mqtt2/database.db'
[2024-09-18 00:01:24] info: 	z2m: Stopped zigbee-herdsman
[2024-09-18 00:01:24] info: 	z2m: Stopped Zigbee2MQTT

after above log the addon just stops. and needs to restarted manually or by my automation

im not sure what the RC for this

the LAN network is stable AFAIK

any idea how to debug this?

[petruchito]

I have a similar crash when I try to transfer OTA updates. Sonoff dongle E, ncp 230400 baud v7.4.4 firmware (I tried quite a few). Latest z2m. So it can relate to the amount of bytes transferred in a frame.

Log

[2024-10-19 20:48:58] info:     z2m:mqtt: MQTT publish: topic 'zigbee2mqtt/Термометр спальня', payload '{"battery":100,"comfort_display":"hide","comfort_humidity_max":60,"comfort_humidity_min":40,"comfort_temperature_max":25,"comfort_temperature_min":20,"display":"on","humidity":53.06,"humidity_calibration":0,"last_seen":"2024-10-19T20:48:58+03:00","linkquality":232,"measurement_interval":60,"temperature":28.62,"temperature_calibration":0,"temperature_display_mode":"celsius","update":{"installed_version":19017729,"latest_version":19083265,"progress":82.95,"remaining":0,"state":"updating"},"update_available":false}'
[2024-10-19 20:48:59] error:    zh:ember:uart:ash: Received ERROR from adapter, with code=ERROR_EXCEEDED_MAXIMUM_ACK_TIMEOUT_COUNT.
[2024-10-19 20:48:59] error:    zh:ember:uart:ash: ASH disconnected | Adapter status: ASH_NCP_FATAL_ERROR
[2024-10-19 20:48:59] error:    zh:ember:uart:ash: Error while parsing received frame, status=ASH_NCP_FATAL_ERROR.
[2024-10-19 20:48:59] error:    zh:ember: Adapter fatal error: HOST_FATAL_ERROR
[2024-10-19 20:48:59] info:     zh:ember:uart:ash: ASH COUNTERS since last clear:
[2024-10-19 20:48:59] info:     zh:ember:uart:ash:   Total frames: RX=219, TX=305
[2024-10-19 20:48:59] info:     zh:ember:uart:ash:   Cancelled   : RX=0, TX=0
[2024-10-19 20:48:59] info:     zh:ember:uart:ash:   DATA frames : RX=213, TX=88
[2024-10-19 20:48:59] info:     zh:ember:uart:ash:   DATA bytes  : RX=4370, TX=2388
[2024-10-19 20:48:59] info:     zh:ember:uart:ash:   Retry frames: RX=3, TX=3
[2024-10-19 20:48:59] info:     zh:ember:uart:ash:   ACK frames  : RX=0, TX=213
[2024-10-19 20:48:59] info:     zh:ember:uart:ash:   NAK frames  : RX=0, TX=0
[2024-10-19 20:48:59] info:     zh:ember:uart:ash:   nRdy frames : RX=0, TX=0
[2024-10-19 20:48:59] info:     zh:ember:uart:ash:   CRC errors      : RX=0
[2024-10-19 20:48:59] info:     zh:ember:uart:ash:   Comm errors     : RX=0
[2024-10-19 20:48:59] info:     zh:ember:uart:ash:   Length < minimum: RX=0
[2024-10-19 20:48:59] info:     zh:ember:uart:ash:   Length > maximum: RX=0
[2024-10-19 20:48:59] info:     zh:ember:uart:ash:   Bad controls    : RX=0
[2024-10-19 20:48:59] info:     zh:ember:uart:ash:   Bad lengths     : RX=0
[2024-10-19 20:48:59] info:     zh:ember:uart:ash:   Bad ACK numbers : RX=0
[2024-10-19 20:48:59] info:     zh:ember:uart:ash:   Out of buffers  : RX=0
[2024-10-19 20:48:59] info:     zh:ember:uart:ash:   Retry dupes     : RX=3
[2024-10-19 20:48:59] info:     zh:ember:uart:ash:   Out of sequence : RX=0
[2024-10-19 20:48:59] info:     zh:ember:uart:ash:   ACK timeouts    : RX=3
[2024-10-19 20:48:59] info:     zh:ember:uart:ash: Port closed.
[2024-10-19 20:48:59] info:     zh:ember:uart:ash: ======== ASH stopped ========
[2024-10-19 20:48:59] info:     zh:ember:ezsp: ======== EZSP stopped ========
[2024-10-19 20:48:59] info:     zh:ember: ======== Ember Adapter Stopped ========
[2024-10-19 20:48:59] error:    z2m: Adapter disconnected, stopping
[2024-10-19 20:48:59] info:     z2m:mqtt: MQTT publish: topic 'zigbee2mqtt/bridge/state', payload 'offline'
[2024-10-19 20:48:59] info:     z2m: Disconnecting from MQTT server
[2024-10-19 20:48:59] info:     z2m: Stopping zigbee-herdsman...
[2024-10-19 20:48:59] info:     z2m: Stopped zigbee-herdsman
[2024-10-19 20:48:59] info:     z2m: Stopped Zigbee2MQTT