Multicast command failing without reason

[dcmeglio]

Checklist

• I have read and followed the above instructions

• I have checked the troubleshooting section and my problem is not described there.

• I have read the changelog and my problem was not mentioned there or the fix did not work.

Describe the issue

What is happening?
I was having an issue where I'd send an "on" command to 7 switches one after the other and was experiencing multiple failures. Someone suggested using multicast commands. Even with this some devices turn on, others do not respond. Feels like something is getting overloaded but 7 commands doesn't seem like a lot.

What did you expect to happen instead?
The switches should turn on.

Steps to reproduce the behavior:

1. Using HA send a multicast command to 7 switch devices, only some will respond.

Anything else we should know?

Software versions

Driver (node-zwave-js): 13.2.0

Z-Wave JS UI: 9.18.1

Home Assistant Z-Wave Integration: 2024.9.2

Home Assistant Z-Wave JS Addon: 3.11.0

ioBroker.zwave2 Adapter: I don't know what this is.

If you are using something non-standard, tell us here: ...

Device information

Does not appear to be a specific device but I was controlling these nodes 118, 80, 141, 26, 129, 58, 83

Checklist

• I made sure to provide a driver log on level debug.

• The log includes a re-interview of the problematic device (if applicable).

• The log includes the problematic interaction with the device (if applicable).

• I provided the node ID of the problematic device (if applicable).

Upload Logfile

zwavejs_current.log

[AlCalzone]

Which Z-Wave stick is that? Which firmware version is it on?

[dcmeglio]

SILabs SLUSB7000A UZB-7 running firmware 7.17.2 I have been considering updating the firmware but discussing with some people on the HASS Discord it sounded questionable if that's a good idea since the version I'm on is generally considered very stable, but I can do that if you suggest.

[dcmeglio]

FYI I also just updated Zwave JS just now since a new version is out, though I don't see anything in the release notes that seems related to my issue.

Driver (node-zwave-js): 13.3.1
Z-Wave JS UI: 9.20.0
Home Assistant Z-Wave JS Addon: 3.12.0

[AlCalzone]

Okay, so this is a difficult question to answer. 7.17.2 is indeed considered pretty stable, but there is something funky going on according to your log. Fail usually indicates that the controller was not able to transmit anything, but in this case it did. Some nodes respond, but apparently the controller no longer ACKs their commands. So obviously the firmware isn't perfect.

I think I saw the discussion and you were also having issues with non-multicast commands?

You can try upgrading, but you have to know there's no easy way back to 7.17.2.
The later 7.18.x versions are generally okay. If you want to go higher, go for 7.21.4 directly. Everything below that is a clusterfuck in one way or another.

[dcmeglio]

Yup that was me. Tell me what you think of this plan:
I have a new Zooz 800 stick coming tomorrow, I will upgrade that to 7.21.4, then I will do an NVM backup from my current SI Labs stick and restore on the Zooz (it should be ok that they're on different firmware to do the NVM backup/restore or no?)
That should rule out faulty hardware and firmware issues (I hope) and I get a nice new 800 controller too

[AlCalzone]

Sound good, except that you use Zooz's firmware 1.50, which is based on SDK 7.22.1 (exclusive to 800 series).
Doing the NVM backup/restore in Z-Wave JS UI makes sure the backup is converted to the correct format.

[dcmeglio]

Well the NVM backup/restore was incredibly smooth (kudos for that!) but
Unfortunately this did not resolve my issue. I'm now on an 800 series Zooz so I think we can rule out faulty hardware, and if it is firmware I'm now on a completely different firmware, Zooz 1.50

Note that same transmit fail at 2024-09-25T23:15:35.280Z

zwavejs_current.log

Anything else I can provide to figure out what's going on? Any work arounds you can think of? I can reproduce it pretty consistently

I ssh'ed into the addon container and have:
tail -f zwavejs_current.log |grep "transmit status: Fail"

I can get it to happen 100% of the time when I run my automation. I have a workaround of adding some delays but I'd love to find out if there's something I can do to track it down

[dcmeglio]

Closest thing I have is a ZWP-TBX (https://www.zwaveproducts.com/products/zwp-tbx-z-wave-toolbox) that I've added as a secondary controller in the past to troubleshoot zwave issues when I was on hubitat. Will that work?

Happy to share my NVM backup, since it will have things like my encryption keys is there an email or somewhere I can send it instead of posting on github? Edit: Sent you the NVM backup via HA Discord

[AlCalzone]

Looks like that thing already has a Zniffer built in ("Packet Analyzer"). I'm not sure if it can export in a usable format, but that would be worth a shot.

I didn't spot anything weird in the NVM for the nodes you're communicating with, except that some have a hop in their LWR, so the initial multicast frame might not reach them all - they seem to react to the singlecast followup though.

What strikes me as odd, is that some nodes which you didn't send a command to are responding with their new state: Node 89 and 101. Do some of the targeted devices have an association to forward the command to others?

[dcmeglio]

Sorry I thought I grabbed enough of the log but I didn't :( the 101 and 89 are from a prior multicast command in the same automation.

I'll setup the packet analyzer this weekend and also grab better logs. I've used the packet analyzer before, it's pretty easy to use, I just join it to the network as a secondary controller. I'll see what format I can get data out in.

[dcmeglio]

Also is it worth switching back to single cast? It definitely seems to have something to do with the volume of commands. If I add delays in the automation it works perfectly. I can also share the automation config that is causing the issues if it helps.

[AlCalzone]

Also is it worth switching back to single cast?

Multicast is mainly meant for reducing/eliminating the popcorn effect when controlling multiple devices. So if you were already having issues with singlecast, it may be worth looking at that first.

Oh, and just to be sure: Do you have the controller on a USB extension?
https://zwave-js.github.io/node-zwave-js/#/troubleshooting/first-steps?id=controller-placement

[Botched1]

Here for the "me too" that I'm having multicast issues. I do have a UZB3 flashed with sniffer firmware, but it is not hooked up right now. I can do that this weekend - is there any preferred way to get the sniffer logs in a format that is unable for you?

[AlCalzone]

Just use either the official (Windows) Zniffer application, or Z-Wave JS UI. Both can export in the same .zlf format.

[Botched1]

Here's a zniffer log of the error, captured in zwave js ui (native zniffer support was such a GREAT addition!).

ZNIFFER.zip

Last error I saw was at 09:32am my local time (US central). Not sure how that translates to log time as the timestamps look really weird to me in the zlf.

@AlCalzone In case it is useful info, all 4 devices are of type:

I have been planning on replacing them with Zwave LR devices (have them on hand), but haven't yet in case this would be useful for troubleshooting.

Thanks,
Jason

[AlCalzone]

@Botched1 can you tell me which firmware (SDK version) is running on the controller and the end devices? You should be able to see the latter in the Version dropdown for each node in Z-Wave JS UI.

This Zniffer log looks like neither the end devices, nor the controller are acknowledging the routed singlecast frames following the multicast.

[AlCalzone]

Also, does this exact issue always happen with these devices? Or does it sometimes work? If so, a Zniffer log of how a successful case looks would be good aswell.

[Botched1]

can you tell me which firmware (SDK version) is running on the controller and the end devices?

Controller = 7.22.1
End Devices = See below - I wasn't sure which number you needed.

Also, does this exact issue always happen with these devices? Or does it sometimes work? If so, a Zniffer log of how a successful case looks would be good as well.

It does sometimes work - actually works something like 8 out of 10 times. I'll get a zniffer log with that.

[dcmeglio]

Multicast is mainly meant for reducing/eliminating the popcorn effect when controlling multiple devices. So if you were already having issues with singlecast, it may be worth looking at that first.

Yeah I am trying to eliminate popcorning too, but we can try with singlecast first. I'll make changes to my automation. I'll get the secondary controller setup this weekend too and see what I can capture.

Oh, and just to be sure: Do you have the controller on a USB extension?

Yup, other than not as centrally located as would be ideal, all other guidelines are met. The USB extender has it at least 3-4ft from any other USB port or device and outside my server rack.

[dcmeglio]

I'm still working on this. I know it's just coincidence but I can't seem to reproduce it when I have the zniffer running. But unfortunately the issue is occurring every day in production use.

[dcmeglio]

I think I’m at the point where I need to just downgrade zwave js. Almost every zwave automation I have fails if it involves controlling 3 or more devices roughly 75% of the time. Not specific devices. Not consistent. Just random failure. Zwave has pretty much become unusable for me after working flawlessly for 2+ years. I haven’t been able to really capture much useful with the zniffer but I had 4 separate failures today alone. Other than zniffer logs is there anything else useful I can provide? The only workaround I’ve found is to add 5 second delays between every zwave command

[dcmeglio]

I rewrote all of my automations to be singlecast and I have had no issues. I was previously having issues but that was before I swapped out my controller. So I wonder if I'm having the same issue as @Botched1 now. I'll do some further troubleshooting

[dcmeglio]

I've been running singlecast automations for a week now and it's working perfectly. My original singlecast issues were when I was still on the 700 controller. I can only reproduce problems on 800 with multicast. I haven't found the secret to reproduce consistently, but I do think it is directly related to multicast.

[AlCalzone]

I haven't found the secret to reproduce consistently, but I do think it is directly related to multicast.

Did you manage to get your traffic analyzer working? If you could provide a Zniffer trace which shows a working multicast and a failing multicast, plus a bit of time before each one, that might help.

[dcmeglio]

Yes, but the only export option it has is CSV. If you need some specific format for you to be able to analyze, just let me know what device I need to buy and I'll get it setup.

[AlCalzone]

Happy to look at one of those to see if it helps at all.

[dcmeglio]

zwave_logs.zip
zwave js logs during a success and failure and the network analyzer logs during the same time windows

In case it's helpful this is what I ran:

action: zwave_js.multicast_set_value
data:
  device_id:
# node 83
    - 060fca1a3da43092516b3f8a6468e541
# node 26
    - 47a51fb1c5eefe6acbda20d1430ecbb0
# node 118
    - 6db2e43b0dc4aebd47c85838db4bd09c
# node 80
    - dc32aa0c77f8996bf3a5693c336ece5c
 # node 141
    - b7e51fbfca1a052721858e5791bf4b87
  # node 58
    - 0462015716f9e0bf1ab11a448cb784ae
  command_class: "37"
  property: targetValue
  value: true