Replies: 5 comments 10 replies
-
|
Further investigations actually showed that the route information of another Thread border router even got to be processed when no OTBR was running (and the kernel configurations were disabled). It turns out that the NetworkManager daemon processes IPv6 Neighbor Discovery packets as well. In particular the code in src/core/ndisc/nm-lndp-ndisc.c#L240-265 processes RFC 4191 Route Information packets. So in other words, the kernel configurations are not really necessary for this to work. Also, the kernel configuration by default sets the max prefix len to 0, which essentially disables the feature. A sysctl configuration to set the |
Beta Was this translation helpful? Give feedback.
-
|
If that is enough what NM does, we should keep it like it is. NM knows which interface he can touch and which he should not touch. It feel more safe to me on a docker env |
Beta Was this translation helpful? Give feedback.
-
|
Yeah agree, I don't see a good reason why we should enable it in the kernel right now, especially since it seems that the NetworkManager disables the kernel feature anyways. |
Beta Was this translation helpful? Give feedback.
-
|
I am still working through something on my end, but I wanted to post the current situation as I was the trigger for this issue existing. I am currently helping a HAOS user who has 5 border routers. They are all Apple (mixture of ATV and HomePod). mDNS discovery is working, but user see's a timeout when trying to pair their Eve accessories. Based on the users who I have helped so far, this usually means a problem with IPv6 connectivity. My baseline environment is similar to theirs - I have a single HomePod border router, and I have functional Eve devices. I am using the official HA container, but I am not running HAOS yet (my yellow just arrived, though). My first step is to confirm the HA container sees the icmp6 packet with the Option 24 RI, using tcpdump. This lets me rule out things like VLAN's or strange WiFi repeaters breaking icmp6. The container does see the RI: However at first glance, there was no route information being added to the kernels route table. On other systems the solution has been to set We tried to do this here, but it was not possible because With @agners help, we realised that something in HAOS was processing the route info that wasn't the kernel. As above, this turned out to be NM. User had ipv6 turned off in HA, which i didn't notice because link local addressing was working (this is expected since home-assistant/supervisor#3829). When we turned it on routes apper in Note that the thread ULA would normally have a via. For example, here is the equivalent entries from my environment. I am using the same HA container to generate this table, and the same Apple hardware, so I would expect to see a route in his environment like this but I don't. I am wondering if the busybox version of ip is to blame here - it is worth calling out that the same route table entry on a Debian box with "full" iproute2 looks like this: So either NM isn't adding correct routes, or the default alpine I wasn't expecting this to work, because the route table looks "wrong" to me, but I did try So, working theory was So take away from this is:
|
Beta Was this translation helpful? Give feedback.
-
Yes, in the Thread Border Router - Bidirectional IPv6 Connectivity and DNS-Based Service Discovery. However, as by my findings in #2333 (comment), it seems that at least It seems that NetworkManager also has a way to push routes into a specific routing table ( I agree with your take aways. I'd also like to add full |
Beta Was this translation helpful? Give feedback.
-
|
I was wondering if NetworkManager's IPv6 link-local address configuration By experimentation it seems that NetworkManager does not process IPv6 Neighbor Discovery packets. And looking into NetworkManager source code supports that: It also seems that the kernels routing advertising processing is disabled by the NetworkManager in src/core/devices/nm-device.c#L11446. |
Beta Was this translation helpful? Give feedback.
-
|
So since our last discussion I have been working with a number of users in the community to try to find the cause of instability with their Thread setups. One case that I've been looking into is a user with"ghost" routes. When I first helped them Whilst the exact cause of that is not known for them, I have now been able to force my own mesh into the same state. And it is fairly trivial. Working setup (Debian + HA Container)First of all, here is my production Home Assistant. This environment is showing what I consider to be expected. It's a Debian machine running containers. It is connected to my iot vlan. Both of these have expiry data visible. At this point I unplugged one of the HomePod's. I left it alone for 30 minutes and when I came back: As expected, one route had expired. The other had continued to see new RA's so the expires time has actually gone up since I last checked. I checked As expected, the border router I unplugged has been removed. And pinging my accessories works: And for completeness: System with Ghosts (HAOS)Now here is a HAOS 9.5 system. Based on the conversation in this thread, we know that NetworkManager is in charge of processing RA's. We start off in the same state as the Debian system: However it's worth calling out at this stage how differently formatted the output is, despite using the same version of After half an hour has passed I returned and checked. Whilst the Debian system no longer had the stale route, HAOS does:
My accessory cannot be pinged: Confusingly As soon as we delete the stale route: ping works: |
Beta Was this translation helpful? Give feedback.
-
|
Debian, no NM: For completeness, the resulting RT: HAOS / NM And the new route table: |
Beta Was this translation helpful? Give feedback.
-
|
It looks like NM and the kernel get out of sync. Right now I see this: But I see this: I should add that right now If I look at the logs for when I turned the 2nd HomePod on I see this: It adds a route via b569 (office homepod) and tries to delete 85e5 (bedroom HomePod). We don't get any more RA's from b569 (cos I turned it back off) and now 85e5 is still sending RA's but they are hitting the EEXIST error. Then I turned the office one back on. Now 9226 is a new route so it can be added: But it is immediately invisible: I am wondering if there might be 2 problems:
|
Beta Was this translation helpful? Give feedback.
-
|
Sounds like this is fixed in or before 1.40 as Debian doesn't hit this. In fact, that error doesn't even exist any more. So the buildroot update you mentioned might fix "ghost routes" However, the new behaviour is not that great. It looks like the last RA wins. Only a single route at any time. So for my user with 8 HomePods, where the RA's go out every 3 minutes, his default route to the mesh is going to churn 2-3 times a minute. |
Beta Was this translation helpful? Give feedback.
-
|
Further investigations with @Jc2k brought new facts to daylight. Test bed was again Apple BR devices. With OS 10.0.rc1 and NetworkManager 1.40.10, the behavior is different: Only a single route gets added, each time a new RIO comes in the route of the previous BR gets overwritten. This effectively leads to the routing table to flip between two states: Router Advertisements are sent every 180s, so on average the routes flip every 90s. It is unclear if this is problematic, for a simple ping it doesn't seem to matter. However, if one of the routers fails, it can take up to 180s for the others advertisement to re-add itself and therefor "fix" the routing table. The following BRs are used:
The reason is that only a single route to a given destination network with a given prefix length is used. Each subsequent RIO is updating that same route, see: Adding a check to match the gateway in this if statement changes NetworkManger's behavior such that it creates two routes: Note that the routes don't carry an However, with two routes now, whichever route is active, will be the one which is chosen (in my testing it was always the lower). Which one it is can also be checked with When I now unplug the Apple HomePod device (breaking the lower route), the Thread device is no longer reachable. In practice, this leads to not working routes between 27 and 30min (depending on when the RIO came in, until the bad route is purged by NetworkManager)! A little Ubuntu 22.04.1 LTS Server Side quest, open up details if interested.
On a Ubuntu 22.04.1 LTS Server systemd-networkd 249 is running. It seems that this device is suffering a similar problem. However, in this case the routes are grouped. Presumably they then also expire together (which might be a problem of systemd-networkd):
Shouldn't Linux detect non-working routes somehow? @Jc2k pointed me to With routing disabled, the route availability check seems to work flawlessly! Route selection is almost instant, at least using ping no big hick-up can be detected. Note With these findings, it is probably advisable to:
|
Beta Was this translation helpful? Give feedback.
-
|
Instead of globally disable IPv6 routing again I came up with a patch to force enable router availability probing. I've also sent an RFC PATCH to the mailing list, maybe this is something which Linux can support upstream. As for the NetworkManager change: It seems to work reliable, and I suggested the change in the NetworkManager issue previously opened to discuss issues we've been seen: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/1232#note_1841630. Depending on outcome of that discussion, I'll try to upstream this change as well. PR #2434 applies the two changes to HAOS. |
Beta Was this translation helpful? Give feedback.
-
|
I have one similar problem with my 3 OTBR on 3 machines.
Other thread devices:
With HA adding possibility for home kit controller Provision devices from BT to thread i was getting 2 EVE Eanergy and one Nanoleaf RGBW light joining the tread network with the 3 OTBR and one OTFD-CLI device but HA is having problem keeping contact to them. By disabling the OTBR and only running the Zigbeed on the first ans last host the second one is working great and its fast as one flash then sending and receiving command to and from my tread device !!! If the multi OTBR was OK my thread network have being moved to my production system but is not critical devices on it so its staying until its working OK. If its something i can helping doing diagnostic test and patches i one good setup for doing the test but dont have very deep knowledge in IPV6 and Linux but can doing it with little help. And one great thanks you both for getting all thins working great so fare with all work being done !! |
Beta Was this translation helpful? Give feedback.
-
Currently Home Assistant OS does not have
CONFIG_IPV6_ROUTER_PREForCONFIG_IPV6_ROUTE_INFOset.These options add support for ICMPv6 Router Advertisements with Route Information Option (Type 24). This type of Router Information are sent by Thread Border Routers.
Note: It seems that the
otbr-agentprocesses such advertisements on its own as well:This means that a Home Assistant OS installation which runs a OTBR actually learns about routes to other Thread border routers. However, Home Assistant OS installations without a(see belowotbr-agentrunning do not learn the routes to other BRs and therefore can not communicate with Thread devices connected to the Thread network behind those BRs.See also this discussion on Discord.
Beta Was this translation helpful? Give feedback.
All reactions