Cannot Switch to Auto Mode on Second Autoware Instance over LAN with Zenoh Bridge

[Justin-Xiang]

I am running two Autoware instances on two separate machines within the same LAN. The first machine (IP 192.168.1.1) can successfully switch its vehicle to Auto mode, while the second machine (IP 192.168.1.2) cannot — even though it can successfully plan routes. Additionally, there are error messages in the zenoh_bridge_ros2dds.log file that I have not encountered before.

Setup Details

Machine 1

• IP: 192.168.1.1
• Actions:
  • Starts Carla
  • Builds and runs the Zenoh bridge and Autoware using:
    • run-bridge-two-vehicles.sh
    • run-autoware.sh v1

Machine 2

• IP: 192.168.1.2
• Actions:
  • Builds Autoware
  • Changes in run-autoware.sh:

    export ZENOH_CARLA_IP_PORT="${2:-'192.168.1.1:7447'}" # previously 127.0.0.1
    export ZENOH_FMS_IP_PORT="${3:-'192.168.1.1:7887'}"  # previously 127.0.0.1

  • Runs run-autoware.sh v2

Expected Behavior

• Both Autoware instances should be able to plan routes and switch the vehicle to Auto mode in Carla.

Actual Behavior

• Machine 1: Can switch to Auto mode and plan routes without issues.
• Machine 2: Can plan routes but cannot switch to Auto mode.

Logs and Error Messages

Below are excerpts from zenoh_bridge_ros2dds.log on Machine 2:

�[2m2025-02-05T01:06:43.328667Z�[0m �[31mERROR�[0m                 rx-1 ThreadId(26) �[2mzenoh_codec::common::extension�[0m�[2m:�[0m Unknown OAM ext: ZExtUnknown { Id: 4, Mandatory: true, Encoding: "Unit" }
�[2m2025-02-05T01:06:47.305967Z�[0m �[31mERROR�[0m                 rx-0 ThreadId(25) �[2mzenoh_codec::common::extension�[0m�[2m:�[0m Unknown OAM ext: ZExtUnknown { Id: 3, Mandatory: true, Encoding: "Unit" }
�[2m2025-02-05T01:06:47.450742Z�[0m �[32m INFO�[0m tokio-runtime-worker ThreadId(12) �[2mzenoh_plugin_ros2dds::discovered_entities�[0m�[2m:�[0m Discovered ROS Node /transform_listener_impl_7d0c340e4ec0
�[2m2025-02-05T01:06:47.450813Z�[0m �[32m INFO�[0m tokio-runtime-worker ThreadId(12) �[2mzenoh_plugin_ros2dds�[0m�[2m:�[0m Node /transform_listener_impl_7d0c340e4ec0 declares Subscriber /clock: rosgraph_msgs/msg/Clock - Allowed
�[2m2025-02-05T01:06:47.618595Z�[0m �[32m INFO�[0m tokio-runtime-worker ThreadId(18) �[2mzenoh_plugin_ros2dds�[0m�[2m:�[0m Node /transform_listener_impl_7d0c340e4ec0 undeclares Subscriber /clock: rosgraph_msgs/msg/Clock - Allowed
�[2m2025-02-05T01:06:47.652328Z�[0m �[32m INFO�[0m tokio-runtime-worker ThreadId(18) �[2mzenoh_plugin_ros2dds::discovered_entities�[0m�[2m:�[0m Undiscovered ROS Node /transform_listener_impl_7d0c340e4ec0
�[2m2025-02-05T01:06:48.787616Z�[0m �[31mERROR�[0m                 rx-1 ThreadId(26) �[2mzenoh_codec::common::extension�[0m�[2m:�[0m Unknown OAM ext: ZExtUnknown { Id: 3, Mandatory: true, Encoding: "Unit" }

...

[2m2025-02-05T01:01:41.664775Z�[0m �[31mERROR�[0m                 rx-1 ThreadId(26) �[2mzenoh_codec::common::extension�[0m�[2m:�[0m Unknown OAM ext: ZExtUnknown { Id: 5, Mandatory: true, Encoding: "ZBuf", Value: ZBuf { slices: [[83, c9, 3e

It seems to be the problem of zenoh-bridge-ros2dds. Any insights or guidance would be highly appreciated!

[Justin-Xiang]

Or perhaps the issue is caused by using the same zenoh-bridge-ros2dds-conf.json5 file on both machines? I’m not entirely sure, but I’ll investigate further and try to resolve it.

[minseokim521]

Hello @Justin-Xiang !

I'm a college student studying autoware-based v2x. Having been through an issue similar to yours, I wrote down my issue right before your issue.

I'm running autoware on one machine, and I've experienced an error that the auto button sometimes activates or does not activate after specifying 2d goal pose, and if not, sometimes it's resolved by shutting down the terminal and running it again.

My problem is that in the process of running two autoware instruments, when you turn on the second autoware after the first autoware is set up, the two autowares are experiencing issues that collide with each other.

If there's anything I set up differently, in the run-autoware-docker.sh file, I added the --network host setting so that the docker container is set to communicate in host mode, is my problem related to this? I'm studying the docker container's network, ros2 dds, zenoh bridge, so I don't know the communication structure.

And can I put the ip address of host1 on run-autoware.sh ?

The os on my computer are uutntu 22.04 and ROS humble, and I'm using nvidia rtx 3080. And I've been cloning the main branch of GitHub. Could you also share your development environment and settings?

Lastly, I'm also looking into it to solve the problem, but if you solve the problem, it would be very helpful if you could share the method with me. If it's possible additionally, could you share the settings related to your container settings sh file and the sh file running autoware and zenoh bridge, or the ROS_DOMAIN_ID / ROS_LOCALHOST_ONLY settings for ROS communication? I wonder if there are any corrections you made in the github data.

I will also research more and study to solve the problem.

Thank you so much.

[Justin-Xiang]

@minseokim521 Hi, I came across your issue but didn’t look into it deeply since you’re using V2X and running everything on a single machine. I haven’t worked with V2X before, but I do have experience running two Autoware instances on a single machine.

From my experience, I didn’t make any code modifications—just followed the documentation—and everything worked as expected. You might want to try running it without modifications to see if that makes a difference.

Since our development environments seem quite similar, I don’t think hardware resources should be a constraint. Let me know if you have any other questions!

[Justin-Xiang]

Found this from zenoh issues:
eclipse-zenoh/zenoh-plugin-ros2dds#305 (comment)

it states: zenoh-bridge-ros2dds doesn't support two hosts having different namespaces communicate with each other.

@evshary Could you confirm whether this means the setup I described in this issue is currently not achievable? Additionally, would it be possible to run two zenoh_carla_bridge instances on separate machines while sharing the same Carla simulator?

[evshary]

Hi @Justin-Xiang

I tried your setup before and it should be available. However, the error message is quite weird and worth investigating.
I have some questions:

• Are you available to run all of them on the same machine? That is, Carla, bridges, two Autoware instances.
• While the second one is unable to switch to Auto, can it get the camera image from Carla bridge? If yes, the communication should be okay. I think there is something wrong with a certain topic.

Anyway, I don't think your issue is related to eclipse-zenoh/zenoh-plugin-ros2dds#305 (comment), since we don't need two Autoware to talk to each other directly.
It should be something else.

[Justin-Xiang]

Hi @evshary Thanks for your response.

1. Yes, I can run it on the same machine, but it’s not stable. I’m not sure if this is due to hardware constraints (I’m using an RTX 3090, 32GB RAM, and an i7-12700KF) or a connection issue. Out of every five attempts, only one successfully sets both vehicles to Auto and allows them to move. Sometimes, one vehicle fails to switch to Auto and sometimes two, and in the worst cases, both fail to initialize. I haven’t made any modifications—I’m strictly following the documentation.
2. Yes, the system receives images from the Carla bridge and successfully plans routes (I can see the green routes in RViz). The issue is that the vehicles can’t be set to Auto. I also tried swapping Machine 1 and Machine 2, changing which one runs what, but the problem persists. The machine running zenoh-bridge works fine, while the other one still can’t switch to Auto.

Let me know if you need more details-I’d be happy to provide additional information.

[evshary]

Thank you! It didn't happen on my machine, but I indeed heard someone else face the issue before. I will take a look. Feel free to provide any other information if you find something weird.