COMM and BATT leds turn off when switched to discovery server

[rohitthampy]

Robot Model

Turtlebot4 Standard

ROS distro

Humble

Networking Configuration

Discovery Server

OS

Ubuntu 22.04

Built from source or installed?

Installed

Package version

turtlebot4-node

Version: 1.0.5-1jammy.20240731.002306

turtlebot4-base

Version: 1.0.3-1jammy.20240731.003557

turtlebot4-robot

Version: 1.0.3-1jammy.20241015.140422

turtlebot4-bringup

Version: 1.0.3-1jammy.20241015.135733

Type of issue

Other

Expected behaviour

To successfully setup discovery server configuration based on the instructions given here.

All the leds must be on and I should be able to get a complete list of topics and the data within from both the rpi4 and the create3.

Actual behaviour

COMM and BATT leds are not on after switching to discovery server configuration as shown in the image below. I'm still able to view topics such as /battery_statuson the rpi4 but unable to get any data from the topic.

Error messages

### Error messages from create3

Dec  5 12:50:50 iRobot-D583C8E6A82C448890E1780CE5A94DF3 daemon.err network[1463]: [ERROR] (src/aos_kvs_helper.c:51 errno:No such file or directory) Failed to stat file product.regdom
Dec  5 12:50:52 iRobot-D583C8E6A82C448890E1780CE5A94DF3 daemon.err network[1463]: [ERROR] (src/main.c:366 errno:No such file or directory) failed to set regdomain
Dec  5 12:50:52 iRobot-D583C8E6A82C448890E1780CE5A94DF3 daemon.err network[1463]: [ERROR] (src/main.c:372 errno:No such file or directory) Could not find product.regdom key
Dec  5 12:50:59 iRobot-D583C8E6A82C448890E1780CE5A94DF3 daemon.err network[1463]: [ERROR] (src/aos_kvs_helper.c:51 errno:No such file or directory) Failed to stat file network.ssid
Dec  5 12:50:59 iRobot-D583C8E6A82C448890E1780CE5A94DF3 daemon.err network[1463]: [ERROR] (src/aos_misc.c:754 errno:No such file or directory) Failed to stat file /opt/irobot/persistent/opt/irobot/data/bbk/user_wpa_supplicant.conf
Dec  5 12:50:59 iRobot-D583C8E6A82C448890E1780CE5A94DF3 daemon.err network[1463]: [ERROR] (src/aos_misc.c:801 errno:No such file or directory) Failed to open file: /opt/irobot/persistent/opt/irobot/data/bbk/user_wpa_supplicant.conf
Dec  5 12:51:00 iRobot-D583C8E6A82C448890E1780CE5A94DF3 daemon.err network[1463]: [ERROR] (src/aos_kvs_helper.c:51 errno:No such file or directory) Failed to stat file robot.disc_reconnect_sleep_sec
Dec  5 12:51:00 iRobot-D583C8E6A82C448890E1780CE5A94DF3 daemon.err network[1463]: [ERROR] (src/kvs_json.c:21 errno:No such file or directory) Failed to retrieve key "robot.disc_reconnect_sleep_sec"
Dec  5 12:51:00 iRobot-D583C8E6A82C448890E1780CE5A94DF3 daemon.err network[1463]: [ERROR] (src/aos_kvs_helper.c:51 errno:No such file or directory) Failed to stat file network.watchdog_interval_min
Dec  5 12:51:00 iRobot-D583C8E6A82C448890E1780CE5A94DF3 daemon.err network[1463]: [ERROR] (src/kvs_json.c:21 errno:No such file or directory) Failed to retrieve key "network.watchdog_interval_min"
Dec  5 12:51:00 iRobot-D583C8E6A82C448890E1780CE5A94DF3 daemon.err network[1463]: [ERROR] (src/aos_misc.c:86 errno:No such file or directory)  MODULE : network_manager CMD (sudo network_dhcp_arp_control.sh disable) failed with status = 1, ret = 256
Dec  5 12:51:05 iRobot-D583C8E6A82C448890E1780CE5A94DF3 daemon.err network[1463]: [ERROR] (src/mqtt_client.c:785 errno:Resource temporarily unavailable) firr_transfer failed
Dec  5 12:51:05 iRobot-D583C8E6A82C448890E1780CE5A94DF3 daemon.err network[1463]: [ERROR] (src/main.c:203 errno:Resource temporarily unavailable) Error publishing sta:0 FIRR request.
Dec  5 12:51:05 iRobot-D583C8E6A82C448890E1780CE5A94DF3 daemon.err network[1463]: [ERROR] (src/aos_kvs_helper.c:51 errno:No such file or directory) Failed to stat file network.addr_type
Dec  5 12:51:05 iRobot-D583C8E6A82C448890E1780CE5A94DF3 daemon.err network[1463]: [ERROR] (src/kvs_json.c:21 errno:No such file or directory) Failed to retrieve key "network.addr_type"
Dec  5 12:51:05 iRobot-D583C8E6A82C448890E1780CE5A94DF3 daemon.err network[1463]: [ERROR] (src/aos_kvs_helper.c:51 errno:No such file or directory) Failed to stat file network.ssid
Dec  5 12:51:05 iRobot-D583C8E6A82C448890E1780CE5A94DF3 daemon.err network[1463]: [ERROR] (src/network_utils.c:262 errno:No such file or directory) cannot load ssid
Dec  5 12:51:30 iRobot-D583C8E6A82C448890E1780CE5A94DF3 daemon.err network[1463]: [ERROR] (src/mqtt_client.c:785) firr_transfer failed
Dec  5 12:51:30 iRobot-D583C8E6A82C448890E1780CE5A94DF3 daemon.err network[1463]: [ERROR] (src/discovery.c:407) falied to get cap from tsprime

To Reproduce

1. run turtlebot-setup on the rpi4
2. Enter discovery server menu via ROS setup and set enabled to true
3. Apply settings and leave setup
4. Call turtlebot4-source and restart the daemon using turtlebot4-daemon-restart

Other notes

When I disable the discovery server from the turtllebot4-setup all the led lights are on. However, I would like to setup the discovery server configuration as the wifi in my university only supports 5GHz.

Please let me know if any more details are required to resolve this.

[smatarCPR]

Hello @rohitthampy ,
Thank you for reaching out on the Turtlebot 4 github page. I am very sorry to hear that you are having issues with one of our products.

Regarding this behavior, my recommendation would be to add a namespace to the Discovery server environment. This should cause the COMM and BATT LEDs to come back on after applying the new namespace environment.

Let me know how this recommendation works for you. Looking forward to hearing from you soon.
Best Regards,
Saif

[rohitthampy]

Hi Saif,

Thanks for getting back to me.
I shall try it and let you know if it works.

[rohitthampy]

Hi @smatarCPR,

I have followed your advice and added a namespace (/tbot4_1) through the bash setup within the ROS setup. I can confirm that this has resolved the issue. I can see all the leds on the turtlebot 4 are on and I can also get a list of all the topics in the turtlebot through ssh.

[rohitthampy]

However, now I am facing a slightly different issue where I am unable to get the list of topics on my PC. As in I have followed the discovery server setup on the user pc and when I try ros2 topic list, I only get /parameter_events and /rosout as the output instead of the entire list of topics from the turtlebot. Should I create a new issue for this?

[smatarCPR]

Hello @rohitthampy
Glad to hear you have been able to make progress with regards to the Turtlebot 4 and apologies for the late reply on this. In terms of the new issue:

• If you run the ros 2 topic list command twice does the list update or does it remain the same?
• If you source your bashrc file source ~/.bashrc then restart the ROS Daemon ros2 daemon stop; ros2 daemon start does the topic list change?
• Are you able to ping the Turtlebot 4 using its IP address from your remote computer?
• Check that the directory /etc/turtlebot4_discovery/ exists and there is a setup.bash with the correct Discovery server information present. If it does not exist or the information is incorrect, recreate the setup.bash file using wget -qO - https://raw.githubusercontent.com/turtlebot/turtlebot4_setup/humble/turtlebot4_discovery/configure_discovery.sh | bash <(cat) </dev/tty

Looking forward to hearing from you soon.
Best Regards,
Saif

[rohitthampy]

Hi @smatarCPR ,

Happy New Year! And thanks for getting back to me.

I will be back in my robotics lab on Thursday and I shall try out your suggestions and get back to you.

[rohitthampy]

Hi @smatarCPR ,

Please find my responses to your points below.

• If you run the ros 2 topic list command twice does the list update or does it remain the same?

  On the rpi the list updates but on my machine it only shows /parameter_events and /rosout even after calling ros2 topic list twice.

• If you source your bashrc file source ~/.bashrc then restart the ROS Daemon ros2 daemon stop; ros2 daemon start does the topic list change?

  It still only shows the two topics: /parameter_event and /rosout

• Are you able to ping the Turtlebot 4 using its IP address from your remote computer?

  Yes, I am able to successfully ping the turtlebot4 from my remote computer

• Check that the directory /etc/turtlebot4_discovery/ exists and there is a setup.bash with the correct Discovery server information present. If it does not exist or the information is incorrect, recreate the setup.bash file using wget -qO - https://raw.githubusercontent.com/turtlebot/turtlebot4_setup/humble/turtlebot4_discovery/configure_discovery.sh | bash <(cat) </dev/tty

  I can confirm that the directory /etc/turtlebot4_discovery/ exists.
  Given below are the contents within the setup.bash
source /opt/ros/humble/setup.bash
export RMW_IMPLEMENTATION=rmw_fastrtps_cpp
[ -t 0 ] && export ROS_SUPER_CLIENT=True || export ROS_SUPER_CLIENT=False
export ROS_DOMAIN_ID=0
export ROS_DISCOVERY_SERVER="10.19.208.47:11811

Please let me know if there are any more details I can provide or anything else I can try to resolve this issue.

[smatarCPR]

Hello @rohitthampy ,
Thank you for providing these details. From this I can confirm that: the script to setup is indeed working as intended in terms of generating the necessary setup.bash file. I have the following test/recommendations I would like for your to perform:

• After calling source ~/.bashrc run the command printenv | grep ROS to print out the set of ROS environment variables. Ensure that environment variables are indeed being set as described in the setup.bash file found in /etc/turtlebot4_discovery/
• If they are not being set try manually sourcing the file using source /etc/turtlebot4_discovery/setup.bash then rerunning the printenv to see if the environment variables have been updated.

Let me know the results of these checks. Looking forward to hearing from you soon.
Best Regards,
Saif

[rohitthampy]

Hi @smatarCPR ,

Given below is the output after running printenv | grep ROS.

ROS_VERSION=2
ROS_PYTHON_VERSION=3
ROS_SUPER_CLIENT=True
ROS_DOMAIN_ID=0
ROS_LOCALHOST_ONLY=0
ROS_DISTRO=humble
ROS_DISCOVERY_SERVER=10.19.208.47:11811;

I wanted to point out something that might be of interest.
I was trying to run a simple publisher today that publishes a message "Hello World" using a topic called "/publisher_topic". However, when I ran ros2 topic list, only the topics /parameter_events and ``/rosoutwere returned. Also when I ranros2 node list`, I didn't get any return. Perhaps something wrong in my machine?

[smatarCPR]

Hello @rohitthampy ,
Thank you for providing these details. From the details shared it appears that on your remote computer is operating as expected. Let's focus our efforts on the Turtlebot side of things:

• Would it be possible to provide screenshots of the Turtlebot 4 settings from the turtlebot4-setup tool?
• Double check the server ID or port being used for the discovery settings.
• A quick note about the results of your last test, when setting up a discovery server it's settings will be applied to all ROS systems in on your computer. Did you run this test with the Turtlebot 4 off? The results may simply be that computer unable to communicate with the established discovery server on the Turtlebot.

Let me know the results of these checks. Looking forward to hearing from you soon.
Best Regards,
Saif

[rohitthampy]

Hi @smatarCPR ,

Thank you for getting back to me. Please find the screenshots of the Turtlebot 4 settings from turtlebot4-setup below.

For the server ID and port, I'm using the default values which are 0 and 11811 respectively.

With regards to my previous test, you are absolutely right, I was unaware that the settings were applied to all ROS systems. However, even after switching on the Turtlebot4, I am still unable to get any of the topics from it on my PC. I've also pasted the environment variables from my turtlebot4 and my PC below.

Turtlebot4 environment variables:
ROS_VERSION=2
ROS_PYTHON_VERSION=3
ROS_SUPER_CLIENT=True
ROS_DOMAIN_ID=0
ROS_LOCALHOST_ONLY=0
ROS_DISTRO=humble
ROS_DISCOVERY_SERVER=127.0.0.1:11811;

PC's environment variables:
ROS_VERSION=2
ROS_PYTHON_VERSION=3
ROS_SUPER_CLIENT=True
ROS_DOMAIN_ID=0
ROS_LOCALHOST_ONLY=0
ROS_DISTRO=humble
ROS_DISCOVERY_SERVER=192.168.161.199:11811;

Please let me know if there is anything else I can try or if you need any more details to resolve this issue.