Neural Control Mode #24366

SindreMHegre · 2025-02-19T12:23:11Z

Solved Problem

Using neural networks for various tasks on UAVs in research is becoming more and more commonplace. My aim is to provide a module that can be used to replace the MC controller with NN to lower the barrier for using RL and NN on UAVs in general, and give researchers and hobbyist a place to start for NN use.

Solution

Add TensorFlow Lite Micro as an inference lib into PX4
Make custom board configs with neural controllers
Make a module for NN control

Changelog Entry

For release notes:

Feature/Bugfix XYZ
New parameter: XYZ_Z
Documentation: Need to clarify page ... / done, read docs.px4.io/...

Alternatives

We could also look into using Eigen or executorch

Test coverage

Built and tested SITL with the x500
https://review.px4.io/plot_app?log=52502d02-009e-4d1f-baac-a49dd51cfdef (check neural_control topic)
Built and tested HITL with pixracerpro and jMAVSim Hitl, note that the sim itself seemed a bit unstable and the network is not optimized for this model.
https://review.px4.io/plot_app?log=43d2ba6c-3d05-4d7f-a31b-490a57b3e647
Plan on testing on the x500 v2 kit with the 6C

Context

Docs:
PX4/PX4-user_guide#3617

Help:

I'm not very experienced with CMakeLists.txt, so any pointers in how to improve the code quality there is much appreciated.
Also for the TFLM submodule, should I make a PX4 branch for it? With the dependencies already downloaded and the standard c++ header files extracted. Should it be on a fork or try to get it upstream?
Would also like it to have documentation on the main docs, where do I start writing that?
Should it be merged into main or as a separate branch upstream?

Other feature requests:

Put in different controller parameters with possibilities to replace different parts: Full e2e, only control, only allocation, Allocation and control as two separate networks.
Script to give thrust and delay parameters from a flight log

Info:
The networks I'm flying with right now is not mine and is being used for research, so they will be open sourced at some point, but can't give them away at the moment

Inference times on the pixracer pro:
control net: 70 microseconds
allocation net: 13 microseconds
Full NN controller: 141 microseconds

DronecodeBot · 2025-02-19T12:28:29Z

This pull request has been mentioned on Discussion Forum for PX4, Pixhawk, QGroundControl, MAVSDK, MAVLink. There might be relevant details there:

https://discuss.px4.io/t/px4-sync-q-a-feb-19-2025/43827/2

DronecodeBot · 2025-02-19T14:50:09Z

This pull request has been mentioned on Discussion Forum for PX4, Pixhawk, QGroundControl, MAVSDK, MAVLink. There might be relevant details there:

https://discuss.px4.io/t/px4-sync-q-a-feb-19-2025/43827/1

- Add TensorFlow Lite Micro(TFLM) as a library in px4 - Make a module that uses neural network inference for control, which uses TFLM for inference - Make board config files for PX4 with neural module

.gitmodules

ROMFS/px4fmu_common/init.d/rc.mc_apps

msg/versioned/VehicleControlMode.msg

msg/versioned/VehicleStatus.msg

Jaeyoung-Lim · 2025-02-22T05:40:26Z

platforms/nuttx/cmake/Toolchain-arm-none-eabi.cmake

+# 	if(NOT ${TOOL})
+# 		message(FATAL_ERROR "could not find ${tool}")
+# 	endif()
+# endforeach()


Not needed?

Jaeyoung-Lim · 2025-02-22T05:40:49Z

platforms/nuttx/cmake/Toolchain-arm-none-eabi.cmake

-set(CMAKE_CXX_COMPILER ${TOOLCHAIN_PREFIX}-g++)
-set(CMAKE_CXX_COMPILER_TARGET ${triple})
+# Define the path to the new toolchain
+set(NEW_TOOLCHAIN_PATH "/home/sindre/Dropbox/Studier/2025_Var/dev/PX4-Autopilot-public/src/lib/tflm/tflite_micro/tensorflow/lite/micro/tools/make/downloads/gcc_embedded/bin")


This has your local user name in the path

When I change it to use a variable like ${PX4_SOURCE_DIR}, I get errors when building uxrce_dds_client. Not sure how to fix unless the standard toolchain downloaded when calling
bash ./Tools/setup/ubuntu.sh
is upgraded.

- Switch ssh link to https link in submodule - Remove mc_nn_control from startup

…4-Autopilot-public into neural_control_with_tflm

Jaeyoung-Lim · 2025-02-24T16:18:45Z

src/modules/mc_nn_control/.gitignore

@@ -0,0 +1,2 @@
+control_net.cpp
+allocation_net.cpp


There should be no gitignore here

It's there temporarily as I don't own the network I'm currently flying it, so I can't publish it in a public repo yet. But I'm in the process of fixing this

Jaeyoung-Lim · 2025-02-24T16:19:40Z

src/modules/mc_nn_control/mc_nn_control.hpp

@@ -0,0 +1,138 @@
+/****************************************************************************
+ *
+ *   Copyright (c) 2013-2025 PX4 Development Team. All rights reserved.


Suggested change

* Copyright (c) 2013-2025 PX4 Development Team. All rights reserved.

* Copyright (c) 2025 PX4 Development Team. All rights reserved.

Pedro-Roque · 2025-02-25T14:24:33Z

.gitmodules

@@ -89,3 +89,7 @@
 [submodule "src/drivers/uavcan/libdronecan/libuavcan/dsdl_compiler/pydronecan"]
 	path = src/drivers/uavcan/libdronecan/libuavcan/dsdl_compiler/pydronecan
 	url = https://github.com/dronecan/pydronecan
+[submodule "src/lib/tflm/tflite_micro"]


It would be nice if this wasn't pulled for all builds.

how do I fix this?

@SindreMHegre Why do you even need the submodule?

@Jaeyoung-Lim I have no idea, it just seemed like the standard way to include a big library. What other options do I have? Just adding the needed parts of the lib directly in the code?

@SindreMHegre Can we install it on the system? submodules only make sense if you need to compile the library with the source

This is the part of this PR where I'm the least experienced. What do you mean installing it on the system? I'm unsure if the library needs to compile on it's own first

DronecodeBot · 2025-03-03T14:30:12Z

This pull request has been mentioned on Discussion Forum for PX4, Pixhawk, QGroundControl, MAVSDK, MAVLink. There might be relevant details there:

https://discuss.px4.io/t/how-to-integrate-eigen-library-with-px4-firmware/44065/2

DronecodeBot · 2025-03-03T14:42:41Z

This pull request has been mentioned on Discussion Forum for PX4, Pixhawk, QGroundControl, MAVSDK, MAVLink. There might be relevant details there:

https://discuss.px4.io/t/adding-an-external-flight-mode-from-the-fcu/44137/1

SindreMHegre changed the title ~~Add tflm to px4 with module~~ Neural Control Mode Feb 19, 2025

SindreMHegre force-pushed the neural_control_with_tflm branch from f5cefaa to 33189a3 Compare February 19, 2025 17:26

Jaeyoung-Lim marked this pull request as draft February 19, 2025 18:18

SindreMHegre force-pushed the neural_control_with_tflm branch 2 times, most recently from 600755d to 2eb97f2 Compare February 20, 2025 14:40

Add tflm to px4 with module

ff4254a

- Add TensorFlow Lite Micro(TFLM) as a library in px4 - Make a module that uses neural network inference for control, which uses TFLM for inference - Make board config files for PX4 with neural module

SindreMHegre force-pushed the neural_control_with_tflm branch from 2eb97f2 to ff4254a Compare February 20, 2025 14:50

SindreMHegre added 2 commits February 20, 2025 16:05

Added neural flight mode

8fe4e32

Add posibility to read of inference times

51ae6ae

Jaeyoung-Lim requested a review from dagar February 21, 2025 10:06

SindreMHegre added 3 commits February 21, 2025 15:21

Add tflm to px4 with module

a09e1a5

- Add TensorFlow Lite Micro(TFLM) as a library in px4 - Make a module that uses neural network inference for control, which uses TFLM for inference - Make board config files for PX4 with neural module

Added neural flight mode

a817473

Add posibility to read of inference times

4ceaaa9

mrpollo force-pushed the neural_control_with_tflm branch from 51ae6ae to 4ceaaa9 Compare February 21, 2025 23:21

Jaeyoung-Lim reviewed Feb 22, 2025

View reviewed changes

.gitmodules Outdated Show resolved Hide resolved

Jaeyoung-Lim requested changes Feb 22, 2025

View reviewed changes

SindreMHegre added 3 commits February 24, 2025 14:16

Fix comments from review:

564b79a

- Switch ssh link to https link in submodule - Remove mc_nn_control from startup

Merge branch 'neural_control_with_tflm' of github.com:SindreMHegre/PX…

3c54162

…4-Autopilot-public into neural_control_with_tflm

Remove auto start

202b292

Pedro-Roque self-requested a review February 24, 2025 15:54

Jaeyoung-Lim reviewed Feb 24, 2025

View reviewed changes

Add logging from neural control module

222f2c7

Pedro-Roque reviewed Feb 25, 2025

View reviewed changes

Fix automatic startup to only be when module is included

d0369ac

SindreMHegre force-pushed the neural_control_with_tflm branch 2 times, most recently from fe9ca99 to aa83e4e Compare March 3, 2025 13:04

SindreMHegre mentioned this pull request Mar 4, 2025

[WIP] Create documentation for NN support with TFLM PX4/PX4-user_guide#3617

Draft

1 task

SindreMHegre force-pushed the neural_control_with_tflm branch from aa83e4e to b08dba2 Compare March 7, 2025 08:45

Switch to flight mode registration

2243d9b

SindreMHegre force-pushed the neural_control_with_tflm branch from b08dba2 to 2243d9b Compare March 7, 2025 08:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Neural Control Mode #24366

Neural Control Mode #24366

SindreMHegre commented Feb 19, 2025 •

edited

Loading

DronecodeBot commented Feb 19, 2025

DronecodeBot commented Feb 19, 2025

Jaeyoung-Lim Feb 22, 2025

Jaeyoung-Lim Feb 22, 2025

SindreMHegre Feb 24, 2025

Jaeyoung-Lim Feb 24, 2025 •

edited

Loading

SindreMHegre Feb 25, 2025

Jaeyoung-Lim Feb 24, 2025

SindreMHegre Feb 28, 2025

Pedro-Roque Feb 25, 2025

SindreMHegre Feb 26, 2025

Jaeyoung-Lim Feb 26, 2025

SindreMHegre Feb 26, 2025

Jaeyoung-Lim Feb 26, 2025

SindreMHegre Feb 26, 2025

DronecodeBot commented Mar 3, 2025

DronecodeBot commented Mar 3, 2025

	* Copyright (c) 2013-2025 PX4 Development Team. All rights reserved.
	* Copyright (c) 2025 PX4 Development Team. All rights reserved.

		@@ -0,0 +1,2 @@
		control_net.cpp
		allocation_net.cpp

Neural Control Mode #24366

Are you sure you want to change the base?

Neural Control Mode #24366

Conversation

SindreMHegre commented Feb 19, 2025 • edited Loading

Solved Problem

Solution

Changelog Entry

Alternatives

Test coverage

Context

DronecodeBot commented Feb 19, 2025

DronecodeBot commented Feb 19, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Jaeyoung-Lim Feb 24, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DronecodeBot commented Mar 3, 2025

DronecodeBot commented Mar 3, 2025

SindreMHegre commented Feb 19, 2025 •

edited

Loading

Jaeyoung-Lim Feb 24, 2025 •

edited

Loading