CPU usage

[lucasw]

I built this in release and started several instances of the turtlebot node to see if they could communicate with each other over localhost and it worked, the keyboard input into one node while that terminal is in focus is received by the read back subscribers on all the other instances- I'll try this out with multiple computers on the same network later. (It's especially nice because I don't have any C++/python ros2 packages and dependencies installed on this system, having a lightweight rust-only ability to make ros nodes that communicate with regular ros2 is fantastic, though now I want to make a rust ros2 topic list and some other basic tools...)

One thing I notice is that each node takes ~5% cpu according to top, and this cpu usage goes up a little with each instance I start, and doubles while I'm pressing keys repeatedly. They aren't exact matches but the ros1 turtlebot_teleop_joy and turtlebot_teleop_key python and C++ nodes in ros1 take ~1.0% cpu to accomplish transmit and receive of the similar messages (I had to reduce the joy update rate rosrun joy joy_node _coalesce_interval:=0.03, the default value is 0.001 and moving the gamepad a lot does start to eat cpu, and neither of those nodes is doing the readback like this rustdds one is).

I know this keyboard input node wasn't made with speed tests in mind, but what's the general feeling for where the cpu is being taken up, could the use of mio and polling be tuned up (I'm not familiar with it), or being compliant with dds is inherently going to consume so much more cpu- and the better route for me if I want to save cpu are IPC or any of the methods I've seen mentioned elsewhere that avoid some or all of dds while still preserving some interoperability with other ros2 nodes?

[jhelovuo]

It is not immediately apparent where the CPU time is consumed, but I have some questions.

Did you try any code profiling?
Did you measure how many RTPS network packets are processed per second?
Did you test with a debug or release build?

[lucasw]

It was a release build, but just now pulled the latest rustdds, added

[profile.release]
opt-level = 3

(but that should have been the default already)

and then also ran a rustup update (went from cargo 1.56 to 1.57). Building with release now and I see lower cpu usage, now each node is staying < 2% cpu with 4 running, If I bring up 6 or 7 more the cpu usage per node does go up to 3 or 4% (more for the one doing the transmitting).

So right now performance looks reasonable but it's not much of a stress test- the test that matters to me is sending 1-2+ megabyte images at 15 or 30 Hz over a few channels, and in general having 20+ nodes sending mostly smaller messages between 5 and 100 Hz in every direction - that's a normal ros1 system to me (maybe with rust and/or dds I ought to go more monolithic, but I really have gotten used to the convenience of spinning up many small single task nodes to prototype features, and consolidate later as an optimization if/when needed). I'll likely set up a repo for that at some point.

I haven't done profiling in rust before (haven't done a whole lot in rust in general), it would be interesting to learn.

Same goes for measuring rtps network monitoring (do I need to install rtps network tools to get started with that, or wireshark or similar generic tools work well?)- I'm less interested in that but willing to try it.

[jhelovuo]

It is good to hear that you got some performance improvements.

If you need to send megabyte-sized images as single objects over DDS, then RustDDS is not yet capable of doing that. Sending over 64 kbyte objects is not doable with UDP fragmentation alone, but requires the DDS implementation to support data fragmentation also, namely handling submessages DATAFRAG, ACKFRAG, and HEARTBEATFRAG.

Issue 20 tracks that feature. If you really need that, you are of course free to make a pull request implementing the missing bits.

Wireshark is useful for diagnosing all sorts of network problems. RTPS-specific tools could give more insight into what is wrong with RTPS.