Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REP-2011] Evolving Message Types #358

Open
wants to merge 22 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
df394d9
WIP
wjwwood Feb 15, 2022
c355ca6
still working
wjwwood Mar 9, 2022
05ac47e
added more about the enforcing of the type versions
wjwwood Apr 13, 2022
6dcb876
Increase readability, fix typos, and reduce ambiguities
methylDragon May 13, 2022
42cf0de
Add clarification for field semantics
methylDragon May 16, 2022
d3449f0
Implement more fixups
methylDragon Jun 1, 2022
98081d9
Methyl dragon/type description distribution (#2)
wjwwood Jun 10, 2022
d7628e7
rename from REP-2010 to REP-2011
wjwwood Jun 10, 2022
22f6cb7
Add runtime type introspection section (#3)
methylDragon Jul 8, 2022
8dccc0f
Add nested type description section (#4)
methylDragon Jul 23, 2022
b6a7086
Add alternatives and on_inconsistent_topic section (#5)
youliangtan Jul 23, 2022
a030033
fix rst errors, revise a few sections
wjwwood Jul 26, 2022
04306e4
touch ups, rename to Run-Time Interface Reflection, reorganize
wjwwood Jul 27, 2022
d5e27d5
diagram typo
wjwwood Jul 27, 2022
fbc8810
fixup
wjwwood Aug 1, 2022
01edda8
change title to a better wording
wjwwood Aug 2, 2022
9ea2a20
fixups, reordering, tools section
wjwwood Aug 22, 2022
24492e5
add feature progress section
wjwwood Sep 6, 2022
bd5cdbc
cleanup feature progress list
wjwwood Sep 9, 2022
ae78cf9
Update Type Version Hash to use SHA-256, and details updates based on…
emersonknapp Jan 31, 2023
ba2f96c
Update RIHS notes based on latest discussion (#8)
emersonknapp Mar 29, 2023
a4e1ef5
Remove Type Description/Hash content (moving to REP-2016) (#9)
emersonknapp Sep 12, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Implement more fixups
Signed-off-by: methylDragon <[email protected]>

Co-authored-by: William Woodall <[email protected]>
methylDragon and wjwwood committed Jun 2, 2022
commit d3449f0a9807e18e7fcacb0c136a123fbc6eb110
25 changes: 11 additions & 14 deletions rep-2010.rst
Original file line number Diff line number Diff line change
@@ -16,11 +16,8 @@ This REP proposes patterns and approaches for evolving message, service, and act
The proposed patterns use the existing default serialization technology, i.e. CDR, in combination with new tooling.
However, technologies provided by different, perhaps future, serialization technologies can always be used in addition.

Specifically, this REP proposes that the following is a good way to achieve backwards compatibility in messages over long periods of time and in a variety of scenarios, e.g. over the wire, converting bag files, or in specialized tools:

- Interpreting older versions of the message at runtime;
- Using its type information;
- And converting it to the current version of the message using user defined transfer functions
Specifically, this REP proposes that interpreting older versions of the message at runtime can be achieved by using its type information and then converting it to the current version of the message using user-defined transfer functions.
This approach is a good way to achieve backwards compatibility in messages over long periods of time and in a variety of scenarios, e.g. over the wire, converting bag files, or in specialized tools.

This approach does not rely on specific features of the serialization technology and instead relies on the ability to communicate the type information of the messages on the wire, beyond their name and version, and use that to dynamically interpret it at runtime.

@@ -104,15 +101,15 @@ So, if you have rosbag2 publishing ``Temperature`` messages and a program consum
The "transfer function" can be user-defined, or for simple changes (like changing the field type to a compatible type) it can be done automatically.
We already do something like this for the ROS 1 to ROS 2 bridge in order to handle changes between message types in ROS 1 and ROS 2, and something like this was also done for rosbags in ROS 1 with the "bag migration rules" feature.

Furthermore, the "transfer function" approach also allows for runtime transformation of messages that only change in field semantics (e.g., a change of ``float distance`` to mean centimeters instead of meters, but with no change in field name or type.) Although in those cases, users will likely have to define the transfer function themselves.
Furthermore, the "transfer function" approach also allows for conversions between messages which are only semantically different (e.g., a change in semantics for a field ``float distance`` to mean centimeters instead of meters, but with no change in field name or type).
However, the user would have to write the transfer function for these changes, because it could not be made automatically as they are not captured in a structured way.

.. TODO:: cite the above

This approach requires a few features, like the ability to have a single application read old and new versions of a message at the same time, and it requires more infrastructure and tooling to make it work, but it has the advantage of keeping both the publishing and subscribing code simple, i.e agnostic to the fact that there are other versions of the message, and it keeps the message type from being cluttered with vestigial fields.

Either way, a problem can usually be solved by changing a message in some of, if not all, of the of the above mentioned ways, and is often influenced by what the underlying technology allows for or encourages.
ROS 2 has special considerations on this topic because it can support different serialization technologies, though CDR is the default and most common right now, and those technologies have different capabilities.

It is neither desirable to depend on features of a specific technology, therefore tying ROS 2 to a specific technology, nor is it desirable suggest patterns that rely on features that only some serialization technologies provide, again tying ROS 2 to some specific technologies through their features.

We will require some features from the middleware and serialization technology, however, to handle evolving interfaces, but we should try to choose approaches which give ROS 2 the broadest support across middleware implementations, ideally while not limiting users from using specific features of the underlying technology when that suites them.
@@ -169,29 +166,29 @@ In order to calculate the type version hashes so that they are stable and are no

The data structure includes:

- A list of field names and types, but not default values
- The serialization format
- The serialization format version
- An optional user-defined interface version, or 0 if not provided
- a list of field names and types, but not default values
- the serialization format
- the serialization format version
- an optional user-defined interface version, or 0 if not provided

The resulting data structure is hashed using a standard SHA-1 method, resulting in a standard 160-bit (20-byte) hash value which is also generally known as a "message digest".
This hash is combined with a "type version hash standard version", the first of which will be ``IDLHASH-1``, with an ``@`` symbol, resulting in a complete type version hash like ``IDLHASH-1@<160-bit SHA-1 of data structure>``.
This allows the tooling to know if a hash mismatch is due to a change in this standard (what is being hashed) or due to a difference in the interface types themselves.

Notably, the user-defined interface version being included in the hash allows for messages that only change in field semantics (i.e., without changing field names or types) to be picked up as discrepancies when they would not before, allowing users to be prompted to write "transfer functions" to resolve them.
The optional user-defined interface version included in the hash allows the user to indicate that there was a change in message semantics (i.e. no change in the field names or types), causing the versions to mismatch and allowing the user to write a "transfer functions" to resolve the semantic differences.

.. TODO:: is the list of field names and types sufficient? how to capture things like .idl annotations, etc... I'm thinking of serialization format specific entries can be added to this data structure, but need to sketch it out a bit more

Enforcing Type Version
~~~~~~~~~~~~~~~~~~~~~~

If the type version hash is available, it can be used as an additional constraint to determine if two endpoints (publishers and subscriptions) on a topic should communicate.
The type version hash can be used as an additional constraint to determine if two endpoints (publishers and subscriptions) on a topic should communicate.

When creating a publisher or subscription, the caller normally provides: a topic name, QoS settings, and a topic type.
The topic type is represented as a string and is automatically deduced based on the type given to the create function, e.g. as a template parameter in C++ or the message type as an argument in Python.
For example, creating a publisher for ``std_msgs::msg::String`` in C++, may result in a topic type like ``std_msgs/msg/String``.
All of these items are used by the middleware to determine if two endpoints should communicate or not, and this REP proposes that the type version be added to this list of provided information.
From the user's perspective, nothing needs to change, as the type version can be extracted based on the topic type given either at the ``rcl`` layer or in the ``rmw`` implementation itself.
Nothing needs to change from the user's perspective, as the type version can be extracted automatically based on the topic type given, either at the ``rcl`` layer or in the ``rmw`` implementation itself.
However, the type version would become something that the ``rmw`` implementation is provided and aware of in the course of creating a publisher or subscription, and therefore the job of using that information to enforce type compatibility would be left to the middleware, rather than implementing it as logic in ``rcl`` or other packages above the ``rmw`` API.

The method for implementing the detection and enforcement of type version mismatches is left up to the middleware, as some middlewares will have tools to make this efficient and others will implement something like what would be possible in the ``rcl`` and above layers.