RFC: partial reorganization of the CPS format. #32
Conversation
Header: - changed the magic field from 64 to 32 bit - extented the version number from 16 to 32 bit Channel: - changed transmit power from dBm to mW, extending the field from 8 to 32 bit - dropped the scan list index field - dropped the group list index field - changed the longitude field from 8 to 16 bit to fix #30
Contacts: - extended info field to 15 bytes, padding the structure to 48 byte and to leave space for future extensions Channels: - extended info to 9 bytes, padding the structure to 96 byte and to leave space for future extensions - dropped the 20kHz channel bandwidth M17 channel info block: - dropped split TX and RX CAN and moved to a unique CAN field - merged CAN and channel mode into the same 8 bit field - merged encryption type and GNSS transmission in the same 8 bit field
n1zzo
force-pushed
the
master
branch
3 times, most recently
from
53eaf93 to
2b94b43
Compare
|
HTML version available here: https://github.com/OpenRTX/openrtx.github.io/blob/cps_refactor/binary_cps_format.md |
|
LGTM!
Are we sure that's going to be enough? 😆 |
n1zzo
force-pushed
the
master
branch
2 times, most recently
from
a238c9a to
e53f25a
Compare
|
To do:
|
Updated the format used to store transmitter location: - removed the +500m offset for altitude, converted the value to a signed type - fix an error with negative coordinates close to zero, see #40.
|
Hi! The solution I first suggested was to include a short table at the beginning of the codeplug that would look like this (numbers are just random numbers and this example only considers contacts).
And then, for each entry you would have something looking like this (for one analog contact and 2 custom contact)
The advantage of this is that you do not really need to update the standard at all to include new modes, but you have to define each mode you want to support, including very ubiquitous ones like analog. Also, you raised the concern that managing strings can be quite computationally heavy for the smallest processors. The final solution we came up with was to reserve the 128 first IDs for pre-defined modes. Which would mean that in all CPS using this standard, ID 0 would always be analog, 1 would always be DMR, 2 would always be M17, and so on... The IDs 127 and up could be defined as explained above, but we could also not put the name of the mode, assuming that if the radio does not know about it, it does not need to know the name of the mode. The risk could be that you may encounter a mode with a "known ID" but whose definition is not what you expect (i.e. if I use id 218 for MyCustomMode and you use id 218 for YourCustomMode). However, it would be required for the table to contain the length of the various entries (contacts, channels, ...). That way you can at least check that the IDs above 127 have the expected length. You also know what is the length of the entries if you do not know about the mode. This solution also means that the radios are expected to keep a table of the lengths of all the modes included in the standard because it might encounter any of those including some it does not support. Not having a table for the IDs below 127 also means that any update to the standard including a new protocol would not be backward compatible, because if I don't know about ID 17 yet, then I have no idea what is the length of the entry and so the rest of the codeplug becomes unreadable. Anyway, that is what I remember about what we talked about more than a month ago. Any remarks are welcome from anybody that might read this comment 😄 |
|
Would it make sense for contacts to be sharable between modes? A contact could look like: {
"callsign": "N1ADJ",
"type": ["DMR","M17"],
"name": "Jim Ancona",
"dmr_id": 3127978,
"location": "Brunswick, ME, US"
}Then the radio could leverage DMR contacts for M17. |
|
I'm doing a comparison between the OpenRTX Binary Codeplug Format (OBCF) defined in this document and the QDMR Extensible Codeplug File Format. QDMR is trying to support all the functionality in multiple supported radios, while OBCF only need to support OpenRTX's functionality, so OBCF doesn't need to support everything QDMR does. Also OBCF is a binary format while QDMR is YAML which will be translated to radio binary formats. I started with contacts. I'll do channels in a follow-up. ContactsOBCF specifies two contact types, DMR and M17. QDMR has also has two types DMR and Analog DTMF. Presmably it would add an M17 type to support OpenRTX. DTMF contacts "can be used to store commonly used DTMF sequences. For example, it may be used to control the EchoLink feature of a repeater." DMR Contacts
Differences
All the fields in common are compatible. |
|
More comparison of this document and the QDMR specification, this time for channels. ChannelsCommon Attributes
Differences
FM channel attributes
Differences
DMR channel attributes
Differences
|
|
Here I take the lists of differences I compiled above and talk about what, if anything, should change in OpenRTX. I'm a big believer in YAGNI, so I'm in favor of deferring differences related to functionality OpenRTX doesn't have and is unlikely to implement soon until we need them. I included an "OpenRTX Feature?" column in the tables but it's mostly blank. I don't know what contact and channel features OpenRTX has because my CS7000 doesn't have any yet. ;-) Feel free to comment and I'll update the table. Also, if we are adding features to the codeplug format over time, it needs to be extensible. Contacts
Comments
|
|
I think such a format should either be self-describing or using a versioned formal schema, and also directly in the form for putting into memory in one block and accessing at runtime (thus, memory-aligned). It's one of my life goals to invent an all-purpose format like that, and use it for as many things as possible. For example I'm thinking of it as a way that a Scheme implementation could have a "sandbox" like Smalltalk implementations usually do. (Such languages of course are often out of scope for uC firmware, but helps to motivate what I want to do.) Also, to be able to write programs in any language that do not do serialization at all, but just mmap files from ssd and expect to keep their data continuously persistent. So my approach is to try to fit most data types into 64-bit words, because most powerful processors are 64-bit so that's the alignment that is needed. (But here, 32-bit alignment would be good enough, and you probably want support for smaller data types too. But multiple smaller values can be packed into 64 bits.) But strings tend to be variable-length. A long time ago I implemented my own data format, perhaps a bit like MsgPack, which was not concerned with alignment, and there I started with a string table at the beginning of the file, at first just to have reusable string "tags" for data. So when converting from xml for example, each tag name gets replaced with an 8-bit value and the string added to the string table. I think that's an idea that I will need to reuse in the memory-aligned format. Another thing I've tried is to borrow something like Baudot encoding: if each character is only 5 bits, you can pack a good number of characters into 64-bits. (Borrowing from FT8 encoding could also be interesting.) And from Scheme implementations I borrow the idea of data type tagging / NaN boxing: 3 or so bits in each word tell you its type (and so the numeric range of 64-bit ints is reduced slightly, but it doesn't have much impact on other data types). I've got a prototype over here: https://git.sr.ht/~ecloud/dscm/tree/main/item/expt/vector-scheme It uses a memory pool borrowed from Plan 9 (the one behind malloc) whereas that would be fixed-size on a microcontroller. And there's a string pool and a multi-type vector implementation. To create a dictionary, a vector of keys has an aux "pointer" to a vector of values. But "pointers" are really just small offsets within the same memory block. I think a codeplug format should be based on some sort of multi-purpose standard; and I would be eager to hear if another such format already exists, so that I don't reinvent the wheel. (I see that too many wheels have been reinvented in the world of ham radio already.) But so far I've checked into most of the formats listed in https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats and have not been satisfied that any of them do quite what I want. I would suggest that you at least do the same: think about whether protobuf or capnproto or asn.1 or flatbuffers or some such might be suitable for this purpose. For most of those, you just have to write a schema, and there is a code generator that writes your s11n code. The trouble as I see it is that I don't want to do s11n at all, and I expect to be able to write back to the data structure in-situ. (That is of course quite hard with variable-length strings, but I suppose that's why SQL databases tend to have both char and varchar data types. If you can fix the length, everything gets easier. But you probably have fixed-size anyway right?) But I think that some of those formats could allow in-situ modification: it's just a matter of writing code for that. I never got into Newton development, but I read that it had a data format called a "soup", meant for continuous persistence of user data on battery-backed SRAM. With such a size constraint I imagine it had to be quite efficient, and maybe also suitable for today's microcontroller applications. I just haven't gone down that rabbit hole yet, and wonder if it could be worthwhile still. |
Header:
Channel:
See also #31