Writing settings during production

[allard-potma]

I'm working on a production programming/testing/calibration rig and software for a product containing a MCU running ZephyrOS. Every device has its own default settings, containing device specific calibration, these settings are stored with settings subsytem with NVS backend. I want to write these settings with the application.
I was wondering if there is already a tool available that can prepare a sector with settings in the right format? Or documentation of the settings on-flash format?

[RomainPelletant]

Hi @allard-potma,

Facing the same requirement. For now I am implementing custom shell dedicated to production/programming mode.
I hope to see better/other suggestions in that thread! :)

[kartben]

@nordicjm

[nordicjm]

There is an upcoming settings management MCUmgr system that would allow settings to be set via a MCUmgr transport: #59345 the original idea did use NVS but switched it to settings.

For NVS queries, these should be directed to @Laczen

[Laczen]

@allard-potma, @RomainPelletant, there are no plans to make a tool that can write a nvs file outside of a device.

Years ago I created a little native_posix application that would be usable to write nvs settings to a bin file (by using the flash simulator): https://github.com/Laczen/zephyrmodules/tree/master/utils/native_posix/settings_preload.

I don't know if this still works, but the principle should still be ok: Make a storage on native posix that replicates the flash, write to this storage and then copy. This utility never made it to the zephyr tree because there was no proper place to put it.

[JPHutchins]

Agreed, that's why we normally use OTP for such settings. I see people use a tiny external EEPROM for chips that don't have much OTP.

By definition, the storage partitions are not flashed - this is really what you're looking for, right? An option to flash to the storage? A quick approach would be to configure the NVS as desired on a device, image it to BIN, then write that at factory.

It sorta feels like NVS is not meant for this though, as it is designed to be a mutable key-value lookup. It also costs at least 2 sectors because of this, which you do not need if the data is immutable.

[JPHutchins]

@Laczen Is there a way to make an NVS partition readonly?

[rettichschnidi]

This is what we do: husqvarnagroup@c188fdb - the code is not in best shape, and we use only the FCB implementation.

Basically like the settings module, but with less overhead because it is append only (during manufacturing) and read only once the productive FW has been flashed.

We also have a simple script to convert from and to FCB images:

#!/usr/bin/env python3
#
# coding: utf-8
#
# SPDX-FileCopyrightText: Copyright (c) 2022 GARDENA GmbH
# SPDX-License-Identifier: Apache-2.0

"""
Library/tool to interact with FCB factory data images.

Known limitations:
- No support for non-0xff erasable flash (FCB has some magic to support such flashes)
- Assumes a little endian machine (e.g. for the FCB magic value serialization)
- No support to deal with multiple sectors
- Testing coverage is low
"""

import argparse
import dataclasses
import json
import logging
import struct
import sys
import traceback
from copy import deepcopy
from functools import cached_property
from io import StringIO
from pathlib import Path
from typing import BinaryIO, Dict, Generator, Iterable, Optional, TextIO

import crcmod

try:
    from intelhex import IntelHex
except ModuleNotFoundError:
    print("Missing module intelhex, please install (e.g. pip3 install intelhex)")
    sys.exit(-1)

log = logging.getLogger('factory_data')
crc8_ccitt = crcmod.predefined.mkPredefinedCrcFun("crc-8")


def intel_hex_as_str(ih: IntelHex) -> str:
    """Intel Hex content as string."""
    out = StringIO()
    ih.write_hex_file(out)
    out.seek(0)
    return out.read()


class InvalidInputError(RuntimeError):
    """Base class of errors to be raised on any kind of invalid input."""


class CRCError(InvalidInputError):
    """Stored CRC8 value does not match calculated one."""


class AlignmentError(InvalidInputError):
    """Data does ot respect flash alignment requirements."""


@dataclasses.dataclass(frozen=True)
class FCBConfig:
    """Values describing the settings of a FCB partition."""

    magic: int = 0xfac70fcb
    version: int = 1


@dataclasses.dataclass(frozen=True)
class FlashType:
    """Values describing the (relevant) characteristics of a flash partition."""

    align: int = 4
    erase_value: int = 0xFF
    size: int = 4096


class FCB:
    """Reading and writing of FCB partitions."""

    DEFAULT_FLASH_PARTITION_OFFSET = 0xFF000

    ENTRY_MAX_LEN = (255 | 255 << 7)

    def __init__(self, flash_type: Optional[FlashType] = None, config: Optional[FCBConfig] = None,
                 data: bytes = None, entries: Iterable[bytes] = None):
        self.__flash_type: FlashType = flash_type if flash_type else FlashType()
        self.__fcb_config: FCBConfig = config if config else FCBConfig()
        self.__write_offset = 0  # 0 when unknown, greater than zero when valid
        if data:
            self.__data: bytearray = bytearray(data)
        elif entries:
            self.set_entries(entries)
        else:
            self.erase()

    def erase(self):
        """Erase all entries."""
        self.__data = bytearray(
            struct.pack("<IBBH", self.__fcb_config.magic, self.__fcb_config.version, 0xFF, 1))
        self.__data += bytearray(
            [self.__flash_type.erase_value] * (self.__flash_type.size - len(self.__data)))
        self.__write_offset = 8

    @classmethod
    def from_image(cls, data: bytes, flash_type: FlashType) -> 'FCB':
        """Create object from a factory data image in FCB format."""
        magic = int.from_bytes(data[0:4], byteorder='little')
        version = data[4]
        padding = data[5]
        id_ = int.from_bytes(data[6:8], byteorder='little')
        log.debug(f"extracted magic: {magic:#08x}")
        log.debug(f"extracted version: {version}")
        log.debug(f"extracted padding: {padding:#02x}")
        log.debug(f"extracted id: {id_:#02x}")
        log.debug(f"extracted data: {data[8:].hex(' ')}")
        return FCB(flash_type, FCBConfig(magic, version), data)

    @classmethod
    def from_image_file(cls, file: BinaryIO, fc: FlashType) -> 'FCB':
        """Create object from a factory data file in FCB format."""
        return cls.from_image(file.read(), fc)

    @classmethod
    def __align_up(cls, num: int, align: int) -> int:
        """Copied from bootloader/mcuboot/scripts/imgtool/image.py."""
        assert (align & (align - 1) == 0) and align != 0
        return (num + (align - 1)) & ~(align - 1)

    @property
    def entries(self) -> Generator[bytes, None, None]:
        """Retrieve all existing entries in a FCB partition."""
        offset = 8  # Skip header
        # Iterate over entries consisting of length information (1-2 bytes), data, and crc8 (1 byte)
        while len(self.__data[offset:]) >= 2:
            if offset % self.__flash_type.align:
                raise AlignmentError("FCB entry size storage location must respect flash alignment")
            low_length_byte = self.__data[offset]
            offset += 1
            if low_length_byte >= 0x80:
                high_length_byte = self.__data[offset]
                offset += 1
                if low_length_byte == high_length_byte == self.__flash_type.erase_value:
                    # Rest of partition is not (yet) used
                    break
                fcb_entry_len = (high_length_byte << 7) + (low_length_byte & ~0x80)
                fcb_entry_len_bytes = bytes([low_length_byte, high_length_byte])
            else:
                fcb_entry_len = low_length_byte
                fcb_entry_len_bytes = bytes([fcb_entry_len])

            # Data is aligned with flash requirements
            offset = FCB.__align_up(offset, self.__flash_type.align)
            data = self.__data[offset:offset + fcb_entry_len]

            # CRC8 value is aligned with flash requirements
            offset = FCB.__align_up(offset + fcb_entry_len, self.__flash_type.align)
            stored_crc8 = self.__data[offset]
            actual_crc8 = crc8_ccitt(fcb_entry_len_bytes + data, 0xFF)
            if stored_crc8 != actual_crc8:
                raise CRCError(f"Data at offset {offset} has a CRC mismatch:"
                               f" '{stored_crc8:#04x}' stored, but {actual_crc8:#04x} calculated")

            # Header is aligned with flash requirements
            offset = FCB.__align_up(offset + 1, self.__flash_type.align)

            yield data

    def _write_offset_advance_aligned(self, offset: int = 0):
        self.__write_offset = FCB.__align_up(self.__write_offset + offset, self.__flash_type.align)

    def set_entries(self, entries: Iterable[bytes]) -> None:
        """Set FCB entries, erase previously existing ones."""
        self.erase()
        for entry in entries:
            length = len(entry)
            if length == 0:
                raise InvalidInputError("A FCB entry must be at last one byte long")
            if length > FCB.ENTRY_MAX_LEN:
                raise InvalidInputError(f"A FCB entry must be no bigger than {FCB.ENTRY_MAX_LEN}")
            if length < 0x80:
                fcb_entry_len_bytes = bytes([length])
                self.__data[self.__write_offset] = length
                self._write_offset_advance_aligned(1)
            else:
                fcb_entry_len_bytes = bytes([(length | 0x80) & 0xFF, length >> 7])
                self.__data[self.__write_offset] = fcb_entry_len_bytes[0]
                self.__data[self.__write_offset + 1] = fcb_entry_len_bytes[1]
                self._write_offset_advance_aligned(2)
            prefix = self.__data[:self.__write_offset]
            postfix = self.__data[self.__write_offset + length:]
            self.__data = prefix + entry + postfix
            self._write_offset_advance_aligned(length)
            crc8 = crc8_ccitt(fcb_entry_len_bytes + entry, 0xFF)
            self.__data[self.__write_offset] = crc8
            self._write_offset_advance_aligned(1)
        pass

    @property
    def image(self) -> bytes:
        """Bytes as they exist in the factory FCB partition."""
        return bytes(self.__data)

    def image_as_hex(self, offset: int = DEFAULT_FLASH_PARTITION_OFFSET) -> IntelHex:
        """Bytes as they exist in the factory FCB partition."""
        ih = IntelHex()
        ih.frombytes(self.image, offset)
        return ih

    @cached_property
    def config(self) -> FCBConfig:
        return self.__fcb_config


class FactoryData:
    """Reading and writing of (FCB based) factory data."""

    def __init__(self):
        # Note: Code relies on insertion ordering being preserved. Python 3.6+, hence all supported
        # versions, guarantee this.
        self.__data: Dict[str, bytes] = {}

    def add(self, name: str, value: bytes) -> None:
        if name in self.__data:
            raise KeyError(f"Entry '{name}' already exist")
        self.__data[name] = value

    def get(self, name: str) -> bytes:
        if name not in self.__data:
            raise KeyError(f"Entry '{name}' does not exist")
        return self.__data[name]

    @property
    def as_dict(self) -> Dict[str, bytes]:
        return deepcopy(self.__data)

    @classmethod
    def from_image(cls, data: bytes, flash: FlashType) -> 'FactoryData':
        """Create object from a factory data file in FCB format."""
        factory_data = FactoryData()
        fcb = FCB.from_image(data, flash)
        for name_and_value in fcb.entries:
            null_byte_offset = name_and_value.find(b'\0')
            if null_byte_offset < 0:
                raise InvalidInputError(f"Invalid value name limiter offset: {null_byte_offset}")
            name = name_and_value[:null_byte_offset]
            value = name_and_value[null_byte_offset + 1:]
            factory_data.add(name.decode("utf-8"), value)
            log.debug(f"name: {name.decode('utf-8')}")
            log.debug(f"value: {value}")
        return factory_data

    @classmethod
    def from_image_file(cls, file: BinaryIO, flash: FlashType) -> 'FactoryData':
        """Create object from a factory data file in FCB format."""
        return cls.from_image(file.read(), flash)

    @classmethod
    def from_json_file(cls, file: TextIO) -> 'FactoryData':
        """Create object from a factory data file in JSON format."""
        factory_data = FactoryData()
        factory_data_source = json.load(file)['factory_data']
        for name, value in factory_data_source.items():
            if value["type"] == "hex":
                # length of hex values be multiple of two
                if len(value["value"]) % 2 != 0:
                    log.error(f"Hex values must be multiple of two: {name}: {value}")
                    raise InvalidInputError("Hex values must be multiple of two")
                value_as_bytes = bytes.fromhex(value["value"])
            elif value["type"] == "string":
                value_as_bytes = bytes(value["value"], 'utf-8')
            else:
                raise InvalidInputError("value type must be either of 'string' or 'hex'")
            factory_data.add(name, value_as_bytes)
        return factory_data

    def to_fcb(self, fcb: Optional[FCB] = None) -> FCB:
        entries = [key.encode('utf-8') + b'\x00' + value for key, value in self.as_dict.items()]
        if not fcb:
            return FCB(entries=entries)
        fcb.set_entries(entries)
        return fcb


def main():
    fcb_config_default = FCBConfig()
    flash_type_default = FlashType()
    parser = argparse.ArgumentParser(description="Read and write FCB based factory data images")
    parser.add_argument("--input", type=argparse.FileType('rb'), default=sys.stdin,
                        help="input file - either in JSON or FCB format (default: stdin)")
    parser.add_argument("--input-type", choices=['json', 'fcb'], default='json',
                        help="input file's type (default: %(default)s)")
    parser.add_argument("--base-image", type=argparse.FileType('r'),
                        help="Intel HEX image (likely a Zephyr application) to enrich")
    parser.add_argument("--output", type=argparse.FileType('wb'), default=sys.stdout.buffer,
                        help="output file for FCB image (default: stdout)")
    parser.add_argument("--output-type", type=str, choices=['fcb', 'hex', 'dump'], default='fcb',
                        help="output file's type (default: %(default)s)")
    parser.add_argument("--flash-image-size", type=int, default=flash_type_default.size,
                        help="size of the to-be-generated FCB image (default: %(default)s)")
    parser.add_argument("--flash-align", type=int, default=flash_type_default.align,
                        help="alignment in FCB (default: %(default)s)")
    parser.add_argument("--flash-erase-value", type=int, default=flash_type_default.erase_value,
                        help="flash erase value (default: %(default)s)")
    parser.add_argument("--flash-partition-offset", type=lambda x: int(x, 0),
                        default=FCB.DEFAULT_FLASH_PARTITION_OFFSET,
                        help="flash partition offset - required for output type 'hex'")
    parser.add_argument("--fcb-magic", type=int, default=fcb_config_default.magic,
                        help=f"factory data magic number of FCB partition (default: {fcb_config_default.magic:#04x})")
    parser.add_argument("--fcb-version", type=int, default=fcb_config_default.version,
                        help="factory data version (default: %(default)s)")
    args = parser.parse_args()

    fcb_config = FCBConfig(args.fcb_magic, args.fcb_version)
    flash_type = FlashType(args.flash_align, args.flash_erase_value, args.flash_image_size)

    if args.input_type == 'fcb':
        factory_data = FactoryData.from_image_file(args.input, flash_type)
    else:
        factory_data = FactoryData.from_json_file(args.input)

    fcb = FCB(flash_type, fcb_config)

    if args.base_image:
        if args.output_type != 'hex':
            log.error("Enriching base images requires the output type to be 'hex'")
            sys.exit(1)

        # Prevent unexpected results due to the output file truncating the input one
        if Path(args.output.name).resolve() == Path(args.base_image.name).resolve():
            log.error("Base image and output file must not be the same file!")
            sys.exit(1)

    if args.output_type == 'hex':
        ih = factory_data.to_fcb(fcb).image_as_hex(args.flash_partition_offset)
        if args.base_image:
            base_image = IntelHex(args.base_image)
            ih.merge(base_image)
        output = intel_hex_as_str(ih).encode('ascii')
    elif args.output_type == 'fcb':
        output = factory_data.to_fcb(fcb).image
    else:
        output = b"\n".join(
            f"{name}: {val.hex(' ')}".encode('ascii') for name, val in factory_data.as_dict.items())
        output += b"\n"
    args.output.write(output)


if __name__ == '__main__':
    try:
        main()
    except KeyboardInterrupt:
        log.info('Keyboard interrupt detected. Exiting...')
    except Exception as e:
        log.error(f'Exception raised: {e}')
        traceback.print_exc()
        sys.exit(1)
    sys.exit(0)

If this would be of any interest, I can share the Python tests and documentation too. Would also be willing to brush it up, fix what was not liked in #45819, and try again to get it upstreamed.

[JPHutchins]

It looks great to me! I'd rather it's available on PyPI than upstreamed to Zephyr. Because FCB could be used in any system. And you don't want to depend on Zephyr, anyway, that's hard for users. Zephyr can always install from PyPI and users can install easily with pipx.

Do you just eat the extra sector or is it possible to create FCB with a single sector?

[Laczen]

@JPHutchins no, NVS cannot be (easily) made read-only. In PR #79665 a system similar to FCB is proposed (storage_area_store) that can be used as a circular buffer with or without persistence, supports read-only, is easy to generate off-line, ... I am looking for some extra reviewers.