feat: Add default thin-edge.io firmware support (using dev thin-edge.…

…io version) (#10) * add thin-edge firmware workflow definition and script * add overlay persistence recipe * move tedge config persistence to thin-edge.io recipe * include firmware update recipe in all images * fixup! add overlay persistence recipe * fixup! add overlay persistence recipe * fix permissions on health check folders * add publish image task to build * refactor workflow to use justfile task * refactor build tasks to support profiles and variants * disable manual downloads by default in favor of streaming * improve c8y url detection * fix handling of stream downloads * configure priority given recipe dependency * add logging to thin-edge.io install script (to debug sporadic build issue)
thin-edge · Nov 29, 2023 · dccf8ac · dccf8ac
1 parent add4197
commit dccf8ac
Show file tree

Hide file tree

Showing 30 changed files with 641 additions and 262 deletions.
diff --git a/.github/workflows/bake-image.yml b/.github/workflows/bake-image.yml
@@ -20,26 +20,17 @@ jobs:
 
       - uses: extractions/setup-just@v1
 
-      - name: Set image name
-        run: echo "IMAGE_NAME=$(just generate_version tedge_rugpi_45)" >> $GITHUB_ENV
-
       - name: Install QEMU
         run: docker run --privileged --rm tonistiigi/binfmt --install arm64
 
-      - name: Set Image
-        run: just set-image images/pi45.toml
-
-      - name: Extract Image
-        run: just extract
-
-      - name: Customize System
-        run: just customize
+      - name: Set image version
+        run: echo "VERSION=$(just generate_version)" >> $GITHUB_ENV
 
-      - name: Bake Image
-        run: just bake
+      - name: Build image
+        run: just build-all-variants
 
       - name: Upload Image
         uses: actions/upload-artifact@v3
         with:
-          name: ${{ env.IMAGE_NAME }}
-          path: build/${{ env.IMAGE_NAME }}*.img
+          name: images
+          path: build/*.xz
diff --git a/README.md b/README.md
@@ -10,23 +10,53 @@ The repository can be used to build custom Raspberry Pi images with thin-edge.io
 * Raspberry Pi 4 (using tryboot)
 * Raspberry Pi 5 (using tryboot)
 
-## Building the image
+## Building an image
 
 To run the build tasks, install [just](https://just.systems/man/en/chapter_5.html).
 
-1. Set which image you want to build
+1. Create the image (including downloading the supported base Raspberry Pi image) using:
 
     ```sh
-    just set-image images/pi45.toml
+    just VARIANT=pi45 build-all
     ```
 
-2. Create the image (including downloading the supported base Raspberry Pi image) using:
+    Possible variants are:
+
+    * pi45
+    * pi4
+    * pi023
 
     ```sh
-    just build-all
+    just PROFILE=wifi VARIANT=pi4 build-all
     ```
 
-3. Using the path to the image shown in the console to flash the image to the Raspberry Pi.
+2. Using the path to the image shown in the console to flash the image to the Raspberry Pi.
 
 
 For further information, checkout the [Rugpi quick start guide](https://oss.silitics.com/rugpi/docs/getting-started).
+
+## Building an image with WIFI credentials
+
+For devices that only support WIFI (e.g. don't have an ethernet adapter), the WIFI credentials are required to be part of the image, otherwise you don't have any way to connect via SSH to your device.
+
+In the future this process will be looked to be improved, and potentially the standard raspberry pi way of using the wpa_supplicant will enable to work out of the box (so that you don't have to bake credentials into the image, and only add them when writing to flash).
+
+The default WIFI credentials are as follows, though it assumes that the given WIFI setup is a non-trusted network that is only used for bootstrapping, and then a secure WIFI network is configured.
+
+|SSID|Password|
+|----|--------|
+|onboarding_jail|onboarding_jail|
+
+1. Create the image (including downloading the supported base Raspberry Pi image) using:
+
+    ```sh
+    just PROFILE=wifi VARIANT=pi023 build-all
+    ```
+
+    Possible variants are:
+
+    * pi023
+    * pi4
+    * pi45
+
+    This profile will use pre-baked credentials for the WIFI which are defined in [profiles/wifi.toml](profiles/wifi.toml).
diff --git a/images/pi023.toml b/images/pi023.toml
@@ -0,0 +1,3 @@
+boot_flow = "u-boot"
+
+include_firmware = "none"
diff --git a/images/pi23.toml b/images/pi23.toml
diff --git a/images/pi4.toml b/images/pi4.toml
@@ -0,0 +1,4 @@
+boot_flow = "tryboot"
+
+# Include firmware as some older pi4 need a firmware update before tryboot will work
+include_firmware = "pi4"
diff --git a/images/pi45.toml b/images/pi45.toml
@@ -1,26 +1,4 @@
-recipes = [
-    # "set-hostname",
-    "persist-root-home",
-    "ssh",
-    "zsh",
-]
+boot_flow = "tryboot"
 
 # Make image generic so it can be used for pi4 and pi5
-# So to either pi4 or pi5 if you want to build pi specific images which include the given firmware (EEPROM)
 include_firmware = "none"
-
-boot_flow = "tryboot"
-
-[parameters.set-hostname]
-hostname = "tedge-rugpi"
-
-[parameters.apt-cleanup]
-autoremove = true
-
-[parameters.ssh]
-root_authorized_keys = """
-ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDfhQGWWw73ponAokdNSRZ5cQc9/CIX1TLQgYlr+BtObKoO4UNFP1YSbgK03GjhjeUid+QPmV+UURqxQTqLQoYWqUFP2CYkILFccVPmTvx9HLwupI+6QQKWfMDx9Djfph9GzInymaA5fT7hKppqittFrC/l3lkKgKTX5ohEOGshIbRgtgOYIaW3ByTx3urnaBbYCIgOyOZzSIyS0dUkwsiLu3XjPspgmn3Fs/+vofT/yhBe1carW0UM3ivV0JFfJzrxbCl/F7I2qwfjZXsypjkwlpNupUMuo3xPMi8YvNvyEu4d+IEAqO1dCcdGcxlkiHxrdITIpVLt5mjJ2LauHE/H bootstrap
-"""
-
-[parameters.rugpi-ctrl]
-rugpi_admin = true  # Enable Rugpi Admin.
diff --git a/justfile b/justfile
@@ -2,32 +2,42 @@
 export IMAGE_URL := "https://downloads.raspberrypi.org/raspios_lite_arm64/images/raspios_lite_arm64-2023-10-10/2023-10-10-raspios-bookworm-arm64-lite.img.xz"
 export RUGPI_IMAGE := "ghcr.io/silitics/rugpi-bakery:latest"
 
-export IMAGE_NAME := env_var_or_default("IMAGE_NAME", replace_regex(file_stem(IMAGE_URL), ".img$", ""))
-export BASE_TAR := "build" / IMAGE_NAME + ".base.tar"
-export CUSTOM_TAR := "build" / IMAGE_NAME + ".tedge.tar"
-export OUTPUT_IMAGE := "build" / IMAGE_NAME + ".tedge.img"
-export BUILD_INFO := file_stem(IMAGE_NAME)
+export PREFIX := "tedge_rugpi_"
+export PROFILE := "default"
+
+export BASE_IMAGE := replace_regex(file_stem(IMAGE_URL), ".img$", "")
+export BASE_TAR := "build" / BASE_IMAGE + ".base.tar"
+export CUSTOM_TAR := "build" / BASE_IMAGE + "." + PROFILE + ".tar"
 
-set-image FILE="images/pi45.toml":
-    rm -f ./rugpi-bakery.toml
-    ln -s {{FILE}} ./rugpi-bakery.toml
+export CUSTOMIZATION_PROFILE := "profiles" / PROFILE + ".toml"
+export VARIANT := "pi45"
+export IMAGE_CONFIG := "images/" + VARIANT + ".toml"
+export VERSION := env_var_or_default("VERSION", `date +'%Y%m%d%H%M'`)
+export IMAGE_NAME := PREFIX + PROFILE + "_" + VARIANT + "_" + VERSION
+export OUTPUT_IMAGE := "build" / IMAGE_NAME + ".img"
+export BUILD_INFO := file_stem(IMAGE_NAME)
 
 # Generate a version name (that can be used in follow up commands)
-generate_version prefix="tedge_rugpi":
-    @echo "{{prefix}}_$(date +'%Y-%m-%d-%H%M')"
+generate_version:
+    @echo "{{VERSION}}"
 
 # Show the install paths
 show:
     @echo "IMAGE_URL: {{IMAGE_URL}}"
     @echo "IMAGE_NAME: {{IMAGE_NAME}}"
+    @echo "CUSTOMIZATION_PROFILE: {{CUSTOMIZATION_PROFILE}}"
+    @echo "IMAGE_CONFIG: {{IMAGE_CONFIG}}"
+
     @echo "BASE_TAR: {{BASE_TAR}}"
     @echo "CUSTOM_TAR: {{CUSTOM_TAR}}"
+
     @echo "OUTPUT_IMAGE: {{OUTPUT_IMAGE}}"
+    @echo "VERSION: {{VERSION}}"
     @echo "BUILD_INFO: {{BUILD_INFO}}"
 
-# Clean build and cache
+# Clean build
 clean:
-    @rm -Rf build/ .rugpi/
+    @rm -Rf build/
 
 # Download and extract the base image
 extract:
@@ -36,25 +46,38 @@ extract:
 # Apply recipes to the base image
 customize:
     echo "{{BUILD_INFO}}" > "{{justfile_directory()}}/recipes/build-info/files/.build_info"
-    ./run-bakery customize "{{BASE_TAR}}" "{{CUSTOM_TAR}}"
+    ./run-bakery --config "{{CUSTOMIZATION_PROFILE}}" customize "{{BASE_TAR}}" "{{CUSTOM_TAR}}"
 
 # Create the image that can be flashed to an SD card or applied using the rugpi interface
 bake:
-    ./run-bakery bake "{{CUSTOM_TAR}}" "{{OUTPUT_IMAGE}}"
+    ./run-bakery --config "{{IMAGE_CONFIG}}" bake "{{CUSTOM_TAR}}" "{{OUTPUT_IMAGE}}"
+    @echo ""
+    @echo "Compressing image"
+    xz -0 -v "{{OUTPUT_IMAGE}}"
+    @echo ""
     @echo ""
     @echo "Image created successfully. Check below for options on how to use the image"
     @echo ""
     @echo "Option 1: Use the Raspberry Pi Imager to flash the image to an SD card"
     @echo ""
-    @echo "    {{justfile_directory()}}/{{OUTPUT_IMAGE}}"
+    @echo "    {{justfile_directory()}}/{{OUTPUT_IMAGE}}.xz"
     @echo ""
     @echo "Option 2: If the device is already running a rugpi image, open the http://tedge-rugpi:8088 website and install the following image:"
     @echo ""
-    @echo "    {{justfile_directory()}}/{{OUTPUT_IMAGE}}"
+    @echo "    {{justfile_directory()}}/{{OUTPUT_IMAGE}}.xz"
     @echo ""
 
 # Build the entire image
 build-all: extract customize bake
 
 # Build the image from an already downloaded image
 build-local: customize bake
+
+# Publish latest image to Cumulocity
+publish:
+    cd {{justfile_directory()}} && ./scripts/upload-c8y.sh
+
+build-all-variants: extract customize
+    # just VARIANT=pi023 bake
+    # just VARIANT=pi4 bake
+    just VARIANT=pi45 bake
diff --git a/images/pi4-firmware.toml → profiles/default.toml b/images/pi4-firmware.toml → profiles/default.toml
@@ -1,18 +1,11 @@
 recipes = [
-    # "set-hostname",
     "persist-root-home",
     "ssh",
     "zsh",
+    "persist-overlay",
+    "tedge-firmware-update",
 ]
 
-# Include firmware as some older pi4 need a firmware update before tryboot will work
-include_firmware = "pi4"
-
-boot_flow = "tryboot"
-
-[parameters.set-hostname]
-hostname = "tedge-rugpi"
-
 [parameters.apt-cleanup]
 autoremove = true
 
@@ -22,4 +15,4 @@ ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDfhQGWWw73ponAokdNSRZ5cQc9/CIX1TLQgYlr+BtO
 """
 
 [parameters.rugpi-ctrl]
-rugpi_admin = true  # Enable Rugpi Admin.
+rugpi_admin = true  # Enable Rugpi Admin.
diff --git a/images/pizero2w.toml → profiles/wifi.toml b/images/pizero2w.toml → profiles/wifi.toml
@@ -1,22 +1,14 @@
 recipes = [
-    # "set-hostname",
     "persist-root-home",
     "ssh",
     "zsh",
 
     # wifi credentials are required as Raspberry Pi Zero's don't have an ethernet adapter
     "set-wifi",
+    "persist-overlay",
+    "tedge-firmware-update",
 ]
 
-# Make image generic so it can be used for pi4 and pi5
-# So to either pi4 or pi5 if you want to build pi specific images which include the given firmware (EEPROM)
-include_firmware = "none"
-
-boot_flow = "u-boot"
-
-[parameters.set-hostname]
-hostname = "tedge-rugpi"
-
 [parameters.apt-cleanup]
 autoremove = true
 
@@ -31,4 +23,4 @@ ssid = "onboarding_jail"
 password = "onboarding_jail"
 
 [parameters.rugpi-ctrl]
-rugpi_admin = true  # Enable Rugpi Admin.
+rugpi_admin = true  # Enable Rugpi Admin.
diff --git a/recipes/persist-overlay/files/ctrl.toml b/recipes/persist-overlay/files/ctrl.toml
@@ -0,0 +1 @@
+overlay = "persist"
diff --git a/recipes/persist-overlay/recipe.toml b/recipes/persist-overlay/recipe.toml
@@ -0,0 +1,2 @@
+description = "Persist general overaly across reboots"
+default = false
diff --git a/recipes/persist-overlay/steps/00-install.sh b/recipes/persist-overlay/steps/00-install.sh
@@ -0,0 +1,3 @@
+#!/bin/sh
+set -eu
+install -D -m 644 "${RECIPE_DIR}/files/ctrl.toml" -t /etc/rugpi/
diff --git a/recipes/persist-tedge-config/files/tedge-config.toml b/recipes/persist-tedge-config/files/tedge-config.toml
diff --git a/recipes/persist-tedge-config/recipe.toml b/recipes/persist-tedge-config/recipe.toml
diff --git a/recipes/persist-tedge-config/steps/00-install.sh b/recipes/persist-tedge-config/steps/00-install.sh
diff --git a/recipes/rugpi-auto-rollback/files/health.d/00-time-sync b/recipes/rugpi-auto-rollback/files/health.d/00-time-sync
@@ -0,0 +1,46 @@
+#!/bin/sh
+#
+# Wait for network to be ready but don't block if still not available as the mender commit
+# might be used to restore network connectivity.
+#
+
+set -e
+
+OK=0
+
+log() {
+    echo "$*" >&2
+}
+
+attempt=0
+max_attempts=10
+
+# Network ready: 0 = no, 1 = yes
+ready=0
+log "Waiting for network to be ready, and time to be synced"
+
+while [ "$attempt" -lt "$max_attempts" ]; do
+    TIME_IN_SYNC=$(timedatectl | awk '/System clock synchronized/{print $NF}')
+    case "${TIME_IN_SYNC}" in
+        yes)
+            ready=1
+            break
+            ;;
+    esac
+    attempt=$((attempt + 1))
+    log "Network not ready yet (attempt: $attempt from $max_attempts)"
+    sleep 30
+done
+
+# Duration can only be based on uptime since the device's clock might not be synced yet, so 'date' will not be monotonic
+duration=$(awk '{print $1}' /proc/uptime)
+
+log "Network: ready=$ready (after ${duration}s)"
+if [ "$ready" = "1" ]; then
+    log "Network is ready after ${duration}s (from startup)"
+else
+    # Don't fail, as the downstream checks might still work
+    log "WARNING: System time is still not in sync but continuing anyway"
+fi
+
+exit ${OK}
diff --git a/...o-rollback/files/health.d/00-tedge-health → ...o-rollback/files/health.d/10-tedge-health b/...o-rollback/files/health.d/00-tedge-health → ...o-rollback/files/health.d/10-tedge-health