Squashed commit of the following:

commit 9a309aa Merge: 68bad9f 655fbe6 Author: Francine Wright <[email protected]> Date: Tue Jul 23 11:04:59 2024 -0700 Merge pull request #26 from groundlight/merge-pi-gen-2024-07 Merge upstream 2024-07 commit 655fbe6 Merge: 68bad9f 48efb5f Author: Francine Wright <[email protected]> Date: Tue Jul 23 09:33:18 2024 -0700 merge upstream commit 68bad9f Merge: c78dafd a267860 Author: Francine Wright <[email protected]> Date: Tue Jul 23 09:19:46 2024 -0700 Merge pull request #25 from groundlight/add-python3-dev Add python3-dev to enable new version of framegrab commit a267860 Author: Francine Wright <[email protected]> Date: Mon Jul 22 17:53:45 2024 -0700 Update comment for python3-dev inclusion Co-authored-by: robotrapta <[email protected]> commit 42c12e8 Author: Francine Wright <[email protected]> Date: Mon Jul 22 13:52:41 2024 -0700 add python3-dev commit c78dafd Author: robotrapta <[email protected]> Date: Mon Feb 5 12:48:29 2024 -0800 Making CLEAN=1 work on dobuild. (#20) commit e555804 Author: robotrapta <[email protected]> Date: Mon Feb 5 09:34:25 2024 -0800 Fixes #16 - makes the .venv environment globally writable (#17) * Making the .venv environment globally writable after it's built. * Fixing readme. * Reverting some inadvertent deprovements to the readme commit c439a2a Author: robotrapta <[email protected]> Date: Mon Feb 5 09:29:14 2024 -0800 Merge from upstream 2024-02 (#19) * Ensure loop device partition nodes are created (#741) Although the loop block device is created before attaching the image to it, the devices for the partition that the image contains are still not created. This patch creates those devices as well, when they are not already available. Fixes #482 Signed-off-by: Vasilis Tsiligiannis <[email protected]> * Update release notes * stage0: Only disable initramfs updates if update-initramfs.conf exists * stage0: Check whether foreign architecture is needed * stage0: prevent kernel packages from creating useless symlinks * stage3: add vulkan support * Remove cmdline.txt and config.txt symlinks * stage2: rc.local drop-in no longer required, since Debian ships its own version * export-image: make sure initramfs is created Fixes #749 * README on how to merge from upstream. --------- Signed-off-by: Vasilis Tsiligiannis <[email protected]> Co-authored-by: Vasilis Tsiligiannis <[email protected]> Co-authored-by: Serge Schneider <[email protected]> commit 66a9d56 Merge: b1cb158 9268aab Author: blaise-muhirwa <[email protected]> Date: Fri Dec 22 11:01:30 2023 -0800 Merge pull request #11 from groundlight/get-qemu-working Emulating the Pi with QEMU commit 9268aab Author: blaise-muhirwa <[email protected]> Date: Fri Dec 22 17:45:12 2023 +0000 check on bad image mounting commit 4afd826 Author: blaise-muhirwa <[email protected]> Date: Fri Dec 22 17:42:16 2023 +0000 clean up the rpistart script commit 7ec13d7 Merge: 794ea6e b1cb158 Author: blaise-muhirwa <[email protected]> Date: Fri Dec 22 16:56:04 2023 +0000 Merge branch 'main' of https://github.com/groundlight/groundlight-pi-gen into get-qemu-working commit b1cb158 Merge: 3754fc5 9576784 Author: blaise-muhirwa <[email protected]> Date: Thu Dec 21 13:13:37 2023 -0800 Merge pull request #14 from groundlight/configure-gh-actions-bigger-runner configure github actions to run on a larger runner commit 9576784 Author: blaise-muhirwa <[email protected]> Date: Thu Dec 21 20:00:12 2023 +0000 explicitly specify runner label commit 09923ea Author: blaise-muhirwa <[email protected]> Date: Thu Dec 21 18:49:48 2023 +0000 configure github actions to run on a larger runner commit 794ea6e Author: blaise-muhirwa <[email protected]> Date: Tue Dec 19 23:19:41 2023 +0000 better documentation commit 3754fc5 Merge: 3eb601e fd08359 Author: blaise-muhirwa <[email protected]> Date: Tue Dec 19 14:21:18 2023 -0800 Merge pull request #12 from groundlight/desktop-version Adding support for Full Desktop Version commit fd08359 Author: blaise-muhirwa <[email protected]> Date: Tue Dec 19 21:18:34 2023 +0000 full desktop version commit b691d7f Author: blaise-muhirwa <[email protected]> Date: Tue Dec 19 19:21:07 2023 +0000 add a one-time qemu setup script commit b52439f Author: blaise-muhirwa <[email protected]> Date: Mon Dec 18 20:53:10 2023 +0000 get the qemu virtualization to work commit 39360cf Author: blaise-muhirwa <[email protected]> Date: Sat Dec 16 01:04:56 2023 +0000 more progress on the bash script for starting the qemu emulator commit 5df35bf Author: blaise-muhirwa <[email protected]> Date: Sat Dec 16 00:07:07 2023 +0000 making progress commit 3eb601e Merge: b384791 8fc2b2a Author: blaise-muhirwa <[email protected]> Date: Wed Dec 13 17:17:15 2023 -0800 Merge pull request #10 from groundlight/fix-release-names Fix small release name bug commit 8fc2b2a Author: Ubuntu <[email protected]> Date: Thu Dec 14 00:44:34 2023 +0000 fix small release name bug commit b384791 Author: robotrapta <[email protected]> Date: Tue Dec 12 17:27:24 2023 -0800 Fixing "externally-managed-environment" problem with pip (#9) * Using a python venv by default for groundlight python SDK stuff. * Fixing cv2 issue, and readme install. * Fixup file location. commit 92449ab Author: robotrapta <[email protected]> Date: Mon Dec 11 14:55:02 2023 -0800 Release assets include tag name instead of datestamp (#7) * Renames release artifacts to include tag name instead of datestamp. * dobuild exits on error. * Fixing yaml indentation. commit d87ad2a Author: robotrapta <[email protected]> Date: Mon Dec 11 13:04:22 2023 -0800 Installs MNS as systemd service (#2) * Basic minimal configuration for NMS * Adding a custom GL1 stage that just installs pip * It appears to install the SDK - does it work? * Renaming the stage to make it easier to read. * Installing missing python library that framegrab needs. * Comment on pip strangeness. * documenting compression level measurements. * Setting to compression 0 * Adding stage-gl2 which installs the code for NMS but doesn't run it yet. * Building again * Pointer to MNS repo * Making fork maintenance easier. Trying 3.10 * Removing redundant readme that was a merge mistake. * Undoing attempts to make the SDK better. For now. It'll be its own PR. * Readme and usability tweaks. * Forking off a "prod" config. * little tweaks * Close to getting systemd service to run I think. * Copies in MNS systemd files using canonical "install" command. * Taking out stage3 commit e92a760 Author: robotrapta <[email protected]> Date: Sat Dec 9 08:00:41 2023 -0800 Builds images for releases (#6) * Updating readme with forward-looking build instructions. * Renaming "glmns" to "gl" and "groundlight" * First crack at GHA to build images and make releases. * Only store artifacts for releases. * Fixing problems found in PR commit bf7420a Author: robotrapta <[email protected]> Date: Thu Dec 7 17:02:28 2023 -0800 General cleanup of build system (#4) * Basic minimal configuration for NMS * Adding a custom GL1 stage that just installs pip * It appears to install the SDK - does it work? * Renaming the stage to make it easier to read. * Installing missing python library that framegrab needs. * Comment on pip strangeness. * documenting compression level measurements. * Setting to compression 0 * Adding stage-gl2 which installs the code for NMS but doesn't run it yet. * Building again * Pointer to MNS repo * Making fork maintenance easier. Trying 3.10 * Removing redundant readme that was a merge mistake. * Undoing attempts to make the SDK better. For now. It'll be its own PR. * Readme and usability tweaks. * Forking off a "prod" config. * little tweaks * Close to getting systemd service to run I think. * Copies in MNS systemd files using canonical "install" command. * Undoing the substantive changes for MNS * Removing reference to stage-gl2 commit 063e8cc Author: robotrapta <[email protected]> Date: Thu Dec 7 14:23:54 2023 -0800 Creates a "lite" OS image that has a working Groundlight SDK (#1) * Headless groundlight SDK with mixed apt/pip * Moving README's around so we can have our own, but not mess with the original file. * Adding symlink into .github/readme. * Cleanup.
groundlight · Jul 23, 2024 · 05160f7 · 05160f7
1 parent dca93d2
commit 05160f7
Show file tree

Hide file tree

Showing 27 changed files with 716 additions and 5 deletions.
diff --git a/.github/README.md b/.github/README.md
@@ -0,0 +1 @@
+../GL-README.md
diff --git a/.github/workflows/build-images.yaml b/.github/workflows/build-images.yaml
@@ -0,0 +1,74 @@
+name: build-images
+
+on:
+  push:
+    branches: [ main ]
+    tags: 
+      - '*'
+  pull_request:
+    # all branches for now while we're testing.
+    #branches: [ main ]
+
+jobs:
+  build:
+    runs-on: 
+      labels: ubuntu-22.04-8core
+
+    steps:
+    - uses: actions/checkout@v4
+
+    - name: install needed packages
+      run: |
+        sudo apt update
+        sudo apt install binfmt-support
+        sudo apt install qemu qemu-user-static
+        sudo update-binfmts --enable
+
+    - name: Detect if release
+      run: |
+        if [[ $GITHUB_REF == refs/tags/* ]]; then
+          echo "IS_RELEASE=1" >> $GITHUB_ENV
+        else
+          echo "IS_RELEASE=0" >> $GITHUB_ENV
+        fi
+
+    - name: Build the full images
+      run: |
+        ./dobuild.sh
+
+    # Print output directory files
+    - name: List output files
+      run: ls -lh deploy
+
+    - name: Upload built image
+      if: env.IS_RELEASE == '1'
+      uses: actions/upload-artifact@v3
+      with:
+        name: rpios-image
+        path: ./deploy/GroundlightPi-*.img.xz
+        if-no-files-found: error
+
+    - name: Upload debug artifacts
+      if: env.IS_RELEASE == '1'
+      uses: actions/upload-artifact@v3
+      with:
+        name: debug-files
+        path: | 
+          ./deploy/*.info
+          ./deploy/build.log
+        if-no-files-found: error
+
+  release:
+    if: startsWith(github.ref, 'refs/tags/')
+    needs: [build]
+    runs-on: 
+      labels: ubuntu-22.04-8core
+    steps:
+      - uses: actions/download-artifact@v3
+        with:
+          name: rpios-image
+      - uses: softprops/action-gh-release@v1
+        with:
+          files: '**/*'
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
diff --git a/.gitignore b/.gitignore
@@ -6,3 +6,10 @@ SKIP
 SKIP_IMAGES
 .pc
 *-pc
+
+
+#
+# Groundlight additions below
+#
+*.swp
+/deleteme*
diff --git a/GL-README.md b/GL-README.md
@@ -0,0 +1,140 @@
+# Groundlight Pi-Gen: OS images for Raspberry PI with Groundlight Tools
+
+This repo builds OS images for Groundlight tools and applications, including the bare python SDK, 
+and the Monitoring Notification Server (MNS).  The OS images are available in the [releases](https://github.com/groundlight/groundlight-pi-gen/releases) and can be installed with [Raspberry Pi imager](https://www.raspberrypi.com/software/).
+
+There are several different images available, depending on your needs, from smallest to largest:
+
+- **`sdk-only`** - A minimal image with just the Python SDK installed.  This is the smallest image, and is suitable for running the SDK on a Raspberry Pi Zero W.  It is also suitable for running the SDK on a Raspberry Pi 3 or 4, if you don't need the GUI.
+- **`mns-headless`** - an image with the [Groundlight Monitoring Notification Server (MNS)](https://github.com/groundlight/monitoring-notification-server) installed for headless use.  "Headless" means it runs the server, which serves HTML pages, but has no browser or GUI to use it from.  You need to connect from another machine to use MNS.  The MNS provides a simple way to configure cameras, Groundlight detectors, and send notifications conditions are met.
+- **`desktop`** - an image with the Groundlight MNS installed, and a desktop GUI with a browser.  This is appropriate for a Raspberry Pi which will have a screen attached to it.
+- **`edge`** - Not available yet.  The Edge Endpoint server is still too resource hungry to run on a Raspberry Pi.  Please [leave a comment](https://github.com/groundlight/groundlight-pi-gen/issues/5) if you'd like to use this.
+
+
+## Source Code
+
+This build system is based on [pi-gen](https://github.com/RPi-Distro/pi-gen).  Refer to its [original README](/README.md) for how everything works.  The (`gl-config`)[gl-config] file is the key source of control.  (What is called "config" in the original.)
+
+Note that we're tracking the `arm64` branch, not main.  (If we build off the main branch, we hit [an issue with missing `arm/v8` docker images](https://github.com/groundlight/monitoring-notification-server/issues/39) and likely others, because we make these funky machines with a 64-bit kernel, but 32-bit applications.)
+
+### Stages
+
+- **`sdk-only`** - Saved after `stage-gl1`
+- **`mns-headless`** - Saved after `stage-gl2`
+- **`desktop`** - Saved after `stage4`
+
+Refer to the [`gl-config`](./gl-config) and [`gl-config-release`](./gl-config-release) files for the how the stages are used.
+
+
+## Building Images
+
+We recommend building on an ARM machine (like an m7g instance in ec2).  You can build on an x86 machine, but it will take significantly longer, and it's not fast to start with.  Building on ARM-powered Macs seems like a good idea, and "should" work, but isn't tested.
+
+### Building directly
+
+On the appropriate machine, run:
+
+```
+./dobuild.sh
+```
+
+This has code for both 
+- **local builds** which are faster, but require sudo, and maybe leak resources in a way that require rebooting the build machine.
+- **docker builds** which are slower, but don't require sudo, and don't leak resources.
+
+To re-use the cache from docker builds, run
+
+```
+CONTINUE=1 ./dobuild.sh
+```
+
+### But it's so SLOW!
+
+A full build can take 10s of minutes. But partial builds get cached in a docker volume and will speed things up dramatically. Beware that the caches don't get invalidated automatically or sensibly, so it will lead to problems as you're working on it.
+
+Also, the best way to speed things up is to skip building stages you don't care about (e.g. `stage3` the desktop environment).  You can do this without editing the `glmns-config` file with:
+
+```
+touch stage3/SKIP
+```
+
+Also, you can get a bit of a boost by skipping export of the `sdk` variant image:
+
+```
+touch stage-gl1/SKIP_IMAGES
+```
+
+
+### Troubleshooting
+
+To start over try
+
+```
+CLEAN=1 ./dobuild.sh
+```
+
+**Unmount errors** - try `sudo mv work deleteme-work`
+
+## Using the images
+
+After ~10 minutes, and then look in the `deploy/` for a file with a name like
+`image_2023-12-06-GroundlightMNS-sdk-qemu.img.xz` which will be ~1GB for now.
+(See the `COMPRESSION_LEVEL` setting in (`gl-config`)[gl-config] to trade speed vs size.)
+
+Copy this to your laptop, and then you can burn it to an SD card using the [Raspberry Pi Image](https://github.com/raspberrypi/rpi-imager).
+
+TODO: set up some tests inside `qemu` that things are working.
+TODO: write those tests into CI/CD actions.
+
+
+## Running the Raspberry Pi Image with QEMU 
+To emulate the Raspberry Pi image with QEMU, we will first need to install the Linux kernel for QEMU on the 
+host machine and install other necessary packages required for QEMU to work. To do this, run 
+
+```shell 
+./setup-qemu-kernel.sh 
+```
+
+This needs to be done only once. It will download a linux kernel(v6.6.8) of 40MB. 
+If you haven't decompressed the image you wan to use, you can do so by running
+
+```shell
+xz -d <path-to-compressed-image> 
+```
+
+To run the emulator, go ahead and run 
+
+```shell 
+./rpistart.sh -i <absolute-path-to-image>
+```
+
+You can SSH into it by running 
+
+```shell
+ssh -l pi localhost -p 2222
+```
+
+By default, the username and password are `pi` and `raspberry` respectively. You can overrride these by setting 
+the following environment variables before running the `rpistart.sh` script. 
+
+```shell
+export USERNAME=<username>
+export PASSWORD=<password>
+```
+
+
+## What's up with this file?
+
+This file is called `GL-README.md` and is elevated to the github repo homepage by a symlink `.github/README.md`.
+
+## Merging from upstream
+
+Here's the process
+
+```
+git remote add upstream https://github.com/RPi-Distro/pi-gen
+git fetch upstream
+git merge upstream/arm64
+```
+
+Then test test test.
diff --git a/build-docker.sh b/build-docker.sh
@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
-# Note: Avoid usage of arrays as MacOS users have an older version of bash (v3.x) which does not supports arrays
+# Note: Avoid usage of arrays as MacOS users have an older version of bash (v3.x) which does not support arrays
 set -eu
 
 DIR="$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)"
@@ -38,6 +38,12 @@ do
 	esac
 done
 
+# check that CONFIG_FILE exists and is readable
+if [ ! -r "${CONFIG_FILE}" ]; then
+  echo "Configuration file '${CONFIG_FILE}' does not exist or is not readable"
+  exit 1
+fi
+
 # Ensure that the configuration file is an absolute path
 if test -x /usr/bin/realpath; then
 	CONFIG_FILE=$(realpath -s "$CONFIG_FILE" || realpath "$CONFIG_FILE")

diff --git a/build.sh b/build.sh
@@ -3,7 +3,8 @@
 # shellcheck disable=SC2119
 run_sub_stage()
 {
-	log "Begin ${SUB_STAGE_DIR}"
+	log " "
+	log "---====> substage ${SUB_STAGE_DIR} <====---"
 	pushd "${SUB_STAGE_DIR}" > /dev/null
 	for i in {00..99}; do
 		if [ -f "${i}-debconf" ]; then
@@ -81,7 +82,9 @@ EOF
 
 
 run_stage(){
-	log "Begin ${STAGE_DIR}"
+	log " "
+	log "        --------========> STAGE: ${STAGE_DIR} <========----------"
+	log " "
 	STAGE="$(basename "${STAGE_DIR}")"
 
 	pushd "${STAGE_DIR}" > /dev/null

diff --git a/dobuild.sh b/dobuild.sh
@@ -0,0 +1,62 @@
+#!/bin/bash
+
+# local build - seems like maybe it's leaking system resources sometimes?
+# time sudo $@ ./build.sh -c gl-config
+# Instead we'll use docker build, which is slower but more reliable.
+# docker build - slower but more reliable.
+
+set -e  # exit on error
+
+# Move to this script's directory
+cd "$(dirname "$0")"
+
+# See if CLEAN=1 flag was set
+if [ "$CLEAN" = "1" ]; then
+    echo "Clearing out all previous build files"
+    sudo rm -rf ./deploy ./work
+    docker rm -v pigen_work || echo "Failed to remove pigen_work, because it doesn't exist, which is fine."
+fi
+
+# check if the IS_RELEASE variable is set to "1"
+if [ "$IS_RELEASE" = "1" ]; then
+    echo "Building release version"
+    CONFIG_FILE=gl-config-release
+    rm -rf ./deploy
+else
+    echo "Building dev version"
+    CONFIG_FILE=gl-config
+fi
+
+time PRESERVE_CONTAINER=1 $@ ./build-docker.sh -c $CONFIG_FILE
+
+# If it's a release, rename the image files to include the tag name
+# and take out the date.
+if [ "$IS_RELEASE" = "1" ]; then
+    TAG_NAME=${GITHUB_REF#refs/tags/}
+    # See if tag name is set
+    if [ -z "$TAG_NAME" ]; then
+        echo "No tag name set.  Expecting TAG_NAME for release buid."
+        exit 1
+    fi
+
+    cd deploy
+    # Loop over every file named "image_*.img.xz"
+    for file in ./image_*.img.xz; do
+        # Get the filename without the path
+        filename=$(basename -- "$file")
+
+        # Files are named like image_2023-12-10-GroundlightPi-sdk.img.xz
+        # Figure out the variant name by removing the image_date- prefix
+        variant=$(echo $filename | cut -d '-' -f5-)
+        # check that $variant is not empty, and does not include "Groundlight"
+        if [ -z "$variant" ] || [[ "$variant" == *"Groundlight"* ]]; then
+            echo "Failed to parse filename $filename - got $variant"
+            exit 1
+        fi
+        new_file="GroundlightPi-$TAG_NAME-$variant"
+
+        # Rename the file
+        echo "Renaming $file to $new_file"
+        mv "$file" "$new_file"
+    done
+fi
diff --git a/gl-config b/gl-config
@@ -0,0 +1,40 @@
+# The config file defining the Groundlight MNS image.
+IMG_NAME='GroundlightPi'
+TARGET_HOSTNAME=GroundlightPi
+FIRST_USER_NAME=groundlight
+
+#
+# core functionality
+#
+
+ENABLE_SSH=1
+RELEASE=bookworm
+# bookworm is newer and works. bullseye is older, and I think really doesn't.
+# Why is this even here?  Just as a reminder not to try bullseye without being careful.
+
+# stage0,1 set up basic linux.
+# stage2 gets us a working "lite" headless distribution w/o desktop
+# stage-gl1 adds the python SDK and supporting libraries like numpy, opencv, framegrab
+# stage-gl2 adds the MNS running in docker
+# stage3 adds basic desktop infra
+# stage4 makes the desktop usable with apps and such
+STAGE_LIST="stage0 stage1 stage2 stage-gl1 stage-gl2 stage3 stage4"
+
+#
+# Options used for development.
+# (These get overridden by gl-config-release)
+#
+
+# Makes a qemu-compatible image - this will help testing when we figure it out.
+USE_QEMU=1
+
+# faster but "experimental" and crashed when I tried it.
+#USE_QCOW2=1
+
+# Dev settings - compressing is quite slow unless you have lots of cores. (xz parallelizes nicely)
+# But big images are slow to download.
+DEPLOY_COMPRESSION=xz
+COMPRESSION_LEVEL=1
+#DEPLOY_COMPRESSION=none
+#COMPRESSION_LEVEL=0
+
diff --git a/gl-config-release b/gl-config-release
@@ -0,0 +1,16 @@
+source gl-config
+
+# Turn off QEMU for release builds
+USE_QEMU=0
+
+# Higher compression takes longer, so it's turned down during dev.
+# We should turn it back up when we're ready to publish.
+# For reference, with a 3.3GB base .img file 
+# Times are on a 4 core m7g instance, and include cached build.sh time
+# xz level 0 took 1m50s to compress to 1.1GB
+# xz level 1 took 2m20s to compress to 986MB
+# xz level 3 took 3m30s to compress to 950MB
+# xz level 9 took 7m30s to compress to 733MB
+# Publish settings:
+DEPLOY_COMPRESSION=xz
+COMPRESSION_LEVEL=9