Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update build system for Linux, macOS, overhaul GitHub Actions, remove serial driver #37

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
7 changes: 7 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
version: 2
updates:
# Dependabot updates for GitHub Actions
- package-ecosystem: github-actions
directory: "/"
schedule:
interval: "weekly"
193 changes: 150 additions & 43 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
@@ -1,96 +1,203 @@
name: Build

on:
push:
branches:
- 'main'
pull_request:
branches:
- '*'
[push, pull_request]

concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
cancel-in-progress: false

defaults:
run:
shell: bash

jobs:
# Use native runners to build for Windows, Linux x86-64, and macOS; use WPI provided Docker images to build for Linux ARM32 and ARM64 platforms
build:
timeout-minutes: 15
strategy:
fail-fast: false
matrix:
include:
- os: windows-latest
container: ''
name: windows64
name: "build-${{ matrix.name }}"
name: Win64
build-options: ""
platform-type: windowsx86-64
- os: ubuntu-latest
name: Linux64
platform-type: linuxx86-64
build-options: ""
- os: macos-latest
name: macOS
platform-type: osxuniversal
build-options: ""
- container: wpilib/aarch64-cross-ubuntu:bullseye-22.04
os: ubuntu-latest
name: LinuxARM64
build-options: "-Ponlylinuxarm64"
platform-type: linuxarm64
arch: arm64
- container: wpilib/raspbian-cross-ubuntu:bullseye-22.04
os: ubuntu-latest
name: LinuxARM32
build-options: "-Ponlylinuxarm32"
platform-type: linuxarm32
arch: arm32
name: "Build - ${{ matrix.name }}"
runs-on: ${{ matrix.os }}
container: ${{ matrix.container }}
steps:
- name: Checkout
uses: actions/checkout@v3
uses: actions/checkout@v4
with:
ref: ${{ github.sha }}

- name: Setup Java
uses: actions/setup-java@v3
uses: actions/setup-java@v4
with:
distribution: 'zulu'
java-version: 11

- name: Setup Gradle
uses: gradle/actions/setup-gradle@v4

- name: Build
run: |
./gradlew outputVersions publish ${{ matrix.build-options }} -PreleaseMode

- name: Download WPILib HAL artifacts and headers, gather all needed headers
- name: Download WPILib HAL artifacts and headers for ${{ matrix.platform-type }}
run : |
halVersion=$(cat wpiHalVersion.txt)

sharedHalPlatformUrl=https://frcmaven.wpi.edu/artifactory/release/edu/wpi/first/hal/hal-cpp/"$halVersion"/hal-cpp-"$halVersion"-${{ matrix.platform-type }}.zip
sharedUtilPlatformUrl=https://frcmaven.wpi.edu/artifactory/release/edu/wpi/first/wpiutil/wpiutil-cpp/"$halVersion"/wpiutil-cpp-"$halVersion"-${{ matrix.platform-type }}.zip
curl -L -o sharedHalPlatform.zip "$sharedHalPlatformUrl"
curl -L -o sharedUtilPlatform.zip "$sharedUtilPlatformUrl"

# Download WPILib artifacts from Artifactory
halWindowsUrl=https://frcmaven.wpi.edu/artifactory/release/edu/wpi/first/hal/hal-cpp/"$halVersion"/hal-cpp-"$halVersion"-windowsx86-64.zip
staticHalPlatformUrl=https://frcmaven.wpi.edu/artifactory/release/edu/wpi/first/hal/hal-cpp/"$halVersion"/hal-cpp-"$halVersion"-${{ matrix.platform-type }}static.zip
staticUtilPlatformUrl=https://frcmaven.wpi.edu/artifactory/release/edu/wpi/first/wpiutil/wpiutil-cpp/"$halVersion"/wpiutil-cpp-"$halVersion"-${{ matrix.platform-type }}static.zip
curl -L -o staticHalPlatform.zip "$staticHalPlatformUrl"
curl -L -o staticUtilPlatform.zip "$staticUtilPlatformUrl"

- name: Unzip WPILib HAL artifacts and headers
run: |
unzip sharedHalPlatform.zip -d sharedHalPlatform
unzip sharedUtilPlatform.zip -d sharedUtilPlatform
unzip staticHalPlatform.zip -d staticHalPlatform
unzip staticUtilPlatform.zip -d staticUtilPlatform
mkdir -p CANBridge-artifacts/static
mkdir -p CANBridge-artifacts/shared

# Put release files together in one directory based on platform
- name: Create Artifact
run: |
mkdir -p CANBridge-artifacts
if [[ "${{ matrix.platform-type }}" == "windowsx86-64" ]]; then
cp build/libs/cANBridge/shared/windowsx86-64/release/CANBridge.dll CANBridge-artifacts/shared/
cp build/libs/cANBridge/shared/windowsx86-64/release/CANBridge.lib CANBridge-artifacts/shared/
cp sharedHalPlatform/windows/x86-64/shared/wpiHal.dll CANBridge-artifacts/shared/
cp sharedHalPlatform/windows/x86-64/shared/wpiHal.lib CANBridge-artifacts/shared/
cp sharedUtilPlatform/windows/x86-64/shared/wpiutil.dll CANBridge-artifacts/shared/
cp sharedUtilPlatform/windows/x86-64/shared/wpiutil.lib CANBridge-artifacts/shared/

cp build/libs/cANBridge/static/windowsx86-64/release/CANBridge.lib CANBridge-artifacts/static/
cp staticHalPlatform/windows/x86-64/static/wpiHal.lib CANBridge-artifacts/static/
cp staticUtilPlatform/windows/x86-64/static/wpiutil.lib CANBridge-artifacts/static/
elif [[ "${{ matrix.platform-type }}" == "linuxx86-64" ]]; then
cp build/libs/cANBridge/shared/linuxx86-64/release/libCANBridge.so CANBridge-artifacts/shared/
cp sharedHalPlatform/linux/x86-64/shared/libwpiHal.so CANBridge-artifacts/shared/
cp sharedUtilPlatform/linux/x86-64/shared/libwpiutil.so CANBridge-artifacts/shared/

cp build/libs/cANBridge/static/linuxx86-64/release/libCANBridge.a CANBridge-artifacts/static/
cp staticHalPlatform/linux/x86-64/static/libwpiHal.a CANBridge-artifacts/static/
cp staticUtilPlatform/linux/x86-64/static/libwpiutil.a CANBridge-artifacts/static/
elif [[ "${{ matrix.platform-type }}" == "osxuniversal" ]]; then
cp build/libs/cANBridge/shared/osxuniversal/release/libCANBridge.dylib CANBridge-artifacts/shared/
cp sharedHalPlatform/osx/universal/shared/libwpiHal.dylib CANBridge-artifacts/shared/
cp sharedUtilPlatform/osx/universal/shared/libwpiutil.dylib CANBridge-artifacts/shared

cp build/libs/cANBridge/static/osxuniversal/release/libCANBridge.a CANBridge-artifacts/static/
cp staticHalPlatform/osx/universal/static/libwpiHal.a CANBridge-artifacts/static/
cp staticUtilPlatform/osx/universal/static/libwpiutil.a CANBridge-artifacts/static/
elif [[ "${{ matrix.platform-type }}" == "linuxarm32" || "${{ matrix.platform-type }}" == "linuxarm64" ]]; then
cp build/libs/cANBridge/shared/release/libCANBridge.so CANBridge-artifacts/shared/libCANBridge.so
cp sharedHalPlatform/linux/${{ matrix.arch }}/shared/libwpiHal.so CANBridge-artifacts/shared/libwpiHal.so
cp sharedUtilPlatform/linux/${{ matrix.arch }}/shared/libwpiutil.so CANBridge-artifacts/shared/libwpiutil.so

cp build/libs/cANBridge/static/release/libCANBridge.a CANBridge-artifacts/static/libCANBridge.a
cp staticHalPlatform/linux/${{ matrix.arch }}/static/libwpiHal.a CANBridge-artifacts/static/libwpiHal.a
cp staticUtilPlatform/linux/${{ matrix.arch }}/static/libwpiutil.a CANBridge-artifacts/static/libwpiutil.a
fi

# Upload build artifact
- name: Upload build artifact
uses: actions/upload-artifact@v4
with:
name: CANBridge-${{ matrix.platform-type }}-${{ github.sha}}
path: CANBridge-artifacts/

# Upload combined headers for WPILib from HAL and WPIUtil
wpi-headers:
runs-on: ubuntu-latest
name: "WPILib Headers"
steps:
- name: Checkout
uses: actions/checkout@v4
with:
ref: ${{ github.sha }}
- name: Download WPILib HAL artifacts and headers
run : |
halVersion=$(cat wpiHalVersion.txt)

halHeadersUrl=https://frcmaven.wpi.edu/artifactory/release/edu/wpi/first/hal/hal-cpp/"$halVersion"/hal-cpp-"$halVersion"-headers.zip
utilWindowsUrl=https://frcmaven.wpi.edu/artifactory/release/edu/wpi/first/wpiutil/wpiutil-cpp/"$halVersion"/wpiutil-cpp-"$halVersion"-windowsx86-64.zip
utilHeadersUrl=https://frcmaven.wpi.edu/artifactory/release/edu/wpi/first/wpiutil/wpiutil-cpp/"$halVersion"/wpiutil-cpp-"$halVersion"-headers.zip
curl -L -o halWindows.zip "$halWindowsUrl"

curl -L -o halHeaders.zip "$halHeadersUrl"
curl -L -o utilWindows.zip "$utilWindowsUrl"
curl -L -o utilHeaders.zip "$utilHeadersUrl"
unzip halWindows.zip -d halWindows

- name: Unzip WPILib HAL artifacts and headers
run: |
unzip halHeaders.zip -d halHeaders
unzip utilWindows.zip -d utilWindows
unzip utilHeaders.zip -d utilHeaders

# Gather all of the the needed headers
- name: Gather all needed headers
run: |
mkdir headers-for-artifact
cp -r halHeaders/hal headers-for-artifact
cp -r utilHeaders/wpi headers-for-artifact
cp -r src/main/native/include/* headers-for-artifact

# Zip the needed headers and put them in the appropriate location for artifact upload
mkdir -p CANBridge-artifacts
7z a CANBridge-artifacts/headers.zip ./headers-for-artifact/*

# Put release files together in one directory
- name: Create Artifact
run: |
mkdir -p CANBridge-artifacts
cp build/libs/cANBridge/static/windowsx86-64/release/CANBridge.lib CANBridge-artifacts/CANBridge-static.lib
cp build/libs/cANBridge/shared/windowsx86-64/release/CANBridge.dll CANBridge-artifacts/CANBridge.dll
cp build/libs/cANBridge/shared/windowsx86-64/release/CANBridge.lib CANBridge-artifacts/CANBridge.lib
cp halWindows/windows/x86-64/shared/wpiHal.dll CANBridge-artifacts/wpiHal.dll
cp halWindows/windows/x86-64/shared/wpiHal.lib CANBridge-artifacts/wpiHal.lib
cp utilWindows/windows/x86-64/shared/wpiutil.dll CANBridge-artifacts/wpiutil.dll
cp utilWindows/windows/x86-64/shared/wpiutil.lib CANBridge-artifacts/wpiutil.lib

# Upload build artifact
- name: Upload build artifact
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: CANBridge-${{ github.sha }}
path: CANBridge-artifacts/
path: headers-for-artifact
name: headers

# Upload version file, used for versioning
version:
runs-on: ubuntu-latest
name: "Version"
steps:
- name: Checkout
uses: actions/checkout@v4
with:
ref: ${{ github.sha }}

- name: Setup Java
uses: actions/setup-java@v4
with:
distribution: 'zulu'
java-version: 11

- name: Setup Gradle
uses: gradle/actions/setup-gradle@v4

- name: Build
run: |
./gradlew outputVersions -PreleaseMode

# Upload version.txt
- name: Upload version artifact
uses: actions/upload-artifact@v3
- name: Upload build artifact
uses: actions/upload-artifact@v4
with:
name: version
path: build/allOutputs/version.txt
66 changes: 40 additions & 26 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,72 +3,86 @@ name: Create release
on:
push:
tags:
- 'v*'
- 'v**'

permissions:
contents: write

concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
cancel-in-progress: false

defaults:
run:
shell: bash

jobs:
check-versions:
# Checks previous build workflow and gets the version from publish.gradle
check-build:
name: Check build and publish versions
runs-on: ubuntu-latest
outputs:
TAG_NAME: ${{ env.TAG_NAME }}
VERSION: ${{ steps.get_version.outputs.version }}
steps:
- name: Wait for build to finish
uses: lewagon/[email protected].1
- name: Wait for build workflow to finish
uses: lewagon/[email protected].4
with:
ref: ${{ github.ref }}
check-name: 'build-windows64'
check-regexp: 'Build|WPILib Headers|Version'
repo-token: ${{ secrets.GITHUB_TOKEN }}
wait-interval: 10
- name: Get tag name
run: |
echo "TAG_NAME=${GITHUB_REF#refs/*/}" >> $GITHUB_ENV

# Download artifacts from build workflow
- name: Download workflow artifacts
uses: dawidd6/action-download-artifact@v2
uses: dawidd6/action-download-artifact@v6
with:
workflow: build.yml
commit: ${{ github.sha }}
path: '.'

# Get publish.gradle version
- name: Get publish.gradle version
id: get_version
id: get-version
run: |
echo "version=$(cat version/version.txt)" >> $GITHUB_OUTPUT
echo "expectedTagName=v$(cat version/version.txt)" >> $GITHUB_OUTPUT

# Check publish.gradle version
- name: publish.gradle version check FAILED
if: ${{ steps.get_version.outputs.expectedTagName != env.TAG_NAME }}
# Check if the publish.gradle version matches the tag name
- name: Check version
run: |
echo Tag name: ${{ env.TAG_NAME }}
echo publish.gradle version: ${{ steps.get_version.outputs.version }}
exit 1
if [[ "${{ steps.get-version.outputs.expectedTagName }}" != "${{ env.TAG_NAME }}" ]]; then
echo "Version mismatch: ${{ steps.get-version.outputs.expectedTagName }} != ${{ env.TAG_NAME }}"
exit 1
fi

# Creates a release draft with the artifacts from the build workflow
prepare-release:
name: Prepare release
runs-on: ubuntu-latest
needs: check-versions
needs: check-build
steps:
# Download API, docs, and version.txt
- name: Download workflow artifacts
uses: dawidd6/action-download-artifact@v2
uses: dawidd6/action-download-artifact@v6
with:
workflow: build.yml
commit: ${{ github.sha }}
path: '.'
skip_unpack: true

# This step is to check what files are downloaded and how they are structured, as well as binary sizes for releases
- name: List files
run: |
ls -Rlh

# Create new release draft
- name: Create release
id: create_release
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
VERSION=${{ needs.check-versions.outputs.version }}
TAG=v$VERSION
ls --recursive -l
gh release create $TAG CANBridge-${{ github.sha }}/* --repo $GITHUB_REPOSITORY --draft --title "Version $VERSION"
uses: softprops/action-gh-release@v2
with:
draft: true
generate_release_notes: true
tag_name: v${{ steps.get-version.outputs.version }}
name: Version ${{ steps.get-version.outputs.version }}
files: |
**/**
3 changes: 3 additions & 0 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,9 @@ model {
}
}
binaries.all {
if (it.targetPlatform.name == 'osxuniversal') {
linker.args '-framework', 'IOKit'

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this necessary with the removal of the serial driver?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not necessary as the serial driver is removed. However, building back with SocketCAN or some other framework with CAN would be beneficial to leave in. There is no hurting with keeping it here.

Copy link
Contributor

@ItsHarper ItsHarper Nov 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that true if we end up using DriverKit instead of IOKit? According to this page, DriverKit is an IOKit replacement: https://developer.apple.com/system-extensions/

DriverKit provides a fully modernized replacement for IOKit to create device drivers.

Copy link
Contributor

@ItsHarper ItsHarper Nov 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

USBDriverKit certainly sounds like what we want: https://developer.apple.com/documentation/usbdriverkit

Copy link
Contributor

@ItsHarper ItsHarper Nov 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at the transcript of the video in that first link, there's a lot of emphasis on DriverKit being better than kernel extensions, which I'd think never would have been necessary in the first place for what we're doing.

It seems like DriverKit would require a separate System Extension that would handle all direct communication with the USB device, and then CANBridge would communicate with our DriverKit driver. That would be fine, but I think we'd have to design a whole API between our driver and CANBridge, which does sound like more work.

IOKit (and specifically IOUSBLib.h) on the other hand looks like a model more similar to what we're doing on Windows (raw USB transactions through WinUSB). The "Important" box on this page makes me a bit nervous, but it's really not clear what it's trying to say:

Devices supported on macOS 11 and later require DriverKit. Use IOKit in your apps and services to discover and use devices.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just noticed this in the DriverKit presentation transcript:

The DriverKit APIs are an extension of the IOKit APIs to userspace. And we have collected them into a new DriverKit SDK that is separate to the macOS SDK. This SDK has a limited API surface for reliability and security and there is no direct access to the file system, networking, or mock messaging. This allows Apple to tailor the userspace process to running drivers and can give it elevated priority and increased capabilities. So, let's talk about some of the classes in the DriverKit SDK. First, the IOService class exists in DriverKit and is very similar to the IOKit class. There also IOMemoryDescriptor and IOBufferMemoryDescriptor classes available that are, again, very similar to IOKit. We also have replacements for the IOWorkLoop and EventSource classes in IOKit. And finally, there's a new class called OSAction that is required to represent a C Function Pointer.

So yeah, I think IOKit is probably the way to go if we don't need any kind of extra OS-level permissions like a kext would have.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So from reading about macOS drivers as a whole when I made the PR, its a mess at the moment on which framework to use.

IOKit is only available on macOS but is being given the "deprecated but not deprecated" treatment. Those drivers run as kernel extensions and because in the name of security and stability those are getting moved over to DriverKit which solves the security and stability issues. Lots of documentation for how to do things since its been around for so much longer, but Apple has other ideas on what to do.

If we want maintained support for the future, IOKit will be discouraged from developing for in favor of using DriverKit.

Could be totally misinterpreting this as there was a whole macOS update that happened that added more things to DriverKit that I haven't dived into the docs for but the writing on the wall looks to be there.

}
if (it.targetPlatform.name == nativeUtils.wpi.platforms.roborio) {
it.buildable = false
}
Expand Down
3 changes: 1 addition & 2 deletions config.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,12 @@ nativeUtils {
// When updating WPILib, be sure to also update wpiHalVersion.txt
wpiVersion = "2023.+"
niLibVersion = "2023.3.0"
googleTestVersion = "1.11.0-3"
googleTestVersion = "1.11.0-4"
}
}
}

nativeUtils.wpi.addWarnings()
nativeUtils.wpi.addWarningsAsErrors()

nativeUtils.setSinglePrintPerPlatform()

Expand Down
Empty file modified gradlew
100644 → 100755
Empty file.
Loading