Add an aarch64 linux binary to all new releases

[afinetooth]

Closes:

• Can not use on Linux Arm #137
• Support GitHub Actions ARM runners github-action#218

---

Add an aarch64 linux binary to all new releases

Cross-compiling coverage-reporter for aarch64 proved challenging. We tried several approaches from the Crystal community, but settled on this crystal-xbuild containers method, which produced our first success.

---

☑️ Checklist for crystal-xbuild containers method:

• Start with example repo: https://github.com/luislavena/crystal-xbuild-container
• Modify components as required for coverage-reporter context
• Extend approach to CI/CD for releases

🛠️ Usage:

1. Build the xbuild-container:

make build-xbuild-container

2. Run the build command(s):

These commands cross-compile coverage-reporter for different architectures; right now just: x86_64 and aarch64.

For convenience, a single command lets us compile and strip all the linux binaries we need:

make compile-and-strip-all

• These sub-targets compile coverage-reporter for specific architectures:

  make compile-x86_64
  make compile-aarch64
  

• Similar targets strip each binary:

  make strip-x86_64
  make strip-aarch64
  

3. Package the binaries for distribution

This command packages the binaries for a release:

make package

Note: The release procedure itself is handled by the release job in our .github/workflows/build.yml workflow, which is triggered when we push a new tag.

✅ Testing the binaries:

While you can examine the binaries on your local machine:

% file coveralls-linux-aarch64 
coveralls-linux-aarch64: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, stripped

You probably can't run them due to an architecture mismatch:

For instance, on MacOS (arm64) I get an exec format error when I try to run the linux aarch64 executable:

% ./coveralls-linux-aarch64 
zsh: exec format error: ./coveralls-linux-aarch64

So, to create a test environment, you can run the original crystal-xbuild container, with this command:

make run-xbuild-container

Or, better, run one of these two (2) architecture-specific Ubuntu containers, with these commands:

make ubuntu-x86_64
make ubuntu-aarch64

Example:

Testing the aarch64 executable in an Ubuntu 22.04 (aarch64) container:

% make ubuntu-aarch64
docker run -it --rm --platform linux/aarch64 -v .:/app -w /app ubuntu:22.04 bash -i

root@a5037a257192:/app# ./build/aarch64-linux-musl/coveralls-linux-aarch64
 
⠀⠀⠀⠀⠀⠀⣿
⠀⠀⠀⠀⠀⣼⣿⣧⠀⠀⠀⠀⠀⠀⠀ ⣠⣶⣾⣿⡇⢀⣴⣾⣿⣷⣆ ⣿⣿⠀⣰⣿⡟⢸⣿⣿⣿⡇ ⣿⣿⣿⣷⣦⠀⠀⢠⣿⣿⣿⠀⠀⣿⣿⠁⠀⣼⣿⡇⠀⢀⣴⣾⣿⡷
⠶⣶⣶⣶⣾⣿⣿⣿⣷⣶⣶⣶⠶  ⣸⣿⡟ ⠀⢠⣿⣿⠃⠈⣿⣿⠀⣿⣿⢠⣿⡿⠀⣿⣿⣧⣤⠀⢸⣿⡇⣠⣿⡿⠀⢠⣿⡟⣿⣿⠀⢸⣿⡿⠀⠀⣿⣿⠃⠀⢸⣿⣧⣄
⠀⠀⠙⢻⣿⣿⣿⣿⣿⡟⠋⠁⠀⠀ ⣿⣿⡇⠀ ⢸⣿⣿⠀⣸⣿⡟⠀⣿⣿⣾⡿⠁ ⣿⣿⠛⠛⠀⣿⣿⢿⣿⣏⠀⢀⣿⣿⣁⣿⣿⠀⣾⣿⡇⠀⢸⣿⡿⠀⠀⡀⠙⣿⣿⡆
⠀⠀⢠⣿⣿⣿⠿⣿⣿⣿⡄⠀⠀⠀ ⠙⢿⣿⣿⠇⠈⠿⣿⣿⡿⠋⠀⠀⢿⣿⡿⠁⠀⢸⣿⣿⣿⡇⢸⣿⣿⠀⣿⣿⣄⣾⣿⠛⠛⣿⣿⢠⣿⣿⣿ ⣼⣿⣿⣿ ⣿⣿⡿⠋⠀
⠀⢀⣾⠟⠋⠀⠀⠀⠙⠻⣷⡀⠀⠀
 
  v0.6.14

[...]

Note:

In our final release artifacts, tarballs of each architecture-specific binary will untar to a binary named coveralls so it's ready to use with commands, coveralls report, coveralls done, etc.

---

📖 About the crystal-xbuild containers method

The approach was created by Crystal community member, luislavena, and is covered in his 2-part blog post (Part 1 | Part 2). It aligns with best practices offered by the official Crystal Doc on Static Linking (which highlights known difficulties in compiling static binaries of Crystal apps).

Note: This approach does not make use of Docker's buildx extension for cross-compilation. Instead, "xbuild" here is a generic term that stands for "cross-build."

Components:

1. Cross-compilation via Docker: The crystal-xbuild container is a pre-configured environment for cross-compiling multi-architecture builds, targeting x86_64, aarch64, and arm64.

   • Base image: crystal-xbuild is based on a custom docker image named hydrofoil-crystal:
     • An "[o]pinionated, Alpine-based development container for Crystal" --luislavena
     • It comes pre-installed with the Crystal compiler and the zig cross-compilation tool.

2. Key scripts:

   • xbuild.sh: Handles the core cross-compilation logic with a modular approach:
     • By architecture: Ensures architecture-specific configurations, libraries, and linking requirements are handled separately, minimizing risks of compilation errors. Makes platform-specific adjustments based on known limitations and requirements, (e.g., including -lunwind for musl, libiconv for macOS).
     • By stage: The generation of object files before final linking allows for cleaner architecture-specific management, with fewer errors during cross-compilation. The use of zig cc simplifies linking multiple architecture-specific libraries
   • Dockerfile: Installs the dependencies required for cross-compilation into the base image (zig, etc.)
   • Makefile: Holds convenient targets for building and testing our new cross-platform binaries.
   • homebrew-downloader.cr: Not used. Fetches the MacOS libraries required for cross-compliation from Homebrew's API.
     • We still rely on our current homebrew release process leveraging the homebrew-bump-formula action.

   Core differentiator: Zig

   Zig is the standout differentiator for this approach. It has robust support for cross-compilation on multiple platforms and simplifies the build process. Read more here and here.

⏭️ Pecadillos

Known Issues / Opportunities:

1. Unavoidable warning on compilation of aarch64 binary - Harmless but annoying. Relates to clang version used in zig right now. Tracking issue here. See comment on compile-aarch64 target in Makefile for more.
2. crystal build vs. shards build - We would prefer to use of shards build --production instead of crystal build --release as used currently in scripts/xbuild.sh, but it's a low priority for now since our stripped binaries are ~7.5MB each.
3. Registry dependency - While hydrofoil-crystal is hosted at GitHub Packages and is not likely to disappear anytime soon, it might be wise to maintain our own registry of base images.

[coveralls-official]

Pull Request Test Coverage Report for Build 10966290363

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 93.983%

---

Totals
Change from base Build 10604607943:	0.0%
Covered Lines:	906
Relevant Lines:	964

---

💛 - Coveralls