Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alpine/Musl with Pact Rust based core, in Pact client libraries #30

Closed
47 of 53 tasks
YOU54F opened this issue Feb 8, 2024 · 2 comments
Closed
47 of 53 tasks

Alpine/Musl with Pact Rust based core, in Pact client libraries #30

YOU54F opened this issue Feb 8, 2024 · 2 comments

Comments

@YOU54F
Copy link
Member

YOU54F commented Feb 8, 2024

Status

Rust Libraries

library alpine support version
pact_ffi - x86_64 .a from 0.2.4
- aarch64 .a from 0.4.15
- .so from 0.4.17
pact_mock_server_cli - 1.0.5
pact_verifier_cli - 1.1.1
pact-stub-server - 0.6.0
pact-plugin-cli - 0.1.2
pact-protobuf-plugin - 0.3.15
pact-csv-plugin - 0.0.6

Note:- With exception to the pact_ffi library, the other libraries are provided as executables, statically linked to musl libc, which means they should work on all Linux distributions, both musl based (Alpine, Void, Chimera etc) & glibc based (Debian, RHEL etc)

Client Libraries

library alpine support version
pact-js PR 13.1.0
pact-php PR 10.0.0-beta1
pact-net WIP PR
pact-go WIP PR

Background

Many teams choose to run builds on Alpine Linux due to its tiny footprint, and smaller security surface area.

Guidance for Alpine users, has been stored on our docs site here

For Pact implementations that use the Ruby shared core

Support is available by installing glibc compatibility layers due the underlying runtime being built on centos7

https://github.com/pact-foundation/pact-ruby-standalone/wiki/Using-the-pact-ruby-standalone-with-Alpine-Linux-Docker

For Pact implementations that use the Rust shared core

Alpine or other musl based distros have not be supported due to

  1. Lack of musl based artifacts from pact-rust based projects
  2. Potential issues requiring the project to possibly be rebuilt against specific versions of Alpine / musl.

Point 1 is now addressed, partially for the FFI library.

Pact-Reference is now publishing musl based versions of the Pact-FFI as of 0.4.15

We would like to investigate creating musl variants of all the pact-reference (rust) based artifacts, and incorporating them into client libraries

Support Level

Any of the musl based artifacts should be considered somewhat experimental and you may encounter issues, as they have not been used for a long time. I would strongly advise these are not used in production workflows, unless you are handy and don't mind investigating issues yourself

We would gratefully ask that if issues are raised, that two things are provided

  1. A minimal reproducible example of a Docker image/file based on Alpine that recreates the issue
  2. A minimal reproducible example of a Docker image/file based on Debian that recreates the issue

Note - The debian image, may not show the same behaviour, this will be useful to determine where there are bugs/issues or gaps in the musl implementation which can be communicated to others.

How to participate?

Drop a comment in this thread, or reach out via slack in #libpact_ffi-users

FFI

Investigate the possibility of incorporating into client libraries

Pact-Js-Core

Snags

  1. pact ruby standalone arm64 fail under qemu in GHA which fails test suite
  2. some inconsistent behaviour in node 16 (set /tmp cache) / 18 ( max sockets 1)
  3. prebuilds take a good while, but these could be limited to only running when the ffi, or native bindings are updated, or failing that just run on pull requests. regular test runs could just grab the latest pre build

Pact-JS

  • Pact-JS

Pact-PHP

Snags

  1. PHP can only load .so files, so requires converted .a musl library
  2. AMD64 musl images require shared-mime-info installing for consistent content type matching, ARM64 doesn't. If installed, they flip behaviour.
  3. Needs musl detection method in downloader/

Pact-Go

  • Pact-Go
    • Support for musl - delta
    • CI Test runs

Snags

  1. Some seg fauls, also occur on vanilla x86_64 build although not consistently reproducible.

Pact-Net

Snags

  1. .NET can only load .so files, so requires converted .a musl library
  2. ARM64 musl on GHA fails to build project, but passes on Cirrus
  3. Needs musl detection method in csproj file

add others

Mock Server CLI

Provider Verifier CLI

Stub Server CLI

Plugins

pact plugin cli

  • fork release
  • issue building existing x86_64 linux executable in CI (build with cross to retain known glibc version)
  • pact plugin driver rust
  • pact plugin driver jvm
  • support downloading non musl variants if not avail, via pact-plugin-cli (rust and jvm driver)

Pact-Protobuf-Plugin

Pact-csv-Plugin

Pact-matt plugin

Build with goreleaser and linked with musl, so existing binaries work with alpine.

  • support downloading non musl variants if not avail, via pact-plugin-cli (rust and jvm driver)

Client language FFI bindings that only support dynamic *.so libaries

For some languages, it is not possible to load a *.a static library.

  • PHP
  • Ruby
  • Others?

It is possible however to convert a .a file to a .so file.

ar -x libpact_ffi.a
gcc -shared *.o -o libpact_ffi.so
rm -f *.o

This shrinks our .a file from 115mb to a circa 41mb .so file, which looks inline with the size of the musl node prebuild for pact-js

@YOU54F
Copy link
Member Author

YOU54F commented Mar 16, 2024

So that was a successful endeavour however, we are going to try a different approach.

The musl built artifacts with work on systems based on glibc, therefore we technically only need to build a single linux binary linked to musl.

This will reduce the need for determining glibc/musl for at least rust built executables.

It is still to be tested with the FFI, to determine if we can get away with just one set (musl based for all) We can't - we require independent ffi based linux artifacts for glibc and musl based targets

YOU54F added a commit to YOU54F/pact-stub-server that referenced this issue Apr 25, 2024
- create linux executables linked statically to musl
  - drop gnu (glibc) target, executable will work on alpine & glibc based distros (debian etc)
- create windows aarch64 target
  - required update to Cargo.lock to update ring to v0.17 due to build failures on v0.16
- refactor release script & workflow to match pact-reference
  - builds on pull request to master/main branch, in debug mode
- add Docker automated pipeline during release
  - publish multi-arch amd64/arm64
  - publish to DockerHub
  - publish to GitHub container registry

Related epics

- musl all the things
  - pact-foundation/roadmap#30
- publish to github container registry
  - pact-foundation/roadmap#29
YOU54F added a commit to pact-foundation/pact-stub-server that referenced this issue Apr 25, 2024
- create linux executables linked statically to musl
  - drop gnu (glibc) target, executable will work on alpine & glibc based distros (debian etc)
- create windows aarch64 target
  - required update to Cargo.lock to update ring to v0.17 due to build failures on v0.16
- refactor release script & workflow to match pact-reference
  - builds on pull request to master/main branch, in debug mode
- add Docker automated pipeline during release
  - publish multi-arch amd64/arm64
  - publish to DockerHub
  - publish to GitHub container registry

Related epics

- musl all the things
  - pact-foundation/roadmap#30
- publish to github container registry
  - pact-foundation/roadmap#29
YOU54F added a commit to pact-foundation/pact-plugins that referenced this issue Apr 25, 2024
- create linux executables linked statically to musl
  - drop gnu (glibc) target, executable will work on alpine & glibc based distros (debian etc)
- create windows aarch64 target
- refactor release script & workflow to match pact-reference
  - builds on pull request to master/main branch, in debug mode

Related epics

- musl all the things
  - pact-foundation/roadmap#30
YOU54F added a commit to YOU54F/pact-protobuf-plugin that referenced this issue Apr 30, 2024
- create linux executables linked statically to musl
  - drop gnu (glibc) target, executable will work on alpine & glibc based distros (debian etc)
- create windows aarch64 target
- refactor release script & workflow to match pact-reference
  - builds on pull request to master/main branch, in debug mode

Related epics

- musl all the things
  - pact-foundation/roadmap#30
YOU54F added a commit to YOU54F/pact-protobuf-plugin that referenced this issue Apr 30, 2024
- create linux executables linked statically to musl
  - drop gnu (glibc) target, executable will work on alpine & glibc based distros (debian etc)
- create windows aarch64 target
- refactor release script & workflow to match pact-reference
  - builds on pull request to master/main branch, in debug mode

Related epics

- musl all the things
  - pact-foundation/roadmap#30
YOU54F added a commit to YOU54F/pact-protobuf-plugin that referenced this issue Apr 30, 2024
- create linux executables linked statically to musl
  - drop gnu (glibc) target, executable will work on alpine & glibc based distros (debian etc)
- create windows aarch64 target
- refactor release script & workflow to match pact-reference
  - builds on pull request to master/main branch, in debug mode

Related epics

- musl all the things
  - pact-foundation/roadmap#30
YOU54F added a commit to YOU54F/pact-net that referenced this issue May 29, 2024
## Rationale

pact-reference has introduced musl and arm64 based ffi libraries for linux

- pact-foundation/pact-reference#416

Tracking Issue

- pact-foundation/roadmap#30

## Issues Resolved

fixes pact-foundation#498
fixes pact-foundation#496
fixes pact-foundation#500
fixes pact-foundation#374
fixes pact-foundation#387

## Backwards Compatibility

Linux glibc based hosts take precedence, so if any error occurs during musl
detection. I do not anticipate breaking changes for users

## Implementation notes

### .NET notes

- Docs
  - [Uses MSBuild Exec task](https://learn.microsoft.com/en-us/visualstudio/msbuild/exec-task?view=vs-2022)
- MSBuild Blog Posts
  - [Cross-Platform Build Events in .NET Core using MSBuild](https://jeremybytes.blogspot.com/2020/05/cross-platform-build-events-in-net-core.html)
  - [MSBuild 101: Using the exit code from a command](https://www.creepingcoder.com/2020/06/01/msbuild-101-using-the-exit-code-from-a-command/)
- Stack OverFlow
  - [Set PropertyGroup property to Exec output](https://stackoverflow.com/questions/76583824/set-propertygroup-property-to-exec-output)
- .NET runtime musl detection code
  - https://github.com/dotnet/runtime/blob/a50ba0669353893ca8ade8568b0a7d210b5a425f/src/mono/llvm/llvm-init.proj\#L7
  - https://github.com/dotnet/runtime/blob/a50ba0669353893ca8ade8568b0a7d210b5a425f/src/libraries/Common/tests/TestUtilities/System/PlatformDetection.Unix.cs\#L78t

### Conditions for execution

musl detection will run if

- if linux
- if /lib/ld-musl-(x86_64|aarch64).so.1 exists
- if ldd bin/sh | grep musl is true (musl lib is loaded, rather than glibc)

will continue on error, reverting back to glibc based libaries.

### Supported musl targets

should work for multiple musl based distroes if

- /lib/ld-musl-(x86_64|aarch64).so.1 exists
- ldd is available (available by default in alpine images)

Tested on Alpine ARM64 / AMD64.

## Caveats

- [.NET does not run under QEMU](https://github.com/dotnet/core/blob/main/release-notes/8.0/supported-os.md#qemu) affecting the ability to test multi-arch from a single system
- .NET restore can take a long time when running under containers.
  - [Workaround](NuGet/Home#13062 (comment)): Set `DOTNET_NUGET_SIGNATURE_VERIFICATION` to `false`

## Compatibility

### Operating System

Due to using a shared native library instead of C# for the main Pact logic only certain OSs are supported:

| OS           | Arch        | Support                                                            |
| ------------ | ----------- | -------------------------------------------------------------------|
| Windows      | x86         | ❌ No                                                              |
| Windows      | x64         | ✔️ Yes                                                              |
| Linux (libc) | x86         | ❌ No                                                              |
| Linux (libc) | x64         | ✔️ Yes                                                              |
| Linux (musl) | x64         | ✔️ Yes (Tier 2)*                                                    |
| Linux (libc) | ARM         | ✔️ Yes (Tier 3)*                                                    |
| Linux (musl) | ARM         | ✔️ Yes (Tier 3)*                                                    |
| OSX          | x64         | ✔️ Yes                                                              |
| OSX          | ARM (M1/M2) | ✔️ Yes                                                              |

#### Support

- Tier 1
  - Established
  - Full CI/CD support.
  - Users should not encounter issues
  - Full reproducible examples running in CI, should be provided by users raising issues
  - If using musl targets, users should attempt the same test on a libc target (such as debian)
- Tier 2
  - Recently introduced
  - Full CI/CD support.
  - Users may encounter issues
  - Full reproducible examples running in CI, should be provided by users raising issues
  - If using musl targets, users should attempt the same test on a libc target (such as debian)
- Tier 3
  - Recently introduced, No/limited CI/CD support.
  - Users may encounter issues
  - Full reproducible examples which can be run by maintainers locally, should be provided by users raising issues
@YOU54F
Copy link
Member Author

YOU54F commented Sep 16, 2024

Only two left to go are pact-net and pact-go, with RFC's/PR's raised.

Will close this off

@YOU54F YOU54F closed this as completed Sep 16, 2024
YOU54F added a commit to YOU54F/pact-net that referenced this issue Sep 24, 2024
## Rationale

pact-reference has introduced musl and arm64 based ffi libraries for linux

- pact-foundation/pact-reference#416

Tracking Issue

- pact-foundation/roadmap#30

## Issues Resolved

fixes pact-foundation#498
fixes pact-foundation#496
fixes pact-foundation#500
fixes pact-foundation#374
fixes pact-foundation#387

## Backwards Compatibility

Linux glibc based hosts take precedence, so if any error occurs during musl
detection. I do not anticipate breaking changes for users

## Implementation notes

### .NET notes

- Docs
  - [Uses MSBuild Exec task](https://learn.microsoft.com/en-us/visualstudio/msbuild/exec-task?view=vs-2022)
- MSBuild Blog Posts
  - [Cross-Platform Build Events in .NET Core using MSBuild](https://jeremybytes.blogspot.com/2020/05/cross-platform-build-events-in-net-core.html)
  - [MSBuild 101: Using the exit code from a command](https://www.creepingcoder.com/2020/06/01/msbuild-101-using-the-exit-code-from-a-command/)
- Stack OverFlow
  - [Set PropertyGroup property to Exec output](https://stackoverflow.com/questions/76583824/set-propertygroup-property-to-exec-output)
- .NET runtime musl detection code
  - https://github.com/dotnet/runtime/blob/a50ba0669353893ca8ade8568b0a7d210b5a425f/src/mono/llvm/llvm-init.proj\#L7
  - https://github.com/dotnet/runtime/blob/a50ba0669353893ca8ade8568b0a7d210b5a425f/src/libraries/Common/tests/TestUtilities/System/PlatformDetection.Unix.cs\#L78t

### Conditions for execution

musl detection will run if

- if linux
- if /lib/ld-musl-(x86_64|aarch64).so.1 exists
- if ldd bin/sh | grep musl is true (musl lib is loaded, rather than glibc)

will continue on error, reverting back to glibc based libaries.

### Supported musl targets

should work for multiple musl based distroes if

- /lib/ld-musl-(x86_64|aarch64).so.1 exists
- ldd is available (available by default in alpine images)

Tested on Alpine ARM64 / AMD64.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant