Setup pipeline that builds release artefacts

[bowd]

Summary

As part of our Release Process we need to setup a pipeline that automatically outputs release artefacts based on the code in our release/<version> and master branches.

Details

The current proposal is that we do this in Google Cloud Build, in a project managed by cLabs with binaries stored in Cloud Storage and Docker images published to Container Registry.

Potential tension: Maybe we should think about using OSS tools for our CI pipeline and have that all out in the open. It looks like there are various ways of automatic this with travis/circle/github actions that also sets up the actual release on github with the binaries.

Binaries

Binaries should be created for the following systems:

• Linux
• Darwin
• Windows

And the following platforms:

• 386
• amd64

Resulting in 6 binary artefacts of the form:

celo-blockchain-vX.Y.Z-stable-{system}-{plaftorm}.{exe?}.tar.gz

Optional/future tasks:

• Extend celotooljs functionality to allow signing release binaries and uploading the signatures for ease of use.

Docker Images

Docker images will be built and pushed to our registry and tags will be updated.

Optional/future tasks:

• Extend celotooljs functionality to allow signing of docker releases via the tar export mechanism (docker save us.gcr.io/celo-testnet/celo-node:vX.Y.Z)

[nategraf]

A couple of clarifications on what I was thinking:

• I propose we continue to use Cloud Build for now because it's the system that cLabs engineers are familiar with, and it is very low maintenance. In the future, we developers from outside cLabs are more involved, we will definitely want to migrate to an OSS build pipeline, but I would recommend against trying to migrate now.
• On creating a celotool command for build signing, I would say that likely is not needed, but if it turns out to be more complex than I am thinking we could. With the image or binary downloaded, a signature can be produced with a single gpg command. I've included some more info in my PR Add OpenPGP keys for [email protected] and add publishing guide celo-monorepo#4089
• Docker tags for the automated image builds, like we have now, should be just the commit hash. Tags for release versions will be added to the commit hash tagged image as part of the release process.

[bowd]

👍 on first two points.

Regarding Docker tags just to make sure I understand, when you say the release version will be added to the image as part of the release process you mean manually right? I.e. the automated build that happens will just upload the image tagged with the commit hash, which is something that's already happening.

Also, where is this current build happening at the moment? Is there a specific google cloud project that already has some CI steps setup for this which I should extend? found it

[bowd]

I've been finding my way through the jungle here and I wanted to document some findings / current issues I'm working through so that this information lives somewhere.

XGo

XGo is a tool used for cross-compiling golang packages, it was developed by karalabe (geth maintainer) specifically for geth and hasn't been really updated. I've spent a while to understand how xgo is used and it works. This tool is a bit poorly designed. If you're running it inside of a specific container it compiles in that context, but if it detects that it's not in the specific container it tries to use docker to execute itself in a container. So essentially you have a binary that when run either does the work or tries to spin up a docker container in which it runs itself with the same arguments. This duality makes it confusing and it's not really well explained. People on the internet had been trying to fix this with a docker-in-docker setup, I also attempted that until I really understood how everything works.

There's also now a fork of the package that's a bit more maintained because karalabe has kinda abandoned it, but the old version is still used inside of geth a lot in the way the build/ci.go script is setup.

There's some extra issues that came up because of how we do the build/env.sh workspace with symlinks.

Compiling for Windows

Ut looks like geth hasn't been using xgo to compile for windows in a while, there are some issues around this and I haven't managed to fix this even with updated dependencies, but I'm still working through this. They have been using appveoyor instead for their official binaries.

Compiling for Darwin

I've been hitting some issues here as well because of a fork of a dependency we're maintaining that includes some iOS specific headers when compiling for darwin. My intuition is that we need to treat compiling for darwin-ios and darwin-macos differently but I don't have experience with this.

After talking with @jeanregisser he pointed out that we can get rid of that workaround if we apply this patch from upstream geth. It would allows to revert to using the main gosigar instead of our fork. And that would probably fix this issues as well.

What has worked so far, what's left

I did manage to setup a Dockerfile that builds the bls rust package with all 6 compilation targets, and then create an image that builds for linux/386 and linux/amd64.

Once I manage to hopefully fix the the other compilation targets this Dockerfile will be able to build all binaries. At that point I would need to:

• Setup a cloud-build pipeline that runs the Dockerfile and creats a container with all 6 binaries
• Copy the binaries out from the container and push them to a cloud storage bucket.

[bowd]

Current* status of cross-compilation:

• ✅linux-386
• ✅darwin-386
• ❌windows-386
• ✅linux-amd64
• ✅darwin-amd64
• ✅windows-amd64

*once these PRs are resolved: celo-org/celo-bls-go#9 and #1082

We're still having problems with 32bit windows but I think we can live with this for now.