Building with SDK v0.37.0 fails fetch-vendored task due to missing GOPROXY

[markusboehme]

Platform I'm building on: Fedora 36 on x86_64

What I expected to happen: With the current develop branch after merging #3690, I expected cargo make -e BUILDSYS_VARIANT=metal-dev to succeed.

What actually happened: The fetch-vendored task fails with github.com/aws/aws-sdk-go/aws: GOPROXY list is not the empty string, but contains no entries and similar entries:

[cargo-make][1] INFO - Running Task: fetch-vendored
go: downloading github.com/containerd/containerd v1.6.26
go: downloading github.com/aws/aws-sdk-go v1.49.13
go: downloading github.com/awslabs/amazon-ecr-containerd-resolver v0.0.0-20231213164356-22a8023e9be3
go: downloading github.com/urfave/cli/v2 v2.27.1
go: downloading google.golang.org/grpc v1.60.1
go: downloading github.com/aws/aws-sdk-go v1.49.13
go: downloading github.com/urfave/cli/v2 v2.27.1
go: downloading github.com/awslabs/amazon-ecr-containerd-resolver v0.0.0-20231213164356-22a8023e9be3
go: downloading github.com/containerd/containerd v1.6.26
go: downloading google.golang.org/grpc v1.60.1
go: host-ctr/cmd/host-ctr imports
        github.com/aws/aws-sdk-go/aws: GOPROXY list is not the empty string, but contains no entries
go: host-ctr/cmd/host-ctr imports
        github.com/aws/aws-sdk-go/aws/session: GOPROXY list is not the empty string, but contains no entries
go: host-ctr/cmd/host-ctr imports
        github.com/aws/aws-sdk-go/service/ecrpublic: GOPROXY list is not the empty string, but contains no entries
go: host-ctr/cmd/host-ctr imports
        github.com/awslabs/amazon-ecr-containerd-resolver/ecr: GOPROXY list is not the empty string, but contains no entries
go: host-ctr/cmd/host-ctr imports
        github.com/containerd/containerd: GOPROXY list is not the empty string, but contains no entries
go: host-ctr/cmd/host-ctr imports
        github.com/containerd/containerd/cio: GOPROXY list is not the empty string, but contains no entries
go: host-ctr/cmd/host-ctr imports
        github.com/containerd/containerd/containers: GOPROXY list is not the empty string, but contains no entries
go: host-ctr/cmd/host-ctr imports
        github.com/containerd/containerd/contrib/seccomp: GOPROXY list is not the empty string, but contains no entries
go: host-ctr/cmd/host-ctr imports
        github.com/containerd/containerd/errdefs: GOPROXY list is not the empty string, but contains no entries
go: host-ctr/cmd/host-ctr imports
        github.com/containerd/containerd/log: GOPROXY list is not the empty string, but contains no entries
go: host-ctr/cmd/host-ctr imports
        github.com/containerd/containerd/namespaces: GOPROXY list is not the empty string, but contains no entries
go: host-ctr/cmd/host-ctr imports
        github.com/containerd/containerd/oci: GOPROXY list is not the empty string, but contains no entries
go: host-ctr/cmd/host-ctr imports
        github.com/containerd/containerd/pkg/cri/server: GOPROXY list is not the empty string, but contains no entries
go: host-ctr/cmd/host-ctr imports
        github.com/containerd/containerd/remotes/docker: GOPROXY list is not the empty string, but contains no entries
go: host-ctr/cmd/host-ctr imports
        github.com/containerd/containerd/runtime/v2/runc/options: GOPROXY list is not the empty string, but contains no entries
go: host-ctr/cmd/host-ctr imports
        github.com/urfave/cli/v2: GOPROXY list is not the empty string, but contains no entries
go: host-ctr/cmd/host-ctr imports
        k8s.io/cri-api/pkg/apis/runtime/v1 imports
        google.golang.org/grpc: GOPROXY list is not the empty string, but contains no entries
go: host-ctr/cmd/host-ctr imports
        k8s.io/cri-api/pkg/apis/runtime/v1 imports
        google.golang.org/grpc/codes: GOPROXY list is not the empty string, but contains no entries
go: host-ctr/cmd/host-ctr imports
        k8s.io/cri-api/pkg/apis/runtime/v1 imports
        google.golang.org/grpc/status: GOPROXY list is not the empty string, but contains no entries
[cargo-make][1] ERROR - Error while executing command, exit code: 1

How to reproduce the problem: Have no GO* environment variables set (ensure env | grep ^GO is empty), no go.env file is present, and then cargo make.

[markusboehme]

I haven't looked yet at what GOPROXY is set at during the build or who sets it (is not the empty string). I also don't know why it looks to affect the dependencies of host-ctr only. In the meantime, explicitly export GOPROXY=direct in the build environment does the trick, but wasn't needed before the update to SDK v0.37.0/Go v1.21.0.

[markusboehme]

SDK v0.36.1 has a default GOPROXY set:

$ docker run --rm public.ecr.aws/bottlerocket/bottlerocket-sdk-x86_64:v0.36.1 /usr/libexec/go/bin/go env | grep GOPROXY
GOPROXY="https://proxy.golang.org,direct"

SDK v0.37.0 does not:

$ docker run --rm public.ecr.aws/bottlerocket/bottlerocket-sdk-x86_64:v0.37.0 /usr/libexec/go/bin/go env | grep GOPROXY
GOPROXY=''

Wondering if the error message (GOPROXY list is not the empty string) is misleading or whether buildsys or twoliter perhaps misinterpret an empty GOPROXY.

[markusboehme]

From https://cs.opensource.google/go/go/+/7aa85e01376d840acc8bb931156d607a00b64a60

Various Linux distributions edit cmd/go/internal/cfg/cfg.go to change
the default settings of GOPROXY and GOSUMDB. Make it possible for
them to do this without editing the go command source code by
introducing GOROOT/go.env and moving those defaults there.

It would seem we need to package go.env during the SDK build then, because there is no such file in the resulting image.

[markusboehme]

This diff for bottlerocket-sdk includes the go.env file with the defaults, and GOPROXY is subsequently set to the old default:

diff --git a/Dockerfile b/Dockerfile
index 0c95d1a..5c3aae5 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1012,6 +1012,7 @@ COPY --chown=0:0 --from=sdk-go /home/builder/sdk-go/bin /usr/libexec/go/bin/
 COPY --chown=0:0 --from=sdk-go /home/builder/sdk-go/lib /usr/libexec/go/lib/
 COPY --chown=0:0 --from=sdk-go /home/builder/sdk-go/pkg /usr/libexec/go/pkg/
 COPY --chown=0:0 --from=sdk-go /home/builder/sdk-go/src /usr/libexec/go/src/
+COPY --chown=0:0 --from=sdk-go /home/builder/sdk-go/go.env /usr/libexec/go/go.env
 COPY --chown=0:0 --from=sdk-go \
   /home/builder/sdk-go/licenses/ \
   /usr/share/licenses/go/

I haven't run the full matrix of SDK builds yet to be able to actually test a build of Bottlerocket, but the output of go env in one of the SDK images looks right to me.