Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build in docker container #250

Open
MichaelSp opened this issue Feb 27, 2023 · 16 comments
Open

build in docker container #250

MichaelSp opened this issue Feb 27, 2023 · 16 comments
Assignees

Comments

@MichaelSp
Copy link

MichaelSp commented Feb 27, 2023

I try to build the firmware inside a docker container (not to mess up my system). But for some reason it does not work. Any idea how to run the build inside docker?

I tried it in master branch:

$ make
make -C buildroot
make[1]: Entering directory '/build/buildroot'
Makefile:592: *** libglib2 is in the dependency chain of bluez5_utils that has added it to its _DEPENDENCIES variable without selecting it or depending on it from Config.in.  Stop.
make[1]: Leaving directory '/build/buildroot'
make: *** [Makefile:3: all] Error 2

and in next branch:

br-user@fab4de019cec:/build$ make
mkdir -p /build/output
mkdir -p /build/output/images
mkdir -p /build/output/initramfs
make -C /build/output/build/initramfs
make[1]: Entering directory '/build/output/build/initramfs'
>>> toolchain-external-custom  Configuring
mkdir -p /build/output/build/initramfs/per-package/toolchain-external-custom/host
rsync -a /build/output/build/initramfs/per-package/host-skeleton/host/ /build/output/build/initramfs/per-package/toolchain-external-custom/host
rsync -a /build/output/build/initramfs/per-package/skeleton/host/ /build/output/build/initramfs/per-package/toolchain-external-custom/host
mkdir -p /build/output/build/initramfs/per-package/toolchain-external-custom/target
rsync -a /build/output/build/initramfs/per-package/host-skeleton/target/ /build/output/build/initramfs/per-package/toolchain-external-custom/target
rsync -a /build/output/build/initramfs/per-package/skeleton/target/ /build/output/build/initramfs/per-package/toolchain-external-custom/target
Cannot execute cross-compiler '/arm-linux-gcc'    ################### ERROR
make[2]: *** [package/pkg-generic.mk:285: /build/output/build/initramfs/build/toolchain-external-custom/.stamp_configured] Error 1
make[1]: *** [Makefile:23: _all] Error 2
make[1]: Leaving directory '/build/output/build/initramfs'
make: *** [Makefile:61: initramfs] Error 2

Build root not found. I assume some PATH is missing: Cannot execute cross-compiler '/arm-linux-gcc'
For some reason the toolchain is not found / built.


Appendix

Dockerfile
FROM debian

LABEL description="Container with everything needed to run Buildroot"

# Setup environment
ENV DEBIAN_FRONTEND noninteractive

# The container has no package lists, so need to update first
RUN apt-get -o APT::Retries=3 update -y
RUN apt-get -o APT::Retries=3 install -y --no-install-recommends \
        bc \
        build-essential \
        bzr \
        ca-certificates \
        cmake \
        cpio \
        cvs \
        file \
        git \
        gcc-arm-linux-gnueabi \
        libc6 \
        libncurses5-dev \
        locales \
        mercurial \
        openssh-server \
        python3 \
        python3-flake8 \
        python3-magic \
        python3-nose2 \
        python3-pexpect \
        python3-pytest \
        qemu-system-arm \
        qemu-system-x86 \
        rsync \
        shellcheck \
        subversion \
        unzip \
        wget \
        && \
    apt-get -y autoremove && \
    apt-get -y clean

# To be able to generate a toolchain with locales, enable one UTF-8 locale
RUN sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen && \
    /usr/sbin/locale-gen

RUN useradd -ms /bin/bash br-user && \
    chown -R br-user:br-user /home/br-user

USER br-user
WORKDIR /home/br-user
ENV HOME /home/br-user
ENV LC_ALL en_US.UTF-8

Since I mounted the /build from outside, the hard-link didn't work and I had to patch this file in /build/buildroot

diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index 973eabe437..05b442774e 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -184,7 +184,7 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
 define per-package-rsync
        mkdir -p $(3)
        $(foreach pkg,$(1),\
-               rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \
+               rsync -a \
                $(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \
                $(3)$(sep))
 endef

running it with docker-compose.yaml

services:
  build:
    build: .
    volumes:
      - ./:/build
    working_dir: /build
    command:
    - make
@hypnotoad
Copy link
Collaborator

hypnotoad commented Feb 27, 2023

I think your container needs more than just the build directory. At least board, configs and package are needed.

If you really want to use docker, then I would have the whole git repo as a volume. @ski7777 plans some cleanups so your plan might work one day.

@MichaelSp
Copy link
Author

MichaelSp commented Mar 5, 2023

I actually did mount the entire repo into /build. All mentioned dirs are present (board, configs, packages)

@hypnotoad
Copy link
Collaborator

I guess you checked already that the uid of the br-user is the same as the one who checked out the ftcommunity repo?

Is there any message from docker itself (maybe that it prevented execution of the cross compiler)?

For further analysis please provide the full output of a complete build (not just the error message) so someone can compare how it differs from a regular build.

@MichaelSp
Copy link
Author

thanks for the hint, I also checked the user and permissions.
I create a user br-user and run build as root. I also tried to run a container with docker-compose run --user br-user --rm build bash as the build-user br-user.
I also couldn't find any relevant error messages in docker system environment.

Above provided log is the complete build log.

@hypnotoad
Copy link
Collaborator

I could not reproduce your docker setup - to me it fails before it starts compiling:

...
Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB]
Get:3 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
Get:4 http://deb.debian.org/debian bullseye/main amd64 Packages [8183 kB]
Get:5 http://deb.debian.org/debian-security bullseye-security/main amd64 Packages [236 kB]
Get:6 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [14.6 kB]
Fetched 8642 kB in 2s (3663 kB/s)
Reading package lists...
Removing intermediate container 80a254f5f37f
 ---> 56175832de0a
Step 5/11 : RUN apt-get -o APT::Retries=3 install -y --no-install-recommends         bc         build-essential         bzr         ca-certificates         cmake         cpio         cvs         file         git         gcc-arm-linux-gnueabi         libc6:i386         libncurses5-dev         locales         mercurial         openssh-server         python3         python3-flake8         python3-magic         python3-nose2         python3-pexpect         python3-pytest         qemu-system-arm         qemu-system-x86         rsync         shellcheck         subversion         unzip         wget         &&     apt-get -y autoremove &&     apt-get -y clean
 ---> Running in 82c6696291bf
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package libc6:i386
The command '/bin/sh -c apt-get -o APT::Retries=3 install -y --no-install-recommends         bc         build-essential         bzr         ca-certificates         cmake         cpio         cvs         file         git         gcc-arm-linux-gnueabi         libc6:i386         libncurses5-dev         locales         mercurial         openssh-server         python3         python3-flake8         python3-magic         python3-nose2         python3-pexpect         python3-pytest         qemu-system-arm         qemu-system-x86         rsync         shellcheck         subversion         unzip         wget         &&     apt-get -y autoremove &&     apt-get -y clean' returned a non-zero code: 100

@MichaelSp
Copy link
Author

@hypnotoad yes, sorry libc6:i386 should be libc6 instead.

@hypnotoad
Copy link
Collaborator

To me, the idea to build in a standardized docker environment seems good because that would hopefully rule out external errors and maybe also extend the developer basis. It would be great if we could eventually add the Dockerfile and the docker-compose to the git (maybe in a subdirectory).

Regarding the issue: Did you use the new docker buildkit (export DOCKER_BUILDKIT=1) or the old system which first copies >20GB into the container? I am just trying the DOCKER_BUILDKIT=1 in the master branch and it seems to compile fine.

@hypnotoad
Copy link
Collaborator

build_1  | make[2]: Leaving directory '/build/buildroot/output/build/linux-v4.14.97'
build_1  | # Copy the kernel image(s) to its(their) final destination
build_1  | /usr/bin/install -m 0644 -D /build/buildroot/output/build/linux-v4.14.97/arch/arm/boot/uImage /build/buildroot/output/images/uImage
build_1  | # If there is a .ub file copy it to the final destination
build_1  | test ! -f /build/buildroot/output/build/linux-v4.14.97/arch/arm/boot/uImage.ub || cp /build/buildroot/output/build/linux-v4.14.97/arch/arm/boot/uImage.ub /build/buildroot/output/images
build_1  | >>>   Generating root filesystems common tables
build_1  | rm -rf /build/buildroot/output/build/buildroot-fs
build_1  | mkdir -p /build/buildroot/output/build/buildroot-fs
build_1  | printf '   	named -1 named -1 * /etc/bind - - BIND daemon\n 	dbus -1 dbus -1 * /run/dbus - dbus DBus messagebus user\n 	sshd -1 sshd -1 * /var/empty - - SSH drop priv user\n 	- - sudo -1 - - - -\n 	- - input -1 * - - - Input device group\n	- - render -1 * - - - DRI rendering nodes\n	- - kvm -1 * - - - kvm nodes\n\n' >> /build/buildroot/output/build/buildroot-fs/full_users_table.txt
build_1  | cat /build/board/fischertechnik/TXT/user.tab >> /build/buildroot/output/build/buildroot-fs/full_users_table.txt
build_1  | printf '   	/bin/busybox                     f 4755 0  0 - - - - -\n 	/usr/libexec/dbus-daemon-launch-helper f 4750 0 dbus - - - - -\n 	/usr/bin/fusermount f 4755 0 0 - - - - -\n 	/var/empty d 755 root root - - - - -\n 	/usr/bin/sudo f 4755 0 0 - - - - -\n\n' > /build/buildroot/output/build/buildroot-fs/full_devices_table.txt
build_1  | cat system/device_table.txt /build/board/fischertechnik/TXT/device_table.txt >> /build/buildroot/output/build/buildroot-fs/full_devices_table.txt
build_1  | >>>   Generating filesystem image rootfs.squashfs
build_1  | mkdir -p /build/buildroot/output/images
build_1  | rm -rf /build/buildroot/output/build/buildroot-fs/squashfs
build_1  | mkdir -p /build/buildroot/output/build/buildroot-fs/squashfs
build_1  | rsync -auH --exclude=/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM /build/buildroot/output/target/ /build/buildroot/output/build/buildroot-fs/squashfs/target
build_1  | echo '#!/bin/sh' > /build/buildroot/output/build/buildroot-fs/squashfs/fakeroot
build_1  | echo "set -e" >> /build/buildroot/output/build/buildroot-fs/squashfs/fakeroot
build_1  | echo "chown -h -R 0:0 /build/buildroot/output/build/buildroot-fs/squashfs/target" >> /build/buildroot/output/build/buildroot-fs/squashfs/fakeroot
build_1  | PATH="/build/buildroot/output/host/bin:/build/buildroot/output/host/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" /build/buildroot/support/scripts/mkusers /build/buildroot/output/build/buildroot-fs/full_users_table.txt /build/buildroot/output/build/buildroot-fs/squashfs/target >> /build/buildroot/output/build/buildroot-fs/squashfs/fakeroot
build_1  | echo "/build/buildroot/output/host/bin/makedevs -d /build/buildroot/output/build/buildroot-fs/full_devices_table.txt /build/buildroot/output/build/buildroot-fs/squashfs/target" >> /build/buildroot/output/build/buildroot-fs/squashfs/fakeroot
build_1  | printf '   	rm -rf /build/buildroot/output/build/buildroot-fs/squashfs/target//etc/udev/hwdb.d/\n' >> /build/buildroot/output/build/buildroot-fs/squashfs/fakeroot
build_1  | printf '   \n' >> /build/buildroot/output/build/buildroot-fs/squashfs/fakeroot
build_1  | printf '   \n' >> /build/buildroot/output/build/buildroot-fs/squashfs/fakeroot
build_1  | printf '   	/build/buildroot/output/host/bin/mksquashfs /build/buildroot/output/build/buildroot-fs/squashfs/target /build/buildroot/output/images/rootfs.squashfs -noappend -processors 9 -comp lz4 -Xhc\n' >> /build/buildroot/output/build/buildroot-fs/squashfs/fakeroot
build_1  | chmod a+x /build/buildroot/output/build/buildroot-fs/squashfs/fakeroot
build_1  | PATH="/build/buildroot/output/host/bin:/build/buildroot/output/host/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" FAKEROOTDONTTRYCHOWN=1 /build/buildroot/output/host/bin/fakeroot -- /build/buildroot/output/build/buildroot-fs/squashfs/fakeroot
build_1  | rootdir=/build/buildroot/output/build/buildroot-fs/squashfs/target
build_1  | table='/build/buildroot/output/build/buildroot-fs/full_devices_table.txt'
build_1  | Parallel mksquashfs: Using 9 processors
build_1  | Creating 4.0 filesystem on /build/buildroot/output/images/rootfs.squashfs, block size 131072.
[=============================================================/] 9315/9315 100%
build_1  | 
build_1  | Exportable Squashfs 4.0 filesystem, lz4 compressed, data block size 131072
build_1  | 	compressed data, compressed metadata, compressed fragments,
build_1  | 	compressed xattrs, compressed ids
build_1  | 	duplicates are removed
build_1  | Filesystem size 106079.33 Kbytes (103.59 Mbytes)
build_1  | 	51.87% of uncompressed filesystem size (204490.31 Kbytes)
build_1  | Inode table size 135317 bytes (132.15 Kbytes)
build_1  | 	37.55% of uncompressed inode table size (360348 bytes)
build_1  | Directory table size 136761 bytes (133.56 Kbytes)
build_1  | 	59.21% of uncompressed directory table size (230970 bytes)
build_1  | Number of duplicate files found 723
build_1  | Number of inodes 10335
build_1  | Number of files 8425
build_1  | Number of fragments 665
build_1  | Number of symbolic links  1154
build_1  | Number of device nodes 0
build_1  | Number of fifo nodes 0
build_1  | Number of socket nodes 0
build_1  | Number of directories 756
build_1  | Number of ids (unique uids + gids) 6
build_1  | Number of uids 6
build_1  | 	root (0)
build_1  | 	unknown (1002)
build_1  | 	unknown (1001)
build_1  | 	br-user (1000)
build_1  | 	unknown (1003)
build_1  | 	www-data (33)
build_1  | Number of gids 6
build_1  | 	root (0)
build_1  | 	unknown (1002)
build_1  | 	unknown (1001)
build_1  | 	br-user (1000)
build_1  | 	unknown (1003)
build_1  | 	www-data (33)
build_1  | ln -snf /build/buildroot/output/host/arm-buildroot-linux-gnueabihf/sysroot /build/buildroot/output/staging
build_1  | >>>   Executing post-image script /build/board/fischertechnik/TXT/post-image.sh
build_1  | make[1]: Leaving directory '/build/buildroot'
ftcommunity-txt_build_1 exited with code 0

@hypnotoad hypnotoad self-assigned this Apr 1, 2023
@hypnotoad
Copy link
Collaborator

When I tried to build an updated release, I also had problems. @MichaelSp : Could you try to delete the container and re-build from scratch? I have the impression that building a second time does not work properly.

@MichaelSp
Copy link
Author

Few questions to make our results comparable:

  • Which branch do you use?
  • Do you modify any of the files (for example /package/pkg-utils.mk as shown in my inital post)?
  • Which Dockerfile do you use?

I'm still not able to build with a clean container:

master:
$ docker-compose run --user br-user --rm build bash 
br-user@6aa5bedc1a8a:/build$ make
make -C buildroot
make[1]: Entering directory '/build/buildroot'
Makefile:592: *** libglib2 is in the dependency chain of bluez5_utils that has added it to its _DEPENDENCIES variable without selecting it or depending on it from Config.in.  Stop.
make[1]: Leaving directory '/build/buildroot'
make: *** [Makefile:3: all] Error 2

next:

$ docker-compose run --user br-user --rm build bash
br-user@f4e58c85fb44:/build$  make
mkdir -p /build/output
mkdir -p /build/output/images
mkdir -p /build/output/initramfs
make -C /build/output/build/initramfs
make[1]: Entering directory '/build/output/build/initramfs'
>>> toolchain-external-custom  Configuring
mkdir -p /build/output/build/initramfs/per-package/toolchain-external-custom/host
rsync -a /build/output/build/initramfs/per-package/host-skeleton/host/ /build/output/build/initramfs/per-package/toolchain-external-custom/host
rsync -a /build/output/build/initramfs/per-package/skeleton/host/ /build/output/build/initramfs/per-package/toolchain-external-custom/host
mkdir -p /build/output/build/initramfs/per-package/toolchain-external-custom/target
rsync -a /build/output/build/initramfs/per-package/host-skeleton/target/ /build/output/build/initramfs/per-package/toolchain-external-custom/target
rsync -a /build/output/build/initramfs/per-package/skeleton/target/ /build/output/build/initramfs/per-package/toolchain-external-custom/target
Cannot execute cross-compiler '/arm-linux-gcc'
make[2]: *** [package/pkg-generic.mk:285: /build/output/build/initramfs/build/toolchain-external-custom/.stamp_configured] Error 1
make[1]: *** [Makefile:23: _all] Error 2
make[1]: Leaving directory '/build/output/build/initramfs'
make: *** [Makefile:61: initramfs] Error 2

As you can see, they are the same error messages as above in my initial post.

@hypnotoad
Copy link
Collaborator

hypnotoad commented Apr 16, 2023

So it fails right after starting to compile? To reproduce the problem better, I created a branch "docker" from master with your docker-compose.yml and the Dockerfile, but with no further change. I try this checkout from scratch and report later if it worked:

git clone https://github.com/ftCommunity/ftcommunity-TXT/
cd ftcommunity-TXT/
git checkout docker
export DOCKER_BUILDKIT=1
docker-compose up

@hypnotoad
Copy link
Collaborator

hypnotoad commented Apr 16, 2023

My first build as above worked and created the correct files in buildroot/output/images. Could you try to reproduce my above commands? After that, I could successfully execute another docker-compose up to simulate a recompiling. I use Debian 12.0 with Docker version 20.10.23+dfsg1, build 7155243.

As I had similar problems, I could assume that mixing the docker-compose up with another make command from outside docker could confuse the compilation.

@MichaelSp
Copy link
Author

MichaelSp commented Apr 16, 2023

Thank you very much @hypnotoad for your extensive analysis and the concrete steps you did to build the images.

I still get this error message on my side:

ft-community-txt-docker-build-1 | Makefile:592: *** libglib2 is in the dependency chain of bluez5_utils that has added it to its _DEPENDENCIES variable without selecting it or depending on it from Config.in. Stop.

Here is the full build log: https://gist.github.com/MichaelSp/8b52708516511735d11f0762a7298293

I'm running on MacOS M1 Pro 13.2.1 (22D68) with Docker 20.10.23, build 7155243 (the same build even). I have no idea what else to try. I tried with and without DOCKER_BUILDKIT=1

@hypnotoad
Copy link
Collaborator

I think this here is where things start going wrong:

ftcommunity-txt-build-1  | WARNING: unmet direct dependencies detected for BR2_PACKAGE_PYTHON_MATPLOTLIB
ftcommunity-txt-build-1  |   Depends on [n]: (BR2_PACKAGE_PYTHON [=n] || BR2_PACKAGE_PYTHON3 [=n]) && BR2_INSTALL_LIBSTDCPP [=n] && BR2_PACKAGE_PYTHON3 [=n] && BR2_PACKAGE_PYTHON_NUMPY_ARCH_SUPPORTS [=n] && (BR2_TOOLCHAIN_USES_GLIBC [=n] || BR2_TOOLCHAIN_USES_MUSL [=n])
ftcommunity-txt-build-1  |   Selected by [y]:
ftcommunity-txt-build-1  |   - BR2_PACKAGE_PYTHON_PANDAS [=y]

I don't have this warning. After the compilation, my buildroot/.config has

BR2_PACKAGE_PYTHON3=y

All of your warning lines are due to the external modules. To me, the docker-compose.yml looks ok regarding hardlinking or "mv" problems, but I could imagine that the BR2_EXTERNAL=.. in the Makefile plays a role. Could you try using BR2_EXTERNAL=/build instead?

@ski7777
Copy link
Member

ski7777 commented Jun 28, 2023

What is the state of this development?

@MichaelSp
Copy link
Author

Just tried it on the latest docker branch. It still fails:

[+] Building 0.0s (0/0)
[+] Running 1/0
 ✔ Container ftcommunity-txt-build-1  Created                                                                                                                                                                   0.0s
Attaching to ftcommunity-txt-build-1
ftcommunity-txt-build-1  | make -C buildroot
ftcommunity-txt-build-1  | make[1]: Entering directory '/build/buildroot'
ftcommunity-txt-build-1  | Makefile:592: *** libglib2 is in the dependency chain of bluez5_utils that has added it to its _DEPENDENCIES variable without selecting it or depending on it from Config.in.  Stop.
ftcommunity-txt-build-1  | make[1]: Leaving directory '/build/buildroot'
ftcommunity-txt-build-1  | make: *** [Makefile:3: all] Error 2
ftcommunity-txt-build-1 exited with code 2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

When branches are created from issues, their pull requests are automatically linked.

3 participants