Bootloader Binaries in the scope of x86 / Cross-Compilation Support

[hoeppnerj]

We're currently working on changes to support the build of certain tools for and on x86 (or ARM) machines.
Most prominent tools would be pvattest, genprotimg, zgetdump. These tools would allow to prepare and analyze
boot images (or dumps) in the realm of IBM Secure Execution on a trusted environment such as the laptop of
an admin (which would be an x86 or ARM (e.g. Mac) machine). The idea is to limit the build targets depending on
the defined or detected host architecture.

The main issue we're facing at the moment is the build of the bootloader binaries (stage3a/b for genprotimg).
Those have to be built for s390x no matter what host machine is targeted. So, running the build on x86 should build
the genprotimg tool for x86 and the bootloader files for s390x. This however requires a cross building setup, which
can get complicated to set up.

Our current idea to solve this problem would be to ship pre-built binaries with each release. Recent changes to the
build system would already allow to include pre-build binaries.

@sharkcz @xnox @frank-heimes @ngueorguiev How would packaging look like with this approach? Do you have
other ideas to solve this? Or would a partial cross-compilation as described already work for everyone and we
wouldn't need to handle this in a special way?

On that note, we couldn't find debug info packages for the bootloaders on any distro. Which is probably due to the
fact that we strip those information away during build. Maybe a pre-built approach could solve this issue, too.

[xnox]

Ubuntu will not ship source tarballs with prebuild binaries. Ubuntu will not sign prebuild binaries. Introducing prebuild binaries, will force us to manually repack/recreate orig tarballs and exclude prebuild binaries. Please do not ship prebuild binaries in the source tarball.

In Ubuntu, and any linux distribution, it is trivial to build stage3 bootloader during native s390x source build, but package it as an architecture independent binary package (_all.deb / .noarch.rpm) such that tooling on any other architecture can use it directly. For Ubuntu, it will mean that the signed bootloader portions will be packaged as architecture independent binary package.

If cross-toolchain is available, which one can detect in a universal manner, one can cross-build stage3 portions on non-s390x host architectures if you so wish. This is done often by packages such as qemu & grub2 if you want some working examples. On Ubuntu, installing cross-toolchains / build-depending on cross-toolchains is trivial. Also, we ensure that compilers are available under the same names as either native or cross version, further simplifying build process. In Ubuntu crossbuild-essential-s390x is available on amd64 arm64 ppc64el => meaning all supported 64bit arches can cross-compile code for s390x. And library dependencies can be installed with multiarch. For example, /usr/bin/s390x-linux-gnu-gcc and /usr/bin/x86_64-linux-gnu-gcc are available on both x86 & s390x on Ubuntu, which are native or cross compilers as appropriate, making it easy to devel and build things as needed - irrespective of the host OS. We have additional tooling in ubuntu-dev-tools package to setup chroots ready for cross-compilation.

However, I don't see how this will be required here. Again, distributions will likely expect to build things natively as needed, and make them installable where needed at runtime. Meaning stage3.bin will be build with native toolchain on s390x, Userspace tooling will be build with native toolchains on x86, arm64, ppc64le, s390x. Which once installed will reuse the identical s390x's build produced stage3.bin build + native tooling on each arch.

There is no need to prebuild anything.

In general, Ubuntu policy is to build with debug symbols, strip them from binaries, and ship them in dbgsyms package. If ubuntu .ddeb / debug symbols service are missing any debug symbols for any portions of s390-tools please open a separate bug report, with hints as to where we may have lost them, and we will fix that to introduce them.

[xnox]

Thanks for the quick reply!

So you're saying the best way to support genprotimg and zipl on x86 would be to build the /usr/bin/genprotimg and /usr/sbin/zipl tools with the native toolchain on x86 and the bootloaders (e.g. stage3.bin) with native toolchains on s390x? How will this be packaged then?

A single source package in all Linux distributions can trigger multiple builds. Once this is available, we will simply update debian/control to state that it builds on amd64 arm64 s390x. And will mark s390x as the architecute to build and provide arch independed arch_all package. What you need to do in upstream, is simply compile as much things as you want when running on those architectures. The distro packaging will handle the rest.

To be honest, i'm not sure if you need to change anything upstream. Becuase most of s390-tools does already just builds on x86. If you want us to turn that on, I guess we just need a bug report. We might as well attempt to compile as many tools from s390-tools as possible. Because why not - and it will also help us with running CI & tests in our installers (i.e. python based curtin & subiquity, which do call out into s390-tools).

I can show things in PPA if that would help.

[xnox]

i guess qemu is bad example, as all distributions strip the cross-built / pre-built binaries, and build them natively and repackage them. Despite upstream trying hard to be "nice", in the worst possible way.

[hoeppnerj]

While this all seems to be good news for genprotimg and other tools, at the moment this approach won't work for zipl, I reckon. The bootloader binaries are build time dependencies and therefore zipl would need to be built on the same machine with a cross-compile toolchain.
See

s390-tools/zipl/src/Makefile

Line 22 in e6be1c1

boot.o: ../boot/.loaders

and (Entire section of data import)

s390-tools/zipl/src/boot.c

Lines 29 to 67 in e6be1c1

	/* Import a binary file */
	/* clang-format off */
	#define DATA_NAME(SYM, SUFFIX) _binary_##SYM##_bin##SUFFIX
	#define DATA_SIZE(SYM) ((size_t)(&DATA_NAME(SYM, _end) - &DATA_NAME(SYM, _start)))
	#define DATA_ADDR(SYM) (&DATA_NAME(SYM, _start))
	#define BIN_FILE_PATH(FILE_NAME) STRINGIFY(BUILD_PATH) "/" STRINGIFY(FILE_NAME) ".bin"
	#define IMPORT_DATA(SYM) \
	extern const uint8_t DATA_NAME(SYM, _start); \
	extern const uint8_t DATA_NAME(SYM, _end); \
	asm(".section \".rodata\", \"a\", @progbits\n" \
	".balign 4\n" \
	".global " STRINGIFY(DATA_NAME(SYM, _start)) "\n" \
	STRINGIFY(DATA_NAME(SYM, _start)) ":\n" \
	".incbin \"" BIN_FILE_PATH(SYM) "\"\n" \
	".global " STRINGIFY(DATA_NAME(SYM, _end)) "\n" \
	STRINGIFY(DATA_NAME(SYM, _end)) ":\n" \
	".balign 4\n" \
	".previous\n")
	/* clang-format on */
	/* Stage 0 Loader */
	IMPORT_DATA(eckd0_cdl);
	IMPORT_DATA(eckd0_ldl);
	IMPORT_DATA(fba0);
	IMPORT_DATA(tape0);
	/* Stage 1 Loader */
	IMPORT_DATA(eckd1);
	/* Stage 1b Loader */
	IMPORT_DATA(eckd1b);
	IMPORT_DATA(fba1b);
	/* Stage 2 Loader */
	IMPORT_DATA(eckd2);
	IMPORT_DATA(fba2);
	/* Stage 2 Dump Loader */
	IMPORT_DATA(eckd2dump_mv);
	IMPORT_DATA(eckd2dump_sv);
	IMPORT_DATA(fba2dump);
	IMPORT_DATA(tape2dump);

However, zipl as a candidate for x86 is only planned for the future, because there are some other issues still. I wanted to bring it up anyway. Solving this would be on our end by decoupling the loaders from the tool.

[fweimer-rh]

Couldn't zipl load the files from the file system instead? That would work on all architectures.

[hoeppnerj]

Yes, probably similar to what we already do with the stage3.bin loader file:

s390-tools/zipl/include/zipl.h

Line 37 in e6be1c1

#define ZIPL_STAGE3_PATH util_libdir_path("stage3.bin")

I just wanted to mention it so that everyone is aware, that zipl would need a couple changes to the code. genprotimg only cares about stage3 AFAIK, so that'll work already.
But yes, you're right. Store the files where we store stage3.bin already and load them in a similar fashion.

[fweimer-rh]

For some RPM distributions, relying on noarch package leakage across architectures definitely looks attractive. It avoids the need for separate testing of cross-compiled artifacts.

[fweimer-rh]

The *.bin files are not recognized by our ELF tooling, so this has to be a custom mechnanism anyway?

What would consume the debuginfo for those *.bin files?

[mhartmay]

Currently there are only raw binary files for the bootloader. But this will change in the future (at least I hope so).

We create the raw binary file directly from the ELF file that contains the debug info using objcopy (see https://github.com/ibm-s390-linux/s390-tools/blob/master/zipl/boot/Makefile#L79). One thing that is special here is that these ELF bootloader files (and of course the raw binary bootloader files as well) don't include a build-id for memory layout reasons.

Not sure what would be the best way (or what change on our part) to help you to be able to get the debuginfo for the bootloaders.

[fweimer-rh]

I think this depends on how you intend debuggers consume the separated debuginfo. Given that this isn't a Linux environment, you'd probably need something else than what Linux distributions implement. I guess it's difficult to discuss this in the abstract.

[mhartmay]

For the QEMU firmware (e.g. s390-ccw.img) the same problem exists... The debuginfo is shipped in the debuginfo package and you have to load it manually at a "magic" position - but at least, it's possible.

e.g. using gdb
add-symbol-file s390-ccw.elf 0X3FE00400

[mhartmay]

So what would be the best way to handle the debuginfo for the bootloaders if we were to go the noarch "route"?
There is a similar bug open for QEMU on Ubuntu (see https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/2020624).

[sharkcz]

In Fedora we can cross-compile the s390x bits on non-s390x arches, but that's not possible in RHEL. Also we can't combine artifacts from various arches into a single arch rpm.

[fweimer-rh]

But you can add a dependency on the noarch package. Even in the buildroot, noarch packages built on just one architecture leak to all other architectures.

And I think you should keep Fedora and RHEL builds aligned.

[sharkcz]

You mean having a noarch subpackage built on s390x together with the existing subpackages and then distribute the noarch rpm to non-s390x arches via some tricks in the compose tools? It might work ...

[fweimer-rh]

Yes to the first part, no the tricks. It just happens automatically, even in cases where we don't want to, see glibc-headers-s390 for an example.

[xnox]

I think the ask is to provide .noarch.rpm / _all.deb (single or multiple as needed for signing) which will contain:

• /usr/share/s390-tools/genprotimg/
• /usr/share/s390-tools/genprotimg/README.md
• /usr/share/s390-tools/genprotimg/check_hostkeydoc
• /usr/share/s390-tools/genprotimg/stage3a.bin
• /usr/share/s390-tools/genprotimg/stage3b_reloc.bin
• /lib/s390-tools/stage3.bin
• /lib/s390-tools/stage3.pem (if signing is available)
• compiled natively on s390x, or cross-built

On x86, arm64 provide .x86_64.rpm / .aarch64.rpm / _amd64.deb / _arm64.deb which will contain:

• /sbin/zipl
• /usr/bin/genprotimg
• compiled natively and depend on the above noarch.rpm / _all.deb

Adjust paths to taste (/lib vs /usr/lib, sbin vs usr/sbin vs usr/bin).

At least in Launchpad it is trivial for us to indicate that we want .noarch.rpm / _all.deb to be produced by the s390x arch build (as that's were we will sign stage3.bin as well). And it is trivial for us to limit s390-tools x86/arm build to just the desired tools. Hopefully koji has similar functionality, or it can be emulated by creating new src.rpm which only builds stage3*.bin.

I think on ubuntu we will split out genprotimg-data & zipl into separate sub-packages. Because we try to route as little things via signing as possible. Which all should be transparent to the end user.

[hoeppnerj]

fyi, we've made a couple of changes to our build system with the latest release to hopefully simplify cross and non-s390x builds: 49937f4...e99b32c

[hoeppnerj]

I should add the slightly different usage here, too:
Cross build for s390x (Host Machine) on x86 (Build Machine):
make HOST_ARCH=s390x CROSS_COMPILE=s390x-linux-gnu-
Cross build for x86 (Host Machine) on s390x (Build Machine):
make HOST_ARCH=x86_64 CROSS_COMPILE=x86_64-linux-gnu-

A native build on x86 would limit the build target to tools that are supported by us on other architectures:
https://github.com/ibm-s390-linux/s390-tools/blob/master/Makefile#L9-L24

[mhartmay]

I think the ask is to provide .noarch.rpm / _all.deb (single or multiple as needed for signing) which will contain:

* /usr/share/s390-tools/genprotimg/

* /usr/share/s390-tools/genprotimg/README.md

* /usr/share/s390-tools/genprotimg/check_hostkeydoc

* /usr/share/s390-tools/genprotimg/stage3a.bin

* /usr/share/s390-tools/genprotimg/stage3b_reloc.bin

* /lib/s390-tools/stage3.bin

* /lib/s390-tools/stage3.pem (if signing is available)

* compiled natively on s390x, or cross-built

On x86, arm64 provide .x86_64.rpm / .aarch64.rpm / _amd64.deb / _arm64.deb which will contain:

* /sbin/zipl

* /usr/bin/genprotimg

* compiled natively and depend on the above noarch.rpm / _all.deb

Adjust paths to taste (/lib vs /usr/lib, sbin vs usr/sbin vs usr/bin).

At least in Launchpad it is trivial for us to indicate that we want .noarch.rpm / _all.deb to be produced by the s390x arch build (as that's were we will sign stage3.bin as well). And it is trivial for us to limit s390-tools x86/arm build to just the desired tools. Hopefully koji has similar functionality, or it can be emulated by creating new src.rpm which only builds stage3*.bin.

I think on ubuntu we will split out genprotimg-data & zipl into separate sub-packages. Because we try to route as little things via signing as possible. Which all should be transparent to the end user.

Sounds like a good plan to me! I'm planning to add the following changes to s390-tools (highlighted in bold) - please check if this fits with your solution described above.

1. Package the genprotimg and co. on a x86 system (or using a cross-compiler...): (CHANGE: make all will also build genprotimg by default, but skip building the genprotimg bootloaders on $HOST_ARCH other than s390x.)

   $ make -j
   $ make install -j
   $ # package the /usr/bin/genprotimg, etc. build artifacts in a .x86_64.rpm / .aarch64.rpm / _amd64.deb / _arm64.deb package

2. package the required genprotimg bootloaders in a non-arch package (e.g. genprotimg-data_all.deb, ...)
   On a s390x system (or using a cross-compiler (no stdlib required))

   $ make -j   # optionally -C genprotimg/boot
   $ make install -j  # optionally -C genprotimg/boot
   $ # package the required genprotimg bootloader build artifacts in a .noarch.rpm / _all.deb

@sharkcz
Would this also work for RHEL/Fedora?