VirtualMachine: add workaround cvars for not disabling platform qualification on arm64 or FreeBSD, and more

[illwieckz]

Add workaround cvars that can be set to off to not disable platform qualification on armhf or FreeBSD.

It makes possible to check if the platform qualification works on a specific arm CPU, or to test if Linuxulator has improved enough to support what's required to run platform qualification though the Linux compatibility layer.

Also add a commit to run nacl_loader on FreeBSD Linuxulator the same way it runs on Linux (using the helper).

I don't know why the helper is needed on Linux:

• What is the need for nacl_helper_bootstrap on Linux? #1327

but we better do the same as Linux on Linuxulator for consistency.

[slipher]

I had the impression that the platform qualification failure on ARM was when you use the 32-bit nacl runtime on a 64-bit ARM processor. So it should only be needed if you are building 64-bit arm. Is that wrong?

[illwieckz]

I had the impression that the platform qualification failure on ARM happens when you use the 32-bit nacl runtime on a 64-bit ARM processor. So it should only be needed if you are building 64-bit arm. Is that wrong?

Unfortunately this is not that reliable. I just tested with a freshly installed RasPiOS bullseye, using the exact same SD card on a Raspberry Pi 4 then on a Raspberry Pi 3.

The SD card booted an arm64 kernel on the Raspberry Pi 4 (probably because it has 8G of RAM) while running an armhf userland, and I had to disable qualification:

illwieckz@rpi4:linux-armhf $ lsb_release -a
No LSB modules are available.
Distributor ID:	Raspbian
Description:	Raspbian GNU/Linux 11 (bullseye)
Release:	11
Codename:	bullseye

illwieckz@rpi4:linux-armhf $ uname -a
Linux rpi4 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr  3 17:24:16 BST 2023 aarch64 GNU/Linux

illwieckz@rpi4:linux-armhf $ file /usr/bin/bash
/usr/bin/bash: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, BuildID[sha1]=f12e6d40fb262ad0037b6ec43162208b76d4da71, for GNU/Linux 3.2.0, stripped

illwieckz@rpi4:linux-armhf $ file nacl_helper_bootstrap nacl_loader
nacl_helper_bootstrap: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, BuildID[sha1]=7369b67e86926fb707f9275f806d3bbe24550abf, not stripped
nacl_loader:           ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.26, BuildID[sha1]=ec9a606fd2f2d2fae9b1a5af9fea8564c3341d15, not stripped

illwieckz@rpi4:linux-armhf $ ./nacl_helper_bootstrap ./nacl_loader --r_debug=0xXXXXXXXXXXXXXXXX --reserved_at_zero=0xXXXXXXXXXXXXXXXX -v -B irt_core-armhf.nexe -e -i 100:63 -- cgame-armhf.nexe 
sel_ldr argument list:
./nacl_loader --r_debug=0x0000000040002000 --reserved_at_zero=0x0000000040002000 -v -B irt_core-armhf.nexe -e -i 100:63 -- cgame-armhf.nexe
[9528,4152312640:18:09:27.213043] Error while loading "cgame-armhf.nexe": CPU model is not supported
[9528,4152312640:18:09:27.216506] NaClPerfCounterInterval(SelMain __start__:SnapshotBlob): 4291 microsecs
[9528,4152312640:18:09:27.217224] NACL: Application output follows
[9528,4152312640:18:09:27.217308] NaClAppStartModule: module not loaded
Dumping vmmap.
In PrintVmmap
Done.

illwieckz@rpi4:linux-armhf $ ./nacl_helper_bootstrap ./nacl_loader --r_debug=0xXXXXXXXXXXXXXXXX --reserved_at_zero=0xXXXXXXXXXXXXXXXX -Q -v -B irt_core-armhf.nexe -e -i 100:63 -- cgame-armhf.nexe
PLATFORM QUALIFICATION DISABLED BY -Q - Native Client's sandbox will be unreliable!
sel_ldr argument list:
./nacl_loader --r_debug=0x0000000040002000 --reserved_at_zero=0x0000000040002000 -Q -v -B irt_core-armhf.nexe -e -i 100:63 -- cgame-armhf.nexe
[9710,4151173952:18:20:37.028476] NaClPerfCounterInterval(SelMain __start__:SnapshotBlob): 3061 microsecs
[9710,4151173952:18:20:37.032328] NaClPerfCounterInterval(NaClAppLoadFile __start__:PreAllocAddrSpace): 1685 microsecs
[9710,4151173952:18:20:37.032612] Native Client module will be loaded at base address 0x0000000000000000
[9710,4151173952:18:20:37.032717] NaClPerfCounterInterval(NaClAppLoadFile PreAllocAddrSpace:*AllocAddrSpace): 392 microsecs
[9710,4151173952:18:20:37.168885] NaClPerfCounterInterval(NaClAppLoadFile *AllocAddrSpace:*NaClElfImageLoad): 136169 microsecs
[9710,4151173952:18:20:37.169415] NaClPerfCounterInterval(NaClAppLoadFile *NaClElfImageLoad:*MakeDynText): 535 microsecs
[9710,4151173952:18:20:37.341442] NaClPerfCounterInterval(NaClAppLoadFile *MakeDynText:*ValidateImg): 172021 microsecs
[9710,4151173952:18:20:37.343089] NaClPerfCounterInterval(NaClAppLoadFile __start__:EndLoadFile): 312455 microsecs
[9710,4151173952:18:20:37.343134] NaClPerfCounterInterval(SelMain SnapshotBlob:AppLoadEnd): 314669 microsecs
[9710,4151173952:18:20:37.357861] NaClPerfCounterInterval(NaClTextDyncodeCreate __start__:*DynRegionValidate): 7557 microsecs
[9710,4151173952:18:20:37.360505] NaClPerfCounterInterval(SelMain AppLoadEnd:BlobLoaded): 17368 microsecs
[9710,4151173952:18:20:37.360560] NACL: Application output follows
[9710,4151173952:18:20:37.361136] NaClPerfCounterInterval(SelMain BlobLoaded:CreateMainThread): 603 microsecs
This program is not meant to be invoked directly, it must be invoked by the engine's VM loader.
[9710,4151170112:18:20:37.398006] Exit syscall handler: 1
[9710,4151173952:18:20:37.398093] NaClPerfCounterInterval(SelMain CreateMainThread:WaitForMainThread): 36986 microsecs
[9710,4151173952:18:20:37.398120] NaClPerfCounterInterval(SelMain __start__:SelMainEnd): 372716 microsecs

The exact same SD card booted an armhf kernel on the Raspberry Pi 3 right after that, and yes qualification was working:

illwieckz@rpi3:linux-armhf $ lsb_release -a
No LSB modules are available.
Distributor ID:	Raspbian
Description:	Raspbian GNU/Linux 11 (bullseye)
Release:	11
Codename:	bullseye

illwieckz@rpi3:linux-armhf $ uname -a
Linux rpi3 6.1.21-v7+ #1642 SMP Mon Apr  3 17:20:52 BST 2023 armv7l GNU/Linux

illwieckz@rpi3:linux-armhf $ file /usr/bin/bash
/usr/bin/bash: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, BuildID[sha1]=f12e6d40fb262ad0037b6ec43162208b76d4da71, for GNU/Linux 3.2.0, stripped

illwieckz@rpi3:linux-armhf $ file nacl_helper_bootstrap nacl_loader
nacl_helper_bootstrap: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, BuildID[sha1]=7369b67e86926fb707f9275f806d3bbe24550abf, not stripped
nacl_loader:           ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.26, BuildID[sha1]=ec9a606fd2f2d2fae9b1a5af9fea8564c3341d15, not stripped

illwieckz@rpi3:linux-armhf $ ./nacl_helper_bootstrap ./nacl_loader --r_debug=0xXXXXXXXXXXXXXXXX --reserved_at_zero=0xXXXXXXXXXXXXXXXX -v -B irt_core-armhf.nexe -e -i 100:63 -- cgame-armhf.nexe
sel_ldr argument list:
./nacl_loader --r_debug=0x0000000040002000 --reserved_at_zero=0x0000000040002000 -v -B irt_core-armhf.nexe -e -i 100:63 -- cgame-armhf.nexe
[1255,1991661120:18:25:38.666288] NaClPerfCounterInterval(SelMain __start__:SnapshotBlob): 4652 microsecs
[1255,1991661120:18:25:38.670432] NaClPerfCounterInterval(NaClAppLoadFile __start__:PreAllocAddrSpace): 2018 microsecs
[1255,1991661120:18:25:38.683105] Native Client module will be loaded at base address 0x0000000000000000
[1255,1991661120:18:25:38.683249] NaClPerfCounterInterval(NaClAppLoadFile PreAllocAddrSpace:*AllocAddrSpace): 12821 microsecs
[1255,1991661120:18:25:38.934074] NaClPerfCounterInterval(NaClAppLoadFile *AllocAddrSpace:*NaClElfImageLoad): 250819 microsecs
[1255,1991661120:18:25:38.936094] NaClPerfCounterInterval(NaClAppLoadFile *NaClElfImageLoad:*MakeDynText): 2025 microsecs
[1255,1991661120:18:25:39.262802] NaClPerfCounterInterval(NaClAppLoadFile *MakeDynText:*ValidateImg): 326704 microsecs
[1255,1991661120:18:25:39.264323] NaClPerfCounterInterval(NaClAppLoadFile __start__:EndLoadFile): 595912 microsecs
[1255,1991661120:18:25:39.264377] NaClPerfCounterInterval(SelMain SnapshotBlob:AppLoadEnd): 598095 microsecs
[1255,1991661120:18:25:39.293132] NaClPerfCounterInterval(NaClTextDyncodeCreate __start__:*DynRegionValidate): 13359 microsecs
[1255,1991661120:18:25:39.297357] NaClPerfCounterInterval(SelMain AppLoadEnd:BlobLoaded): 32976 microsecs
[1255,1991661120:18:25:39.297419] NACL: Application output follows
[1255,1991661120:18:25:39.297963] NaClPerfCounterInterval(SelMain BlobLoaded:CreateMainThread): 608 microsecs
This program is not meant to be invoked directly, it must be invoked by the engine's VM loader.
[1255,1991648320:18:25:39.331832] Exit syscall handler: 1
[1255,1991661120:18:25:39.331949] NaClPerfCounterInterval(SelMain CreateMainThread:WaitForMainThread): 33981 microsecs
[1255,1991661120:18:25:39.331980] NaClPerfCounterInterval(SelMain __start__:SelMainEnd): 670350 microsecs

[illwieckz]

I added code to detect arm64 kernel.

[illwieckz]

I added more code, especially I added two generic cvars vm.nacl.qualification and vm.nacl.bootstrap to generically disable platform qualification and not use the helper bootstrap. This may be used for testing purpose, or to make the game run on uncommon systems.

For example, NetBSD provides a similar Linux emulation than FreeBSD, and then the platform qualification is likely to fail, maybe the whole Linux game (engine + nacl) runs without building a native NetBSD engine and without implementing a specific NetBSD workaround just by setting vm.nacl.qualification off.

Another example would be people building a native engine for another platform (like ppc64el), but configuring their system binaries from other platforms (like amd64) to run over qemu-user transparently, or to run the whole game (engine + nacl) this way, such generic cvar may help to unlock the game without requiring us to add specific code.

The cvar about the helper bootstrap is more about making possible to test if it is required and when. For example I noticed that it is required on armhf and arm64 on Debian Bullseye, while it is not required on amd64 on Ubuntu Noble…

[slipher]

I added code to detect arm64 kernel.

Great!