Static linking libraries built in release mode into a debug build shared library or executable results in corrupt debug symbol table

[pal1000]

For example let's try a trimmed down Mesa3D 22.3.0-devel-9f3eb63878bbfd73e0c9112b5cec04caf930eb4d 32-bit GCC build with Microsoft CLC only:

Install dependencies and build

$ pacman -S bison flex git mingw-w64-i686-{cc,clang,gdb,libclc,meson,pkgconf,python-mako,opencl-icd,spirv-llvm-translator,spirv-tools,zstd} --needed --noconfirm
$ cd ~
$ git clone -b 9f3eb63878bbfd73e0c9112b5cec04caf930eb4d --depth=1 https://gitlab.freedesktop.org/mesa/mesa.git mesa
$ curl https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18117.diff -o static-link-clang-if-static-link-llvm.patch
$ cd mesa
$ git apply -v ../static-link-clang-if-static-link-llvm.patch
/mingw32/bin/meson setup build/gcc-x86 --libdir="lib/x86" --pkgconfig.relocatable -Db_ndebug=true --prefer-static -Dzstd=enabled --backend=ninja --buildtype=debug -Dcpp_rtti=true -Dllvm=enabled -Dshared-llvm=disabled -Dgallium-drivers= -Dvulkan-drivers= -Dgallium-d3d10umd=false -Dspirv-to-dxil=true -Dgles1=auto -Dgles2=auto -Dshared-glapi=auto -Degl=disabled -Dosmesa=false -Dbuild-tests=false -Dmicrosoft-clc=enabled -Dgallium-opencl=disabled -Dstatic-libclc=all -Dc_args="-march=core2 -pipe" -Dcpp_args="-march=core2 -pipe" -Dc_link_args="-static -ltre -lintl -liconv" -Dcpp_link_args="-static -ltre -lintl -liconv" -Ddraw-use-llvm=true -Dmin-windows-version=8
/mingw32/bin/ninja -C build/gcc-x86

Other dependencies

dxil.zip
openclon12.zip

Notes

• openclon12.dll is the OpenCL ICD and it only builds with MSVC as it makedepends on WDK and Ninja is unable to build it;
• dxil.dll is a redistributable binary blob from Windows SDK that all Microsoft developed Mesa3D components depend on at runtime.

GDB output

cd "c:\Software\systools\GPU/GPU_Caps_Viewer";gdb -ex 'target exec GPU_Caps_Viewer.exe' -ex 'run'

GNU gdb (GDB) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-w64-mingw32".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Starting program: C:\Software\systools\GPU\GPU_Caps_Viewer\GPU_Caps_Viewer.exe
[New Thread 7136.0xee8]
[New Thread 7136.0x1390]
[New Thread 7136.0x19b0]
[New Thread 7136.0xc30]
[New Thread 7136.0x1434]
[New Thread 7136.0x82c]
[New Thread 7136.0xeb4]
warning: onecore\net\netprofiles\service\src\nsp\dll\namespaceserviceprovider.cpp(550)\nlansp_c.dll!730885D4: (caller: 7715CD00) LogHr(1) tid(eb4) 8007277C No such service is known. The service cannot be found in the specified name space.
[New Thread 7136.0x1e74]
[Thread 7136.0x1e74 exited with code 0]
[Thread 7136.0xeb4 exited with code 0]
[New Thread 7136.0x2798]
[New Thread 7136.0x12dc]
[New Thread 7136.0x2358]
[New Thread 7136.0x20a0]
[New Thread 7136.0x5bc]
[New Thread 7136.0x220c]
[New Thread 7136.0x20bc]
[New Thread 7136.0x680]
[New Thread 7136.0x14b0]
BFD: C:\Software\Development\projects\mesa-dist-win\bin\x86\clon12compiler.dll: bad string table size 2983591936
Error while mapping shared library sections:
`C:\Software\Development\projects\mesa-dist-win\bin\x86\clon12compiler.dll': not in executable format: file format not recognized
[New Thread 7136.0x9f4]
[New Thread 7136.0x11a8]
[New Thread 7136.0x23c8]
[New Thread 7136.0x1c28]
[New Thread 7136.0x2430]
[Thread 7136.0x2358 exited with code 0]
[Thread 7136.0x19b0 exited with code 0]
[Thread 7136.0xee8 exited with code 0]
[Thread 7136.0x1390 exited with code 0]
[Thread 7136.0x82c exited with code 0]
[Thread 7136.0x11a8 exited with code 0]
[Thread 7136.0xc30 exited with code 0]
[Thread 7136.0x23c8 exited with code 0]
[Thread 7136.0x2430 exited with code 0]
[Thread 7136.0x1c28 exited with code 0]
[Thread 7136.0x12dc exited with code 0]
[Thread 7136.0x2798 exited with code 0]
[New Thread 7136.0xc4c]
[New Thread 7136.0xb04]
[Thread 7136.0x9f4 exited with code 0]
[Thread 7136.0xb04 exited with code 0]
[Thread 7136.0xc4c exited with code 0]
[New Thread 7136.0x1130]
[Thread 7136.0x1130 exited with code 0]
[New Thread 7136.0xd20]
[New Thread 7136.0x1760]
[New Thread 7136.0x778]
[New Thread 7136.0x1940]
[New Thread 7136.0x1c60]
[New Thread 7136.0x1ae8]
[New Thread 7136.0xa84]
[New Thread 7136.0x1220]
[Thread 7136.0x778 exited with code 0]
[Thread 7136.0x1220 exited with code 0]
[Thread 7136.0xa84 exited with code 0]
[Thread 7136.0x1ae8 exited with code 0]
[Thread 7136.0x1c60 exited with code 0]
[Thread 7136.0x1940 exited with code 0]
[Thread 7136.0xd20 exited with code 0]
[Thread 7136.0x220c exited with code 0]
[Thread 7136.0x1434 exited with code 0]
[Thread 7136.0x1760 exited with code 0]
[Thread 7136.0x14b0 exited with code 0]
[Thread 7136.0x20a0 exited with code 0]
[Thread 7136.0x5bc exited with code 0]
[Thread 7136.0x20bc exited with code 0]
[Thread 7136.0x680 exited with code 0]
[Inferior 1 (process 7136) exited normally]
(gdb) exit

[pal1000]

This issue had a different cause entirely. LLVM JIT.

[Biswa96]

Just curious. Would you like to explain the issue or the actual cause?

[pal1000]

So here what I an trying to do; I am trying to get a backtrace from a Mesa3D llvmpipe and softpipe crash when running any OpenGL 1.x or 2.x GPU Caps Viewer 3D demo that only occurs with MinGW and only if Mesa3D is built statically.

If I don't strip binaries at link time with -s in -Dc_link_args and -Dcpp_link_args I get stuff like this in GDB:

BFD: C:\Software\systools\GPU\GPU_Caps_Viewer\opengl32.dll: bad string table size 16777216
Error while mapping shared library sections:
`C:\Software\systools\GPU\GPU_Caps_Viewer\opengl32.dll': not in executable format: file format not recognized

Of course this destroys debug info so it's not an option.
However I can extract debug info from such binary with objcopy --only-keep-debug <src> <dest>
Unfortunately using <dest> as symbol file results in this GDB abort:

Unable to read JIT descriptor from remote memory
Warning:
Cannot insert breakpoint -4.
Cannot access memory at address 0x6b6e3850

Command aborted.

Final attempt involved doing 2 builds. I extracted debug info from a dynamic linked debug build and used resulting symbol file with a static linked stripped release build in GDB. The resulted backtrace was unhelpful:

cd "c:\Software\systools\GPU/GPU_Caps_Viewer";gdb -ex 'symbol-file "c:/Software/Development/projects/mesa-dist-win/debugsymbols/x86/libgallium_wgl.dll.sym"' -ex 'target exec GPU_Caps_Viewer.exe' -ex 'run'

GNU gdb (GDB) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-w64-mingw32".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Reading symbols from c:/Software/Development/projects/mesa-dist-win/debugsymbols/x86/libgallium_wgl.dll.sym...

Starting program: C:\Software\systools\GPU\GPU_Caps_Viewer\GPU_Caps_Viewer.exe
[New Thread 5936.0x1fc]
[New Thread 5936.0x1ae8]
[New Thread 5936.0x1410]
[New Thread 5936.0x558]
[New Thread 5936.0x2574]
[New Thread 5936.0x2654]
[New Thread 5936.0x258c]
[New Thread 5936.0x2258]
[New Thread 5936.0x188c]
[New Thread 5936.0x104]
[New Thread 5936.0x1d7c]
[New Thread 5936.0x1ef4]
[New Thread 5936.0x2054]
[New Thread 5936.0xad4]
[New Thread 5936.0x1ecc]
warning: onecore\net\netprofiles\service\src\nsp\dll\namespaceserviceprovider.cpp(550)\nlansp_c.dll!70B585D4: (caller: 76A0CD00) LogHr(1) tid(1ecc) 8007277C No such service is known. The service cannot be found in the specified name space.
[New Thread 5936.0x10dc]
[Thread 5936.0x10dc exited with code 0]
[Thread 5936.0x1ecc exited with code 0]
[New Thread 5936.0x1338]
[New Thread 5936.0xe14]
[Thread 5936.0x1ae8 exited with code 0]
[Thread 5936.0x1fc exited with code 0]
[Thread 5936.0x1410 exited with code 0]
[New Thread 5936.0x2194]

Thread 20 received signal SIGSEGV, Segmentation fault.
[Switching to Thread 5936.0x2194]
0x739f1e01 in libgallium_wgl!DrvValidateVersion () from C:\Software\systools\GPU\GPU_Caps_Viewer\libgallium_wgl.dll
(gdb) bt
#0  0x739f1e01 in libgallium_wgl!DrvValidateVersion ()
   from C:\Software\systools\GPU\GPU_Caps_Viewer\libgallium_wgl.dll
#1  0x07427f80 in ?? ()
#2  0x739f302c in libgallium_wgl!DrvValidateVersion ()
   from C:\Software\systools\GPU\GPU_Caps_Viewer\libgallium_wgl.dll
#3  0x07427f80 in ?? ()
#4  0x76926739 in KERNEL32!BaseThreadInitThunk () from C:\Windows\SysWOW64\kernel32.dll
#5  0x77e68fd2 in ntdll!RtlGetFullPathName_UEx () from C:\Windows\SysWOW64\ntdll.dll
#6  0x77e68f9d in ntdll!RtlGetFullPathName_UEx () from C:\Windows\SysWOW64\ntdll.dll
#7  0x00000000 in ?? ()
(gdb) continue
Continuing.

Thread 20 received signal SIGSEGV, Segmentation fault.
0x739f1e01 in libgallium_wgl!DrvValidateVersion () from C:\Software\systools\GPU\GPU_Caps_Viewer\libgallium_wgl.dll
(gdb) bt
#0  0x739f1e01 in libgallium_wgl!DrvValidateVersion ()
   from C:\Software\systools\GPU\GPU_Caps_Viewer\libgallium_wgl.dll
#1  0x07427f80 in ?? ()
#2  0x739f302c in libgallium_wgl!DrvValidateVersion ()
   from C:\Software\systools\GPU\GPU_Caps_Viewer\libgallium_wgl.dll
#3  0x07427f80 in ?? ()
#4  0x76926739 in KERNEL32!BaseThreadInitThunk () from C:\Windows\SysWOW64\kernel32.dll
#5  0x77e68fd2 in ntdll!RtlGetFullPathName_UEx () from C:\Windows\SysWOW64\ntdll.dll
#6  0x77e68f9d in ntdll!RtlGetFullPathName_UEx () from C:\Windows\SysWOW64\ntdll.dll
#7  0x00000000 in ?? ()
(gdb) continue
Continuing.
[Thread 5936.0x2194 exited with code 3221225477]
[Thread 5936.0x1338 exited with code 3221225477]
[Thread 5936.0xfd8 exited with code 3221225477]
[Thread 5936.0x558 exited with code 3221225477]
[Thread 5936.0x2574 exited with code 3221225477]
[Thread 5936.0xe14 exited with code 3221225477]
[Thread 5936.0x104 exited with code 3221225477]
[Thread 5936.0x1d7c exited with code 3221225477]
[Thread 5936.0x1ef4 exited with code 3221225477]
[Thread 5936.0x2054 exited with code 3221225477]
[Thread 5936.0xad4 exited with code 3221225477]
[Thread 5936.0x2654 exited with code 3221225477]
[Thread 5936.0x258c exited with code 3221225477]
[Thread 5936.0x188c exited with code 3221225477]

Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.
(gdb) q

[mati865]

Isn't it a linker issue?

[pal1000]

It may be.

[pal1000]

I made some more discoveries. It seams that these errors are caused by me adding version information to binaries with Resource Hacker:

BFD: C:\Software\systools\GPU\GPU_Caps_Viewer\opengl32.dll: bad string table size 4176805888
Error while mapping shared library sections:
`C:\Software\systools\GPU\GPU_Caps_Viewer\opengl32.dll': not in executable format: file format not recognized

While not fatal they could make things harder for me to separate debug info and symbols and for debug link to still work seamlessly.

Now the interesting part. I discovered that enabling asserts ( -Db_ndebug=false) avoids both the crash in GPU Caps Viewer and the abort related to LLVM JIT when running under GDB.

Both the crash and debugging abort can be reproduced when building only gallium swrast and nothing else:
/mingw32/bin/meson setup build/gcc-x86 --libdir="lib/x86" --pkgconfig.relocatable -Dzstd=enabled --backend=ninja --buildtype=debug -Db_ndebug=true --prefer-static --force-fallback-for= -Dcpp_rtti=true -Dllvm=enabled -Dshared-llvm=disabled --cmake-prefix-path= -Dgallium-drivers=swrast -Dvulkan-drivers= -Dgallium-d3d10umd=false -Dspirv-to-dxil=false -Dgles1=auto -Dgles2=auto -Dshared-glapi=enabled -Degl=disabled -Dosmesa=false -Dbuild-tests=false -Dmicrosoft-clc=disabled -Dgallium-opencl=disabled -Dc_args="-march=core2 -pipe" -Dcpp_args="-march=core2 -pipe" -Dc_link_args="-static -ltre -lintl -liconv" -Dcpp_link_args="-static -ltre -lintl -liconv" -Ddraw-use-llvm=true -Dmin-windows-version=7
Try building 22.3.0-devel-daafeb9893620dcfaeffda44d525dee2f9c97dbe in MINGW32.

[mati865]

Can you try with LLD (-fuse-ld=lld)?
That could help pinpoint the problem.

[pal1000]

Okay, I finally figured out what's going on here. I have to make sure I extract debug information before adding version info with RH otherwise objcopy --only-keep-debug for example fails horribly without warning.

Anyway it would help to know how to extract both debug info and symbols to external .debug file.