Clang 16 and 17 segfault while compiling OpenBLAS

[giordano]

With a x86_64-linux-musl build of Clang, compilation of OpenBLAS silently dies while compiling some x86_64 kernels, returning exit code 255. For example, with OpenBLAS 0.3.28 + OpenMathLib/OpenBLAS#4859 (which resolves some warnings issues by Clang) I get

sandbox:${WORKSPACE}/srcdir/OpenBLAS-0.3.28/kernel # cc -c -O2 -DSMALL_MATRIX_OPT -DMAX_STACK_ALLOC=2048 -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DDYNAMIC_ARCH -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=512 -DMAX_PARALLEL_NUMBER=1 -DBUILD_BFLOAT16 -DBUILD_SINGLE=1 -DBUILD_DOUBLE=1 -DBUILD_COMPLEX=1 -DBUILD_COMPLEX16=1 -DVERSION=\"0.3.28\" -msse3 -mssse3 -msse4.1 -mavx -mavx2 -march=cooperlake -mavx2 -UASMNAME -UASMFNAME -UNAME -UCNAME -UCHAR_NAME -UCHAR_CNAME -DASMNAME=_sbgemv_t_COOPERLAKE -DASMFNAME=_sbgemv_t_COOPERLAKE_ -DNAME=sbgemv_t_COOPERLAKE_ -DCNAME=sbgemv_t_COOPERLAKE -DCHAR_NAME=\"sbgemv_t_COOPERLAKE_\" -DCHAR_CNAME=\"sbgemv_t_COOPERLAKE\" -DNO_AFFINITY -DTS=_COOPERLAKE -I.. -DBUILD_KERNEL -DTABLE_NAME=gotoblas_COOPERLAKE -march=cooperlake -DBFLOAT16 -UDOUBLE  -UCOMPLEX -UCOMPLEX ../kernel/x86_64/sbgemv_t.c -o sbgemv_t_COOPERLAKE.o; echo $?
255

It's particularly worrying that Clang just dies without printing any error message, it's even hard to tell what's the issue.

This happens with any x86_64 target, this was first observed in JuliaPackaging/Yggdrasil#9234 with x86_64-linux-gnu, x86_64-apple-darwin, and x86_64-unknown-freebsd.

Clang versions (here showing x86_64-apple-darwin, but it's the same version for all different targets)

sandbox:${WORKSPACE}/srcdir/OpenBLAS-0.3.28 # clang --version
clang version 17.0.6 (/home/tim/.cache/BinaryBuilder/downloads/clones/llvm-project.git-1df819a03ecf6890e3787b27bfd4f160aeeeeacd50a98d003be8b0893f11a9be 6009708b4367171ccdbf4b5905cb6a803753fe18)
Target: x86_64-apple-darwin14
Thread model: posix
InstalledDir: /opt/x86_64-linux-musl/bin
[...]
sandbox:${WORKSPACE}/srcdir/OpenBLAS-0.3.28 # clang --version
clang version 16.0.6 (/home/gbaraldi/.julia/dev/BinaryBuilderBase/deps/downloads/clones/llvm-project.git-1df819a03ecf6890e3787b27bfd4f160aeeeeacd50a98d003be8b0893f11a9be 7cbf1a2591520c2491aa35339f227775f4d3adf6)
Target: x86_64-apple-darwin14
Thread model: posix
InstalledDir: /opt/x86_64-linux-musl/bin

Unfortunately I don't have builds of Clang 18 or nightly at hand.

[giordano]

A little bit more of information: the exit code 255 mentioned above was coming from the compiler wrapper (Edit: actually, the culprit was ccache, used inside the compiler wrapper) we use in our builds, if I call the actual clang compiler directly I get:

sandbox:${WORKSPACE}/srcdir/OpenBLAS-0.3.28/kernel # /opt/x86_64-linux-musl/bin/clang -target x86_64-apple-darwin14 --sysroot=/opt/x86_64-apple-darwin14/x86_64-apple-darwin14/sys-root -nostdinc++ -isystem /opt/x86_64-apple-darwin14/x86_64-apple-darwin14/sys-root/usr/include/c++/v1 -Wno-unused-command-line-argument -mmacosx-version-min=10.10 -c -O2 -DSMALL_MATRIX_OPT -DMAX_STACK_ALLOC=2048 -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DDYNAMIC_ARCH -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=512 -DMAX_PARALLEL_NUMBER=1 -DBUILD_BFLOAT16 -DBUILD_SINGLE=1 -DBUILD_DOUBLE=1 -DBUILD_COMPLEX=1 -DBUILD_COMPLEX16=1 -DVERSION="0.3.28" -msse3 -mssse3 -msse4.1 -mavx -mavx2 -march=cooperlake -mavx2 -UASMNAME -UASMFNAME -UNAME -UCNAME -UCHAR_NAME -UCHAR_CNAME -DASMNAME=_sbgemv_t_COOPERLAKE -DASMFNAME=_sbgemv_t_COOPERLAKE_ -DNAME=sbgemv_t_COOPERLAKE_ -DCNAME=sbgemv_t_COOPERLAKE -DCHAR_NAME="sbgemv_t_COOPERLAKE_" -DCHAR_CNAME="sbgemv_t_COOPERLAKE" -DNO_AFFINITY -DTS=_COOPERLAKE -I.. -DBUILD_KERNEL -DTABLE_NAME=gotoblas_COOPERLAKE -march=cooperlake -DBFLOAT16 -UDOUBLE -UCOMPLEX -UCOMPLEX ../kernel/x86_64/sbgemv_t.c -o sbgemv_t_COOPERLAKE.o; echo $?
Segmentation fault
139

With gdb I can see the following stack trace:

(gdb) run
Starting program: /opt/x86_64-linux-musl/bin/clang -target x86_64-apple-darwin14 --sysroot=/opt/x86_64-apple-darwin14/x86_64-apple-darwin14/sys-root -nostdinc++ -isystem /opt/x86_64-apple-darwin14/x86_64-
apple-darwin14/sys-root/usr/include/c++/v1 -Wno-unused-command-line-argument -mmacosx-version-min=10.10 -c -O2 -DSMALL_MATRIX_OPT -DMAX_STACK_ALLOC=2048 -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DDYNAMIC_ARCH
 -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=512 -DMAX_PARALLEL_NUMBER=1 -DBUILD_BFLOAT16 -DBUILD_SINGLE=1 -DBUILD_DOUBLE=1 -DBUILD_COMPLEX=1 -DBUILD_COMPLEX16=1 -DVERSION=0.3.28 -msse3 -mssse3 -msse4.1 -ma
vx -mavx2 -march=cooperlake -mavx2 -UASMNAME -UASMFNAME -UNAME -UCNAME -UCHAR_NAME -UCHAR_CNAME -DASMNAME=_sbgemv_t_COOPERLAKE -DASMFNAME=_sbgemv_t_COOPERLAKE_ -DNAME=sbgemv_t_COOPERLAKE_ -DCNAME=sbgemv_t
_COOPERLAKE -DCHAR_NAME=sbgemv_t_COOPERLAKE_ -DCHAR_CNAME=sbgemv_t_COOPERLAKE -DNO_AFFINITY -DTS=_COOPERLAKE -I.. -DBUILD_KERNEL -DTABLE_NAME=gotoblas_COOPERLAKE -march=cooperlake -DBFLOAT16 -UDOUBLE -UCO
MPLEX -UCOMPLEX ../kernel/x86_64/sbgemv_t.c -o sbgemv_t_COOPERLAKE.o

Program received signal SIGSEGV, Segmentation fault.
0x00007fffecd21fa2 in llvm::EVT::getEVT(llvm::Type*, bool) () from /opt/x86_64-linux-musl/bin/../lib/libLLVM-16.so
(gdb) where
#0  0x00007fffecd21fa2 in llvm::EVT::getEVT(llvm::Type*, bool) () from /opt/x86_64-linux-musl/bin/../lib/libLLVM-16.so
#1  0x00007ffff02b9b3c in llvm::TargetLoweringBase::getValueType(llvm::DataLayout const&, llvm::Type*, bool) const [clone .part.0] () from /opt/x86_64-linux-musl/bin/../lib/libLLVM-16.so
#2  0x00007ffff02bc930 in llvm::BasicTTIImplBase<llvm::X86TTIImpl>::getTypeLegalizationCost(llvm::Type*) const () from /opt/x86_64-linux-musl/bin/../lib/libLLVM-16.so
#3  0x00007ffff02bee16 in llvm::X86TTIImpl::getVectorInstrCost(unsigned int, llvm::Type*, llvm::TargetTransformInfo::TargetCostKind, unsigned int, llvm::Value*, llvm::Value*) ()
   from /opt/x86_64-linux-musl/bin/../lib/libLLVM-16.so
#4  0x00007ffff02c0550 in llvm::X86TTIImpl::getShuffleCost(llvm::TargetTransformInfo::ShuffleKind, llvm::VectorType*, llvm::ArrayRef<int>, llvm::TargetTransformInfo::TargetCostKind, int, llvm::VectorType*, llvm::ArrayRef<llvm::Value const*>) () from /opt/x86_64-linux-musl/bin/../lib/libLLVM-16.so
#5  0x00007ffff02bf75d in llvm::X86TTIImpl::getVectorInstrCost(unsigned int, llvm::Type*, llvm::TargetTransformInfo::TargetCostKind, unsigned int, llvm::Value*, llvm::Value*) ()
   from /opt/x86_64-linux-musl/bin/../lib/libLLVM-16.so
#6  0x00007ffff02c0550 in llvm::X86TTIImpl::getShuffleCost(llvm::TargetTransformInfo::ShuffleKind, llvm::VectorType*, llvm::ArrayRef<int>, llvm::TargetTransformInfo::TargetCostKind, int, llvm::VectorType*, llvm::ArrayRef<llvm::Value const*>) () from /opt/x86_64-linux-musl/bin/../lib/libLLVM-16.so
#7  0x00007ffff02bf75d in llvm::X86TTIImpl::getVectorInstrCost(unsigned int, llvm::Type*, llvm::TargetTransformInfo::TargetCostKind, unsigned int, llvm::Value*, llvm::Value*) ()
   from /opt/x86_64-linux-musl/bin/../lib/libLLVM-16.so
#8  0x00007ffff02c0550 in llvm::X86TTIImpl::getShuffleCost(llvm::TargetTransformInfo::ShuffleKind, llvm::VectorType*, llvm::ArrayRef<int>, llvm::TargetTransformInfo::TargetCostKind, int, llvm::VectorType*, llvm::ArrayRef<llvm::Value const*>) () from /opt/x86_64-linux-musl/bin/../lib/libLLVM-16.so
#9  0x00007ffff02bf75d in llvm::X86TTIImpl::getVectorInstrCost(unsigned int, llvm::Type*, llvm::TargetTransformInfo::TargetCostKind, unsigned int, llvm::Value*, llvm::Value*) ()
   from /opt/x86_64-linux-musl/bin/../lib/libLLVM-16.so
#10 0x00007ffff02c0550 in llvm::X86TTIImpl::getShuffleCost(llvm::TargetTransformInfo::ShuffleKind, llvm::VectorType*, llvm::ArrayRef<int>, llvm::TargetTransformInfo::TargetCostKind, int, llvm::VectorType*, llvm::ArrayRef<llvm::Value const*>) () from /opt/x86_64-linux-musl/bin/../lib/libLLVM-16.so
#11 0x00007ffff02bf75d in llvm::X86TTIImpl::getVectorInstrCost(unsigned int, llvm::Type*, llvm::TargetTransformInfo::TargetCostKind, unsigned int, llvm::Value*, llvm::Value*) ()
   from /opt/x86_64-linux-musl/bin/../lib/libLLVM-16.so
#12 0x00007ffff02c0550 in llvm::X86TTIImpl::getShuffleCost(llvm::TargetTransformInfo::ShuffleKind, llvm::VectorType*, llvm::ArrayRef<int>, llvm::TargetTransformInfo::TargetCostKind, int, llvm::VectorType*, llvm::ArrayRef<llvm::Value const*>) () from /opt/x86_64-linux-musl/bin/../lib/libLLVM-16.so
#13 0x00007ffff02bf75d in llvm::X86TTIImpl::getVectorInstrCost(unsigned int, llvm::Type*, llvm::TargetTransformInfo::TargetCostKind, unsigned int, llvm::Value*, llvm::Value*) ()
   from /opt/x86_64-linux-musl/bin/../lib/libLLVM-16.so
#14 0x00007ffff02c0550 in llvm::X86TTIImpl::getShuffleCost(llvm::TargetTransformInfo::ShuffleKind, llvm::VectorType*, llvm::ArrayRef<int>, llvm::TargetTransformInfo::TargetCostKind, int, llvm::VectorType*, llvm::ArrayRef<llvm::Value const*>) () from /opt/x86_64-linux-musl/bin/../lib/libLLVM-16.so
#15 0x00007ffff02bf75d in llvm::X86TTIImpl::getVectorInstrCost(unsigned int, llvm::Type*, llvm::TargetTransformInfo::TargetCostKind, unsigned int, llvm::Value*, llvm::Value*) ()
   from /opt/x86_64-linux-musl/bin/../lib/libLLVM-16.so
#16 0x00007ffff02c0550 in llvm::X86TTIImpl::getShuffleCost(llvm::TargetTransformInfo::ShuffleKind, llvm::VectorType*, llvm::ArrayRef<int>, llvm::TargetTransformInfo::TargetCostKind, int, llvm::VectorType*, llvm::ArrayRef<llvm::Value const*>) () from /opt/x86_64-linux-musl/bin/../lib/libLLVM-16.so
#17 0x00007ffff02bf75d in llvm::X86TTIImpl::getVectorInstrCost(unsigned int, llvm::Type*, llvm::TargetTransformInfo::TargetCostKind, unsigned int, llvm::Value*, llvm::Value*) ()
   from /opt/x86_64-linux-musl/bin/../lib/libLLVM-16.so
#18 0x00007ffff02c0550 in llvm::X86TTIImpl::getShuffleCost(llvm::TargetTransformInfo::ShuffleKind, llvm::VectorType*, llvm::ArrayRef<int>, llvm::TargetTransformInfo::TargetCostKind, int, llvm::VectorType*, llvm::ArrayRef<llvm::Value const*>) () from /opt/x86_64-linux-musl/bin/../lib/libLLVM-16.so
#19 0x00007ffff02bf75d in llvm::X86TTIImpl::getVectorInstrCost(unsigned int, llvm::Type*, llvm::TargetTransformInfo::TargetCostKind, unsigned int, llvm::Value*, llvm::Value*) ()
   from /opt/x86_64-linux-musl/bin/../lib/libLLVM-16.so
#20 0x00007ffff02c0550 in llvm::X86TTIImpl::getShuffleCost(llvm::TargetTransformInfo::ShuffleKind, llvm::VectorType*, llvm::ArrayRef<int>, llvm::TargetTransformInfo::TargetCostKind, int, llvm::VectorType*, llvm::ArrayRef<llvm::Value const*>) () from /opt/x86_64-linux-musl/bin/../lib/libLLVM-16.so
#21 0x00007ffff02bf75d in llvm::X86TTIImpl::getVectorInstrCost(unsigned int, llvm::Type*, llvm::TargetTransformInfo::TargetCostKind, unsigned int, llvm::Value*, llvm::Value*) ()
   from /opt/x86_64-linux-musl/bin/../lib/libLLVM-16.so

[EugeneZelenko]

Could you please try 19 or main branch? https://godbolt.org should be helpful.

[llvmbot]

@llvm/issue-subscribers-backend-x86

Author: Mosè Giordano (giordano)

With a `x86_64-linux-musl` build of Clang, compilation of OpenBLAS silently dies while compiling some x86_64 kernels, returning exit code 255. For example, with OpenBLAS 0.3.28 + https://github.com/OpenMathLib/OpenBLAS/pull/4859 (which resolves some warnings issues by Clang) I get ```console sandbox:${WORKSPACE}/srcdir/OpenBLAS-0.3.28/kernel # cc -c -O2 -DSMALL_MATRIX_OPT -DMAX_STACK_ALLOC=2048 -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DDYNAMIC_ARCH -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=512 -DMAX_PARALLEL_NUMBER=1 -DBUILD_BFLOAT16 -DBUILD_SINGLE=1 -DBUILD_DOUBLE=1 -DBUILD_COMPLEX=1 -DBUILD_COMPLEX16=1 -DVERSION=\"0.3.28\" -msse3 -mssse3 -msse4.1 -mavx -mavx2 -march=cooperlake -mavx2 -UASMNAME -UASMFNAME -UNAME -UCNAME -UCHAR_NAME -UCHAR_CNAME -DASMNAME=_sbgemv_t_COOPERLAKE -DASMFNAME=_sbgemv_t_COOPERLAKE_ -DNAME=sbgemv_t_COOPERLAKE_ -DCNAME=sbgemv_t_COOPERLAKE -DCHAR_NAME=\"sbgemv_t_COOPERLAKE_\" -DCHAR_CNAME=\"sbgemv_t_COOPERLAKE\" -DNO_AFFINITY -DTS=_COOPERLAKE -I.. -DBUILD_KERNEL -DTABLE_NAME=gotoblas_COOPERLAKE -march=cooperlake -DBFLOAT16 -UDOUBLE -UCOMPLEX -UCOMPLEX ../kernel/x86_64/sbgemv_t.c -o sbgemv_t_COOPERLAKE.o; echo $? 255 ``` It's particularly worrying that Clang just dies without printing any error message, it's even hard to tell what's the issue.

This happens with any x86_64 target, this was first observed in JuliaPackaging/Yggdrasil#9234 with x86_64-linux-gnu, x86_64-apple-darwin, and x86_64-unknown-freebsd.

Clang versions (here showing x86_64-apple-darwin, but it's the same version for all different targets)

sandbox:${WORKSPACE}/srcdir/OpenBLAS-0.3.28 # clang --version
clang version 17.0.6 (/home/tim/.cache/BinaryBuilder/downloads/clones/llvm-project.git-1df819a03ecf6890e3787b27bfd4f160aeeeeacd50a98d003be8b0893f11a9be 6009708b4367171ccdbf4b5905cb6a803753fe18)
Target: x86_64-apple-darwin14
Thread model: posix
InstalledDir: /opt/x86_64-linux-musl/bin
[...]
sandbox:${WORKSPACE}/srcdir/OpenBLAS-0.3.28 # clang --version
clang version 16.0.6 (/home/gbaraldi/.julia/dev/BinaryBuilderBase/deps/downloads/clones/llvm-project.git-1df819a03ecf6890e3787b27bfd4f160aeeeeacd50a98d003be8b0893f11a9be 7cbf1a2591520c2491aa35339f227775f4d3adf6)
Target: x86_64-apple-darwin14
Thread model: posix
InstalledDir: /opt/x86_64-linux-musl/bin

Unfortunately I don't have builds of Clang 18 or nightly at hand.

[DimitryAndric]

Maybe with musl you are running out of stack earlier than with e.g. glibc? From the stack trace you showed, it looks a lot like it is going into a rather deep recursion, which might blow out the stack. In such cases, clang's driver cannot produced a preprocessed test case.

[giordano]

I don't think it's musl-specific, I can reproduce the segmentation fault on Godbolt (using a locally preprocessed file as input, preprocessing works): https://godbolt.org/z/eKfh1vjzc. However it seems to be working with Clang 18: https://godbolt.org/z/dW6Pz8PdY. @martin-frbg would you be able to tell if this is the assembly you'd expect for the sbgemv_n_COOPERLAKE.o object file?