If ExifTool is used, binary compiled using perlcc depends on Config.so and cwd.so, even if --static/--staticxs is used. #423

jiucenglou · 2020-08-28T06:45:18Z

If ExifTool is used, binary compiled using perlcc depends on Config.so and Cwd.so, even if --static/--staticxs is used.
The reason might be that ExifTool utilizes Config.so .
The problem is that the binary compiled requires cperl installation and is not portable any more.
That is to say, the binary cannot run on another machine without cperl at the expected position.

Output of strace is shown below

stat("/usr/local/lib/cperl/5.30.0/x86_64-linux/auto/Config/Config.so", 0x7f4f8d872ba0) = -1 ENOENT (No such file or directory)
write(2, "Error: load_file /usr/local/lib/"..., 90Error: load_file /usr/local/lib/cperl/5.30.0/x86_64-linux/auto/Config/Config.so not found
) = 90

Could you suggest how to build the "true" static binary under this circumstance ? Many thanks !

The text was updated successfully, but these errors were encountered:

rurban · 2020-08-28T11:48:57Z

A true static binary needs the Perl to be compiled static, with all modules be compiled static also. It is possible, but I never heard that anyone did that.

jiucenglou · 2020-08-28T19:01:16Z

Is it possible to compile everything statically base on the install instruction for cperl (shown below) ? :D If yes, could you suggest which switches one should use ? Many thanks !

http://perl11.org/cperl/

INSTALL:

Simple:

    ./Configure -sder -Dusedevel -Dusecperl
    make -s -j4
    make -s -j4 test
    sudo make install

Experts:

    ./Configure -sder -Dusedevel -Dusecperl \
      -Accflags='-msse4.2 -DPERL_FAKE_SIGNATURE' --optimize='-O3 -g' \
      -Dinstallman1dir=none -Dinstallman3dir=none -Dinstallsiteman1dir=none \
      -Dinstallsiteman3dir=none
    make -s -j4 ECHO=true
    make -s -j4 ECHO=true test
    sudo make install

vadimkantorov · 2020-10-10T19:11:05Z

Also interested if one can compile cperl statically with its modules - for the goal of embedding the interpreter statically to run some scripts, similar to what might (?) be accomplished with staticperl. Maybe it even can be a lightweight alternative to WebPerl and compile fully into WebAssembly. It would also be very useful for perlcc

vadimkantorov · 2020-10-13T02:35:52Z

Here's how I tried configuring cperl to compile statically: bash +x ./Configure -sde -Uusedl -Dprefix=$PREFIX
log.zip

On my system it breaks with:

Run make depend now? [y]
`sh  cflags "optimize='-O3 --pipe'" generate_uudmap.o`  generate_uudmap.c
ld -o generate_uudmap -fstack-protector-strong -L/usr/local/lib -flto=4 -O3 --pipe generate_uudmap.o -lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
ld: unrecognized option '--pipe'
ld: use the --help option for usage information
Makefile:367: recipe for target 'generate_uudmap' failed
make: *** [generate_uudmap] Error 1

vadimkantorov · 2020-10-13T12:24:19Z

@rurban I tried to compile cperl with all modules in static way: -Dstatic_ext="B B/C Compress/Raw/Bzip2 Compress/Raw/Zlib Config Cpanel/JSON/XS Cwd Data/Dumper Devel/NYTProf Devel/PPPort Devel/Peek Digest/MD5 Digest/SHA Encode Fcntl File/DosGlob File/Glob Filter/Util/Call Hash/Util Hash/Util/FieldHash I18N/Langinfo IO IPC/SysV Internals/DumpArenas List/Util MIME/Base64 Math/BigInt/FastCalc Opcode POSIX PerlIO/encoding PerlIO/mmap PerlIO/scalar PerlIO/via SDBM_File Socket Storable Sys/Hostname Sys/Syslog Term/ReadKey Tie/Hash/NamedCapture Time/HiRes Time/Piece Unicode/Collate Unicode/Normalize XS/APItest XS/Typemap YAML/Safe mro re threads threads/shared"

At the end of compilation (linking time of the final cperl executable?) I got this error:

cc -o perl -fstack-protector-strong -L/usr/local/lib -flto=4 -O3 --pipe -Wl,-E perlmain.o  lib/auto/B/B.a lib/auto/B/C/C.a lib/auto/Compress/Raw/Bzi
p2/Bzip2.a lib/auto/Compress/Raw/Zlib/Zlib.a lib/auto/Config/Config.a lib/auto/Cpanel/JSON/XS/XS.a lib/auto/Cwd/Cwd.a lib/auto/Data/Dumper/Dumper.a 
lib/auto/Devel/NYTProf/NYTProf.a lib/auto/Devel/PPPort/PPPort.a lib/auto/Devel/Peek/Peek.a lib/auto/Digest/MD5/MD5.a lib/auto/Digest/SHA/SHA.a lib/auto/Encode/Encode.a lib/auto/Fcntl/Fcntl.a lib/auto/File/DosGlob/DosGlob.a lib/auto/File/Glob/Glob.a lib/auto/Filter/Util/Call/Call.a lib/auto/Hash/Util/Util.a lib/auto/Hash/Util/FieldHash/FieldHash.a lib/auto/I18N/Langinfo/Langinfo.a lib/auto/IO/IO.a lib/auto/IPC/SysV/SysV.a lib/auto/Internals/DumpArenas/DumpArenas.a lib/auto/List/Util/Util.a lib/auto/MIME/Base64/Base64.a lib/auto/Math/BigInt/FastCalc/FastCalc.a lib/auto/Opcode/Opcode.a lib/auto/POSIX/POSIX.a lib/auto/PerlIO/encoding/encoding.a lib/auto/PerlIO/mmap/mmap.a lib/auto/PerlIO/scalar/scalar.a lib/auto/PerlIO/via/via.a lib/auto/SDBM_File/SDBM_File.a lib/auto/Socket/Socket.a lib/auto/Storable/Storable.a lib/auto/Sys/Hostname/Hostname.a lib/auto/Sys/Syslog/Syslog.a lib/auto/Term/ReadKey/ReadKey.a lib/auto/Tie/Hash/NamedCapture/NamedCapture.a lib/auto/Time/HiRes/HiRes.a lib/auto/Time/Piece/Piece.a lib/auto/Unicode/Collate/Collate.a lib/auto/Unicode/Normalize/Normalize.a lib/auto/XS/APItest/APItest.a lib/auto/XS/Typemap/Typemap.a lib/auto/YAML/Safe/Safe.a lib/auto/mro/mro.a lib/auto/re/re.a lib/auto/threads/threads.a lib/auto/threads/shared/shared.a lib/auto/Encode/Byte/Byte.a lib/auto/Encode/CN/CN.a lib/auto/Encode/EBCDIC/EBCDIC.a lib/auto/Encode/JP/JP.a lib/auto/Encode/KR/KR.a lib/auto/Encode/Symbol/Symbol.a lib/auto/Encode/TW/TW.a lib/auto/Encode/Unicode/Unicode.a libperl.a `cat ext.libs` -lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
utf8.o (symbol from plugin): In function `nonchar_cp_format':
(.text+0x0): multiple definition of `isExclusion'
Normalize.o (symbol from plugin):(.text+0x0): first defined here
utf8.o (symbol from plugin): In function `nonchar_cp_format':
(.text+0x0): multiple definition of `isSingleton'
Normalize.o (symbol from plugin):(.text+0x0): first defined here
utf8.o (symbol from plugin): In function `nonchar_cp_format':
(.text+0x0): multiple definition of `isNonStDecomp'
Normalize.o (symbol from plugin):(.text+0x0): first defined here
utf8.o (symbol from plugin): In function `nonchar_cp_format':
(.text+0x0): multiple definition of `isComp2nd'
Normalize.o (symbol from plugin):(.text+0x0): first defined here
../../perl.h:7634:19: warning: type of ‘PL_nan’ does not match original declaration [-Wlto-type-mismatch]
perl.h:7602:19: note: ‘PL_nan’ was previously declared here
 INFNAN_U8_NV_DECL PL_nan = { { DOUBLENANBYTES } };

This is not so many errors, so hopefully can be fixed easily

vadimkantorov · 2020-10-13T12:29:48Z

Maybe it's Unicode/Normalize module to blame. I'll try to confirm.

PL_nan warning is concerning though: it seems that two ifdef branches trigger on my system (WSLv1, Ubuntu 18.04) and it gets defined twice:

cperl/perl.h

Line 7602 in 071f7ab

INFNAN_U8_NV_DECL PL_nan = { { DOUBLENANBYTES } };
cperl/perl.h

Line 7634 in 071f7ab

INFNAN_NV_U8_DECL PL_nan;

rurban · 2020-10-13T12:34:28Z

The duplicate Normalize symbols are easy to fix, and should be in a seperate ticket.

The nan stuff is an upstream problem, and indeed a severe bug

vadimkantorov · 2020-10-13T12:35:52Z

Also, regular expression module still got put into some dynamic_ext_re

vadimkantorov · 2020-10-13T12:50:05Z

A more lightweight setup:

./Configure -sde -Dprefix=$PREFIX -Dstatic_ext="Encode Config IO Fcntl Cwd Storable Digest/MD5 List/Util Time/HiRes File/Glob PerlIO/scalar"
bash +x Makefile.SH
make -j4
# succeeds

# remove installman line
make install
# fails

make[1]: Leaving directory '/mnt/c/Users/user/wiptlmgr/cperl-5.30.0/cpan/Devel-NYTProf'
Manifying 1 pod document
make[1]: Leaving directory '/mnt/c/Users/user/wiptlmgr/cperl-5.30.0/cpan/Math-BigInt-FastCalc'
./perl -Ilib -I. -f pod/buildtoc -q



        Everything is up to date. Type 'make test' to run test suite.
./perl -Ilib -I. installperl --destdir=
Use of uninitialized value in XS subroutine entry at lib/File/Glob.pm line 68.
Use of uninitialized value in XS subroutine entry at lib/File/Glob.pm line 68.
Invalid version format (non-numeric data) at lib/File/Glob.pm line 68.
Compilation failed in require at installperl line 394.
BEGIN failed--compilation aborted at installperl line 394.
Makefile:589: recipe for target 'install-all' failed
make: *** [install-all] Error 255

lib/File/Glob.pm:68 has XSLoader::load();

vadimkantorov · 2020-10-13T12:51:48Z

What is XS and how do I link it statically?

vadimkantorov · 2020-10-13T14:40:55Z

Basically, static perl compiles fine (except for NaN issue), but Glob module fails:

vadimkantorov@DESKTOP-4UF8FID:/mnt/c/Users/user/wiptlmgr/cperl-5.30.0$ ./perl -Ilib -I. installperl --destdir=
Use of uninitialized value in XS subroutine entry at lib/File/Glob.pm line 68.
Use of uninitialized value in XS subroutine entry at lib/File/Glob.pm line 68.
Invalid version format (non-numeric data) at lib/File/Glob.pm line 68.
Compilation failed in require at installperl line 394.
BEGIN failed--compilation aborted at installperl line 394.

vadimkantorov · 2020-10-13T14:44:17Z

It seems that the reason is that XSLoader module and XSLoader::load() doesn't work well.

I found XSLoader in Makefile:

DL_SRC = ext/DynaLoader/DynaLoader.xs ext/DynaLoader/dlboot.c ext/DynaLoader/dlutils.c          ext/DynaLoader/XSLoader.c

So it seems it still wants to do dynamic loading for something. What does it try to load dynamically for globbinb? libc?

vadimkantorov · 2020-10-13T15:17:07Z

Here lives that Glob.pm file: https://github.com/perl11/cperl/tree/0f5b639e779cc36fc203afefe79c7810906fac65/ext/File-Glob

Is XSLoader FFI supposed to support XSLoader::load() even if https://github.com/perl11/cperl/blob/0f5b639e779cc36fc203afefe79c7810906fac65/ext/File-Glob/Glob.xs is compiled statically?

Disalbing usedl completely, unfortunately leads to:

/bin/ln -s xsutils.c xsutilsmini.c
`sh  cflags "optimize='-O3 --pipe'" regexec.o`  regexec.c
`sh  cflags "optimize='-O3 --pipe'" utf8.o`  utf8.c
`sh  cflags "optimize='-O3 --pipe'" universal.o`  universal.c
`sh  cflags "optimize='-O3 --pipe'" perlapi.o`  perlapi.c
ld -o generate_uudmap -fstack-protector-strong -L/usr/local/lib -flto=4 -O3 --pipe generate_uudmap.o -lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
ld: unrecognized option '--pipe'
ld: use the --help option for usage information
Makefile:367: recipe for target 'generate_uudmap' failed

vadimkantorov@DESKTOP-4UF8FID:/mnt/c/Users/user/wiptlmgr$ /usr/bin/ld --version
GNU ld (GNU Binutils for Ubuntu) 2.30
Copyright (C) 2018 Free Software Foundation, Inc.

vadimkantorov · 2020-10-13T16:12:40Z

It seems that all static modules that use XSLoader have this problem: during make install or just in runtime. Here's a log of a similar problem of PerlIO/Scalar while running perlcc:

/mnt/c/Users/user/wiptlmgr/cperlprefix/bin/perlcc: Unexpected compiler output
Use of uninitialized value $cvname in string eq at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/B/C.pm line 4981.
 Use of uninitialized value $cvname in string eq at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/B/C.pm line 4981.
 Use of uninitialized value $cvname in string eq at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/B/C.pm line 4981.
 Use of uninitialized value $cvname in string eq at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/B/C.pm line 4981.
 Use of uninitialized value $cvname in string eq at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/B/C.pm line 4981.
 Use of uninitialized value $cvname in string eq at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/B/C.pm line 4981.
 Use of uninitialized value $cvname in string eq at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/B/C.pm line 4981.
 Use of uninitialized value $cvname in string eq at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/B/C.pm line 4981.
 Use of uninitialized value $cvname in string eq at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/B/C.pm line 4981.
 Use of uninitialized value $cvname in string eq at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/B/C.pm line 4981.
 Use of uninitialized value $cvname in string eq at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/B/C.pm line 4981.


 Use of uninitialized value in XS subroutine entry at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/PerlIO/scalar.pm line 4, <__ANONIO__> line 2459.
 Use of uninitialized value in XS subroutine entry at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/PerlIO/scalar.pm line 4, <__ANONIO__> line 2459.
 Invalid version format (non-numeric data) at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/PerlIO/scalar.pm line 4, <__ANONIO__> line 2459.
 Compilation failed in require at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/B/C.pm line 6225, <__ANONIO__> line 2459.
 CHECK failed--call queue aborted.

vadimkantorov · 2020-10-13T17:03:48Z

It seems that static modules also misbehave in perlcc-produced source code:

Compiling without File/Glob and PerlIO/Scalar leads to following errors while compiling tlmgr.c produced by perlcc. It seems that these boot function names are not formatted well in this case.


tlmgr.c:77:26: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘/’ token
 EXTERN_C void boot_Digest/MD5 (pTHX_ CV* cv);
                          ^
tlmgr.c:78:24: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘/’ token
 EXTERN_C void boot_List/Util (pTHX_ CV* cv);
                        ^
tlmgr.c:79:24: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘/’ token
 EXTERN_C void boot_Time/HiRes (pTHX_ CV* cv);
                        ^
tlmgr.c:80:26: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘/’ token
 EXTERN_C void boot_Encode/Byte (pTHX_ CV* cv);
                          ^
tlmgr.c:81:26: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘/’ token
 EXTERN_C void boot_Encode/CN (pTHX_ CV* cv);
                          ^
tlmgr.c:82:26: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘/’ token
 EXTERN_C void boot_Encode/EBCDIC (pTHX_ CV* cv);
                          ^
tlmgr.c:83:26: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘/’ token
 EXTERN_C void boot_Encode/JP (pTHX_ CV* cv);

tlmgr.c source code: tlmgr.zip

vadimkantorov · 2020-10-13T17:14:19Z

To summarize the encountered problems:

XSLoader-using modules fail when compiled statically with Use of uninitialized value in XS subroutine entry
XSLoader is used in File/Glob which is used at install-time, leading to failing install if File/Glob is compiled statically
perlcc produces malformed code for static modules init functions
disabling dynamic module loading by -Uusedl fails very badly during cperl compilation

XSLoader seems to be inside DynaLoader.o package which by default is linked into libperl.a.

vadimkantorov mentioned this issue Oct 12, 2020

[discussion] Static Perl compilation without PAR + make optional / drop more dependencies plk/biber#338

Closed

vadimkantorov mentioned this issue Oct 13, 2020

Unicode/Normalize module produces duplicate symbols under static linking #425

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

If ExifTool is used, binary compiled using perlcc depends on Config.so and cwd.so, even if --static/--staticxs is used. #423

If ExifTool is used, binary compiled using perlcc depends on Config.so and cwd.so, even if --static/--staticxs is used. #423

jiucenglou commented Aug 28, 2020 •

edited

Loading

rurban commented Aug 28, 2020

jiucenglou commented Aug 28, 2020 •

edited

Loading

vadimkantorov commented Oct 10, 2020 •

edited

Loading

vadimkantorov commented Oct 13, 2020 •

edited

Loading

vadimkantorov commented Oct 13, 2020 •

edited

Loading

vadimkantorov commented Oct 13, 2020 •

edited

Loading

rurban commented Oct 13, 2020 •

edited

Loading

vadimkantorov commented Oct 13, 2020

vadimkantorov commented Oct 13, 2020

vadimkantorov commented Oct 13, 2020

vadimkantorov commented Oct 13, 2020

vadimkantorov commented Oct 13, 2020

vadimkantorov commented Oct 13, 2020 •

edited

Loading

vadimkantorov commented Oct 13, 2020

vadimkantorov commented Oct 13, 2020 •

edited

Loading

vadimkantorov commented Oct 13, 2020 •

edited

Loading

If ExifTool is used, binary compiled using perlcc depends on Config.so and cwd.so, even if --static/--staticxs is used. #423

If ExifTool is used, binary compiled using perlcc depends on Config.so and cwd.so, even if --static/--staticxs is used. #423

Comments

jiucenglou commented Aug 28, 2020 • edited Loading

rurban commented Aug 28, 2020

jiucenglou commented Aug 28, 2020 • edited Loading

vadimkantorov commented Oct 10, 2020 • edited Loading

vadimkantorov commented Oct 13, 2020 • edited Loading

vadimkantorov commented Oct 13, 2020 • edited Loading

vadimkantorov commented Oct 13, 2020 • edited Loading

rurban commented Oct 13, 2020 • edited Loading

vadimkantorov commented Oct 13, 2020

vadimkantorov commented Oct 13, 2020

vadimkantorov commented Oct 13, 2020

vadimkantorov commented Oct 13, 2020

vadimkantorov commented Oct 13, 2020

vadimkantorov commented Oct 13, 2020 • edited Loading

vadimkantorov commented Oct 13, 2020

vadimkantorov commented Oct 13, 2020 • edited Loading

vadimkantorov commented Oct 13, 2020 • edited Loading

jiucenglou commented Aug 28, 2020 •

edited

Loading

jiucenglou commented Aug 28, 2020 •

edited

Loading

vadimkantorov commented Oct 10, 2020 •

edited

Loading

vadimkantorov commented Oct 13, 2020 •

edited

Loading

vadimkantorov commented Oct 13, 2020 •

edited

Loading

vadimkantorov commented Oct 13, 2020 •

edited

Loading

rurban commented Oct 13, 2020 •

edited

Loading

vadimkantorov commented Oct 13, 2020 •

edited

Loading

vadimkantorov commented Oct 13, 2020 •

edited

Loading

vadimkantorov commented Oct 13, 2020 •

edited

Loading