Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-99108: Build the hashlib HACL* code as a static library. (fix wasm builds) #101917

Merged
merged 4 commits into from
Feb 14, 2023

Conversation

gpshead
Copy link
Member

@gpshead gpshead commented Feb 14, 2023

A followup to #101707 which broke some builds. This builds HACL* as a library in one place.

This doesn't solve all the problems, the wasm-wasi build is fixed by this. The wasm-enscripten build is not. Because it has a linker that can't cope with the same .a file being listed twice on its command line and decides to ignorantly process that as two inputs with duplicate symbols.

@gpshead
Copy link
Member Author

gpshead commented Feb 14, 2023

!buildbot wasm

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @gpshead for commit 3c7e18e 🤖

The command will test the builders whose names match following regular expression: wasm

The builders matched are:

  • wasm32-wasi PR
  • wasm32-emscripten node (dynamic linking) PR
  • wasm32-emscripten node (pthreads) PR
  • wasm32-emscripten browser (dynamic linking, no tests) PR

@gpshead
Copy link
Member Author

gpshead commented Feb 14, 2023

The wasm-wasi bot using llvm-ar to build libpython3.12.a appears happy with this.
The wasm-enscripten ones that use emscripten/emar to build libpython3.12.a still does not.

is this the problem? the .a file is showing up on the emcc linking command line
twice, is that really the source of the "duplicate" symbols?  is emcc that
dumb?
@gpshead
Copy link
Member Author

gpshead commented Feb 14, 2023

!buildbot wasm

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @gpshead for commit a5bb63e 🤖

The command will test the builders whose names match following regular expression: wasm

The builders matched are:

  • wasm32-wasi PR
  • wasm32-emscripten node (dynamic linking) PR
  • wasm32-emscripten node (pthreads) PR
  • wasm32-emscripten browser (dynamic linking, no tests) PR

@gpshead
Copy link
Member Author

gpshead commented Feb 14, 2023

Emscripten emcc which calls wasm-ld winds up with a duplicate of -LModules/_hacl -lHacl_Streaming_SHA2 on its command line or a direct Modules/_hacl/libHacl_Streaming_SHA2.a on its command line (the emcc script turns the first into the latter regardless). And whatever version of enscripten wasm-ld this is... is brain dead and does not deduplicate input files. Thus reporting "duplicate symbol" because the same .a is listed twice. 💩

Real linkers (like clang and thus llvm's ld.lld) do not have a problem with this.

@gpshead gpshead changed the title Build the hashlib HACL* code as a static library. gh:99108: Build the hashlib HACL* code as a static library. Feb 14, 2023
@gpshead gpshead changed the title gh:99108: Build the hashlib HACL* code as a static library. gh-99108: Build the hashlib HACL* code as a static library. Feb 14, 2023
@gpshead gpshead marked this pull request as ready for review February 14, 2023 22:44
@gpshead
Copy link
Member Author

gpshead commented Feb 14, 2023

Realistically: We should probably just combine _sha256 and _sha512 into a single _sha2 module as another way to work around this. The .so files for each of those are carrying the code for both regardless given how we link the shared libraries.

@gpshead gpshead changed the title gh-99108: Build the hashlib HACL* code as a static library. gh-99108: Build the hashlib HACL* code as a static library. (fix wasm builds) Feb 14, 2023
@gpshead
Copy link
Member Author

gpshead commented Feb 14, 2023

Realistically: We should probably just combine _sha256 and _sha512 into a single _sha2 module as another way to work around this. The .so files for each of those are carrying the code for both regardless given how we link the shared libraries.

I'll do that as its own follow on PR. This one at least addresses 2 of the 4 bot failures and defines the static library as we want regardless.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants