Dynamic selection of extended instructions sets at runtime for x86 architecture #1261
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I didn't review the code implementation, but the feature is really important for shipping build apps for end-customers. Great work, hope it will get merged! |
didzis
force-pushed
the
dynamic-arch
branch
3 times, most recently
from
46cfc26
to
add74c2
Compare
|
Any update on getting this merged? |
didzis
force-pushed
the
dynamic-arch
branch
3 times, most recently
from
1a6124e
to
996c1da
Compare
didzis
force-pushed
the
dynamic-arch
branch
3 times, most recently
from
baecb36
to
af4d524
Compare
didzis
force-pushed
the
dynamic-arch
branch
2 times, most recently
from
ae614fa
to
8b5da8e
Compare
…86 architecture This avoids illegal instruction set error when moving compiled binary between x86 machines but retains almost full performance available for each particular target machine. Implemented employing per-function selection of target instruction sets (sse3, ssse3, f16c, fma, avx, avx2, avx512). To alleviate application of of this patch, some parts are moved out to a separate header file. Due to certain limitations and to keep max performance in each setting, there is a degree of code redundancy. There still might be few runtime overheads because of the dynamic selection of optimized function which might disable inlining. To reduce the possible performance hit there are optimized functions introduced for some combinations of instruction sets. Enable runtime selection of extended instruction sets by setting DYNAMIC_ARCH directive. Works on Linux and macOS (x86_64) using GCC and Clang compilers. CPU feature detection is placed into separate file, but can be easily inlined into ggml.c
|
Hi, your implementation of |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR aims to solve the problem of distributing a single, highly performant binary executable to x86 target machines by dynamically identifying the available extended instruction sets at runtime. With this approach, there's no need to compromise between delivering optimal performance on each target machine (having to select only the common shared instruction sets) and ensuring broad distribution across as many target machines as possible without encountering illegal instruction errors.
Unfortunately, due to some limitations, the solution is not very clean. Specifically, there are copies of the same functionality for different instruction sets, as well as combinations of instruction sets, to achieve the best performance in the context of whisper.cpp.
However, I believe this might offer significant practical value for some. Therefore, I've submitted this PR for your consideration.