Compiler: enable parallel codegen with MT #14748
Conversation
|
This looks great. But it also seems to be a mix of different changes. Could we extract the independent refactorings (such as extracting |
ysbaddaden
force-pushed
the
feature/compiler-mt-codegen-2
branch
from
3a08d9e
to
ac91f7c
Compare
|
Rebased on top of #14760. |
Refactors `Crystal::Compiler`: 1. extracts `#sequential_codegen`, `#parallel_codegen` and `#fork_codegen` methods; 2. merges `#codegen_many_units` into `#codegen` directly; 3. stops collecting reused units: `#fork_codegen` now updates `CompilationUnit#reused_compilation_unit?` state as reported by the forked processes, and `#print_codegen_stats` now counts & filters the reused units. Prerequisite for #14748 that will introduce `#mt_codegen`.
When compiled with -Dpreview_mt the compiler will take advantage of the MT environment to codegen the compilation units in parallel, avoiding fork (that's not supported with MT) and allowing parallel codegen on Windows.
ysbaddaden
force-pushed
the
feature/compiler-mt-codegen-2
branch
from
5303d59
to
96b6f77
Compare
|
Rebased from master that merged #14760 (prerequisite) and ready for review. |
Isn't this addressed (using |
|
@beta-ziliani this could be a compile time ENV to change the default number of threads/schedulers. It's tangential to this pull request. |
|
The RFC-2 link in the OP here points to a non-existing URL. Think it was supposed to be crystal-lang/rfcs#2? |
Implements parallel codegen of object files when MT is enabled in the compiler (
-Dpreview_mt).It only impacts codegen for compilations with more than one compilation unit (module), that is when neither of
--single-module,--releaseor--cross-compileis specified. This behavior is identical to the fork based codegen.Advantages:
The main points are increased portability and simpler logic, despite having to take care of LLVM thread safety quirks (see comments).
Issues:
The
threadsarg actually depicts the number of fibers, not threads, which is confusing and problematic: increasingthreadsbut notCRYSTAL_WORKERSwill lead to more fibers than threads, with fibers being sheduled on the same threads, which won't bring any improvement.In fact
CRYSTAL_WORKERSdefaults to 4, whenthreadsdefaulted to 8. With this patch it defaults toCRYSTAL_WORKERS, so MT can end up being slower if we don't specifyCRYSTAL_WORKERS=8.This is still not as efficient as it could be. The main fiber (that feeds the worker fibers) can get blocked by a worker fiber doing codegen, leading the other workers to starve. This is easily noticeable when compiling with
-O1for example.Both issues will be fixable with RFC 2 where we can start an explicit context to run the worker fibers or start N isolated contexts (maybe a better idea). Until then, one should increase
CRYSTAL_WORKERS.Supersedes #14227 and doesn't segfault (so far) with LLVM 12 or LLVM 18.1 🤞
TODO:
mt_parallel(units, n_threads)CRYSTAL_CONFIG_WORKERSto configure the default number of workers at compile time instead of the hardcoded 4 (in a distinct PR)