Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

llvm-project: link against jemalloc #38

Merged
merged 1 commit into from
Jul 1, 2022
Merged

llvm-project: link against jemalloc #38

merged 1 commit into from
Jul 1, 2022

Conversation

nickdesaulniers
Copy link
Member

@nickdesaulniers nickdesaulniers commented Jul 1, 2022

The following symbols from musl's malloc-ng implementation were
appearing in a perf profile of a kernel build within the top 20 hottest
symbols by cycle count.

malloc-ng:
Samples: 589K of event 'cycles:ppu', Event count (approx.): 11053523397362
  Overhead  Shared Object              Symbol
+    1.57%  clang                     [.] __libc_malloc_impl
     0.69%  clang                     [.] __libc_free
+    0.65%  clang                     [.] alloc_slot
+    0.58%  clang                     [.] get_meta

$ /usr/bin/time -v make CC=clang LD=ld.lld -j72 -s
Command being timed: "make CC=clang LD=ld.lld -j72 -s"
  User time (seconds): 2801.78
  System time (seconds): 395.48
  Percent of CPU this job got: 5212%
  Elapsed (wall clock) time (h:mm:ss or m:ss): 1:01.33

jemalloc:
Samples: 569K of event 'cycles:ppu', Event count (approx.): 10681965442287
  Overhead  Shared Object              Symbol
+    0.55%  clang                     [.] free
+    0.41%  clang                     [.] malloc

$ /usr/bin/time -v make CC=clang LD=ld.lld -j72 -s
Command being timed: "make CC=clang LD=ld.lld -j72 -s"
User time (seconds): 2626.97
System time (seconds): 248.37
Percent of CPU this job got: 4774%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:00.22

And these symbols are no longer in the top 20 hottest symbols by cycle
count.

Replace musl's malloc with jemalloc. Nothing major to write home about,
but gives a ~2.5% speedup by cycle count (N=1).

Fixes #36

Copy link
Member

@compnerd compnerd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems fine procedurally. I do have to ask - which malloc in musl was being used? There is oldmalloc and malloc-ng IIRC, and I'm curious if switching between those has any impact on performance. It would be nice to have that sort of information available to better explain the reasoning for the switch (and perhaps expand on that a bit more in the commit message).

@nickdesaulniers
Copy link
Member Author

Based on the disassembly, it looked like malloc-ng IIRC.

Let me confirm, then collect some more number, then update the commit message.

@nickdesaulniers
Copy link
Member Author

jemalloc:

$ PATH=/android2/containers/llvm-project:$PATH /usr/bin/time -v make CC=clang LD=ld.lld -j72 -s        
	Command being timed: "make CC=clang LD=ld.lld -j72 -s"
	User time (seconds): 2626.97
	System time (seconds): 248.37
	Percent of CPU this job got: 4774%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:00.22
 0.55%  clang                     [.] free
 0.41%  clang                     [.] malloc
so malloc and free are still hot BUT they fall out of the top 20 hottest functions like we have with malloc-ng, and they take less time.  Maybe `-Xclang -disable-free` for another slight improvement...

malloc-ng:
```sh
$ PATH=/android2/containers/llvm-project:$PATH /usr/bin/time -v make CC=clang LD=ld.lld -j72 -s
	Command being timed: "make CC=clang LD=ld.lld -j72 -s"
	User time (seconds): 2801.78
	System time (seconds): 395.48
	Percent of CPU this job got: 5212%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:01.33

In the container, I can see that ${BUILD_DIR}/obj/src/malloc/ contains mallocng/ but not oldmalloc/.

The following symbols from musl's malloc-ng implementation were
appearing in a perf profile of a kernel build within the top 20 hottest
symbols by cycle count.

malloc-ng:
Samples: 589K of event 'cycles:ppu', Event count (approx.): 11053523397362
  Overhead  Shared Object              Symbol
+    1.57%  clang                     [.] __libc_malloc_impl
     0.69%  clang                     [.] __libc_free
+    0.65%  clang                     [.] alloc_slot
+    0.58%  clang                     [.] get_meta

$ /usr/bin/time -v make CC=clang LD=ld.lld -j72 -s
Command being timed: "make CC=clang LD=ld.lld -j72 -s"
  User time (seconds): 2801.78
  System time (seconds): 395.48
  Percent of CPU this job got: 5212%
  Elapsed (wall clock) time (h:mm:ss or m:ss): 1:01.33

jemalloc:
Samples: 569K of event 'cycles:ppu', Event count (approx.): 10681965442287
  Overhead  Shared Object              Symbol
+    0.55%  clang                     [.] free
+    0.41%  clang                     [.] malloc

$ /usr/bin/time -v make CC=clang LD=ld.lld -j72 -s
Command being timed: "make CC=clang LD=ld.lld -j72 -s"
User time (seconds): 2626.97
System time (seconds): 248.37
Percent of CPU this job got: 4774%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:00.22

And these symbols are no longer in the top 20 hottest symbols by cycle
count.

Replace musl's malloc with jemalloc. Nothing major to write home about,
but gives a ~2.5% speedup by cycle count (N=1).

Fixes #36
@nickdesaulniers nickdesaulniers merged commit b82a764 into main Jul 1, 2022
@nickdesaulniers nickdesaulniers deleted the jemalloc branch July 1, 2022 21:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants