Dissecting pacquet frozen lockfile performance

[KSXGitHub]

Scope

This discussion is centered around the frozen-lockfile feature of pacquet only. It is well known that pacquet algorithm without lockfile is not good enough, but we are not focusing on that.

Problem

It is easy to see that pacquet algorithm is not parallelized to the fullest, as there are many points where unrelated IO/network operations still happen in serial order. Yet, when I try to parallelize it either by using rayon or tokio::task::spawn, pacquet becomes slower than before.

Resources

The branches:

• The main branch is at pnpm/pacquet@6b00165 at the time of writing this.
• optimization-attempt-rayon is an attempt at using rayon to parallelize some operations. It is at pnpm/pacquet@1f4b10f at the time of writing this.
• optimization-attempt-tokio-tasks is an attempt at using tokio tasks to parallelize some operations. It is at pnpm/pacquet@92d5949 at the time of writing this.

The manifest files and lockfiles are located in tasks/integrated-benchmark/src/fixtures/.

• Small manifest file and lockfile: Are the default.
• Medium-sized manifest file and lockfile: pnpm/pacquet@8289e36

To compare benchmark against the main branch, run:

just integrated-benchmark -s frozen-lockfile $TARGET_BRANCH main -V

NOTE: The benchmarks are using a small lockfile by default. You should manually copy the medium-size lockfile and manifest file for a more objective measurement. Remember to manually set autoInstallPeers in the lockfile to false.

NOTE: You can add --with-pnpm to compare pacquet against pnpm. Be aware that startup time will hold back pnpm if using the default lockfile, so a bigger lockfile is necessary to be objective.

[kdy1]

I want to attach the profiler to the command. What are just and just integrated-benchmark?

[KSXGitHub]

just is a task runner. All the tasks are stored in justfile in the root of the repo.

It runs a Rust program at tasks/integrated-benchmark.

[kdy1]

I think we need a benchmark that is more reliable.

[KSXGitHub]

It varies too much. It seems like it has some network operations.

All network operations only connect to localhost which is served by verdaccio.

[kdy1]

But it still varies from 150ms to 650ms.

[KSXGitHub]

My machine (Linux, 56 cores, 48GB of RAM) is quite stable.

bench-work-env/BENCHMARK_REPORT.json

{
  "results": [
    {
      "command": "pacquet@HEAD",
      "mean": 0.13035584404,
      "stddev": 0.008067993213068583,
      "median": 0.12837198564,
      "user": 0.6484349066666666,
      "system": 0.5851663733333333,
      "min": 0.11587404964,
      "max": 0.14220072864,
      "times": [
        0.13805841264000002,
        0.13784145564,
        0.14220072864,
        0.13894517864,
        0.12523665964,
        0.13909586864,
        0.12507481564,
        0.12837198564,
        0.12144884164,
        0.12672492064000002,
        0.12493211264000001,
        0.11587404964,
        0.12384233564,
        0.13879491864000001,
        0.12889537664
      ],
      "exit_codes": [
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0
      ]
    },
    {
      "command": "pacquet@main",
      "mean": 0.12817658916941177,
      "stddev": 0.007999823319031376,
      "median": 0.12709307764,
      "user": 0.4304567694117647,
      "system": 0.46009327529411764,
      "min": 0.11347365664,
      "max": 0.13927691164,
      "times": [
        0.12102636863999999,
        0.12533512664000002,
        0.11468248464,
        0.12001481564000001,
        0.12709307764,
        0.11347365664,
        0.12586417164,
        0.12999639964,
        0.12615296564,
        0.13927691164,
        0.13159887164,
        0.13921117664000002,
        0.13567371164,
        0.13893893164,
        0.13319960764,
        0.12477131363999999,
        0.13269242464
      ],
      "exit_codes": [
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0
      ]
    }
  ]
}

Anyway, the result observed in your machine could indicate a problem with the implementation itself. Do you think mixing tokio and rayon would create unpredictable results?

[kdy1]

I think the difference here is mac vs linux problem.

Many project mixes rayon and tokio, and itself is fine. But we need to restrict the scope of operations of each threadpool to make it performant. The characteristics of tasks that each thread pool should be similar (blocking without cpu / cpu-heavy / network io) and the pool size should be dtermined based on it.

[KSXGitHub]

Does this mean that we have to balance between tokio thread pool and rayon thread pool? But then how could one determine the right balance? Maybe we should migrate the codebase to only use tokio or rayon, not both?