ci: Compare performance to msquic

[larseggert]

In progress

[github-actions]

Benchmark results

Performance differences relative to 50876af.

• drain a timer quickly time: [391.85 ns 398.79 ns 405.11 ns]
  change: [-3.5688% -2.0497% -0.5018%] (p = 0.01 < 0.05)
  Change within noise threshold.

• coalesce_acked_from_zero 1+1 entries
  time: [193.77 ns 194.19 ns 194.66 ns]
  change: [-0.9878% -0.3625% +0.2183%] (p = 0.25 > 0.05)
  No change in performance detected.

• coalesce_acked_from_zero 3+1 entries
  time: [235.95 ns 236.50 ns 237.09 ns]
  change: [-0.4647% -0.0168% +0.4699%] (p = 0.94 > 0.05)
  No change in performance detected.

• coalesce_acked_from_zero 10+1 entries
  time: [236.81 ns 237.90 ns 239.06 ns]
  change: [-0.4510% +0.1024% +0.6451%] (p = 0.72 > 0.05)
  No change in performance detected.

• coalesce_acked_from_zero 1000+1 entries
  time: [214.16 ns 214.39 ns 214.69 ns]
  change: [-1.0684% -0.3058% +0.5524%] (p = 0.46 > 0.05)
  No change in performance detected.

• RxStreamOrderer::inbound_frame()
  time: [117.34 ms 117.43 ms 117.52 ms]
  change: [-1.2006% -1.0891% -0.9835%] (p = 0.00 < 0.05)
  Change within noise threshold.

• transfer/Run multiple transfers with varying seeds
  time: [118.14 ms 118.41 ms 118.67 ms]
  thrpt: [33.707 MiB/s 33.782 MiB/s 33.858 MiB/s]
  change:
  time: [+0.8201% +1.1651% +1.5168%] (p = 0.00 < 0.05)
  thrpt: [-1.4941% -1.1517% -0.8134%]
  Change within noise threshold.

• transfer/Run multiple transfers with the same seed
  time: [119.00 ms 119.16 ms 119.33 ms]
  thrpt: [33.520 MiB/s 33.567 MiB/s 33.613 MiB/s]
  change:
  time: [+0.8034% +0.9808% +1.1608%] (p = 0.00 < 0.05)
  thrpt: [-1.1475% -0.9712% -0.7970%]
  Change within noise threshold.

Client/server transfer results

Transfer of 134217728 bytes over loopback.

Client	Server	CC	Pacing	Mean [ms]	Min [ms]	Max [ms]	Relative
msquic	msquic			408.4 ± 62.6	350.9	513.9	1.00
neqo	msquic	reno	on	1912.2 ± 27.1	1879.0	1959.9	1.00
neqo	msquic	reno		1899.1 ± 41.2	1826.5	1970.3	1.00
neqo	msquic	cubic	on	1908.7 ± 47.5	1858.6	2001.2	1.00
neqo	msquic	cubic		1923.9 ± 56.6	1859.3	2067.1	1.00
msquic	neqo	reno	on	4366.5 ± 148.7	4157.5	4598.2	1.00
msquic	neqo	reno		4303.2 ± 144.3	4108.4	4542.4	1.00
msquic	neqo	cubic	on	4463.2 ± 161.3	4226.4	4664.8	1.00
msquic	neqo	cubic		4379.4 ± 169.1	4156.6	4643.9	1.00
neqo	neqo	reno	on	3616.2 ± 103.1	3418.1	3721.7	1.00
neqo	neqo	reno		3457.4 ± 175.1	3210.3	3731.3	1.00
neqo	neqo	cubic	on	4214.0 ± 146.4	3979.5	4415.5	1.00
neqo	neqo	cubic		4192.3 ± 159.8	3915.8	4396.9	1.00

⬇️ Download logs

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93.11%. Comparing base (50876af) to head (3798f0b).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1750      +/-   ##
==========================================
+ Coverage   93.08%   93.11%   +0.02%     
==========================================
  Files         117      117              
  Lines       36422    36422              
==========================================
+ Hits        33903    33913      +10     
+ Misses       2519     2509      -10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[larseggert]

I looked at why we're so much slower then msquic, and the neqo server spends an insane amount of time in neqo-common::timer:

       │      /// Take the next item, unless there are no items with                                                   ▒
       │      /// a timeout in the past relative to `until`.                                                           ▒
       │      pub fn take_next(&mut self, until: Instant) -> Option<T> {                                               ▒
       │      for i in 0..self.items.len() {                                                                           ▒
       │      let idx = self.bucket(i);                                                                                ▒
       │      if !self.items[idx].is_empty() && self.items[idx][0].time <= until {                                     ▒
       │        test      %rbx,%rbx                                                                                    ▒
       │        jne       2444                                                                                         ▒
       │        jmp       23f0                                                                                         ▒
       │      neqo_common::timer::Timer<T>::bucket:                                                                    ▒
       │        nop                                                                                                    ▒
       │      (self.cursor + delta) % self.items.len()                                                                 ▒
  0.03 │2430:   mov       %ecx,%eax                                                                                    ▒
       │        xor       %edx,%edx                                                                                    ▒
 29.50 │        div       %esi                                                                                         ▒
       │      <usize as core::slice::index::SliceIndex<[T]>>::index:                                                   ▒
  1.27 │        lea       (%rdx,%rdx,2),%rax                                                                           ▒
       │      alloc::vec::Vec<T,A>::len:                                                                               ▒
 22.02 │        mov       0x10(%r8,%rax,8),%rbx                                                                        ▒
       │      neqo_common::timer::Timer<T>::take_next:                                                                 ▒
       │      if !self.items[idx].is_empty() && self.items[idx][0].time <= until {                                     ▒
  0.04 │        test      %rbx,%rbx                                                                                    ▒
  7.64 │        je        23f0                                                                                         ▒

       │      neqo_common::timer::Timer<T>::bucket:                                                                    ▒
      │      (self.cursor + delta) % self.items.len()                                                                 ▒
 0.11 │2bf0:   mov       %ecx,%eax                                                                                    ▒
 0.09 │        xor       %edx,%edx                                                                                    ▒
 8.55 │        div       %edi                                                                                         ▒
      │      <usize as core::slice::index::SliceIndex<[T]>>::index:                                                   ▒
 0.12 │        lea       (%rdx,%rdx,2),%rax                                                                           ▒
      │      core::slice::<impl [T]>::first:                                                                          ▒
 4.36 │        inc       %rcx                                                                                         ▒
 4.53 │        cmpq      $0x0,0x10(%rsi,%rax,8)                                                                       ▒
 2.20 │        jne       2c32                                                                                         ▒

(Numbers in the first column are percent of total cycles spent in that line.)