[cbindlist/mergelist] New seqexp helper

[MichaelChirico]

Towards #4370.

[MichaelChirico]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• [cbindlist/mergelist] New seqexp helper #6434 👈
• [cbindlist/mergelist] Add some C utilities: NROW/NCOL/isDataTable/isDataList/perhapsDataTable #6433 : 1 other dependent PR (#6435 )
• [cbindlist/mergelist] Add ERR to the mult enum in bmerge.c #6432
• [cbindlist/mergelist] Whitespace changes #6431
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @MichaelChirico and the rest of your teammates on Graphite

[MichaelChirico]

@jangorecki AFAICT this is just rep(seq_along(x), x), do we actually need this?

[github-actions]

Generated via commit f1ee884

Download link for the artifact containing the test results: ↓ atime-results.zip

Time taken to finish the standard R installation steps: 11 minutes and 36 seconds

Time taken to run atime::atime_pkg on the tests: 5 minutes and 55 seconds

[jangorecki]

Don't remember now but comments says something else

[MichaelChirico]

Using what's described in the comment:

seqexp = function(x) unlist(lapply(seq_along(x), function(i) rep(i, x[[i]])))
baseonly = function(x) rep(seq_along(x), x)

for (ii in 1e4) {
  input = sample(0:10, sample(10), TRUE)
  stopifnot(identical(seqexp(input), baseonly(input)))
}

So I think we can just use rep(seq_along(x), x) and drop this PR.

[jangorecki]

If it doesn't sacrifice performance then make sense

[MichaelChirico]

I think your fully custom function is bound to be a bit faster -- I am seeing ~2x speedup vs. the rep() approach. But both are leagues faster (50 or 100x) than the original suggested unlist(lapply(...)) approach:

slow_seqexp = function(x) unlist(lapply(seq_along(x), function(i) rep(i, x[[i]])))
rep_seqexp = function(x) rep(seq_along(x), x)
seqexp = function(x) .Call(Cseqexp, x)

library(microbenchmark)
for (n in 10^(0:7)) {
  x = sample(0:9, n, TRUE)
  message("length(x)=", n)
  print(microbenchmark(times = 50L,
    slow_seqexp(x),
    rep_seqexp(x),
    seqexp(x)
  ))
}

Output:

length(x)=1
Unit: nanoseconds
           expr  min   lq    mean median   uq   max neval
 slow_seqexp(x) 2785 2925 5158.56   3086 5380 55624    50
  rep_seqexp(x)  470  530 1013.98    611  912  8286    50
      seqexp(x)  361  400  867.44    481  591 16491    50
length(x)=10
Unit: nanoseconds
           expr   min    lq     mean  median    uq   max neval
 slow_seqexp(x) 10029 10310 11207.58 10535.0 10800 36198    50
  rep_seqexp(x)   852  1022  1319.74  1097.0  1262  9969    50
      seqexp(x)   441   571  1238.62   671.5   791 18615    50
length(x)=100
Unit: nanoseconds
           expr   min    lq     mean  median    uq    max neval
 slow_seqexp(x) 80481 81802 83225.96 82344.0 83146 104756    50
  rep_seqexp(x)  2846  3166  4221.00  3451.0  3727  36428    50
      seqexp(x)   741   922  1228.06  1071.5  1272   7243    50
length(x)=1000
Unit: microseconds
           expr     min      lq      mean   median      uq      max neval
 slow_seqexp(x) 802.197 841.701 954.25018 868.2700 920.979 2569.766    50
  rep_seqexp(x)  22.803  23.765  28.58256  24.6160  27.992  120.465    50
      seqexp(x)   3.246   3.436   5.82878   3.8175   4.649   41.337    50
length(x)=10000
Unit: microseconds
           expr      min       lq        mean    median        uq       max neval
 slow_seqexp(x) 8350.744 8811.453 10005.48454 9305.6555 11389.805 13688.665    50
  rep_seqexp(x)  269.052  270.976   278.48738  274.4725   278.309   399.265    50
      seqexp(x)   56.044   74.980    87.70662   88.2995    99.846   116.808    50
length(x)=1e+05
Unit: microseconds
           expr        min         lq       mean     median         uq        max neval
 slow_seqexp(x) 101143.434 125845.171 152667.150 146076.487 168876.313 306177.552    50
  rep_seqexp(x)   2672.197   2705.428   3196.966   3375.609   3408.150   5426.176    50
      seqexp(x)    946.787    988.405   1316.806   1053.977   1650.169   2147.478    50
length(x)=1e+06
Unit: milliseconds
           expr         min         lq      mean     median         uq       max neval
 slow_seqexp(x) 1096.518670 1319.06547 1386.3892 1366.99251 1453.26510 1817.7871    50
  rep_seqexp(x)   26.939532   29.19514   41.5856   33.94697   38.86142  141.5816    50
      seqexp(x)    9.680785   12.03521   26.4762   16.21189   21.59973  141.1328    50
1e+07
Unit: milliseconds
           expr         min         lq       mean     median         uq        max neval
 slow_seqexp(x) 11887.56058 13621.8274 15053.2152 14740.4736 16888.6295 18849.8520    50
  rep_seqexp(x)   268.52191   296.4584   470.7072   348.7749   459.7034  1117.5036    50
      seqexp(x)    96.75193   130.7713   323.5593   173.1193   327.9886   945.9111    50

Even at an output of 45,000,000 elements, the difference is <.2 seconds. I think we can drop this helper.

It's also conceivable in the future base R would use ALTREP to speed this up -- we could do the same but probably better to have such logic live in base.