Enable protoboeuf-encode benchmark on TruffleRuby

[andrykonchin]

The benchmark was disabled in #325. Now the issue is fixed (in oracle/truffleruby#3674).

[k0kubun]

Is truffleruby-head not updated yet?

Failed benchmarks:
  ruby: protoboeuf-encode

[k0kubun]

TBH we want to test yjit-bench here instead of the TruffleRuby project. Given how unstable this job has been so far and the fact that it has been mostly due to TruffleRuby's bugs, I think this job should live in oracle/truffleruby and not in this repository.

If our interest is not to test the edge TruffleRuby but make sure yjit-bench works with TruffleRuby, we should run a released stable version of TruffleRuby instead of the HEAD TruffleRuby. yjit-bench's CI shouldn't randomly fail when nothing has been changed in yjit-bench.

[andrykonchin]

Yeah, looks like a build for the fix isn't ready yet (the fix was merged today). Will retry jobs tomorrow.

[eregon]

Yeah, looks like a build for the fix isn't ready yet (the fix was merged today). Will retry jobs tomorrow.

FWIW that's easily visible at https://github.com/ruby/truffleruby-dev-builder which shows which commit the latest build used.

[eregon]

TBH we want to test yjit-bench here instead of the TruffleRuby project. Given how unstable this job has been so far and the fact that it has been mostly due to TruffleRuby's bugs,

In this case at least it was not a bug, protoboeuf is using a method from Ruby 3.3, so that was not available yet on other Ruby implementations. For most gems out there they would support older Ruby versions and so it's quite a special case.

Given how unstable this job has been so far

The only case I recall is #262, were there many others?

I also think it is important to test another Ruby than just CRuby, otherwise the harness is likely to become accidentally CRuby-specific.

If our interest is not to test the edge TruffleRuby but make sure yjit-bench works with TruffleRuby, we should run a released stable version of TruffleRuby instead of the HEAD TruffleRuby. yjit-bench's CI shouldn't randomly fail when nothing has been changed in yjit-bench.

That's fair, we should try switching to a release of TruffleRuby.
I think only protoboeuf-encode* would fail, which seems OK as it's a very CRuby/YJIT-specific gem, considering the generated code it emits (huge methods).

[maximecb]

I think only protoboeuf would fail, which seems OK as it's a very CRuby/YJIT-specific gem, considering the generated code it emits (huge methods).

You should have no problem handling that generated code. It's not CRuby-specific.

[eregon]

It executes correctly, it's just that

yjit-bench/benchmarks/protoboeuf/benchmark_pb.rb

Line 888 in d3ce8da

def decode_from(buff, index, len)

is too big to be compiled by a method JIT (1400 lines long).
Anyway, I shouldn't have mentioned that here, it's pretty much orthogonal.

[k0kubun]

The only case I recall is #262, were there many others?

It had three benchmarks. yjit-bench used to stop at a single benchmark failure, so it was three round-trips, which were already too many for me. We have two others failing right now that happened separately, so five cases in total.

That's fair, we should try switching to a release of TruffleRuby.

👍

[maximecb]

too big to be compiled by a method JIT (1400 lines long).

I think you mean too big for your method JIT. Google's V8 handles this sort of thing just fine 😉

[eregon]

Actually it compiles fine, my bad, probably because it's a lot of very simple arithmetic, and the benchmark likely uses only a small portion of the many branches:
[engine] opt done engine=1 id=2126 ProtoBoeuf::TrunkItem#decode_from |Tier 2|Time 236( 110+126 )ms|AST 11185|Inlined 47Y 0N|IR 4401/ 7516|CodeSize 29163|Addr 0x7f20e00da000|UTC 2024-09-27T10:10:02.328|Src benchmark_pb.rb:888 0x2acaf309

In the source code it certainly looks like a lot of duplication and something the JIT should be able to decide whether to inline or not (which it could if it's a separate method).

---

I'll make a PR to switch to a release of TruffleRuby in CI: #334