Add a Railsbench-style benchmark using latest edge Rails

[noahgibbs]

We should add a benchmark that tracks latest Edge Rails. That way we'll notice if YJIT has a large performance change relative to Rails, and whether Rails has added code we do especially well or poorly on.

Previous issue description

Railbench: update to Rails 6.1

This will require updating the Gemfile and doing a "bundle update" at a minimum. There may be other changes as well.

Rails 6.1 made some changes to template rendering that may be relevant to us (rails/rails#43713). Thanks for pointing this out, @byroot!

[byroot]

Rails 6.1 made some changes to template rendering that may be relevant to us (rails/rails#43713).

No that PR is Rails edge (soon to be 7).

But what I meant is that we should either test the latest rails release, or multiple releases.

And having a test for Rails edge as well would be great because it would allow us to make changes to Rails with YJIT in mind and see the results over time, but I understand how it's more maintenance work that the latest release.

[noahgibbs]

Hm. If we test latest Rails, that's going to mean that there are a lot more changes to speed that have nothing to do with YJIT's performance - there will be changes to Rails that speed us up or slow us down, but hard to tell the difference between ours and theirs.

I suppose we could have 6.0 or 6.1 and then separately edge. The 6.0/6.1 update seems pretty easy at least. Railsbench is very simple and doesn't look like it's affected by any of the API changes (https://edgeguides.rubyonrails.org/upgrading_ruby_on_rails.html#upgrading-from-rails-6-0-to-rails-6-1). I can just update the Gemfile and regenerate Gemfile.lock and it runs fine.

Doing a quick run of all four configurations: it looks like 6.1 is a bit faster than 6.0 without YJIT (around 2500ms vs 2700ms) and also with YJIT (around 1970ms vs 2070ms). It looks like 6.0 optimises slightly better with YJIT -- basically, YJIT seems to duplicate some of whatever optimisation they made for 6.1, so with YJIT the numbers are closer together.

I'll probably just leave Railsbench on 6.0 since that's what our existing body of data is for. Changing the code we're testing is normally a problem, and we should only do it if there's some corresponding advantage.

It probably does make sense to add an edge-rails benchmark for Railsbench-or-similar. I'll update this issue accordingly.

[byroot]

I suppose we could have 6.0 or 6.1 and then separately edge.

Yes, that's what I was suggesting, sorry if it wasn't clear.

6.0 is a bit old now, but I guess It make sense to keep it for historical comparison.

7.0 is around the corner, so might make sense to make in the second version, but yeah the "rails" benchmark value should include the Rails version number as we're likely to make changes in Rails edge to better accomodate YJIT.

[maximecb]

Superseded by #242