-
Notifications
You must be signed in to change notification settings - Fork 22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add fluentd benchmark #243
Conversation
Looks good. What are the stats like currently? |
Here's YJIT stats on Ruby master:
|
Interesting. I'm wondering if we get so little speedup despite such a high percentage because we spend too much time in C, maybe regexp parsing? Otherwise maybe a lot of usage of keyword arguments? |
Parsing time values is probably done by something written in C, but the main LTSV parsing logic is written in Ruby with operations like |
This PR adds a new benchmark
fluentd
. Fluentd is a log collector that Shopify currently uses for services like Core and SFR. When Core callsRails.logger.info("foo=bar")
, our Fluentd config parses it into{"foo":"bar"}
and forwards the log to Splunk, for example.The benchmark code is inspired by the [JA]How fast really is Ruby 3.x? talk in RubyKaigi 2022. According to him, a Fluentd committer, their parser creates a Ruby object(s) for each log, so the parser is a very runtime-intensive part. Benchmarking the forwarding feature could be challenging in a single-process script, so this benchmarks only the parser. Instead of his synthetic ltsv.rb that is shown later in his talk, this PR uses the actual Fluent::Plugin::LabeledTSVParser, while parsing the same thing as his data.ltsv.
Since he said "CRuby performance is very important for Fluentd" in the talk, I thought it'd be interesting to include it as a Ruby benchmark.
Current result