How's the Performance of Ruby 3.0.0-preview1?
The new Ruby 3.0 preview is out! Woo-hoo!
If you’ve heard of me, you know performance is kinda my thing , especially Rails performance on large apps . I do other stuff too, but I got paid to do that for years (thanks, AppFolio!), so I’ve written a lot about it.
How does the new preview’s performance stack up on Rails? And how reliable are these numbers?
Pitfalls
First off, not every gem is ready for Ruby 3. For instance, the latest version of ruby_dep (1.5.0) has a “~>2.2” dependency on Ruby. I’m not trying to pick on it! Tilde-dependencies are usually a really good idea ! And now a bunch of them are going to need to change.
I’m also using ancient code for this benchmark, frankly. I consider myself a pragmatist on this point, and I tried what a pragmatist would do. <whispering>I commented out the check for Ruby version in the local copy of Bundler 1.1.17</whispering>. Um, I mean we’re all respectable Rubyists here who never cut corners to get speed ratings of prerelease software.
It didn’t work, though. It turns out there are several changes in Ruby 3.0 that are going to require gems to upgrade a bit. But if the main problem is just the version number…
I put together a Ruby branch which started out exactly identical to the Git SHA for Ruby 3.0.0-preview1, plus a commit that rolled the reported version back to 2.8 . That’s not as good an idea as updating all the gems to support 3.0, but it requires a lot fewer updates to ancient software that I can’t do for myself. Though I still have to revert a few deprecations so that old code warns instead of crashing…
And now, using version 3.0 with the serial numbers filed off, off we go!
Nearly. In fact, it looks like my (badly-patched) version has an incompatibility between (un-reverted) Ruby 3.0 changes and a monkeypatch claiming to be for ActionModel::Serializer 0.8/0.9 (!). So after removing a (very small) bit of functionality in favor of returning a static string, and turning down the number of load threads a bit to avoid an occasional segfault… Now off we go.
Software for Testing
My existing test stack seemed like the way to go: I’ve built this very specifically to time Ruby 3.0 against Ruby 2.0, for many years. Now that Ruby 3 exists, I can finally use it for that! It’s about time!
Rails Ruby Bench runs a copy of Discourse , a common and popular Rails app to host internet forums. It’s one of the biggest available “real” open-source Rails apps, making it a fine choice for “real world” benchmarking. RRB runs a set of simulated pseudorandom user requests against the running Rails app, and times how long they all take to finish. So it’s a throughput test. You can run it yourself if you like , though it’s a bit complex and finicky. You may be gathering that from the “Pitfalls” section above. The dark side of using real-world software is hitting real-world complexity and bugs.
RRB has run against AWS m4.2xlarge instances for nearly its whole existence. That’s what I’m doing again. For now: m4.2xlarge instances, 10 processes, 6 threads/process, the same as I’ve been using for over 3 years for this purpose. Once Ruby 3 comes out, it’ll be time to look at upgrading RRB. After that, it will have fulfilled its purpose: to measure the total speedup from 2.0 to 3.0 of a typical real-world Ruby on Rails application.
And I can finally stop supporting some hideous old hacks to make that possible.
These results are running minimum 30 batches per Ruby version of 10,000 requests each. That’s enough to get pretty solid results, but not to detect tiny differences.
(Due to randomization and a restart after fixing a rare bug, we got 38 batches for Ruby 2.7 and 36 batches for Ruby 3.0.0-preview1.)
Results
How are results? Not dramatic, I’m afraid.
| Ruby 2.7 | Ruby 3.0.0-preview1 | Speedup/Slowdown | |
|---|---|---|---|
| Median Throughput | 165.7 | 160.7 | -3.0% |
| Fastest Run of 30 | 168.7 | 164.0 | -2.8% |
| Slowest Run of 30 | 163.0 | 158.2 | -2.9% |
(I also have the raw data , if that’s of use to you. And technically there’s an even more raw, larger set of results. Hit me up on Twitter if you want them.)
To summarize simply: they’re nearly the same speed, and a bit a pre-release Ruby polish and/or fixes to my testing will probably get them back to exactly equal.
Other Issues…
Keep in mind that preview1 isn’t final. I’m going to investigate this and see what I can see. My code branch is certainly not perfect. We’re seeing ugly interactions between the now-ancient Discourse code I’m using and Ruby 3.0’s deprecations. And I may find some improvements along the way.
Still, what I’m seeing suggests that these results aren’t far off. A severe bug wouldn’t cost a few percent of performance - it would kill it, or boost it to utterly unreasonable levels. That’s not what we’re seeing here. (I’m going to see if I can find where Rails and/or my hackery slows things down, of course.)
This is also a very limited test. I could check Ruby versions (again) much farther back, though you can see the previous results of that , and they shouldn’t change. Ruby 2.6 and 2.7 are functionally the same speed to within 1%-2% . It looks like 2.7 and the 3.0 preview aren’t showing a significant speed boost for Rails.
Is that shocking? It shouldn’t be. I was, frankly, very surprised to see the 72%-ish speed boost from 2.0 to 2.6. Rails spends a lot of its time I/O-bound, waiting on databases, files and the network. The current Ruby JIT champions, JRuby and TruffleRuby can’t easily squeeze more performance out of Rails than CRuby in most cases. The garbage collector, a source of slowdown back in Ruby 2.0 and 2.1, runs extremely solidly for these use cases.
I think this is about the speed that Rails is going to be, for Ruby 3.0 and for some time afterward.
With that said, I think a combination of polishing Ruby 3.0 for release and me making sure my test is in order will return the few percentage points of speed that (my tests claim) Ruby 3.0 has lost versus 2.6 and 2.7.
Hey, you read all the way to the bottom! I’m impressed. If you’re still on a Rails performance kick and you want to stick with it a bit, you could check out more performance articles on this site!