ruby-core

The folks at IBM recently open-sourced the Ruby OMR preview. Matthew
Gaudet posted to this list about it. I was curious about relative
memory use compared to MRI, so I've run a few tests on them
side-by-side. These are my preliminary impressions and results -- I'd
love any comments on them.

The OMR tools have been tested with Ruby 2.2, but not yet updated to
any later Ruby. Not every test passes. Some of this seems to be test
rot from old code (e.g. SSL certificate problems,) but some of it is
probably related to JIT and their modifications.

OMR Ruby currently only builds on Linux, not on Mac. I used a
VirtualBox VM running Ubuntu Xenial for these comparisons. Take the
FPS below with a large grain of salt.

OMR Ruby uses both JIT and OMR's garbage collector. For this reason, I
test OMR Ruby in three configurations: no JIT active, JIT with methods
compiled after 50 calls (count=50), and JIT with methods compiled
immediately (count=0). I tested vanilla Ruby 2.2 in only standard
configuration. I used the ruby_2_ branch (commit a5b324, from SVN
checkin 56370) as vanilla Ruby in these comparisons.

Memory usage below is measured using "/usr/bin/time -f %Mkb" for
minimum resident set size.

All of these numbers were from just a few runs. This has very low
statistical power. It's just an initial evaluation. If we see
something we like, I'll use many more runs and a tool like ABProf to
get more authoritative comparisons for speedups and memory use.

All OMR runs w/ JIT activated use "LD_LIBRARY_PATH=$PWD" and either
"OMR_JIT_OPTIONS=-Xjit:count=50" or "OMR_JIT_OPTIONS=-Xjit:count=0".
So the final optcarrot run for OMR w/ immediate compilation would use
the command line: LD_LIBRARY_PATH=$PWD OMR_JIT_OPTIONS=-Xjit:count=0
/usr/bin/time -f %Mkb ./tool/runruby.rb ../optcarrot/bin/optcarrot
--headless --print-fps --print-video-checksum --frames 18000
../optcarrot/examples/Lan_Master.nes

Matthew Gaudet has mentioned some ways to reduce the memory usage of
OMR. My initial trials haven't seen a difference with his suggestions,
but I'll keep trying -- I may have modified incorrectly, or I may find
a better tweak.

== My impressions

The test data below could be summarized as: JIT gives a small speedup,
maybe between 5% and 10%, but nothing huge. The change is small enough
that optcarrot's noise swamps it. This is still noticeably better
performance than the previous OMR preview benchmark I can find
(https://github.com/mame/optcarrot/blob/master/doc/benchmark.md).

Turning on JIT in small cases, even without ever JITting a method, has
a big memory cost. It's around 3MB for "hello, world" and goes up from
there, to around 10MB-20MB for optcarrot. But perhaps there could be a
way to only turn on JIT at all when it will be used? Right now, JIT
that is never used uses as much memory as JIT when it's used heavily.

Raw data below:

== First Test, mem use: ./miniruby -e "puts 'Hello, world'"

Vanilla Ruby 2.2: about 6,400kb.

Ruby OMR 2.2 w/ no JIT settings: about 7,700kb.

Ruby OMR 2.2 w/ JIT count 50: about 10,800kb, but occasional spikes up
to 14,000-17,000kb.

Ruby OMR 2.2 w/ JIT count 0: 14,000-18,000kb.

== ./tool/runruby.rb ../optcarrot/bin/optcarrot --benchmark
../optcarrot/examples/Lan_Master.nes

Optcarrot FPS is as reported on the console. Note that for only 180
frames, JIT is probably not a great idea. The warm-up time is likely
to be too high for it to help.

Vanilla Ruby 2.2:
about 57,000kb
about 30-32FPS

Ruby OMR 2.2 w/ no JIT settings:
about 77,000kb
about 27-31FPS

Ruby OMR 2.2 w/ JIT count 50:
about 86,000-88,000kb
about 27-29FPS

Ruby OMR 2.2 w/ JIT count 0:
about 86,000-88,000kb
about 28-29FPS

Ruby OMR 2.2mod w/ JIT count 0:
Don't see any noticeable diff in min RSS

== ./tool/runruby.rb ../optcarrot/bin/optcarrot --headless --print-fps
--print-video-checksum --frames 18000
../optcarrot/examples/Lan_Master.nes

This is 100x as many frames per headless run. JIT should be
substantially better in this case. I also only used one run for each
setting because it takes 100x as long to run, so it's already averaged
out ;-)

Vanilla Ruby 2.2:
27.0 FPS
61404kb

Ruby OMR 2.2 w/ no JIT settings:
77160kb
31.03FPS

Ruby OMR 2.2 w/ JIT count 50:
86652kb
29.62FPS

Ruby OMR 2.2 w/ JIT count 0:
88432kb
29.25FPS

Unsubscribe: <mailto:[email protected]?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>