Feature #21797
open
Make Etc.nprocessors cgroup-aware on Linux
Description
Currently, Etc.nprocessors ignores cgroup CPU quotas. This causes issues in containers where CPU limits differ from the host CPU count.
I have written a gem for this purpose (https://github.com/moznion/maxprocs-ruby), but it would be preferable if the Ruby core implementation respected cgroup configuration.
Additionally, concurrent-ruby provides similar functionality, but if the language itself offered this capability, language users would not need to implement it individually.
And some parts of the Ruby language handle the number of processors using hard-coded values (e.g., https://github.com/ruby/ruby/blob/8efaf5e6b6a25e0d237f3d71b75865661ae98268/thread_pthread.c#L1737-L1738), so this could also be useful for Ruby language development.
Extending Etc.nprocessors to respect cgroups is one option, but that would be a breaking change, so adding a new API (e.g., Etc.cpu_quota or something?) might be a better approach.
Prior Art¶
Updated by moznion (Taiki Kawakami) 2 days ago
- Description updated (diff)
Updated by hsbt (Hiroshi SHIBATA) 2 days ago
RubyGems 4.0.x support -j option for building C extension gem. But It causes in container environment like Circle CI.
https://github.com/ruby/rubygems/issues/9170
If cgroup provides the correct number of CPU for Cicle CI or others, I'm positive to support this.
Updated by moznion (Taiki Kawakami) 2 days ago
- Description updated (diff)
Updated by osyoyu (Daisuke Aritomo) 2 days ago
It'd be nice if RUBY_MAX_CPU would be autoconfigured based on this, just like Go 1.25 GOMAXPROCS.
Its default value is currently fixed to 8, but not many cloud containers have 8 cores worth of processors.
https://github.com/ruby/ruby/blob/8efaf5e6b6a25e0d237f3d71b75865661ae98268/thread_pthread.c#L1737-L1738
Updated by
Updated by moznion (Taiki Kawakami) 1 day ago
osyoyu (Daisuke Aritomo) wrote in #note-4:
It'd be nice if
RUBY_MAX_CPUwould be autoconfigured based on this, just like Go 1.25GOMAXPROCS.Its default value is currently fixed to 8, but not many cloud containers have 8 cores worth of processors.
https://github.com/ruby/ruby/blob/8efaf5e6b6a25e0d237f3d71b75865661ae98268/thread_pthread.c#L1737-L1738
This sounds very good, but it should probably be a separate ticket though.
nobu (Nobuyoshi Nakada) wrote in #note-5:
How about https://github.com/ruby/etc/tree/linux-cgroup-from-maxprocs?
Thank you for your quick action. I left a comment on the pull request; overall, it looks good.
Updated by Eregon (Benoit Daloze) about 20 hours ago
CPU quotas can be floating point numbers though, not just integers.
But Etc.nprocessors returns an Integer.
So it seems having a new method is worth it.
I think it's good if Etc.nprocessors rounds it though for convenience and so it applies to existing usages of Etc.nprocessors.
FYI concurrent-ruby already has Concurret.cpu_quota (returns Float or nil):
https://github.com/ruby-concurrency/concurrent-ruby/blob/129cf004294af68ac53e53a2f1197621b303570a/lib/concurrent-ruby/concurrent/utility/processor_counter.rb#L198-L211
Updated by moznion (Taiki Kawakami) about 17 hours ago
Thank you for your comments. Let me summarize the discussion.
1. Introducing a new API under Etc
¶
Introducing a method such as Etc.cpu_quota, which returns Float | nil, seems reasonable.
2. Changing the behavior of Etc.nprocessors
¶
Changing the behavior of Etc.nprocessors to respect cgroup-based values and fall back to the original logic when cgroups are unavailable.
This would be a breaking change, but it would likely be beneficial for most use cases, and the negative impact should be minimal.
3. Injecting the number of actually available CPUs into RUBY_MAX_CPU
¶
This would be useful, but it should be handled as a separate ticket after this one is resolved.
4. Implementation approach¶
Which is preferred: pure Ruby as a part of ruby/etc or C?
What do you think?