Bug #4272
closed
rb_enc_str_new() causes segmentfault when using threads in parallel
Description
=begin
Hi, I attach a custom C extension which shows how rb_enc_str_new() causes segmentfault when running multiple threads. The bug is reproducible in 1.9.2-p0, 1.9.2-p136 and 1.9 today's snapshot (2011-01-12).
In order to test the bug compile the extension and use the included test_bug.rb script passing a long number (iterations) as argument (1000, 10000...).
~# ruby extconf.rb && make && ruby test_bug.rb 5000
The provided C extension is very simple and just contains two methods:
TestEncoding::Tester#test_rb_str_new => returns a string encoded in ASCII-8BIT (generated with "rb_str_new").
TestEncoding::Tester#test_rb_enc_str_new => returns a string encoded in UTF-8 (generated with "rb_enc_str_new").
When running threads in parallel with the second method, after some iterations Ruby crashses (as the provided test_bug.rb shows).
=end
Files
Updated by ibc (Iñaki Baz Castillo) over 14 years ago
=begin
I've checked the provided test script under ruby-1.9.1-p376 and ruby-1.9.1-p430 and it DOES NOT crash.
=end
Updated by ibc (Iñaki Baz Castillo) over 14 years ago
- File test_bug.rb test_bug.rb added
=begin
=end
Updated by ibc (Iñaki Baz Castillo) over 14 years ago
=begin
(please use the latest test_bug.rb attachment, it seems is not possible to remove an attachment in this Redmine)
=end
Updated by ibc (Iñaki Baz Castillo) over 14 years ago
=begin
It's interesting the fact that (using my attached C extension) the following code doesn't crash (I mean under ruby 1.9.2-xxxx as under 1.9.1 it never crashes):
times.times do |num|
tester.test_rb_str_new
threads << Thread.new { }
end
but the following codes DO crash:
times.times do |num|
tester.test_rb_str_new.inspect
threads << Thread.new { }
end
times.times do |num|
a = tester.test_rb_str_new
threads << Thread.new { }
end
=end
Updated by ibc (Iñaki Baz Castillo) over 14 years ago
=begin
The segmentfault just depends on the version of Ruby running the ruby code, and not on the version of Ruby compiling the C extension. This is:
a) C extension compiled with Ruby 1.9.1 and test_bug.rb executed with Ruby 1.9.2 => CRASH
b) C extension compiled with Ruby 1.9.2 and test_bug.rb executed with Ruby 1.9.1 => doesn't crash
=end
Updated by ibc (Iñaki Baz Castillo) over 14 years ago
=begin
The bug also occurs if I use rb_ascii8bit_encoding(), rb_usascii_encoding() or rb_locale_encoding() so it must be a bug in rb_enc_str_new() function.
=end
Updated by nagachika (Tomoyuki Chikanaga) over 14 years ago
=begin
Hi,
In my environment script just like the following cause "[BUG] rb_register_sigaltstack. malloc error".
It's because all Thread object is stored in `threads' variable and cannot be GC'ed.
Could you show us your error messages?
threads = []
300000.times do |i|
puts i
threads << Thread.new{}
end
threads.each {|t| t.join}
=end
Updated by matz (Yukihiro Matsumoto) over 14 years ago
=begin
Hi,
I could not reproduce your problem, but your script allocates
thousands of threads by default. Each thread consumes at least 1M of
memory, so that allocating thousands of threads could cause memory
starvation, which may be the source of your problem.
matz.
In message "Re: [ruby-core:34399] [Ruby 1.9-Bug#4272][Open] rb_enc_str_new() causes segmentfault when using threads in parallel"
on Wed, 12 Jan 2011 11:04:21 +0900, Iñaki Baz Castillo [email protected] writes:
|Hi, I attach a custom C extension which shows how rb_enc_str_new() causes segmentfault when running multiple threads. The bug is reproducible in 1.9.2-p0, 1.9.2-p136 and 1.9 today's snapshot (2011-01-12).
|
|In order to test the bug compile the extension and use the included test_bug.rb script passing a long number (iterations) as argument (1000, 10000...).
|
| ~# ruby extconf.rb && make && ruby test_bug.rb 5000
|
|The provided C extension is very simple and just contains two methods:
|
| TestEncoding::Tester#test_rb_str_new => returns a string encoded in ASCII-8BIT (generated with "rb_str_new").
| TestEncoding::Tester#test_rb_enc_str_new => returns a string encoded in UTF-8 (generated with "rb_enc_str_new").
|
|When running threads in parallel with the second method, after some iterations Ruby crashses (as the provided test_bug.rb shows).
=end
Updated by naruse (Yui NARUSE) over 14 years ago
=begin
I can reproduce this on ruby 1.9.3dev (2011-01-12 trunk 30517) [x86_64-freebsd8.2].
Anyway what your environment? (why you didn't copy ruby -v ?)
=end
Updated by nagachika (Tomoyuki Chikanaga) over 14 years ago
=begin
Hi,
I can reproduce SEGV on Mac OS X 10.6.6.
Please add the following line in test_encoding.c
#include <ruby/encoding.h>
Without that header, rb_utf8_encoding() has no prototype declaration and its return value is estimated to be int.
=end
Updated by ibc (Iñaki Baz Castillo) over 14 years ago
- File test_bug2.rb test_bug2.rb added
=begin
Yukihiro:
I could not reproduce your problem
Do you mean under Ruby 1.9.2 64 bits? If so please use a bigger argument (ruby test_bug.rb 100000). I've realized that depending on the machine it crashes sooner or later.
but your script allocates thousands of threads by default. Each thread consumes at least 1M of memory, so
that allocating thousands of threads could cause memory starvation, which may be the source of your problem.
The segmentfault just occurs when using rb_enc_str_new(). If you edit test_bug.rb and remove/comment the first iteration with "tester.test_rb_str_new" the segmentfault also occurs. And it never occurs if you just leave the iteration with "tester.test_rb_str_new".
And more important, I also reproduce always the segmentfault using a fixed pool of 20 threads (in EventMachine). And also in this modified test script (max 20 threads are created) you can run by loading my attached C extension (I attach now test_bug2.rb with this code):
require "./test_encoding"
times = ( ( ARGV[0] and ARGV[0].to_i > 0 ) ? ARGV[0].to_i : 5000 )
threads = []
num_threads = 20
tester = TestEncoding::Tester.new
times.times do |num|
puts "iteration [#{num}]"
str = tester.test_rb_enc_str_new
threads << Thread.new { } unless num_threads == 0
num_threads -= 1
end
Using Ruby 1.9.2- produces different segmentfault (randomly, but it crashes always in my machines):
- Crash on Thread.initialize:
iteration [1981]
iteration [1982]
iteration [1983]
iteration [1984]
test_bug.rb:12: [BUG] Segmentation fault
ruby 1.9.2p0 (2010-08-18 revision 29036) [x86_64-linux]
-- control frame ----------
c:0008 p:---- s:0025 b:0025 l:000024 d:000024 CFUNC :initialize
c:0007 p:---- s:0023 b:0023 l:000022 d:000022 CFUNC :new
c:0006 p:0063 s:0020 b:0019 l:001f08 d:000018 BLOCK test_bug.rb:12
c:0005 p:---- s:0015 b:0015 l:000014 d:000014 FINISH
c:0004 p:---- s:0013 b:0013 l:000012 d:000012 CFUNC :times
c:0003 p:0121 s:0010 b:0010 l:001f08 d:000968 EVAL test_bug.rb:9
c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH
c:0001 p:0000 s:0002 b:0002 l:001f08 d:001f08 TOP¶
-- Ruby level backtrace information ----------------------------------------
test_bug.rb:9:in <main>' test_bug.rb:9:in times'
test_bug.rb:12:in block in <main>' test_bug.rb:12:in new'
test_bug.rb:12:in `initialize'
-- C level backtrace information -------------------------------------------
/usr/lib/libruby-1.9.1.so.1.9(rb_vm_bugreport+0x5f) [0x7f10c71c658f]
/usr/lib/libruby-1.9.1.so.1.9(+0x5a7be) [0x7f10c70cd7be]
/usr/lib/libruby-1.9.1.so.1.9(rb_bug+0xa6) [0x7f10c70ce136]
/usr/lib/libruby-1.9.1.so.1.9(+0xf4a75) [0x7f10c7167a75]
/lib/libpthread.so.0(+0xfb40) [0x7f10c6e65b40]
/usr/lib/libruby-1.9.1.so.1.9(+0x6edc8) [0x7f10c70e1dc8]
/usr/lib/libruby-1.9.1.so.1.9(rb_thread_mark+0x4c) [0x7f10c71b1b8c]
/usr/lib/libruby-1.9.1.so.1.9(+0x6e87a) [0x7f10c70e187a]
/usr/lib/libruby-1.9.1.so.1.9(+0x13ea89) [0x7f10c71b1a89]
/usr/lib/libruby-1.9.1.so.1.9(st_foreach+0xb4) [0x7f10c7170284]
/usr/lib/libruby-1.9.1.so.1.9(rb_vm_mark+0x29) [0x7f10c71b1f09]
/usr/lib/libruby-1.9.1.so.1.9(+0x6e87a) [0x7f10c70e187a]
/usr/lib/libruby-1.9.1.so.1.9(+0x6fd98) [0x7f10c70e2d98]
/usr/lib/libruby-1.9.1.so.1.9(rb_newobj+0x47) [0x7f10c70e37c7]
/usr/lib/libruby-1.9.1.so.1.9(rb_data_typed_object_alloc+0x26) [0x7f10c70e3886]
/usr/lib/libruby-1.9.1.so.1.9(+0x13f97a) [0x7f10c71b297a]
/usr/lib/libruby-1.9.1.so.1.9(rb_vm_make_proc+0x56) [0x7f10c71b44e6]
/usr/lib/libruby-1.9.1.so.1.9(+0x64be3) [0x7f10c70d7be3]
/usr/lib/libruby-1.9.1.so.1.9(+0x154a92) [0x7f10c71c7a92]
/usr/lib/libruby-1.9.1.so.1.9(+0x149ca5) [0x7f10c71bcca5]
/usr/lib/libruby-1.9.1.so.1.9(+0x154cc8) [0x7f10c71c7cc8]
/usr/lib/libruby-1.9.1.so.1.9(+0x14c93e) [0x7f10c71bf93e]
/usr/lib/libruby-1.9.1.so.1.9(+0x143f61) [0x7f10c71b6f61]
/usr/lib/libruby-1.9.1.so.1.9(+0x148b44) [0x7f10c71bbb44]
/usr/lib/libruby-1.9.1.so.1.9(rb_yield+0x238) [0x7f10c71c2408]
/usr/lib/libruby-1.9.1.so.1.9(+0x96571) [0x7f10c7109571]
/usr/lib/libruby-1.9.1.so.1.9(+0x14c93e) [0x7f10c71bf93e]
/usr/lib/libruby-1.9.1.so.1.9(+0x143f61) [0x7f10c71b6f61]
/usr/lib/libruby-1.9.1.so.1.9(+0x148b44) [0x7f10c71bbb44]
/usr/lib/libruby-1.9.1.so.1.9(rb_iseq_eval_main+0xb1) [0x7f10c71c2c81]
/usr/lib/libruby-1.9.1.so.1.9(+0x5ecca) [0x7f10c70d1cca]
/usr/lib/libruby-1.9.1.so.1.9(ruby_exec_node+0x1d) [0x7f10c70d22ed]
/usr/lib/libruby-1.9.1.so.1.9(ruby_run_node+0x1e) [0x7f10c70d3e3e]
ruby(main+0x4b) [0x40092b]
/lib/libc.so.6(__libc_start_main+0xfe) [0x7f10c6229d8e]
ruby() [0x400819]
- Crash on Thread#new:
iteration [1780]
iteration [1781]
iteration [1782]
iteration [1783]
test_bug.rb:12: [BUG] Segmentation fault
ruby 1.9.2p0 (2010-08-18 revision 29036) [x86_64-linux]
-- control frame ----------
c:0007 p:---- s:0023 b:0023 l:000022 d:000022 CFUNC :new
c:0006 p:0063 s:0020 b:0019 l:000048 d:000018 BLOCK test_bug.rb:12
c:0005 p:---- s:0015 b:0015 l:000014 d:000014 FINISH
c:0004 p:---- s:0013 b:0013 l:000012 d:000012 CFUNC :times
c:0003 p:0121 s:0010 b:0010 l:000048 d:0011b8 EVAL test_bug.rb:9
c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH
c:0001 p:0000 s:0002 b:0002 l:000048 d:000048 TOP¶
-- Ruby level backtrace information ----------------------------------------
test_bug.rb:9:in <main>' test_bug.rb:9:in times'
test_bug.rb:12:in block in <main>' test_bug.rb:12:in new'
-- C level backtrace information -------------------------------------------
/usr/lib/libruby-1.9.1.so.1.9(rb_vm_bugreport+0x5f) [0x7f53d34ec58f]
/usr/lib/libruby-1.9.1.so.1.9(+0x5a7be) [0x7f53d33f37be]
/usr/lib/libruby-1.9.1.so.1.9(rb_bug+0xa6) [0x7f53d33f4136]
/usr/lib/libruby-1.9.1.so.1.9(+0xf4a75) [0x7f53d348da75]
/lib/libpthread.so.0(+0xfb40) [0x7f53d318bb40]
/usr/lib/libruby-1.9.1.so.1.9(+0x6edc8) [0x7f53d3407dc8]
/usr/lib/libruby-1.9.1.so.1.9(rb_thread_mark+0x4c) [0x7f53d34d7b8c]
/usr/lib/libruby-1.9.1.so.1.9(+0x6e87a) [0x7f53d340787a]
/usr/lib/libruby-1.9.1.so.1.9(+0x13ea89) [0x7f53d34d7a89]
/usr/lib/libruby-1.9.1.so.1.9(st_foreach+0xb4) [0x7f53d3496284]
/usr/lib/libruby-1.9.1.so.1.9(rb_vm_mark+0x29) [0x7f53d34d7f09]
/usr/lib/libruby-1.9.1.so.1.9(+0x6e87a) [0x7f53d340787a]
/usr/lib/libruby-1.9.1.so.1.9(+0x6fd98) [0x7f53d3408d98]
/usr/lib/libruby-1.9.1.so.1.9(+0x70b4c) [0x7f53d3409b4c]
/usr/lib/libruby-1.9.1.so.1.9(+0x1403af) [0x7f53d34d93af]
/usr/lib/libruby-1.9.1.so.1.9(rb_thread_alloc+0x66) [0x7f53d34e9286]
/usr/lib/libruby-1.9.1.so.1.9(+0x154cb8) [0x7f53d34edcb8]
/usr/lib/libruby-1.9.1.so.1.9(+0x14c93e) [0x7f53d34e593e]
/usr/lib/libruby-1.9.1.so.1.9(+0x143f61) [0x7f53d34dcf61]
/usr/lib/libruby-1.9.1.so.1.9(+0x148b44) [0x7f53d34e1b44]
/usr/lib/libruby-1.9.1.so.1.9(rb_yield+0x238) [0x7f53d34e8408]
/usr/lib/libruby-1.9.1.so.1.9(+0x96571) [0x7f53d342f571]
/usr/lib/libruby-1.9.1.so.1.9(+0x14c93e) [0x7f53d34e593e]
/usr/lib/libruby-1.9.1.so.1.9(+0x143f61) [0x7f53d34dcf61]
/usr/lib/libruby-1.9.1.so.1.9(+0x148b44) [0x7f53d34e1b44]
/usr/lib/libruby-1.9.1.so.1.9(rb_iseq_eval_main+0xb1) [0x7f53d34e8c81]
/usr/lib/libruby-1.9.1.so.1.9(+0x5ecca) [0x7f53d33f7cca]
/usr/lib/libruby-1.9.1.so.1.9(ruby_exec_node+0x1d) [0x7f53d33f82ed]
/usr/lib/libruby-1.9.1.so.1.9(ruby_run_node+0x1e) [0x7f53d33f9e3e]
ruby(main+0x4b) [0x40092b]
/lib/libc.so.6(__libc_start_main+0xfe) [0x7f53d254fd8e]
ruby() [0x400819]
- Sometimes it crashes on "times.times do".
However as I said, in same machine using Ruby 1.9.1 it doesn't crash.
My system data:
- AMD Phenom(tm) II X4 965 Processor
- Ubuntu 10.10 64 bits
- gcc-4.4
- Ruby 1.9.2-p0, 1.9.2-p136, and snapshot version (2011-01-12)
- 4GB RAM
=end
Updated by ibc (Iñaki Baz Castillo) over 14 years ago
=begin
Yui:
I can reproduce this on ruby 1.9.3dev (2011-01-12 trunk 30517) [x86_64-freebsd8.2].
Anyway what your environment? (why you didn't copy ruby -v ?)
Hi, I've tested with these versions of Ruby in my Linux Ubuntu 64 bits:
- CRASH in:
- ruby 1.9.2p0 (2010-08-18 revision 29036) [x86_64-linux]
- ruby 1.9.2p136 (2010-12-25 revision 30365) [x86_64-linux]
- ruby 1.9.3dev (2011-01-11 trunk 30513) [x86_64-linux]
- All ok in:
- ruby 1.9.1p430 (2010-08-16 revision 28998) [x86_64-linux]
=end
- ruby 1.9.1p430 (2010-08-16 revision 28998) [x86_64-linux]
Updated by ibc (Iñaki Baz Castillo) over 14 years ago
=begin
Tomoyuki:
Please add the following line in test_encoding.c
#include <ruby/encoding.h>
Without that header, rb_utf8_encoding() has no prototype declaration and its return value is estimated to be int.
Yes!! that is! no more crashes after adding it! Thanks a lot.
Now just a two questions remain:
- Why doesn't it crash on Ruby-1.9.1-xxx?
- Should "ruby.h" include "ruby/encoding.h"?
=end
Updated by naruse (Yui NARUSE) over 14 years ago
- Status changed from Open to Closed
=begin
"ruby/encoding.h" fixes this on my environment too.
=end