[#40602] [ruby-trunk - Bug #5532][Open] Compile problem for bigdecimal on cygwin — Martin Dürst <duerst@...>

14 messages 2011/11/01

[#40617] [ruby-trunk - Feature #5534][Open] Redefine Range class and introduce RelativeNumeric and RelativeRange — Alexey Muranov <muranov@...>

17 messages 2011/11/01

[#40646] [ruby-trunk - Bug #5541][Open] Better configure error message when llvm-gcc is the default compiler — Eric Hodel <[email protected]>

10 messages 2011/11/01

[#40648] [ruby-trunk - Feature #5543][Open] rb_thread_blocking_region() API is poorly designed — Christopher Huff <cjameshuff@...>

14 messages 2011/11/01

[#40684] [ruby-trunk - Feature #5555][Open] rename #include? to #includes? — Alexey Muranov <muranov@...>

20 messages 2011/11/02

[#40688] [ruby-trunk - Bug #5556][Open] SIGHUP no longer ignored when sent to process group from a subprocess — Brian Ford <brixen@...>

12 messages 2011/11/02

[#40706] [ruby-trunk - Feature #5562][Open] Improvement of Windows IO performance — Hiroshi Shirosaki <h.shirosaki@...>

39 messages 2011/11/03

[#40737] [ruby-trunk - Bug #5570][Open] Encoding of environment variables on Windows — Nikolai Weibull <now@...>

11 messages 2011/11/04

[#40748] Proposal for sustainable branch maintenance — "Yuki Sonoda (Yugui)" <yugui@...>

-----BEGIN PGP SIGNED MESSAGE-----

14 messages 2011/11/05

[#40770] [ruby-trunk - Feature #5578][Open] Embedded YAML for Ruby 2.0 — Thomas Sawyer <transfire@...>

17 messages 2011/11/06

[#40806] [ruby-trunk - Feature #5583][Open] Optionally typing — Yasushi ANDO <andyjpn@...>

21 messages 2011/11/07

[#40824] [ruby-trunk - Feature #5588][Open] add negation flag (v) to Regexp — Suraj Kurapati <sunaku@...>

38 messages 2011/11/08

[#40865] IO.copy_stream creates files with restrictive permissions — Eric Wong <normalperson@...>

I'm not sure if this is a bug or intended as spec.

16 messages 2011/11/09
[#41151] Re: IO.copy_stream creates files with restrictive permissions — Tanaka Akira <akr@...> 2011/11/19

2011/11/9 Eric Wong <[email protected]>:

[#41166] Re: IO.copy_stream creates files with restrictive permissions — KOSAKI Motohiro <kosaki.motohiro@...> 2011/11/20

>> I noticed when a file name argument is passed to the IO.copy_stream, the

[#41168] Re: IO.copy_stream creates files with restrictive permissions — Clifford Heath <clifford.heath@...> 2011/11/20

On 20/11/2011, at 5:09 PM, KOSAKI Motohiro wrote:

[#41176] Re: IO.copy_stream creates files with restrictive permissions — Tanaka Akira <akr@...> 2011/11/21

2011/11/20 Clifford Heath <[email protected]>:

[#41180] Re: IO.copy_stream creates files with restrictive permissions — KOSAKI Motohiro <kosaki.motohiro@...> 2011/11/21

>> I think documentation is the wrong answer. The security defects are not caused

[#40908] [ruby-trunk - Feature #5607][Open] Inconsistent reaction in Range of String — Yen-Nan Lin <redmine@...>

15 messages 2011/11/10

[#40941] [ruby-trunk - Feature #5617][Open] Allow install RubyGems into dediceted directory — Vit Ondruch <v.ondruch@...>

22 messages 2011/11/11

[#40951] [Backport93 - Backport #5621][Open] Please backport thread-safe autoloading patch — Mike Perham <mperham@...>

25 messages 2011/11/12
[#40971] [Backport93 - Backport #5621] Please backport thread-safe autoloading patch — Mike Perham <mperham@...> 2011/11/12

[#40972] Re: [Backport93 - Backport #5621] Please backport thread-safe autoloading patch — Yehuda Katz <wycats@...> 2011/11/12

Unfortunately ruby-head has a deadlock in one of my go-to scenarios for

[#40976] Re: [Backport93 - Backport #5621] Please backport thread-safe autoloading patch — Hiroshi Nakamura <nahi@...> 2011/11/13

-----BEGIN PGP SIGNED MESSAGE-----

[#41128] Re: [Backport93 - Backport #5621] Please backport thread-safe autoloading patch — Charles Oliver Nutter <headius@...> 2011/11/18

On Sat, Nov 12, 2011 at 7:24 PM, Hiroshi Nakamura <[email protected]> wrote:

[#41129] Re: [Backport93 - Backport #5621] Please backport thread-safe autoloading patch — Hiroshi Nakamura <nahi@...> 2011/11/18

-----BEGIN PGP SIGNED MESSAGE-----

[#41142] Re: [Backport93 - Backport #5621] Please backport thread-safe autoloading patch — Charles Oliver Nutter <headius@...> 2011/11/18

On Fri, Nov 18, 2011 at 12:15 AM, Hiroshi Nakamura <[email protected]> wro=

[#40982] [ruby-trunk - Bug #5625][Open] Remove profanity and pejoratives — Andrew Grimm <andrew.j.grimm@...>

30 messages 2011/11/13

[#41004] [ruby-trunk - Feature #5628][Open] Module#basename — Thomas Sawyer <transfire@...>

18 messages 2011/11/14

[#41024] [ruby-trunk - Feature #5632][Open] Attempt to open included class shades it instead. — Boris Stitnicky <boris@...>

12 messages 2011/11/14

[#41025] Proposal to add new methods: positive? negative? natural? — JosFrancisco Calvo Moreno <josefranciscocalvo@...>

Hi all!

11 messages 2011/11/14
[#41027] Re: Proposal to add new methods: positive? negative? natural? — Jeremy Evans <code@...> 2011/11/14

On 11/15 12:58, Jos? Francisco Calvo Moreno wrote:

[#41031] Re: Proposal to add new methods: positive? negative? natural? — JosFrancisco Calvo Moreno <josefranciscocalvo@...> 2011/11/14

Hi Jeremy,

[#41038] [ruby-trunk - Bug #5634][Open] yield and binding — Thomas Sawyer <transfire@...>

17 messages 2011/11/14

[#41086] [ruby-trunk - Feature #5644][Open] add Enumerable#exclude? antonym — Suraj Kurapati <sunaku@...>

14 messages 2011/11/17

[#41175] [ruby-trunk - Feature #5654][Open] Introduce global lock to avoid concurrent require — Hiroshi Nakamura <nakahiro@...>

12 messages 2011/11/21

[#41200] [ruby-trunk - Bug #5659][Open] bug releasing a gem created with rails 3.1 — Vinicius Gati <viniciusgati@...>

14 messages 2011/11/22

[#41212] [ruby-trunk - Feature #5662][Open] inject-accumulate, or Haskell's mapAccum* — Edvard Majakari <edvard.majakari@...>

12 messages 2011/11/22

[#41213] [ruby-trunk - Bug #5663][Open] Combined map/select method — Yehuda Katz <wycats@...>

62 messages 2011/11/22

[#41317] [ruby-trunk - Bug #5676][Open] miniruby linking error: undefined reference to ___stack_chk_guard — Martin Dürst <duerst@...>

10 messages 2011/11/27

[#41404] [ruby-trunk - Bug #5690][Open] Module#qualified_const_get — Yehuda Katz <wycats@...>

31 messages 2011/11/30

[ruby-core:40903] Re: [ruby-trunk - Feature #5605] [PATCH] net/http: use IO.copy_stream for requests using body_stream

From: Eric Wong <normalperson@...>
Date: 2011-11-10 03:25:39 UTC
List: ruby-core #40903
Eric Hodel <[email protected]> wrote:
> For the chunked? half, would looping over
> (({IO.copy_stream f, sock.io, 1024})) be faster?

Yes, but its a rare case, needs to depend on a non-portable ioctl(),
and isn't much faster.

You can't blindly specify 1024 bytes because 'f' isn't guaranteed to
have 1024 readable.  But you need to write the "%x\r\n" size
header before calling copy_stream, so I'm using IO#nread.
Chunked uploads only make sense if:

1) f is a pipe/socket
2) you want to generate something like a Content-MD5 Trailer:
   on-the-fly and don't want to read f twice.

In either case, sendfile() can't be used by IO.copy_stream.  We can
teach IO.copy_stream to splice() on Linux, maybe[1]

I don't know if it's worth it to introduce the following patch, it only
saves 200ms user CPU time on a 1G upload for an uncommon use case.
I'm concerned about the portability of IO#nread.  Even if IO#nread is
available for the IO object, there's no guarantee the FIONREAD ioctl()
is implemented (or implemented correctly) for the underlying file type
given all the OSes Ruby supports.

--- a/lib/net/http.rb
+++ b/lib/net/http.rb
@@ -21,6 +21,7 @@
 
 require 'net/protocol'
 require 'uri'
+require 'io/wait'
 autoload :OpenSSL, 'openssl'
 
 module Net   #:nodoc:
@@ -1965,9 +1966,25 @@ module Net   #:nodoc:
       write_header sock, ver, path
       wait_for_continue sock, ver if sock.continue_timeout
       if chunked?
-        chunker = Chunker.new(sock)
-        IO.copy_stream(f, chunker)
-        chunker.finish
+        io = sock.io
+        if io.respond_to?(:nread)
+          seekable_p = f.stat.file?
+          f.wait
+          len = f.nread
+          while len > 0
+            io.write("#{len.to_s(16)}\r\n")
+            IO.copy_stream(f, io, len)
+            f.pos += len if seekable_p
+            io.write("\r\n")
+            f.wait or break
+            len = f.nread
+          end
+          io.write("0\r\n\r\n")
+        else
+          chunker = Chunker.new(sock)
+          IO.copy_stream(f, chunker)
+          chunker.finish
+        end
       else
         # copy_stream can sendfile() to sock.io unless we use SSL.
         # If sock.io is an SSLSocket, copy_stream will hit SSL_write()
---------------------------------------------------------------
Since chunked encoding is normally used for pipe/sockets
of unknown length, I added the following to my test script to test:

r, w = IO.pipe
tmp.sysseek(0)
pid = fork { IO.copy_stream(tmp.to_io, w) }
w.close
Net::HTTP.start(host, port) do |http|
  put = Net::HTTP::Put.new("/")
  put.body_stream = r
  put["Transfer-Encoding"] = "chunked"
  puts "with (pipe) Transfer-Encoding: chunked"
  bm = Benchmark.measure { res = http.request(put) }
  p [ res, res.body ]
  printf("utime=%0.3f stime=%0.3f\n", bm.utime, bm.stime)
end
Process.waitpid2(pid)

> ------------------------ Results -------------------------------
> after
> -----
> with Content-Length: 1073741825
> [#<Net::HTTPOK 200 OK readbody=true>,
>  "dcdd67a8f07b73796c0485890d48fa697966d09f\n"]
> utime=0.010 stime=0.410
                      ^-- Oops, I realized stime=0.410 is high
                          because I was on a cold cache, even
                          with a sparse file. It should be 0.090

> with Transfer-Encoding: chunked
> [#<Net::HTTPOK 200 OK readbody=true>,
>  "dcdd67a8f07b73796c0485890d48fa697966d09f\n"]
> utime=0.320 stime=0.620

With a regular file, chunked becomes as fast as with Content-Length:
---------------------------------------------------------------
with (file) Transfer-Encoding: chunked
[#<Net::HTTPOK 200 OK readbody=true>, "dcdd67a8f07b73796c0485890d48fa697966d09f\n"]
utime=0.010 stime=0.090

I think a better alternative would be to teach Net::HTTP to
automatically set Content-Length if body_stream is a regular file.
Chunked requests is are still an uncommon feature, I think.


With a pipe, using IO.copy_stream calling Chunker
---------------------------------------------------------------
with (pipe) Transfer-Encoding: chunked
[#<Net::HTTPOK 200 OK readbody=true>, "dcdd67a8f07b73796c0485890d48fa697966d09f\n"]
utime=0.350 stime=0.940

With a pipe, using IO#nread loop calling IO.copy_stream
---------------------------------------------------------------
with (pipe) Transfer-Encoding: chunked
[#<Net::HTTPOK 200 OK readbody=true>, "dcdd67a8f07b73796c0485890d48fa697966d09f\n"]
utime=0.050 stime=0.940


[1] - We have to be careful since there are still old kernels with
      a buggy splice() implementation.

In This Thread