Ruby C10K: High Performance Networking - RubyKaigi '09

Ruby C10K: High Performance Networking - RubyKaigi '09

• 1.
  Ruby C10K: HighPerformance Networkinga case study with EM-ProxyIlya Grigorik@igrigorik
• 2.
  postrank.com/topic/rubyTwitterMy blog
• 3.
  C10K EM-Proxy+ ExamplesBenchmarks+ MiscEventMachine
• 4.
  Proxy Love
• 5.
  “Rails, Django, Seaside,Grails…” cant scale.Myth: Slow Frameworks
• 6.
  The Proxy Solution
• 7.
  The “More” ProxySolution
• 8.
  Transparent Scalability
• 9.
  Load BalancerReverse ProxyAppServerMySQL ProxyArchitecturemiddleware ftw!Shard 1Shard 2
• 10.
  C10K Problem +Rubywhy do we care?
• 11.
  Bottleneck: ~100 req/ sComplexity, Time, and Moneycirca 1995-2000
• 12.
  ReceiveVerifyDispatchAggregateHandle errorsRenderSendApplication BottlenecksI/O+ KernelBottlenecks Kernel + I/O Bottlenecks
• 13.
  C10K Challenge: 10,000Concurrent Connections
• 15.
  No concurrencyBlockingOk resourceutilizationrequire 'rubygems'require 'socket'server = TCPServer.new(80)loop do session = server.acceptsession.print"HTTP/1.1 200 OK\r\n\r\ndone"session.closeendFork!Synchronous + Blocking IO
• 16.
  Fork LatencyLinux 2.6:~200 microseconds
• 17.
  Socket.accept_nonblockBusy-wait CPU cycles
• 18.
  Poll for eachsocketselect( […], nil, nil )1024 FD limit by default
• 19.
  Non linear performanceNon-BlockingIO + Pollconcurrency without threads
• 20.
  Epoll + KqueueBenchmarks
• 21.
  while (1) {intnfds= epoll_wait(fd, arr, 3, timeout);if (nfds < 0) die("Error in epoll_wait!");for(inti = 0; i < nfds; i++) {intfd = events[i].data.fd;handle_io_on_socket(fd); }}and in Ruby…EPoll & KQueueconcurrency without threadsrequire 'eventmachine'EM.epollEM.run { # ...}
• 22.
  while (1) {intnfds= epoll_wait(fd, arr, 3, timeout);if (nfds < 0) die("Error in epoll_wait!");for(inti = 0; i < nfds; i++) {intfd = events[i].data.fd;handle_io_on_socket(fd); }}and in Ruby…EPoll & KQueueconcurrency without threadsrequire 'eventmachine'EM.epollEM.run { # ...}
• 23.
  EventMachine: Speed +Conveniencebuilding high performance network apps in Ruby
• 24.
  p "Starting"EM.run do p "Running in EM reactor"endputs "Almost done"whiletruedo timersnetwork_ioother_ioendEventMachine Reactorconcurrency without threads
• 25.
  p "Starting"EM.rundo p "Running in EM reactor"endputs "Almost done"whiletruedotimersnetwork_ioother_ioendEventMachine Reactorconcurrency without threads
• 26.
  C++ core Easy concurrency without threadingEventMachine Reactorconcurrency without threads
• 27.
  http = EM::HttpRequest.new('http://site.com/').gethttp.callback{ p http.response } # ... do other work, until callback fires. Event = IO event + block or lambda callEventMachine Reactorconcurrency without threads
• 28.
  http=EM::HttpRequest.new('http://site.com/').gethttp.callback{phttp.response}# ... doother work, until callback fires.Screencast: http://bit.ly/hPr3j Event = IO event + block or lambda callEventMachine Reactorconcurrency without threads
• 29.
  EM.rundoEM.add_timer(1) { p"1 second later" }EM.add_periodic_timer(5) { p "every 5 seconds"}EM.defer { long_running_task() }endclass Server < EM::Connection def receive_data(data)send_data("Pong; #{data}") end def unbind p [:connection_completed] endendEM.run doEM.start_server "0.0.0.0", 3000, Serverend
• 30.
  EM.run doEM.add_timer(1) {p "1 second later" }EM.add_periodic_timer(5) { p "every 5 seconds"}EM.defer { long_running_task() }endclass Server < EM::Connection def receive_data(data)send_data("Pong; #{data}") end def unbind p [:connection_completed] endendEM.rundoEM.start_server"0.0.0.0", 3000, ServerendStart Reactor
• 31.
  EM.run doEM.add_timer(1) {p "1 second later" }EM.add_periodic_timer(5) { p "every 5 seconds"}EM.defer { long_running_task() }endclass Server < EM::Connectiondefreceive_data(data)send_data("Pong; #{data}")enddef unbind p [:connection_completed]endendEM.rundoEM.start_server"0.0.0.0", 3000, ServerendConnection HandlerStart Reactor
• 32.
  http://bit.ly/aiderss-eventmachineby Dan Sinclair(Twitter: @dj2sincl)
• 33.
  Profile of querieschanges FailLoad on production changes FailParallel environment FailSlower release cycle FailProblem: Staging Environment Fail
• 34.
  Proxies for Monitoring,Performance and Scalewelcome tothe wonderful world of… (C10K proof)…
• 35.
  Duplex Ruby Proxy,FTW!Real (production) trafficBenchmarking Proxyflash of the obvious
• 36.
  github.com/igrigorik/em-proxyProxy DSL: EM+ EPoll
• 37.
  Proxy.start(:host => "0.0.0.0",:port => 80) do |conn|conn.server:name, :host => "127.0.0.1", :port => 81conn.on_data do |data| # ... endconn.on_response do |server, resp| # ... endconn.on_finish do # ... endendRelay ServerEM-Proxywww.github.com/igrigorik/em-proxy
• 38.
  Proxy.start(:host => "0.0.0.0",:port => 80) do |conn|conn.server:name, :host => "127.0.0.1", :port => 81conn.on_datado |data|# ...endconn.on_response do |server, resp| # ... endconn.on_finish do # ... endendProcess incoming dataEM-Proxywww.github.com/igrigorik/em-proxy
• 39.
  Proxy.start(:host => "0.0.0.0",:port => 80) do |conn|conn.server:name, :host => "127.0.0.1", :port => 81conn.on_datado |data|# ...endconn.on_responsedo |server, resp|# ...endconn.on_finish do # ... endendProcess response dataEM-Proxywww.github.com/igrigorik/em-proxy
• 40.
  Proxy.start(:host => "0.0.0.0",:port => 80) do |conn|conn.server:name, :host => "127.0.0.1", :port => 81conn.on_datado |data|# ...endconn.on_responsedo |server, resp|# ...endconn.on_finishdo# ...endendPost-processing stepEM-Proxywww.github.com/igrigorik/em-proxy
• 41.
  Proxy.start(:host => "0.0.0.0",:port => 80) do |conn|conn.server:srv, :host => "127.0.0.1", :port => 81 # modify / process request streamconn.on_data do |data| p [:on_data, data] data end # modify / process response streamconn.on_response do |server, resp| p [:on_response, server, resp]resp end endExample: Port-Forwardingtransparent proxy
• 42.
  Proxy.start(:host => "0.0.0.0",:port => 80) do |conn|conn.server:srv, :host => "127.0.0.1", :port => 81# modify / process request streamconn.on_datado |data| p [:on_data, data] dataend# modify / process response streamconn.on_response do |server, resp| p [:on_response, server, resp]resp end endExample: Port-Forwardingtransparent proxy
• 43.
  Proxy.start(:host => "0.0.0.0",:port => 80) do |conn|conn.server:srv, :host => "127.0.0.1", :port => 81# modify / process request streamconn.on_datado |data| p [:on_data, data] dataend# modify / process response streamconn.on_responsedo |server, resp| p [:on_response, server, resp]respendendNo data modificationsExample: Port-Forwardingtransparent proxy
• 44.
  Proxy.start(:host => "0.0.0.0",:port => 80) do |conn|conn.server:srv, :host => "127.0.0.1", :port => 81conn.on_datado |data| dataendconn.on_response do |backend, resp|resp.gsub(/hello/, 'good bye') endendExample: Port-Forwarding + Altertransparent proxy
• 45.
  Proxy.start(:host => "0.0.0.0",:port => 80) do |conn|conn.server:srv, :host => "127.0.0.1", :port => 81conn.on_datado |data| dataendconn.on_responsedo |backend, resp|resp.gsub(/hello/, 'good bye')endendAlter responseExample: Port-Forwarding + Altertransparent proxy
• 46.
  Duplicating HTTP Trafficforbenchmarking & monitoring
• 47.
  Proxy.start(:host => "0.0.0.0",:port => 80) do |conn|@start = Time.now@data = Hash.new("")conn.server:prod, :host => "127.0.0.1", :port => 81conn.server:test, :host => "127.0.0.1", :port => 82 conn.on_data do |data|data.gsub(/User-Agent: .*?\r\n/, 'User-Agent: em-proxy\r\n') endconn.on_response do |server, resp| @data[server] += respresp if server == :prod endconn.on_finish do p [:on_finish, Time.now - @start] p @data endendProd + TestDuplex HTTP: BenchmarkingIntercepting proxy
• 48.
  Proxy.start(:host => "0.0.0.0",:port => 80) do |conn| @start = Time.now @data = Hash.new("")conn.server :prod, :host => "127.0.0.1", :port => 81conn.server :test, :host => "127.0.0.1", :port => 82 conn.on_datado |data|data.gsub(/User-Agent: .*?\r\n/, 'User-Agent: em-proxy\r\n')endconn.on_responsedo |server, resp|@data[server] += resprespif server == :prodendconn.on_finish do p [:on_finish, Time.now - @start] p @data endendRespond from productionDuplex HTTP: BenchmarkingIntercepting proxy
• 49.
  Proxy.start(:host => "0.0.0.0",:port => 80) do |conn| @start = Time.now @data = Hash.new("")conn.server :prod, :host => "127.0.0.1", :port => 81conn.server :test, :host => "127.0.0.1", :port => 82 conn.on_data do |data|data.gsub(/User-Agent: .*?\r\n/, 'User-Agent: em-proxy\r\n') endconn.on_response do |server, resp| @data[server] += respresp if server == :prod endconn.on_finishdo p [:on_finish, Time.now - @start] p @dataendendRun post-processingDuplex HTTP: BenchmarkingIntercepting proxy
• 50.
  [ilya@igvita] >ruby examples/appserver.rb81[ilya@igvita] >ruby examples/appserver.rb 82[ilya@igvita] >ruby examples/line_interceptor.rb[ilya@igvita] >curl localhost>> [:on_finish, 1.008561]>> {:prod=>"HTTP/1.1 200 OK\r\nConnection: close\r\nDate: Fri, 01 May 2009 04:20:00 GMT\r\nContent-Type: text/plain\r\n\r\nhello world: 0", :test=>"HTTP/1.1 200 OK\r\nConnection: close\r\nDate: Fri, 01 May 2009 04:20:00 GMT\r\nContent-Type: text/plain\r\n\r\nhello world: 1"}Duplex HTTP: BenchmarkingIntercepting proxy
• 51.
  [ilya@igvita] >ruby examples/appserver.rb81[ilya@igvita] >ruby examples/appserver.rb 82[ilya@igvita] >ruby examples/line_interceptor.rb[ilya@igvita] >curl localhostSTDOUT[:on_finish, 1.008561]{:prod=>"HTTP/1.1 200 OK\r\nConnection: close\r\nDate: Fri, 01 May 2009 04:20:00 GMT\r\nContent-Type: text/plain\r\n\r\nhello world: 0",:test=>"HTTP/1.1 200 OK\r\nConnection: close\r\nDate: Fri, 01 May 2009 04:20:00 GMT\r\nContent-Type: text/plain\r\n\r\nhello world: 1"}Duplex HTTP: BenchmarkingIntercepting proxy
• 52.
  Same response, differentturnaround timeDifferent response body!
• 53.
  Woops!Validating Proxyeasy, real-timediagnostics
• 54.
  Hacking SMTP: Whitelistingforfun and profit
• 55.
  Proxy.start(:host => "0.0.0.0",:port => 2524) do |conn|conn.server:srv, :host => "127.0.0.1", :port => 2525# RCPT TO:<[email protected]>\r\n RCPT_CMD = /RCPT TO:<(.*)?>\r\n/conn.on_data do |data| if rcpt = data.match(RCPT_CMD) if rcpt[1] != "[email protected]"conn.send_data "550 No such user here\n" data = nil end end data endconn.on_responsedo |backend, resp|respendendIntercept AddresseeDefeating SMTP WildcardsIntercepting proxy
• 56.
  Proxy.start(:host => "0.0.0.0",:port => 2524) do |conn|conn.server :srv, :host => "127.0.0.1", :port => 2525 # RCPT TO:<[email protected]>\r\n RCPT_CMD = /RCPT TO:<(.*)?>\r\n/conn.on_datado |data|if rcpt = data.match(RCPT_CMD)if rcpt[1] != "[email protected]"conn.send_data"550 No such user here\n" data = nilendend dataendconn.on_response do |backend, resp|resp endendAllow: [email protected] Error otherwiseDefeating SMTP WildcardsIntercepting proxy
• 57.
  [ilya@igvita] >mailtrap run–p 2525 –f /tmp/mailtrap.log[ilya@igvita] >ruby examples/smtp_whitelist.rb> require 'net/smtp‘> smtp = Net::SMTP.start("localhost", 2524)> smtp.send_message "Hello World!", "[email protected]", "[email protected]" => #<Net::SMTP::Response:0xb7dcff5c @status="250", @string="250 OK\n">> smtp.finish => #<Net::SMTP::Response:0xb7dcc8d4 @status="221", @string="221 Seeya\n">> smtp.send_message "Hello World!", "[email protected]", “[email protected]"=> Net::SMTPFatalError: 550 No such user hereDuplex HTTP: BenchmarkingIntercepting proxy
• 58.
  [ilya@igvita] >mailtrap run–p 2525 –f /tmp/mailtrap.log[ilya@igvita] >ruby examples/smtp_whitelist.rbTo: [email protected]> require 'net/smtp‘> smtp = Net::SMTP.start("localhost", 2524)> smtp.send_message"Hello World!", "[email protected]", "[email protected]" => #<Net::SMTP::Response:0xb7dcff5c @status="250", @string="250 OK\n">> smtp.finish => #<Net::SMTP::Response:0xb7dcc8d4 @status="221", @string="221 Seeya\n">> smtp.send_message"Hello World!", "[email protected]", “[email protected]"=> Net::SMTPFatalError: 550 No such user hereDenied!Duplex HTTP: BenchmarkingIntercepting proxy
• 59.
  : Beanstalkd +EM-Proxy because RAM is still expensive
• 60.
  ~93 Bytes of overhead per job~300 Bytes of data / job x 80,000,000 jobs in memory ~ 30 GB of RAM = 2 X-Large EC2 instancesOi, expensive!BeanstalkdMath
• 61.
  Observations: 1. Each job is rescheduled several times 2. > 95% are scheduled for > 3 hours into the future 3. Beanstalkd does not have overflow page-to-diskMemory is wasted…Extending BeanstalkdWe’ll add it ourselves!
• 62.
  1 “Medium” EC2InstanceIntercepting Proxy@PostRank: “Chronos Scheduler”
• 63.
  Proxy.start(:host => "0.0.0.0",:port => 11300) do |conn|conn.server:srv, :host => "127.0.0.1", :port => 11301 PUT_CMD = /put (\d+) (\d+) (\d+) (\d+)\r\n/conn.on_data do |data| if put = data.match(PUT_CMD) if put[2].to_i > 600 p [:put, :archive] # INSERT INTO ....conn.send_data "INSERTED 9999\r\n" data = nil end end data endconn.on_responsedo |backend, resp|respendendIntercept PUT command
• 64.
  Proxy.start(:host => "0.0.0.0",:port => 11300) do |conn|conn.server :srv, :host => "127.0.0.1", :port => 11301 PUT_CMD = /put (\d+) (\d+) (\d+) (\d+)\r\n/conn.on_datado |data|if put = data.match(PUT_CMD)if put[2].to_i > 600 p [:put, :archive]# INSERT INTO ....conn.send_data"INSERTED 9999\r\n" data = nilendend dataendconn.on_response do |backend, resp|resp endendIf over 10 minutes…Archive & Reply
• 65.
  Overload the protocol PUTput job, 900 RESERVE, PUT, …@PostRank: “Chronos Scheduler”
• 66.
  ~79,000,000 jobs, 4GBRAM400% cheaper + extensible! PUTUpcoming jobs: ~ 1MRESERVE, PUT, …@PostRank: “Chronos Scheduler”
• 67.
  … x 2,5001 process / 1 core~ 5,000 open sockets~ 1200 req/sEM-ProxyBeanstalkdMySQL2x EM-Proxy (dual core)C10K Success!Performance: Beanstalk + EM-Proxyis it “C10K proof”?
• 68.
  C10K: http://www.kegel.com/c10k.html Code: http://github.com/igrigorik/em-proxyTwitter: @igrigorikThanks. Questions?TwitterMy blog