![End to End metrics
● Producer to Router to Sink Average Latencies
○ Batch processing [S3 sink]: ~3 sec
○ Stream processing [Consumer Kafka sink]: ~1 sec
○ Log analysis [Elasticsearch]: ~400 seconds (with back pressure)](https://image.slidesharecdn.com/keystone-hownetflixhandlesdatastreamsupto11meventssec-160224183316/85/Netflix-Keystone-How-Netflix-Handles-Data-Streams-up-to-11M-Events-Sec-52-320.jpg)
Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec
- 1. A NETFLIX ORIGINAL SERVICE
- 2. Peter Bakas | @peter_bakas @ Netflix : Cloud Platform Engineering - Real Time Data Infrastructure @ Ooyala : Analytics, Discovery, Platform Engineering & Infrastructure @ Yahoo : Display Advertising, Behavioral Targeting, Payments @ PayPal : Site Engineering and Architecture @ Play : Advisor to Startups (Data, Security, Containers) Who is this guy?
- 3. common data pipeline to collect, transport, aggregate, process and visualize events Why are we here?
- 4. ● Architectural design and principles for Keystone ● Technologies that Keystone is leveraging ● Best practices What should I expect?
- 5. Let’s get down to business
- 6. Netflix is a logging company
- 7. that occasionally streams movies
- 8. 600+ billion events ingested per day 11 million events (24 GB per second) peak Hundreds of event types Over 1.3 Petabyte / day Numbers Galore!
- 9. But wait, there’s more
- 10. 1+ trillion events processed every day 1 trillion events ingested per day during holiday season Numbers Galore - Part Deux
- 11. How did we get here?
- 12. Chukwa
- 13. Chukwa/Suro + Real-Time Branch
- 15. Kafka Primer Kafka is a distributed, partitioned, replicated commit log service.
- 16. Kafka Terminology ● Producer ● Consumer ● Topic ● Partition ● Broker
- 18. Netflix Kafka Producer ● Best effort delivery ● Prefer msg drop than disrupting producer app ● Wraps Apache Kafka Producer ● Integration with Netflix ecosystem: Eureka, Atlas, etc.
- 19. Producer Impact ● Kafka outage does not disrupt existing instances from serving its purpose ● Kafka outage should never prevent new instances from starting up ● After kafka cluster restored, event producing should resume automatically
- 20. Prefer Drop than Block ● Drop when buffer is full ● Handle potential blocking of first meta data request ● ack=1 (vs 2)
- 21. Sticky Partitioner ● Batching is important to reduce CPU and network I/O on brokers ● Stick to one partition for a while when producing for non-keyed messages ● “linger.ms” works well with sticky partitioner
- 22. Producing events to Keystone ● Using Netflix Platform logging API ○ LogManager.logEvent(Annotatable): majority of the cases ○ KeyValueSeriazlier with ILog#log(String) ● REST endpoint that proxies Platform logging ○ ksproxy ○ Prana sidecar
- 23. Injected Event Metadata ● GUID ● Timestamp ● Host ● App
- 24. Keystone Extensible Wire Protocol ● Invisible to source & sinks ● Backwards and forwards compatibility ● Supports JSON. AVRO on the horizon ● Efficient - 10 bytes overhead per message ○ message size - hundreds of bytes to 10MB
- 25. Keystone Extensible Wire Protocol ● Packaged as a jar ● Why? Evolve Independently ○ event metadata & traceability metadata ○ event payload serialization
- 26. Max message size 10MB ● Keystone drops if > 10MB ○ Immutable event payload
- 29. Fronting Kafka Clusters ● Normal-priority (majority) ● High-priority (streaming activities etc.)
- 30. Fronting Kafka Instances ● 3 ASGs per cluster, 1 ASG per zone ● 3000 d2.xl AWS instances across 3 regions for regular & failover traffic
- 31. Partition Assignment ● All replica assignments zone aware ○ Improved availability ○ Reduce cost of maintenance
- 32. Kafka Fault Tolerance ● Instance failure ○ With replication factor of N, guarantee no data loss with N-1 failures ○ With zone aware replica assignment, guarantee no data loss with multiple instance failures in the same zone ● Sink failure ○ No data loss during retention period ● Replication is the key ○ Data loss can happen if leader dies while follower AND consumer cannot catch up ○ Usually indicated by UncleanLeaderElection metric
- 33. Kafka Auditor as a Service ● Broker monitoring ● Consumer monitoring ● Heart-beat & Continuous message latency ● On-demand Broker performance testing ● Built as a service deployable on single or multiple instances
- 34. Current Issues ● By using the d2-xl there is trade off between cost and performance ● Performance deteriorates with increase of partitions ● Replication lag during peak traffic
- 35. Routing Service
- 38. Broker
- 40. Router Job Manager (Control Plane) EC2 Instances Zookeeper (Instance Id assignment) Job Job Job ksnode Checkpointing Cluster ASG Reconcile every min.
- 41. Routing Layer ● Total of 13,000 containers on 1,300 AWS C3-4XL instances ○ S3 sink: ~7000 Containers ○ Consumer Kafka sink: ~ 4500 Containers ○ Elasticsearch sink: ~1500 Containers
- 42. Routing Layer ● Total of ~1400 streams across all regions ○ ~1000 S3 streams ○ ~250 Consumer Kafka streams ○ ~150 Elasticsearch streams
- 43. Router Job Details ● One Job per sink and Kafka source topic ○ Separate Job each for S3, ElasticSearch & Kafka sink ○ Provides better isolation & better QOS ● Batch processed message requests to sinks ● Offset checkpointed after batch request succeeds
- 44. Processing Semantics Data Loss & Duplicates
- 45. Backpressure Producer ⇐ Kafka Cluster ⇐ Samza job router ⇐ Sink ● Keystone - at least once
- 46. Data Loss - Producer ● buffer full ● network error ● partition leader change ● partition migration
- 47. Data Loss - Kafka ● Lose all Kafka replicas of data ○ Safe guards: ■ AZ isolation / Alerts / Broker replacement automation ■ alerts and monitoring ● Unclean partition leader election ○ ack = 1 could cause loss
- 48. Data Loss - Router ● Lose checkpointed offset & the router was down for retention period duration ● If messages not processed past retention period (8h / 24h) ● Unclean leader election cause offset to go back ● Safe guard ○ alerts for lag > 0.1% of traffic for 10 minutes ● Concerned only if unable to launch router instances
- 49. Duplicates Router - Sink ● Duplicates possible ○ messages reprocessed - retry after batch S3 upload failure ○ Loss of checkpointed offset (message processed marker) ○ Event GUID helps dedup
- 50. Measure Duplicates ● Producer sent count diff with Kafka message received ● Router checkpointed offset monitored over time Note: GUID can be used to dedup at the sink
- 52. End to End metrics ● Producer to Router to Sink Average Latencies ○ Batch processing [S3 sink]: ~3 sec ○ Stream processing [Consumer Kafka sink]: ~1 sec ○ Log analysis [Elasticsearch]: ~400 seconds (with back pressure)
- 53. End to End metrics ● End to End latencies ○ S3: ■ 50 percentile under 1 sec ■ 80 percentile under 8 seconds ○ Consumer Kafka: ■ 50 percentile under 800 ms ■ 80 percentile under 4 seconds ○ Elasticsearch: ■ 50 percentile under 13 seconds ■ 80 percentile under 53 seconds
- 54. Alerts ● Producer drop rate over 1% ● Consumer lag > 0.1% ● Next offset after Checkpointed offset not found ● Consumer stuck on partition level
- 59. And Then???
- 60. There’s more in the pipeline... ● Self service tools ● Better management of scaling Kafka ● More capable control plane ● JSON Support exists, support for Avro on the horizon ● Multi-tenant Messaging as a Service - MaaS ● Multi-tenant Stream Processing as a Service - SPaaS
- 61. ???s