Download to read offline
More Related Content
Similar to Polyglot, Fault Tolerant Event-Driven Programming with Kafka, Kubernetes and gRpc
Polyglot, Fault Tolerant Event-Driven Programming with Kafka, Kubernetes and gRpc
- 1. natansil.com twitter@NSilnitsky linkedin/natansilnitskygithub.com/natansil Polyglot, Fault-Tolerant, and Performant Event-Driven Programming with Kafka, Kubernetes and gRPC Natan Silnitsky Backend Infra Developer, Wix.com
- 2. registered users from 190countries 180M of all internet websites run on Wix 5% About Wix @NSilnitsky
- 3.
- 4.
- 5. Wix 1510M Kafka messages aday @NSilnitsky
- 6. Wix 1,075M Kafka messages aday So, we need our message flows to be performant, fault-tolerant, and polyglot.
- 7. Agenda Event-driven programming withKafka + Performance with Greyhound + Fault-tolerance with Greyhound & Kubernetes + Polyglot with Kubernetes & gRPC
- 8. @NSilnitsky HTTP * coupled Request-Reply Communication New App installed SiteApps Service ECom Catalog Service Classic.
- 9. @NSilnitsky HTTP New App installed What ifNetwork is Unreliable Request-Reply Communication Site Apps Service ECom Catalog Service
- 10. @NSilnitsky * 1500hard HTTP New App installed Request-Reply Communication Cascading Failures can happen.
- 11.
- 12.
- 13. @NSilnitsky Kafka Broker Event-driven Communication Site Apps Topic 01 2 3 45 0 1 2 3 45 0 1 2 3 45 0 1 2 3 45 0 1 2 3 4 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 Where you store messages (events!) A Kafka Broker Looks Like This: * vs message queues
- 14. @NSilnitsky Kafka Broker Event-driven Communication Site Apps Topic 01 2 3 45 0 1 2 3 45 0 1 2 3 45 0 1 2 3 45 0 1 2 3 4 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 Kafka Producer Site Apps Service There’s the Event Producer,
- 15. @NSilnitsky Kafka Broker Event-driven Communication Site Apps Topic 01 2 3 45 0 1 2 3 45 0 1 2 3 45 0 1 2 3 45 0 1 2 3 4 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 Kafka Consumer Ecom Catalog Service and Event Consumer. * scale
- 16. Site Apps Service Kafka Consumer Kafka Producer Greyhound wraps Kafka EcomCatalog Service Kafka Broker Greyhound Producer Greyhound Consumer
- 17. @NSilnitsky Simplify APIs, with additional features Greyhound wrapsKafka Site Apps Service Kafka Consumer Kafka Producer Ecom Catalog Service Kafka Broker
- 18. @NSilnitsky Abstract so that itis easy to change for everyone Kafka Consumer Kafka Producer Kafka Broker Greyhound wraps Kafka
- 19. Performant Event-driven Programming with Kafkaand Greyhound (Wix OSS) Wix 1,075M Kafka messages a day
- 20. @NSilnitsky Kafka Broker Topic Greyhound Consumer Kafka Consumer 0 12 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 Performant message handling A Service Message Handler
- 21. @NSilnitsky Kafka Broker Topic Greyhound Consumer Kafka Consumer SCALA ZIOFIBERS + QUEUES 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 (Thread-safe) Parallel Message Consumption Performant message handling A Service Message Handler
- 22. @NSilnitsky 80 partitions Kafka Broker Topic 01 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 0 1 2 3 4 5 Performant message handling Maximum Throughput: Messages Processing - Non blocking IO (Netty grpc client) with response latency of 100ms
- 23. @NSilnitsky 80 partitions Kafka Broker Topic 01 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 0 1 2 3 4 5 Performant message handling Maximum Throughput: 800 Messages per second Messages Processing - Non blocking IO (Netty grpc client) with response latency of 100ms
- 24. @NSilnitsky Kafka ConsumerKafka ConsumerKafka ConsumerKafka ConsumerKafka ConsumerKafka ConsumerKafka Consumer Performant message handling 80 KafkaConsumers with 80 Java threads 80 partitions Kafka Broker Topic 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 0 1 2 3 4 5 Messages Processing - Non blocking IO (Netty grpc client) with latency of 100ms Kafka Consumer
- 25. @NSilnitsky Greyhound Performant message handling or 1Greyhound Consumer with 80 fibers running on a small thread pool Kafka Consumer 80 partitions Kafka Broker Topic 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 0 1 2 3 4 5 Messages Processing - Non blocking IO (Netty grpc client) with latency of 100ms
- 26. fault-tolerant Event-driven Programming with Kafka,Greyhound & Kubernetes Wix 1,075M Kafka messages a day
- 27. @NSilnitsky Kafka Broker renew-sub-topic 0 12 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 Greyhound Consumer Kafka Consumer Fails To Read Fault-tolerant message handling
- 28. @NSilnitsky renew-sub-topic 0 1 23 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 renew-sub-topic-retry-0 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 renew-sub-topic-retry-1 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 Greyhound Consumer Kafka Consumer RETRY PRODUCER Fault-tolerant message handling Inspired by Uber RETRY! Kafka Broker
- 29. @NSilnitsky Producer Wix Payments Service Subscription renewal Job Scheduler Fault-tolerant message handling UseCase: Guarantee Completion Kafka Broker
- 30. @NSilnitsky + Retry on failure KafkaBroker Producer Consumer Wix Payments Service Subscription renewal Job Scheduler Fault-tolerant message handling Use Case: Guarantee Completion
- 31. @NSilnitsky Producer Consumer Subscription renewal Job Scheduler Fault-tolerant message handling WixPayments Service Use Case: Guarantee Completion Kafka Broker + Retry on failure
- 32. @NSilnitsky Producer Fault-tolerant message handling The ResilientProducer Kafka Broker
- 33. @NSilnitsky Save message to disk Kafka Broker Producer Fault-tolerant messagehandling The Resilient Producer: When failed to produce will save the message
- 34. @NSilnitsky Kafka Broker Producer Fault-tolerant message handling TheResilient Producer: When failed to produce will save the message and retry on failure. + Retry on failure
- 35. @NSilnitsky Kebe Fault-tolerant message handling What ifpod is killed? pod Kube Node
- 36. @NSilnitsky Kebe Fault-tolerant message handling What ifpod is killed? pod Kube Node
- 37. @NSilnitsky Kebe Fault-tolerant message handling What ifpod is killed? pod Kube Node
- 38. Fault-tolerant message handling 38 @NSilnitsky Kubernetes Node 1 DaemonSetpod pod 1 pod 2 Kubernetes Node 2 DaemonSet pod pod 1 pod 2 DaemonSet
- 39.
- 40. @NSilnitsky Kafka Broker pod Scavenger Kube Node DaemonSet Whatif pod is killed? Fault-tolerant message handling Flush out messages * small footprint
- 41. Polyglot Event-driven Programming with Kubernetes& gRPC Wix 1,075M Kafka messages a day
- 42. @NSilnitsky Polyglot message handling Motivation: codereuse Kafka Broker Greyhound Scala/Java services Greynode NodeJS services Producer Consumer Producer Consumer
- 43. @NSilnitsky Polyglot message handling Partial implementationis a problem Kafka Broker Greyhound Greynode NodeJS services Producer Consumer Producer Consumer
- 44. @NSilnitsky Polyglot message handling Experiment #1 Greyhoundon GraalVM Kafka Broker Greyhound NodeJS services GraalVM Producer Consumer
- 45. @NSilnitsky Greyhound Polyglot message handling Experiment #2 GreyhoundSidecar Kafka Broker NodeJS services Consumer gRPC Producer
- 46. @NSilnitsky Kafka Broker Greyhound Polyglot message handling Experiment#2 Greyhound Sidecar gRPC .proto Sidecar written with Scala Service written with JS & TS
- 47. @NSilnitsky Kafka Broker Greyhound Polyglot message handling Experiment#2 Greyhound Sidecar gRPC Sidecar written with Scala .proto Service written with Python gRPC Service written with JS & TS
- 48. 48 @NSilnitsky Polyglot message handling pod Kube Node Containerizedapp Volume The Sidecar resides with the app in the same pod.
- 49. @NSilnitsky Kubernetes pod with mainand sidecar containers Polyglot message handling pod Kube Node pod pod Kafka Broker @NSilnitsky
- 50. @NSilnitsky There’s a memory issue. pod KubeNode pod pod Polyglot message handling Kafka Broker @NSilnitsky
- 51.
- 52. 52 Sidecar Daemonset ✔ Simpledesign (simple state) ✘ Complex Design (Multi-tenant state) ✔ App and sidecar lifecycles are in sync ✘ Daemonset GA means downtime ✔ Failure footprint is small ✘ Failure affects more consumers ✘ Memory overhead/footprint ✔ Memory usage per Kube Node @NSilnitsky Design Dilemmas
- 53. Standalone Producer Service Mitigates DaemonsetGreyhound downtime @NSilnitsky gRPC gRPC * network hop Kafka Broker Greyhound Producer Greyhound Producer
- 54. So, We useKubernetes’ flexibility to deploy Greyhound producers and consumers in different patterns, in order to comply with different requirements. @NSilnitsky
- 55. Wix harnesses Kafka,Kubernetes and gRPC to achieve a polyglot, fault tolerant, scalable event-driven distributed system.
- 56. @NSilnitsky A Java/Scala high-levelSDK for Apache Kafka. 0.1 is out! github.com/wix/greyhound
- 57. @NSilnitsky Thank You natansil.com twitter@NSilnitskylinkedin/natansilnitsky github.com/natansil
- 58.
- 59. @NSilnitsky Q&A natansil.com twitter@NSilnitsky linkedin/natansilnitskygithub.com/natansil