![• Array[Byte]
• Serialization?
• JSON?
• Protocol Buffers
• Binary - Fast
• IDL - Code Generation
• Message evolution
Messages](https://image.slidesharecdn.com/reducingmscomplexitywithkafkaandreactivestreams2016-05-11-160512022534/85/Reducing-Microservice-Complexity-with-Kafka-and-Reactive-Streams-37-320.jpg)
![Publisher[T] Subscriber[T]
onSubscribe(s: Subscription)
onNext(t: T)
onComplete()
onError(t: Throwable)
Subscription
subscribe(s: Subscriber[-T])
request(n: Long)
cancel()](https://image.slidesharecdn.com/reducingmscomplexitywithkafkaandreactivestreams2016-05-11-160512022534/85/Reducing-Microservice-Complexity-with-Kafka-and-Reactive-Streams-40-320.jpg)
![Simple Producer Example
implicit val system = ActorSystem("producer-test")
implicit val materializer = ActorMaterializer()
val producerSettings = ProducerSettings(
system, new ByteArraySerializer, new StringSerializer
).withBootstrapServers("localhost:9092")
Source(1 to 100)
.map(i => s"Message $i")
.map(m => new ProducerRecord[Array[Byte], String]("lower", m))
.to(Producer.plainSink(producerSettings)).run()](https://image.slidesharecdn.com/reducingmscomplexitywithkafkaandreactivestreams2016-05-11-160512022534/85/Reducing-Microservice-Complexity-with-Kafka-and-Reactive-Streams-51-320.jpg)
![val control = Consumer.committableSource(consumerSettings.withClientId("client1"))
.map { msg =>
val upper = msg.value.toUpperCase
Producer.Message(
new ProducerRecord[Array[Byte], String]("upper", upper),
msg.committableOffset)
}.to(Producer.commitableSink(producerSettings)).run()
control.stop()
Combined Example](https://image.slidesharecdn.com/reducingmscomplexitywithkafkaandreactivestreams2016-05-11-160512022534/85/Reducing-Microservice-Complexity-with-Kafka-and-Reactive-Streams-53-320.jpg)
More Related Content
What's hot (20)
Similar to Reducing Microservice Complexity with Kafka and Reactive Streams (20)
Recently uploaded (20)
Reducing Microservice Complexity with Kafka and Reactive Streams
- 1. Reducing Microservice Complexity with Kafka and Reactive Streams Jim Riecken
- 3. Reducing Microservice Complexity with Kafka and Reactive Streams Senior Software Developer Jim Riecken @jimriecken - [email protected]
- 4. @jimriecken
- 8. • Monolith to Microservices + Complexity • Asynchronous Messaging • Kafka • Reactive Streams + Akka Streams Agenda
- 9. • Details on how to set up a Kafka cluster • In-depth tutorial on Akka Streams Anti-Agenda
- 11. M
- 12. Efficiency Time
- 13. M S1 S2
- 14. F S1 S2 S3 S4 S5
- 15. Efficiency Time • Small • Scalable • Independent • Easy to Create • Clear ownership
- 16. Network Calls • Latency • Failure
- 17. ~99.5% Reliability 99.9% 99.9% 99.9% 99.9%
- 18. Coordination • Between services • Between teams
- 22. • Decoupling • Pub/Sub • Less coordination • Additional consumers are easy • Help scale organization Why?
- 23. • Well-defined delivery semantics • High-Throughput • Highly-Available • Durable • Scalable • Backpressure Messaging Requirements
- 24. Kafka
- 25. • Distributed, partitioned, replicated commit log service • Pub/Sub messaging functionality • Created by LinkedIn, now an Apache open-source project What is Kafka?
- 27. 0 | 1 | 2 | 3 | 4 | 5 0 | 1 | 2 | 3 | 4 | 5 | 6 0 | 1 | 2 | 3 P0 P1 P2 New Messages Appended Topic Topics + Partitions
- 28. • Send messages to topics • Responsible for choosing which partition to send to • Round-robin • Consistent hashing based on a message key Producers
- 29. • Pull messages from topics • Track their own offset in each partition Consumers
- 30. P0 P1 P2 1 2 3 4 5 6 Topic Group 1 Group 2
- 31. How does Kafka meet the requirements?
- 32. • Hundreds of MB/s of reads/writes from thousands of concurrent clients • LinkedIn (2015) • 800 billion messages per day (18 million/s peak) • 175 TB of data produced per day • > 1000 servers in 60 clusters Kafka is Fast
- 33. • Brokers • All data is persisted to disk • Partitions replicated to other nodes • Consumers • Start where they left off • Producers • Can retry - at-least-once messaging Kafka is Resilient
- 34. • Capacity can be added at runtime with zero downtime • More servers => more disk space • Topics can be larger than any single node could hold • Additional partitions can be added to add more parallelism Kafka is Scalable
- 35. • Large storage capacity • Topic retention is a Consumer SLA • Almost impossible for a fast producer to overload a slow consumer • Allows real-time as well as batch consumption Kafka Helps with Back-Pressure
- 37. • Array[Byte] • Serialization? • JSON? • Protocol Buffers • Binary - Fast • IDL - Code Generation • Message evolution Messages
- 39. • Standard for async stream processing with non-blocking back- pressure • Subscriber signals demand to publisher • Publisher sends no more than demand • Low-level • Mainly meant for library authors Reactive Streams
- 40. Publisher[T] Subscriber[T] onSubscribe(s: Subscription) onNext(t: T) onComplete() onError(t: Throwable) Subscription subscribe(s: Subscriber[-T]) request(n: Long) cancel()
- 41. Processing Data with Akka Streams
- 42. • Library on top of Akka Actors and Reactive Streams • Process sequences of elements using bounded buffer space • Strongly Typed Akka Streams
- 45. Composition
- 46. • Turning on the tap • Create actors • Open files/sockets/other resources • Materialized values • Source: Actor, Promise, Subscriber • Sink: Actor, Future, Producer Materialization
- 47. Reactive Kafka
- 48. • https://github.com/akka/reactive-kafka • Akka Streams wrapper around Kafka API • Consumer Source • Producer Sink Reactive Kafka
- 49. • Sink - sends message to Kafka topic • Flow - sends message to Kafka topic + emits result downstream • When the stream completes/fails the connection to Kafka will be automatically closed Producer
- 50. • Source - pulls messages from Kafka topics • Offset Management • Back-pressure • Materialization • Object that can stop the consumer (and complete the stream) Consumer
- 51. Simple Producer Example implicit val system = ActorSystem("producer-test") implicit val materializer = ActorMaterializer() val producerSettings = ProducerSettings( system, new ByteArraySerializer, new StringSerializer ).withBootstrapServers("localhost:9092") Source(1 to 100) .map(i => s"Message $i") .map(m => new ProducerRecord[Array[Byte], String]("lower", m)) .to(Producer.plainSink(producerSettings)).run()
- 52. Simple Consumer Example implicit val system = ActorSystem("producer-test") implicit val materializer = ActorMaterializer() val consumerSettings = ConsumerSettings( system, new ByteArrayDeserializer, new StringDeserializer, Set("lower") ).withBootstrapServers("localhost:9092").withGroupId("test-group") val control = Consumer.atMostOnceSource(consumerSettings.withClientId("client1")) .map(record => record.value) .to(Sink.foreach(v => println(v))).run() control.stop()
- 53. val control = Consumer.committableSource(consumerSettings.withClientId("client1")) .map { msg => val upper = msg.value.toUpperCase Producer.Message( new ProducerRecord[Array[Byte], String]("upper", upper), msg.committableOffset) }.to(Producer.commitableSink(producerSettings)).run() control.stop() Combined Example
- 54. Demo
- 55. Wrap-Up
- 56. • Microservices have many advantages, but can introduce failure and complexity. • Asynchronous messaging can help reduce this complexity and Kafka is a great option. • Akka Streams makes reliably processing data from Kafka with back-pressure easy Wrap-Up
- 58. Thank you! Questions? @jimriecken - [email protected] Jim Riecken