How Apache Kafka® Works

Oct 28, 20194 likes11,833 views

Watch this talk here: https://www.confluent.io/online-talks/how-apache-kafka-works-on-demand Pick up best practices for developing applications that use Apache Kafka, beginning with a high level code overview for a basic producer and consumer. From there we’ll cover strategies for building powerful stream processing applications, including high availability through replication, data retention policies, producer design and producer guarantees. We’ll delve into the details of delivery guarantees, including exactly-once semantics, partition strategies and consumer group rebalances. The talk will finish with a discussion of compacted topics, troubleshooting strategies and a security overview. This session is part 3 of 4 in our Fundamentals for Apache Kafka series.

1
Fundamentals for Apache Kafka®
How Apache Kafka Works
Mike Bingham, Technical Trainer
Confluent Certified Developer, Confluent Certified Administrator

2
Session Schedule
● Session 1: Benefits of Stream Processing and Apache Kafka Use Cases
● Session 2: Apache Kafka Architecture & Fundamentals Explained
● Session 3: How Apache Kafka Works
● Session 4: Integrating Apache Kafka into your Environment

3
Learning Objectives
After this module you will be able to:
● Give a high level description of the programming logic
in Kafka producer and consumer clients
● Explain how EOS works to an interested lay person
● List the means with which Kafka provides durability
and HA
● Illustrate on a high level, how you can secure your
Kafka cluster

5
Development:
A Basic
Consumer in
.NET/C#

8
Data Retention Policy
How long do I want or can I store my data?
• How long (default: one week)
• Set globally or per topic
• Business decision
• Cost factor
• Compliance factor (e.g., GDPR)

Most read

18
Troubleshooting
● Confluent Control Center
● Log files
● Special config settings
○ SSL Logging
○ Authorizer debugging

19
Security Overview
● Kafka supports Encryption in Transit
● Kafka supports Authorization and Authentication
● No Encryption at Rest out of the box
● Clients can be mixed with & without Encryption & Authentication

20
Client Side Security Features
● Encryption of Data in Transit
● Client Authentication
● Client Authorization

21
Continue your Apache Kafka Education!
● Confluent Operations for Apache Kafka
● Confluent Developer Skills for Building Apache Kafka
● Confluent Stream Processing using Apache Kafka
Streams and KSQL
● Confluent Advanced Skills for Optimizing Apache Kafka
For more details, seehttp://confluent.io/training

2222
Certifications
Confluent Certified Developer
for Apache Kafka
(aligns to Confluent Developer Skills
for Building Apache Kafka course)
Confluent Certified
Administrator for Apache
Kafka
(aligns to Confluent Operations Skills
for Apache Kafka)
What you Need to Know
○ Qualifications: 6-to-9 months hands-on
experience
○ Duration: 90 mins
○ Availability: Live, online 24/7
○ Cost: $150
○ Register online:
www.confluent.io/certification

2323
cnfl.io/slack
Stay in touch!
cnfl.io/kafka-trainingcnfl.io/download

25
Thank you for attending!
• Thank you for attending thesession!
• Feedback to: training-admin@confluent.io

14
Exactly Once Semantics (EOS)
What is it?
● Strong transactional guarantees for
Kafka
● Prevents clients from processing
duplicate messages
● Handles failures gracefully
Use Cases
● Tracking ad views
● Processing financial transactions
● Stream processing

15
Partitioning Strategies
Why partitioning?
● Consumers need to aggregate or join by some key
● Consumers need ordering guarantee
● Concentrate data for storage efficiency and/or indexing

Watch this talk here: https://www.confluent.io/online-talks/apache-kafka-architecture-and-fundamentals-explained-on-demand This session explains Apache Kafka’s internal design and architecture. Companies like LinkedIn are now sending more than 1 trillion messages per day to Apache Kafka. Learn about the underlying design in Kafka that leads to such high throughput. This talk provides a comprehensive overview of Kafka architecture and internal functions, including: -Topics, partitions and segments -The commit log and streams -Brokers and broker replication -Producer basics -Consumers, consumer groups and offsets This session is part 2 of 4 in our Fundamentals for Apache Kafka series.

Splunk 101Splunk

This document provides an overview and introduction to Splunk, including: 1. It discusses the challenges of machine data including volume, velocity, variety and variability. 2. Splunk's mission is to make machine data accessible, usable and valuable to everyone. 3. It demonstrates how Splunk can unlock critical insights from machine data sources like order processing, social media, customer service systems and more.

Reactive programming introAhmed Ehab AbdulAziz

GraphQL vs RESTGreeceJS

Security Automation & OrchestrationSplunk

Introduction to Kafka connectKnoldus Inc.

Introduction to BioinformaticsAsad Afridi

This document discusses bioinformatics, including its goals and applications. Bioinformatics is defined as applying information technology to store, organize, and analyze vast amounts of biological data, such as sequences and structures of proteins and nucleic acids. It merges biology, mathematics, statistics, computer science, and information technology. Bioinformatics helps analyze gene and protein expression, compare genomic data, and simulate DNA, RNA, and proteins. It has applications in molecular medicine, drug development, microbial genomics, crop improvement, and more. Common bioinformatics tools include BLAST for comparing biological sequences.

NGINX: Basics and Best PracticesNGINX, Inc.

On-demand recording: nginx.com/resources/webinars/nginx-basics-best-practices You’ve heard of NGINX and the benefits it can provide to your web application, but maybe you’re not sure how to get started. There are a lot of tutorials online, but they can be outdated and contradict each other, making things more challenging. In this webinar we’ll cover the basics of NGINX to help you effectively begin using it as part of your existing or new web app. This webinar covers how to: * Install NGINX and verify it's properly running * Create NGINX configurations for reverse proxy, load balancer, etc. * Improve performance using keepalives and other NGINX directives * Debug and troubleshoot using NGINX logs

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...Flink Forward

Flink Forward San Francisco 2022. Flink consumers read from Kafka as a scalable, high throughput, and low latency data source. However, there are challenges in scaling out data streams where migration and multiple Kafka clusters are required. Thus, we introduced a new Kafka source to read sharded data across multiple Kafka clusters in a way that conforms well with elastic, dynamic, and reliable infrastructure. In this presentation, we will present the source design and how the solution increases application availability while reducing maintenance toil. Furthermore, we will describe how we extended the existing KafkaSource to provide mechanisms to read logical streams located on multiple clusters, to dynamically adapt to infrastructure changes, and to perform transparent cluster migrations and failover. by Mason Chen

Introduction to Apache KafkaJeff Holoman

The document provides an introduction and overview of Apache Kafka presented by Jeff Holoman. It begins with an agenda and background on the presenter. It then covers basic Kafka concepts like topics, partitions, producers, consumers and consumer groups. It discusses efficiency and delivery guarantees. Finally, it presents some use cases for Kafka and positioning around when it may or may not be a good fit compared to other technologies.

Introduction to Kafka StreamsGuozhang Wang

Kafka Streams is a new stream processing library natively integrated with Kafka. It has a very low barrier to entry, easy operationalization, and a natural DSL for writing stream processing applications. As such it is the most convenient yet scalable option to analyze, transform, or otherwise process data that is backed by Kafka. We will provide the audience with an overview of Kafka Streams including its design and API, typical use cases, code examples, and an outlook of its upcoming roadmap. We will also compare Kafka Streams' light-weight library approach with heavier, framework-based tools such as Spark Streaming or Storm, which require you to understand and operate a whole different infrastructure for processing real-time data in Kafka.

Kafka Tutorial - Introduction to Apache Kafka (Part 1)Jean-Paul Azar

Why is Kafka so fast? Why is Kafka so popular? Why Kafka? This slide deck is a tutorial for the Kafka streaming platform. This slide deck covers Kafka Architecture with some small examples from the command line. Then we expand on this with a multi-server example to demonstrate failover of brokers as well as consumers. Then it goes through some simple Java client examples for a Kafka Producer and a Kafka Consumer. We have also expanded on the Kafka design section and added references. The tutorial covers Avro and the Schema Registry as well as advance Kafka Producers.

Introduction to Apache KafkaAIMDek Technologies

Apache Kafka is an open-source distributed event streaming platform used for building real-time data pipelines and streaming apps. It was developed by LinkedIn in 2011 to solve problems with data integration and processing. Kafka uses a publish-subscribe messaging model and is designed to be fast, scalable, and durable. It allows both streaming and storage of data and acts as a central data backbone for large organizations.

Apache Kafka - Martin PodvalMartin Podval

Apache Kafka is a distributed messaging system that allows for publishing and subscribing to streams of records, known as topics, in a fault-tolerant and scalable way. It is used for building real-time data pipelines and streaming apps. Producers write data to topics which are committed to disks across partitions and replicated for fault tolerance. Consumers read data from topics in a decoupled manner based on offsets. Kafka can process streaming data in real-time and at large volumes with low latency and high throughput.

kafkaAmikam Snir

Kafka is an open-source distributed commit log service that provides high-throughput messaging functionality. It is designed to handle large volumes of data and different use cases like online and offline processing more efficiently than alternatives like RabbitMQ. Kafka works by partitioning topics into segments spread across clusters of machines, and replicates across these partitions for fault tolerance. It can be used as a central data hub or pipeline for collecting, transforming, and streaming data between systems and applications.

Kafka presentationMohammed Fazuluddin

Kafka is an open source messaging system that can handle massive streams of data in real-time. It is fast, scalable, durable, and fault-tolerant. Kafka is commonly used for stream processing, website activity tracking, metrics collection, and log aggregation. It supports high throughput, reliable delivery, and horizontal scalability. Some examples of real-time use cases for Kafka include website monitoring, network monitoring, fraud detection, and IoT applications.

Kafka 101 and Developer Best Practicesconfluent

Apache Kafka IntroductionAmita Mirajkar

Integrating Apache Kafka Into Your Environmentconfluent

Watch this talk here: https://www.confluent.io/online-talks/integrating-apache-kafka-into-your-environment-on-demand Integrating Apache Kafka with other systems in a reliable and scalable way is a key part of an event streaming platform. This session will show you how to get streams of data into and out of Kafka with Kafka Connect and REST Proxy, maintain data formats and ensure compatibility with Schema Registry and Avro, and build real-time stream processing applications with Confluent KSQL and Kafka Streams. This session is part 4 of 4 in our Fundamentals for Apache Kafka series.

Kafka 101Clement Demonchy

ksqlDB: A Stream-Relational Database Systemconfluent

Speaker: Matthias J. Sax, Software Engineer, Confluent ksqlDB is a distributed event streaming database system that allows users to express SQL queries over relational tables and event streams. The project was released by Confluent in 2017 and is hosted on Github and developed with an open-source spirit. ksqlDB is built on top of Apache Kafka®, a distributed event streaming platform. In this talk, we discuss ksqlDB’s architecture that is influenced by Apache Kafka and its stream processing library, Kafka Streams. We explain how ksqlDB executes continuous queries while achieving fault tolerance and high vailability. Furthermore, we explore ksqlDB’s streaming SQL dialect and the different types of supported queries. Matthias J. Sax is a software engineer at Confluent working on ksqlDB. He mainly contributes to Kafka Streams, Apache Kafka's stream processing library, which serves as ksqlDB's execution engine. Furthermore, he helps evolve ksqlDB's "streaming SQL" language. In the past, Matthias also contributed to Apache Flink and Apache Storm and he is an Apache committer and PMC member. Matthias holds a Ph.D. from Humboldt University of Berlin, where he studied distributed data stream processing systems. https://db.cs.cmu.edu/events/quarantine-db-talk-2020-confluent-ksqldb-a-stream-relational-database-system/

Apache kafka performance(throughput) - without data loss and guaranteeing dat...SANG WON PARK

Apache Kafak의 성능이 특정환경(데이터 유실일 발생하지 않고, 데이터 전송순서를 반드시 보장)에서 어느정도 제공하는지 확인하기 위한 테스트 결과 공유 데이터 전송순서를 보장하기 위해서는 Apache Kafka cluster로 partition을 분산할 수 없게되므로, 성능향상을 위한 장점을 사용하지 못하게 된다. 이번 테스트에서는 Apache Kafka의 단위 성능, 즉 partition 1개에 대한 성능만을 측정하게 된다. 향후, partition을 증가할 경우 본 테스트의 1개 partition 단위 성능을 기준으로 예측이 가능할 것 같다.

Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안SANG WON PARK

Apache Kafak의 빅데이터 아키텍처에서 역할이 점차 커지고, 중요한 비중을 차지하게 되면서, 성능에 대한 고민도 늘어나고 있다. 다양한 프로젝트를 진행하면서 Apache Kafka를 모니터링 하기 위해 필요한 Metrics들을 이해하고, 이를 최적화 하기 위한 Configruation 설정을 정리해 보았다. [Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안] Apache Kafka 성능 모니터링에 필요한 metrics에 대해 이해하고, 4가지 관점(처리량, 지연, Durability, 가용성)에서 성능을 최적화 하는 방안을 정리함. Kafka를 구성하는 3개 모듈(Producer, Broker, Consumer)별로 성능 최적화를 위한 … [Apache Kafka 모니터링을 위한 Metrics 이해] Apache Kafka의 상태를 모니터링 하기 위해서는 4개(System(OS), Producer, Broker, Consumer)에서 발생하는 metrics들을 살펴봐야 한다. 이번 글에서는 JVM에서 제공하는 JMX metrics를 중심으로 producer/broker/consumer의 지표를 정리하였다. 모든 지표를 정리하진 않았고, 내 관점에서 유의미한 지표들을 중심으로 이해한 내용임 [Apache Kafka 성능 Configuration 최적화] 성능목표를 4개로 구분(Throughtput, Latency, Durability, Avalibility)하고, 각 목표에 따라 어떤 Kafka configuration의 조정을 어떻게 해야하는지 정리하였다. 튜닝한 파라미터를 적용한 후, 성능테스트를 수행하면서 추출된 Metrics를 모니터링하여 현재 업무에 최적화 되도록 최적화를 수행하는 것이 필요하다.

Stream processing using KafkaKnoldus Inc.

Benefits of Stream Processing and Apache Kafka Use Casesconfluent

Watch this talk here: https://www.confluent.io/online-talks/benefits-of-stream-processing-and-apache-kafka-use-cases-on-demand This talk explains how companies are using event-driven architecture to transform their business and how Apache Kafka serves as the foundation for streaming data applications. Learn how major players in the market are using Kafka in a wide range of use cases such as microservices, IoT and edge computing, core banking and fraud detection, cyber data collection and dissemination, ESB replacement, data pipelining, ecommerce, mainframe offloading and more. Also discussed in this talk are the differences between Apache Kafka and Confluent Platform. This session is part 1 of 4 in our Fundamentals for Apache Kafka series.

Kafka Streams: What it is, and how to use it?confluent

Kafka Streams is a client library for building distributed applications that process streaming data stored in Apache Kafka. It provides a high-level streams DSL that allows developers to express streaming applications as set of processing steps. Alternatively, developers can use the lower-level processor API to implement custom business logic. Kafka Streams handles tasks like fault-tolerance, scalability and state management. It represents data as streams for unbounded data or tables for bounded state. Common operations include transformations, aggregations, joins and table operations.

Apache Kafka Best PracticesDataWorks Summit/Hadoop Summit

Apache Kafka becoming the message bus to transfer huge volumes of data from various sources into Hadoop. It's also enabling many real-time system frameworks and use cases. Managing and building clients around Apache Kafka can be challenging. In this talk, we will go through the best practices in deploying Apache Kafka in production. How to Secure a Kafka Cluster, How to pick topic-partitions and upgrading to newer versions. Migrating to new Kafka Producer and Consumer API. Also talk about the best practices involved in running a producer/consumer. In Kafka 0.9 release, we’ve added SSL wire encryption, SASL/Kerberos for user authentication, and pluggable authorization. Now Kafka allows authentication of users, access control on who can read and write to a Kafka topic. Apache Ranger also uses pluggable authorization mechanism to centralize security for Kafka and other Hadoop ecosystem projects. We will showcase open sourced Kafka REST API and an Admin UI that will help users in creating topics, re-assign partitions, Issuing Kafka ACLs and monitoring Consumer offsets.

Apache kafkaLong Nguyen

Apache Kafka is a distributed publish-subscribe messaging system that allows for high throughput, low latency data ingestion and distribution. It provides reliability through replication, scalability by partitioning topics across brokers, and durability by persisting messages to disk. Common uses of Kafka include metrics collection, log aggregation, and stream processing using frameworks like Spark Streaming. Kafka's architecture includes brokers that store topics which are partitions distributed across a cluster, with ZooKeeper for coordination. Producers write messages to topics and consumers read messages in a subscriber model.

Apache Kafka Architectures and Fundamentalsconfluent

Apache Kafka is a community distributed event streaming platform capable of handling trillions of events a day. Initially conceived as a messaging queue, Kafka is based on an abstraction of a distributed commit log. Since being created and open-sourced by LinkedIn in 2011, Kafka has quickly evolved from messaging queue to a full-fledged event streaming platform. In this session Henrik Janzon, Solutions Engineer at Confluent, explains Apache Kafka’s internal design and architecture. by attending you will learn why companies like LinkedIn, ING, Dominos Pizza, Nordea, & Royal Bank of Canada are now sending trillions of messages per day to Apache Kafka. In addition you will come to understand the underlying design in Kafka that leads to such high throughput.

Apache KAfkaPedro Alcantara

More Related Content

What's hot (20)

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...Flink Forward

Introduction to Apache KafkaJeff Holoman

Introduction to Kafka StreamsGuozhang Wang

Kafka Tutorial - Introduction to Apache Kafka (Part 1)Jean-Paul Azar

Introduction to Apache KafkaAIMDek Technologies

Apache Kafka - Martin PodvalMartin Podval

kafkaAmikam Snir

Kafka presentationMohammed Fazuluddin

Kafka 101 and Developer Best Practicesconfluent

Apache Kafka IntroductionAmita Mirajkar

Integrating Apache Kafka Into Your Environmentconfluent

Kafka 101Clement Demonchy

ksqlDB: A Stream-Relational Database Systemconfluent

Apache kafka performance(throughput) - without data loss and guaranteeing dat...SANG WON PARK

Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안SANG WON PARK

Stream processing using KafkaKnoldus Inc.

Benefits of Stream Processing and Apache Kafka Use Casesconfluent

Kafka Streams: What it is, and how to use it?confluent

Apache Kafka Best PracticesDataWorks Summit/Hadoop Summit

Apache kafkaLong Nguyen

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...Flink Forward

Introduction to Apache KafkaJeff Holoman

Introduction to Kafka StreamsGuozhang Wang

Kafka Tutorial - Introduction to Apache Kafka (Part 1)Jean-Paul Azar

Introduction to Apache KafkaAIMDek Technologies

Apache Kafka - Martin PodvalMartin Podval

kafkaAmikam Snir

Kafka presentationMohammed Fazuluddin

Kafka 101 and Developer Best Practicesconfluent

Apache Kafka IntroductionAmita Mirajkar

Integrating Apache Kafka Into Your Environmentconfluent

Kafka 101Clement Demonchy

ksqlDB: A Stream-Relational Database Systemconfluent

Apache kafka performance(throughput) - without data loss and guaranteeing dat...SANG WON PARK

Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안SANG WON PARK

Stream processing using KafkaKnoldus Inc.

Benefits of Stream Processing and Apache Kafka Use Casesconfluent

Kafka Streams: What it is, and how to use it?confluent

Apache Kafka Best PracticesDataWorks Summit/Hadoop Summit

Apache kafkaLong Nguyen

Similar to How Apache Kafka® Works (20)

Apache Kafka Architectures and Fundamentalsconfluent

Apache KAfkaPedro Alcantara

Confluent Operations Training for Apache Kafkaconfluent

This three-day hands-on course teaches students how to build, manage, and monitor Apache Kafka clusters using industry best practices. Students will learn how Kafka works, how to set up and manage a cluster, monitor performance, recover from failures, and secure the cluster. The course includes exercises on installing, operating, and tuning a Kafka cluster. It is designed for engineers and operations staff responsible for Kafka clusters.

Confluent Partner Tech Talk with Synthesisconfluent

Confluent Developer Trainingconfluent

This three-day course teaches developers how to build applications that can publish and subscribe to data from an Apache Kafka cluster. Students will learn Kafka concepts and components, how to use Kafka and Confluent APIs, and how to develop Kafka producers, consumers, and streams applications. The hands-on course covers using Kafka tools, writing producers and consumers, ingesting data with Kafka Connect, and more. It is designed for developers who need to interact with Kafka as a data source or destination.

Westpac Bank Tech Talk 1: Dive into Apache Kafkaconfluent

Welcome to Kafka; We’re Glad You’re Here (Dave Klein, Centene) Kafka Summit 2020confluent

Apache Kafka sits at the center of a technology ecosystem that can be a bit overwhelming to someone just getting started. Fortunately, Apache Kafka is also at the heart of an amazing community that is able and eager to help! So, if you are new, or relatively new, to Apache Kafka, welcome! I’d like to introduce you to the Kafka ecosystem, and present you with a plan for how to learn and be productive with it. I’d also like to introduce you to one of the most helpful and welcoming software communities I’ve ever encountered. I’ll take you through the basics of Kafka—the brokers, the partitions, the topics—and then on and up into the different APIs and tools that are available to work with it. Consider it a Kafka 101, if you will. We’ll stay at a high level, but we’ll cover a lot of ground, with an emphasis on where and how you can dig in deeper. I am still learning myself, so I will share with you what and who have helped me in my journey, and then I’ll invite you to continue that journey with me. It’s going to be a great adventure!

Introduction to Apache Kafka and why it matters - MadridPaolo Castagna

This document provides an introduction to Apache Kafka and discusses why it is an important distributed streaming platform. It outlines how Kafka can be used to handle streaming data flows in a reliable and scalable way. It also describes the various Apache Kafka APIs including Kafka Connect, Streams API, and KSQL that allow organizations to integrate Kafka with other systems and build stream processing applications.

JHipster conf 2019 - Kafka EcosystemFlorent Ramiere

kafka for db as postgresPivotalOpenSourceHub

Confluent Enterprise Datasheetconfluent

Being Ready for Apache Kafka - Apache: Big Data Europe 2015Michael Noll

These are the slides of my Kafka talk at Apache: Big Data Europe in Budapest, Hungary. Enjoy! --Michael Apache Kafka is a high-throughput distributed messaging system that has become a mission-critical infrastructure component for modern data platforms. Kafka is used across a wide range of industries by thousands of companies such as Twitter, Netflix, Cisco, PayPal, and many others. After a brief introduction to Kafka this talk will provide an update on the growth and status of the Kafka project community. Rest of the talk will focus on walking the audience through what's required to put Kafka in production. We’ll give an overview of the current ecosystem of Kafka, including: client libraries for creating your own apps; operational tools; peripheral components required for running Kafka in production and for integration with other systems like Hadoop. We will cover the upcoming project roadmap, which adds key features to make Kafka even more convenient to use and more robust in production.

Devoxx university - Kafka de haut en basFlorent Ramiere

Kafka can be used to build real-time streaming applications and process large amounts of data. It provides a simple publish-subscribe messaging model with streams of records. Kafka Connect allows connecting Kafka with other data systems and formats through reusable connectors. Kafka Streams provides a streaming library to allow building streaming applications and processing data in Kafka streams through operators like map, filter and windowing.

What's new in Confluent 3.2 and Apache Kafka 0.10.2 confluent

With the introduction of connect and streams API in 2016, Apache Kafka is becoming the defacto solution for anyone looking to build a streaming platform. The community continues to add additional capabilities to make it the complete solution for streaming data. Join us as we review the latest additions in Apache Kafka 0.10.2. In addition, we’ll cover what’s new in Confluent Enterprise 3.2 that makes it possible for running Kafka at scale.

Monitoring Apache Kafka with Confluent Control Center confluent

Presentation by Nick Dearden, Direct, Product and Engineering, Confluent It’s 3 am. Do you know how your Kafka cluster is doing? With over 150 metrics to think about, operating a Kafka cluster can be daunting, particularly as a deployment grows. Confluent Control Center is the only complete monitoring and administration product for Apache Kafka and is designed specifically for making the Kafka operators life easier. Join Confluent as we cover how Control Center is used to simplify deployment, operability, and ensure message delivery. Watch the recording: https://www.confluent.io/online-talk/monitoring-and-alerting-apache-kafka-with-confluent-control-center/

Introduction to apache kafka, confluent and why they matterPaolo Castagna

Streaming in Practice - Putting Apache Kafka in Productionconfluent

This presentation focuses on how to integrate all these components into an enterprise environment and what things you need to consider as you move into production. We will touch on the following topics: - Patterns for integrating with existing data systems and applications - Metadata management at enterprise scale - Tradeoffs in performance, cost, availability and fault tolerance - Choosing which cross-datacenter replication patterns fit with your application - Considerations for operating Kafka-based data pipelines in production

Data In Motion Paris 2023confluent

Vous apprendrez également à : • Créer plus rapidement des produits et fonctionnalités à l’aide d’une suite complète de connecteurs et d’outils de gestion des flux, et à connecter vos environnements à des pipelines de données • Protéger vos données et charges de travail les plus critiques grâce à des garanties intégrées en matière de sécurité, de gouvernance et de résilience • Déployer Kafka à grande échelle en quelques minutes tout en réduisant les coûts et la charge opérationnelle associés

Beyond the brokers - Un tour de l'écosystème KafkaFlorent Ramiere

What is Apache Kafka and What is an Event Streaming Platform?confluent

Speaker: Gabriel Schenker, Lead Curriculum Developer, Confluent Streaming platforms have emerged as a popular, new trend, but what exactly is a streaming platform? Part messaging system, part Hadoop made fast, part fast ETL and scalable data integration. With Apache Kafka® at the core, event streaming platforms offer an entirely new perspective on managing the flow of data. This talk will explain what an event streaming platform such as Apache Kafka is and some of the use cases and design patterns around its use—including several examples of where it is solving real business problems. New developments in this area such as KSQL will also be discussed.

Apache Kafka Architectures and Fundamentalsconfluent

Apache KAfkaPedro Alcantara

Confluent Operations Training for Apache Kafkaconfluent

Confluent Partner Tech Talk with Synthesisconfluent

Confluent Developer Trainingconfluent

Westpac Bank Tech Talk 1: Dive into Apache Kafkaconfluent

Welcome to Kafka; We’re Glad You’re Here (Dave Klein, Centene) Kafka Summit 2020confluent

Introduction to Apache Kafka and why it matters - MadridPaolo Castagna

JHipster conf 2019 - Kafka EcosystemFlorent Ramiere

kafka for db as postgresPivotalOpenSourceHub

Confluent Enterprise Datasheetconfluent

Being Ready for Apache Kafka - Apache: Big Data Europe 2015Michael Noll

Devoxx university - Kafka de haut en basFlorent Ramiere

What's new in Confluent 3.2 and Apache Kafka 0.10.2 confluent

Monitoring Apache Kafka with Confluent Control Center confluent

Introduction to apache kafka, confluent and why they matterPaolo Castagna

Streaming in Practice - Putting Apache Kafka in Productionconfluent

Data In Motion Paris 2023confluent

Beyond the brokers - Un tour de l'écosystème KafkaFlorent Ramiere

What is Apache Kafka and What is an Event Streaming Platform?confluent

More from confluent (20)

Webinar Think Right - Shift Left - 19-03-2025.pptxconfluent

Migration, backup and restore made easy using Kannikaconfluent

In this presentation, you’ll discover how easily you can migrate data from any Kafka-compatible event hub to Confluent using Kannika’s intuitive self-service interface. We’ll guide you through the process, showing how the same approach can be applied to define specific event data sets and effortlessly spin up secure environments for demos, testing, or other purposes. You’ll also learn how to back up event data in just a few steps by transferring compressed data to the cloud storage location of your choice. In addition, we’ll demonstrate how to restore filtered datasets of topics, ensuring quick recovery and maintaining business continuity when needed.

Five Things You Need to Know About Data Streaming in 2025confluent

Topics that Peter covers: Tapping into the Potential of Data Products: Data drives some of today's most important business use cases. Data products enable instant access to reliable and trustworthy data by eliminating the data mess created by point-to-point connections. The Need to Tap into 'Quick Thinking': The C-level has to reorient itself so it doesn't become the bottleneck to adaptability in a data-driven world. Nine in 10 (90%) business leaders say they must now react in real-time. Learn what you can do to provide executive access to real-time data to enable 'Quick Thinking.' Rise Above Data Hurdles: Discover how to enforce governance at data production. Reestablishing trustworthiness later is almost always harder, so investing in data tools that solve business problems rather than add to them is essential. Paradigm to Shift Left: Shift Left is a new paradigm for processing and governing data at any scale, complexity, and latency. Shift Left moves the processing and governance of data closer to the source, enabling organisations to build their data once, build it right and reuse it anywhere within moments of its creation. The Need for a Strategic View: The positive correlation between data streaming maturity and significant business returns underscores the importance of a long-term, strategic view of data streaming investments. It also highlights the value of advancing beyond initial, siloed use cases to a more integrated approach that leverages data streaming across the enterprise.

Data in Motion Tour Seoul 2024 - Keynoteconfluent

Data in Motion Tour Seoul 2024 - Roadmap Democonfluent

From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...confluent

In this presentation, we’ll demonstrate how Confluent and Lightstreamer come together to tackle the last-mile challenge of extending your Kafka architecture to web and mobile platforms. Learn how to effortlessly build real-time web applications within minutes, subscribing to Kafka topics directly from your web pages, with unmatched low latency and high scalability. Explore how Confluent's leading Kafka platform and Lightstreamer's intelligent proxy work seamlessly to bridge Kafka with the internet frontier, delivering data in real-time.

Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...confluent

Data in Motion Tour 2024 Riyadh, Saudi Arabiaconfluent

Data streaming platforms are becoming increasingly important in today’s fast-paced world. From retail giants who need to monitor inventory levels to ensure stores never run out of items, to new-age, innovative banks who are building out-of-the-box banking solutions for traditional retail banks, data streaming platforms are at the centre, powering these workflows. Data streaming platforms connect all your applications, systems, and teams with a shared view of the most up-to-date, real-time data. From Gen AI, stream governance to stream processing - it’s these cutting edge developments that will be featured during the day.

Build a Real-Time Decision Support Application for Financial Market Traders w...confluent

Strumenti e Strategie di Stream Governance con Confluent Platformconfluent

Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeksconfluent

As businesses strive to stay at the forefront of innovation, the ability to quickly develop scalable Generative AI (GenAI) applications is essential. Join us for an exclusive webinar featuring MIA Platform, MongoDB, and Confluent, where you'll learn how to compose GenAI apps with real-time data integration in a fraction of the time. Discover how these three powerful platforms work together to ensure applications remain responsive, relevant, and adaptive to user preferences and contextual changes. Our experts will guide you through leveraging MIA Platform's microservices architecture and low-code development, MongoDB's flexibility, and Confluent's stream processing capabilities. Experience live demonstrations and practical insights that will transform your approach to AI-driven app development, enabling you to accelerate your development process from weeks to mere minutes. Don't miss this opportunity to keep your business at the cutting edge.

Building Real-Time Gen AI Applications with SingleStore and Confluentconfluent

Unlocking value with event-driven architecture by Confluentconfluent

Sfrutta il potere dello streaming di dati in tempo reale e dei microservizi basati su eventi per il futuro di Sky con Confluent e Kafka®. In questo tech talk esploreremo le potenzialità di Confluent e Apache Kafka® per rivoluzionare l'architettura aziendale e sbloccare nuove opportunità di business. Ne approfondiremo i concetti chiave, guidandoti nella creazione di applicazioni scalabili, resilienti e fruibili in tempo reale per lo streaming di dati. Scoprirai come costruire microservizi basati su eventi con Confluent, sfruttando i vantaggi di un'architettura moderna e reattiva. Il talk presenterà inoltre casi d'uso reali di Confluent e Kafka®, dimostrando come queste tecnologie possano ottimizzare i processi aziendali e generare valore concreto.

Il Data Streaming per un’AI real-time di nuova generazioneconfluent

Per costruire applicazioni di AI affidabili, sicure e governate occorre una base dati in tempo reale altrettanto solida. Ancor più quando ci troviamo a gestire ingenti flussi di dati in continuo movimento. Come arrivarci? Affidati a una vera piattaforma di data streaming che ti permetta di scalare e creare rapidamente applicazioni di AI in tempo reale partendo da dati affidabili. Scopri di più! Non perdere il nostro prossimo webinar durante il quale avremo l’occasione di: • Esplorare il paradigma della GenAI e di come questa nuova tecnnologia sta rimodellando il panorama aziendale, rispondendo alla necessità di offrire un contesto e soluzioni in tempo reale che soddisfino le esigenze della tua azienda. • Approfondire le incertezze del panorama dell'AI in evoluzione e l'importanza cruciale del data streaming e dell'elaborazione dati. • Vedere in dettaglio l'architettura in continua evoluzione e il ruolo chiave di Kafka e Confluent nelle applicazioni di AI. • Analizzare i vantaggi di una piattaforma di streaming dei dati come Confluent nel collegare l'eredità legacy e la GenAI, facilitando lo sviluppo e l’utilizzo di AI predittive e generative.

Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...confluent

As businesses strive to remain at the cutting edge of innovation, the demand for scalable and up-to-date conversational AI solutions has become paramount. Generative AI (GenAI) chatbots that seamlessly integrate into our daily lives and adapt to the ever-evolving nuances of human interaction are crucial. Real-time data plays a pivotal role in ensuring the responsiveness and relevance of these chatbots, empowering them to stay abreast of the latest trends, user preferences, and contextual information.

Break data silos with real-time connectivity using Confluent Cloud Connectorsconfluent

Building API data products on top of your real-time data infrastructureconfluent

This talk and live demonstration will examine how Confluent and Gravitee.io integrate to unlock value from streaming data through API products. You will learn how data owners and API providers can document, secure data products on top of Confluent brokers, including schema validation, topic routing and message filtering. You will also see how data and API consumers can discover and subscribe to products in a developer portal, as well as how they can integrate with Confluent topics through protocols like REST, Websockets, Server-sent Events and Webhooks. Whether you want to monetize your real-time data, enable new integrations with partners, or provide self-service access to topics through various protocols, this webinar is for you!

Speed Wins: From Kafka to APIs in Minutesconfluent

Evolving Data Governance for the Real-time Streaming and AI Eraconfluent

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...confluent