SlideShare a Scribd company logo

Data Pipelines With Streamsets

Jowanza Joseph, profile picture
Jowanza Joseph

This document discusses data pipelines and StreamSets. It provides an overview of StreamSets, including that it is used to build scalable data pipelines for ingesting, transforming, and delivering data from multiple sources to various sinks. It notes StreamSets runs on Mesos and supports ingesting both real-time and batch data from various sources for analytics. The document concludes with an announcement of a demo of StreamSets.

Technology
Related topics:
Apache Spark
1 of 26
Downloaded 24 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
Data Pipelines With Streamsets Jowanza Joseph @jowanza
Agenda About me The Problem Space Streaming StreamSets Demo Questions
About Me Software Engineer at One ClickRetail Scala / Spark / Mesos / Kubernetes Author: Apache Spark Fieldbook Cyclist Husband and father
Data Pipelines With Streamsets
Retail Intelligence
Data Size
Real-Time
Operational Complexity
Data Pipelines With Streamsets
Batch Processing
What Are Data Pipelines?
Data Pipelines With Streamsets
What Problems Do They Solve?
Scalability Complexity Observability Extendability
Lambda Architecture
Kappa Architecture
Data Pipelines With Streamsets
Goals Data Provenance Guaranteed Delivery Configurable Extendable Multi-Protocol Support DAG Distribute
Data Pipelines With Streamsets
Based on Streams
Architecture
Running on Mesos
Analytics Data
Real-Time Data
Our Use Case
Demo

More Related Content

PPTX
Building Data Pipelines with Spark and StreamSets
Pat Patterson
 
PPTX
Webinar: The Modern Streaming Data Stack with Kinetica & StreamSets
Kinetica
 
PDF
Building a data pipeline to ingest data into Hadoop in minutes using Streamse...
Guglielmo Iozzia
 
PDF
Architect’s Open-Source Guide for a Data Mesh Architecture
Databricks
 
PDF
Spark Summit EU talk by Pat Patterson
Spark Summit
 
PDF
Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...
HostedbyConfluent
 
PDF
Simplify and Scale Data Engineering Pipelines with Delta Lake
Databricks
 
PPTX
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...
Data Con LA
 
Building Data Pipelines with Spark and StreamSets
Pat Patterson
 
Webinar: The Modern Streaming Data Stack with Kinetica & StreamSets
Kinetica
 
Building a data pipeline to ingest data into Hadoop in minutes using Streamse...
Guglielmo Iozzia
 
Architect’s Open-Source Guide for a Data Mesh Architecture
Databricks
 
Spark Summit EU talk by Pat Patterson
Spark Summit
 
Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...
HostedbyConfluent
 
Simplify and Scale Data Engineering Pipelines with Delta Lake
Databricks
 
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...
Data Con LA
 

What's hot (20)

PPTX
August 2016 HUG: Open Source Big Data Ingest with StreamSets Data Collector
Yahoo Developer Network
 
PDF
Virtual Flink Forward 2020: Netflix Data Mesh: Composable Data Processing - J...
Flink Forward
 
PDF
Delta Lake: Open Source Reliability w/ Apache Spark
George Chow
 
PDF
SQL Analytics Powering Telemetry Analysis at Comcast
Databricks
 
PDF
Building Sessionization Pipeline at Scale with Databricks Delta
Databricks
 
PDF
Databricks + Snowflake: Catalyzing Data and AI Initiatives
Databricks
 
PDF
Lightning-Fast Analytics for Workday Transactional Data with Pavel Hardak and...
Databricks
 
PDF
Redash: Open Source SQL Analytics on Data Lakes
Databricks
 
PDF
Data Privacy with Apache Spark: Defensive and Offensive Approaches
Databricks
 
PPTX
Big Data Day LA 2016/ NoSQL track - Analytics at the Speed of Light with Redi...
Data Con LA
 
PPTX
Insights Without Tradeoffs Using Structured Streaming keynote by Michael Armb...
Spark Summit
 
PDF
Intro to databricks delta lake
Mykola Zerniuk
 
PDF
Columbia Migrates from Legacy Data Warehouse to an Open Data Platform with De...
Databricks
 
PDF
Power Your Delta Lake with Streaming Transactional Changes
Databricks
 
PDF
Ebooks - Accelerating Time to Value of Big Data of Apache Spark | Qubole
Vasu S
 
PPTX
Spark Streaming with Azure Databricks
Dustin Vannoy
 
PDF
New Developments in the Open Source Ecosystem: Apache Spark 3.0, Delta Lake, ...
Databricks
 
PDF
Democratizing Data
Databricks
 
PDF
Migrating Your Data Platform At a High Growth Startup
Databricks
 
PPTX
Intuit Analytics Cloud 101
DataWorks Summit/Hadoop Summit
 
August 2016 HUG: Open Source Big Data Ingest with StreamSets Data Collector
Yahoo Developer Network
 
Virtual Flink Forward 2020: Netflix Data Mesh: Composable Data Processing - J...
Flink Forward
 
Delta Lake: Open Source Reliability w/ Apache Spark
George Chow
 
SQL Analytics Powering Telemetry Analysis at Comcast
Databricks
 
Building Sessionization Pipeline at Scale with Databricks Delta
Databricks
 
Databricks + Snowflake: Catalyzing Data and AI Initiatives
Databricks
 
Lightning-Fast Analytics for Workday Transactional Data with Pavel Hardak and...
Databricks
 
Redash: Open Source SQL Analytics on Data Lakes
Databricks
 
Data Privacy with Apache Spark: Defensive and Offensive Approaches
Databricks
 
Big Data Day LA 2016/ NoSQL track - Analytics at the Speed of Light with Redi...
Data Con LA
 
Insights Without Tradeoffs Using Structured Streaming keynote by Michael Armb...
Spark Summit
 
Intro to databricks delta lake
Mykola Zerniuk
 
Columbia Migrates from Legacy Data Warehouse to an Open Data Platform with De...
Databricks
 
Power Your Delta Lake with Streaming Transactional Changes
Databricks
 
Ebooks - Accelerating Time to Value of Big Data of Apache Spark | Qubole
Vasu S
 
Spark Streaming with Azure Databricks
Dustin Vannoy
 
New Developments in the Open Source Ecosystem: Apache Spark 3.0, Delta Lake, ...
Databricks
 
Democratizing Data
Databricks
 
Migrating Your Data Platform At a High Growth Startup
Databricks
 
Intuit Analytics Cloud 101
DataWorks Summit/Hadoop Summit
 
Ad

Recently uploaded (20)

PDF
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
PDF
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
PPTX
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
PDF
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
PDF
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
PPTX
ACSFv1EN-58255 AWS Academy Cloud Security Foundations.pptx
23bcla24
 
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
PPTX
Cloud computing and distributed systems.
kkkkkhan
 
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
PDF
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
PDF
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
PDF
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
PDF
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Safe Software
 
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
ACSFv1EN-58255 AWS Academy Cloud Security Foundations.pptx
23bcla24
 
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
Cloud computing and distributed systems.
kkkkkhan
 
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Safe Software
 
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 
Ad

Data Pipelines With Streamsets