Building Data Product Based on Apache Spark at Airbnb with Jingwei Lu and Liyin Tang
11 likes4,338 views
The document outlines Airbnb's approach to building data products using Spark, highlighting their data infrastructure that integrates both streaming and batch processing. It describes the architecture, data sources, computation flows, and technology used, including HBase, Kafka, and various query engines. Key features include a unified API for processing and a shared global state store, enabling efficient data handling across multiple use cases.
![Sources
Liyin Tang and Jingwei Lu
8
Streaming
source: [
{
name: source_example,
type: kafka,
config: {
topic: "example_topic",
}
}
]
Batch
source: [
{
name: source_example,
type: hive,
sql: {
select * from db.table where
ds=‘2017-06-05’;
}
}
]](https://image.slidesharecdn.com/034lutang-170614011240/85/Building-Data-Product-Based-on-Apache-Spark-at-Airbnb-with-Jingwei-Lu-and-Liyin-Tang-8-320.jpg)
![Computation
Liyin Tang and Jingwei Lu
9
Streaming/Batch
process: [{
name = process_example,
type = sql,
sql = """
SELECT listing_id, checkin_date, context.source as source
FROM source_example
WHERE user_id IS NOT NULL """
}]](https://image.slidesharecdn.com/034lutang-170614011240/85/Building-Data-Product-Based-on-Apache-Spark-at-Airbnb-with-Jingwei-Lu-and-Liyin-Tang-9-320.jpg)
![Sinks
Liyin Tang and Jingwei Lu
10
Streaming
sink: [
{
name = sink_example
input = process_example
type = hbase_update
hbase_table_name = test_table
bulk_upload = false
}
]
Batch
sink: [
{
name = sink_example
input = process_example
type = hbase_update
hbase_table_name = test_table
bulk_upload = true
}
]](https://image.slidesharecdn.com/034lutang-170614011240/85/Building-Data-Product-Based-on-Apache-Spark-at-Airbnb-with-Jingwei-Lu-and-Liyin-Tang-10-320.jpg)
![Unified Write API
Liyin Tang and Jingwei Lu
16
DataFrame
HBase
Region 1
Region 2
Region N
Re-partition
<Region 1, [RowKey, Value]>
<Region 2, [RowKey, Value]>
<Region N, [RowKey, Value]>
… …
Puts
HFile
BulkLoad](https://image.slidesharecdn.com/034lutang-170614011240/85/Building-Data-Product-Based-on-Apache-Spark-at-Airbnb-with-Jingwei-Lu-and-Liyin-Tang-16-320.jpg)
More Related Content
Similar to Building Data Product Based on Apache Spark at Airbnb with Jingwei Lu and Liyin Tang (20)
Building Data Product Based on Apache Spark at Airbnb with Jingwei Lu and Liyin Tang
- 1. Building Data Products on Spark at Airbnb LIYIN TANG & JINGWEI LU
- 2. Data Infrastructure at Airbnb
- 3. Event Logs MySQL Dumps Gold Cluster HDFS Hive Kafka Sqoop Silver Cluster Spark Cluster Spark ReAir Airflow Scheduling S3 Presto Cluster AirPal SuperSet Tableau Batch Infrastructure Yarn HDFS Hive Yarn Liyin Tang and Jingwei Lu 3
- 4. Streaming at Airbnb Liyin Tang and Jingwei Lu 4 Cluster Spark Streaming Airflow Scheduling HBase HDFS Sources Kafka S3 HDFS … Sinks Datadog Kafka Dynamo DB Elastic Search …
- 6. Batch AirStream Hive Spark SQL Lambda Architecture Liyin Tang and Jingwei Lu 6 Streaming Kafka Spark Streaming State Storage
- 7. Combine Streaming and Batch Processing
- 8. Sources Liyin Tang and Jingwei Lu 8 Streaming source: [ { name: source_example, type: kafka, config: { topic: "example_topic", } } ] Batch source: [ { name: source_example, type: hive, sql: { select * from db.table where ds=‘2017-06-05’; } } ]
- 9. Computation Liyin Tang and Jingwei Lu 9 Streaming/Batch process: [{ name = process_example, type = sql, sql = """ SELECT listing_id, checkin_date, context.source as source FROM source_example WHERE user_id IS NOT NULL """ }]
- 10. Sinks Liyin Tang and Jingwei Lu 10 Streaming sink: [ { name = sink_example input = process_example type = hbase_update hbase_table_name = test_table bulk_upload = false } ] Batch sink: [ { name = sink_example input = process_example type = hbase_update hbase_table_name = test_table bulk_upload = true } ]
- 11. Streaming Computation Flow Liyin Tang and Jingwei Lu 11 Source Process_A Process_B Process_A1 Sink_A2 Sink_B2 Batch Source Process_A Process_B Process_A1 Sink_A2 Sink_B2
- 12. Liyin Tang and Jingwei Lu Unified API through AirStream • Declarative job configuration • Streaming source vs static source • Computation operator or sink can be shared by streaming and batch job. • Computation flow is shared by streaming and batch • Single driver executes in both streaming and batch mode job 12
- 14. AirStream Shared Global State Store Liyin Tang and Jingwei Lu 14 HBase Tables Spark StreamingSpark StreamingSpark StreamingSpark Streaming Spark BatchSpark BatchSpark BatchSpark Batch
- 15. •Well integrated with Hadoop eco system •Efficient API for streaming writes and bulk uploads •Rich API for sequential scan and point-lookups • Merged view based on version 15 Why HBase
- 16. Unified Write API Liyin Tang and Jingwei Lu 16 DataFrame HBase Region 1 Region 2 Region N Re-partition <Region 1, [RowKey, Value]> <Region 2, [RowKey, Value]> <Region N, [RowKey, Value]> … … Puts HFile BulkLoad
- 17. Rich Read API Liyin Tang and Jingwei Lu 17 HBase Tables Spark Streaming/Batch Jobs Multi-Gets Prefix Scan Time Range Scan
- 18. Merged Views Liyin Tang and Jingwei Lu 18 Row Key R1 V200 TS200 R1 V150 TS150 R1 V01 TS01 … … … … Time Streaming Writes Streaming Writes Streaming Writes
- 19. Merged Views Liyin Tang and Jingwei Lu 19 Row Key R1 V200 TS200 R1 V150 TS150 R1 V01 TS01 Time Streaming Writes Streaming Writes Streaming Writes R1 V100 TS100Batch Bulk Upload
- 20. Liyin Tang and Jingwei Lu Our Foundations •Unify streaming with batch process •Shared global state store 20
- 21. Use Cases
- 22. MySQL DB Snapshot Using Binlog Replay
- 23. • Large amount of data: Multiple large mysql DBs • Realtime-ness: minutes delay/ hours delay • Transaction : Need to keep transaction across different tables • Schema change: Table schema evolves Database Snapshot 23 Move Elephant
- 24. 24 Binlog Replay on Spark 20+ hr 4+ hr AirStream Job 5 mins 15 mins 1 hr spinal tap seed
- 25. • Streaming and Batch shares Logic: Binlog file reader, DDL processor, transaction processor, DML processor. • Merged by binlog position: <filenum, offset> • Idempotent: Log can be replayed multiple times. • Schema changes: Full schema change history. 25 Log Parser Transaction Processor Change Processor Schema Processor HBASE Lambda Architecture Binlog(realtime/history) DML DDL XVID Mysql Instance
- 27. Hive Realtime Indexing Liyin Tang and Jingwei Lu 27 Elastic Search es_version = mutation id AirStream Spark Streaming Spark Batch Table A Event Event Event… … Kafka Table B Table C
- 29. Druid Ingestion Liyin Tang and Jingwei Lu 29 Druid AirStream Spark Streaming Kafka Dimension Metrics Druid Beam
- 30. Superset Powered by Druid 30
- 31. Tips
- 33. Long Window Computation 33 What if window is weeks, months, or even years?
- 34. Distinct in a Large Window 34 I don’t want approximation. What should I do?
- 35. Distinct Count Liyin Tang and Jingwei Lu 35 Row Key Listing 1 Visitor 01 TS100 Listing 1 Visitor 02 TS100 Listing 1 Visitor 04 TS98 Listing 1 Visitor 03 TS99 Prefix Scan with TimeRange Prefix Scan with TimeRange Time
- 36. Moving Average Liyin Tang and Jingwei Lu 36 Row Key Listing 1 Total Review Cnt: 100 TS100 Listing 1 Total Review Cnt: 98 TS99 Listing 1 Total Review Cnt: 01 TS01 Listing 1 Total Review Cnt: 50 TS50 Count Difference/ Time Elapsed Count Difference/ Time Elapsed Time … … … … … … Window 1 Window 2
- 38. Realtime Ingestion and Interactive Query Liyin Tang and Jingwei Lu 38 HBase AirStream Spark Streaming Kafka Query Engine Data Portal Spark SQL Hive SQL Presto SQL
- 39. Interactive Query in SqlLab 39
- 41. Thrift-> DataFrame Liyin Tang and Jingwei Lu 41 Thrift Event https://github.com/airbnb/airbnb-spark-thrift Thrift Class Thrift Object Field Meta Data Struct Type Field Value Row DataFrame
- 42. Summary
- 43. Unify Batch and Streaming Computation 43
- 44. Global State Store Using HBase 44
- 45. 45 We are hiring Happy Hour: 6pm, B Restaurant&Bar, 720 Howard St, SF