Downloaded 19 times
![API
interface RequiresDistributionAndOrdering extends Write {
Distribution requiredDistribution();
SortOrder[] requiredOrdering();
}](https://image.slidesharecdn.com/124antonokolnychyi-210613230331/85/Data-Distribution-and-Ordering-for-Efficient-Data-Source-V2-40-320.jpg)
Uploaded byDatabricks
Data Distribution and Ordering for Efficient Data Source V2
This presentation discusses data distribution and ordering in Apache Iceberg's Data Source V2. It explains that proper distribution and ordering of data is important for performance when writing and reading large datasets. The new version introduces an API for connectors to specify their required distribution and ordering, addressing issues in V1 where connectors could apply arbitrary transformations. Supported distribution options include ordered, clustered, and unspecified, and the API supports batch and streaming writes. Future work includes supporting distribution and ordering in table creation and improving partition handling. Proper data distribution and ordering is key to scaling performance in Iceberg.
More Related Content
Similar to Data Distribution and Ordering for Efficient Data Source V2
Data Distribution and Ordering for Efficient Data Source V2
- 1. Data Distribution andOrdering for Efficient Data Source V2 Anton Okolnychyi This is not a contribution Data + AI Summit 2021
- 2. Presenter • Apache IcebergPMC member • Apache Spark contributor • Data Lakes at Apple • Open source enthusiast
- 3. Agenda • Why V2? •Data distribution and ordering • Future work
- 4.
- 5. Reliability • Behavior ofDataFrameWriter is not defined - Connectors interpret SaveMode differently - SaveIntoDataSourceCommand vs InsertIntoDataSourceCommand
- 6. Reliability • Validation rulesare not consistent - PreprocessTableCreation vs PreprocessTableInsertion - No schema validation for path-based tables
- 7. Design choices • Connectorsinteract with internal APIs - SQLContext - RDD - DataFrame
- 8. Extensibility • Hard tosupport new features - No easy way to extend PrunedFilterScan - Exposing ColumnarBatch instead of Row is challenging
- 9. Features • No StructuredStreaming support • No multi-catalog support • Limited bucketed tables support
- 10.
- 11. Reliability • Predictable andreliable behavior - Clearly defined logical plans for all connectors - Consistent validation rules - Less delegation to connectors
- 12. Design choices • Properabstractions - Connectors interact only with InternalRow and ColumnarBatch - Mix-in traits for optional functionality
- 13. Features • Multi-catalog support •Structured Streaming • Vectorization • Bucketed tables (in progress)
- 14. Data distribution andordering
- 15.
- 16.
- 17.
- 18.
- 19. Why should Icare?
- 20. Impact • Writes - Controlthe number of generated files - Reduce the overall memory consumption - Reduce the actual writing time
- 21. © 2021 AppleInc. All rights reserved. Redistribution or public display not permitted without written permission from Apple. Unspecified distribution
- 22. © 2021 AppleInc. All rights reserved. Redistribution or public display not permitted without written permission from Apple. Unspecified distribution
- 23. © 2021 AppleInc. All rights reserved. Redistribution or public display not permitted without written permission from Apple. Unspecified distribution
- 24. © 2021 AppleInc. All rights reserved. Redistribution or public display not permitted without written permission from Apple. Proper distribution
- 25. © 2021 AppleInc. All rights reserved. Redistribution or public display not permitted without written permission from Apple. Proper distribution
- 26. © 2021 AppleInc. All rights reserved. Redistribution or public display not permitted without written permission from Apple. Proper distribution
- 27. © 2021 AppleInc. All rights reserved. Redistribution or public display not permitted without written permission from Apple. Unspecified ordering
- 28. © 2021 AppleInc. All rights reserved. Redistribution or public display not permitted without written permission from Apple. Unspecified ordering
- 29. © 2021 AppleInc. All rights reserved. Redistribution or public display not permitted without written permission from Apple. Proper ordering
- 30. © 2021 AppleInc. All rights reserved. Redistribution or public display not permitted without written permission from Apple. Proper ordering
- 31. Impact • Reads - Clusterdata on write for faster reads - Enable efficient data skipping
- 32. Impact • Storage footprint -Columnar encodings perform better on sorted data (e.g. dictionary encoding)
- 33. How do connectorscontrol this?
- 34. Data Source V1 •Connectors can apply arbitrary transformations on DataFrame • Built-in connectors sort data within tasks using partition columns
- 35. Data Source V2 •No way to control (SPARK-23889) • Severe performance issues unless explicitly handled by the user • Blocks migration to V2 • Fixed in upcoming Spark 3.2
- 36.
- 37. Use cases • Globalsort • Cluster + sort within tasks • Local sort within tasks • No distribution and sort
- 38. API interface WriteBuilder { Writebuild() }
- 39. API interface Write { BatchWritetoBatch(); StreamingWrite toStreaming(); }
- 40. API interface RequiresDistributionAndOrdering extendsWrite { Distribution requiredDistribution(); SortOrder[] requiredOrdering(); }
- 41. Distributions • OrderedDistribution • ClusteredDistribution •UnspecifiedDistribution
- 42. SortOrder interface SortOrder extendsExpression { Expression expression(); SortDirection direction(); NullOrdering nullOrdering(); }
- 43. Current state • Availableand fully functional in master for batch queries • Structured Streaming support is in progress (SPARK-34183)
- 44. Future work • Distributionand ordering in CREATE TABLE • Ability to control the number of shuffle partitions • Coalesce partitions during adaptive query execution
- 45.
- 46. Summary • Consider migratingto Data Source V2 • Data distribution and ordering is critical at scale
- 47. Feedback • Your feedbackis important to us • Don’t forget to review and rate sessions
- 48.
- 49. TM and ©2021 Apple Inc. All rights reserved.