ETL in Clojure
5 likes•3,653 views
The document discusses ETL processes at Zoomdata using Clojure and Cascalog, highlighting its advantages over traditional SQL methods. It details the challenges faced with Hive and the benefits of using Clojure for interactive development and testing. Additionally, it touches on data science applications, specifically in drug persistence, and offers insights on alternatives to Cascalog.
ETL in Clojure
- 1. ETL in ClojureETL in Clojure Dmitriy Morozov / JEEConf 2015
- 2. Dmitriy MorozovDmitriy Morozov Software engineer at Functional programming junky Occasional cyclist Zoomdata.com @argc
- 3. Plan of attackPlan of attack ETL at ZoomdataETL at Zoomdata CascalogCascalog SparkSpark DemoDemo ConclusionConclusion
- 4. Is a modern BI application focused onIs a modern BI application focused on allowing everyday business users toallowing everyday business users to be able to visually interact andbe able to visually interact and explore their data and discoverexplore their data and discover insight out of that data.insight out of that data.
- 5. What we do at ZoomdataWhat we do at Zoomdata
- 6. What we do at ZoomdataWhat we do at Zoomdata
- 8. We did ETL inWe did ETL in Hive/ImpalaHive/Impala
- 9. Using SQL for ETLUsing SQL for ETL Hive is slow, and so is Hive on Tez SQL is horrible for doing anything complicated Code is hard to maintain, reuse and test Lessons learnedLessons learned
- 10. Why Clojure?Why Clojure? Functional! Runs on JVM Interactive development Zero delta between prototyp code and production code
- 11. CascalogCascalog Datalog DSL in CLojure Built on top of Hadoop and Cascading Query compiles to Hadoop MapReduce jobs Supports local execution for prototyping Great testing story
- 12. DatalogDatalog language Syntactically is a subset of Prolog It is often used as a for deductive databases. Query statements can be stated in any order Logic programming query language
- 13. DatalogDatalog
- 14. Word Count using Hadoop API
- 15. Word count in CascalogWord count in Cascalog
- 16. Cascalog Query StructureCascalog Query Structure
- 17. Cascalog / GeneratorsCascalog / Generators
- 18. Cascalog / OperationsCascalog / Operations
- 19. Cascalog / OperationsCascalog / Operations
- 20. Cascalog / JoinsCascalog / Joins
- 21. Cascalog / OperationsCascalog / Operations
- 22. Cascalog / AggregatorsCascalog / Aggregators
- 23. Cascalog / AggregatorsCascalog / Aggregators
- 24. Cascalog / TroubleshootingCascalog / Troubleshooting
- 25. Cascalog / TestingCascalog / Testing
- 26. Cascalog / TroubleshootingCascalog / Troubleshooting
- 27. Flow Visualisation /Flow Visualisation / DOTDOT
- 28. Flow Visualisation /Flow Visualisation / DrivenDriven
- 29. DEMODEMO
- 30. Cascalog DownsidesCascalog Downsides Hadoop < SparkHadoop < Spark **
- 31. Cascalog DownsidesCascalog Downsides No supportNo support for streamingfor streaming datadata
- 33. What are the alternatives?What are the alternatives? Java API forJava API for FlamboFlambo SparklingSparkling SparkSpark
- 35. Customer X wants to do DataCustomer X wants to do Data Science!Science!
- 36. Drug PersistenceDrug Persistence Determining whether a patient isDetermining whether a patient is persistent or not based on whether shepersistent or not based on whether she refilled the prescription in time.refilled the prescription in time.
- 46. Example: Drug PersistenceExample: Drug Persistence
- 48. Things to check outThings to check out How Yieldbot does Data science in Clojure Cascalog for the Impatient Streaming MapReduce in Clojure Sparkling Flambo