Data Engineering Basics
- 1. DATA ENGINEERING BASICS & GETTING STARTED
- 2. DEFINITIONS
- 3. Data Engineer They build and scale the platforms that enable data collection, processing and storage for data science/business analytics use. Data Scientist They use linear algebra and multivariable calculus to create new insight from existing data.
- 5. Designing, building and scaling systems that organize data for analytics
- 7. Basic architecture of ETL Scaling factor
- 10. Raw data Unprocessed data in format used on source e.g JSON No schema applied Processed data Raw data with schema applied Stored in event tables/destinations in pipelines Cooked data Processed data that has been summarized.
- 12. Volume How much data you have Velocity How fast data is getting to you Variety How different your data is Veracity How reliable your data is
- 14. BATCH PROCESSING
- 15. STREAM PROCESSING Process data on the fly, as it comes in
- 17. At Least Once At Most Once Exactly Once
- 19. MAP REDUCE Key –Value pairing. Organize the data into keys and values, Sort by the key, Combine the data with matching keys Repeat until you have the final key- value outcome.
- 21. DATA STORAGE
- 22. Relational Database (SQL) Document Store (NoSQL)
- 23. THANKYOU
- 24. REFERENCES
- 25. The Data Engineering Cookbook https://github.com/andkret/Cookbook