Myth Busters III: I’m Building a Data Lake, So I Don’t Need Data Virtualization
0 likes90 views
The document discusses the misconception that data lakes eliminate the need for data virtualization, presenting arguments and evidence supporting the value of virtualization in data lake implementations. It explores the evolution of data lakes, their integration with various data sources, and the importance of governance, accessibility, and management. Key takeaways emphasize the complexity of data lakes and the necessity of a virtual layer to enhance data access and integration.
Myth Busters III: I’m Building a Data Lake, So I Don’t Need Data Virtualization
- 1. W E B I N A R S E R I E S I’m Building a Data Lake, So I Don’t Need Data Virtualization
- 2. W E B I N A R S E R I E S I’m Building a Data Lake, So I Don’t Need Data Virtualization Paul Moxon SVP Data Architectures & Chief Evangelist Denodo 20th January 2021
- 3. Paul Moxon SVP Data Architectures & Chief Evangelist, Denodo Speakers
- 4. 1. Today’s Myth 2. Origins of the Myth 3. Just the Facts, Ma’am 4. The Proof is in the Pudding 5. Conclusions 6. Q&A 7. Next Steps Agenda
- 5. 5 Myth #3: I’m building a Data Lake. I don’t need Data Virtualization.
- 6. Origins of the Myth
- 7. 7 A Bit of History – Etymology of “Data Lake” https://jamesdixon.wordpress.com/2010/10/14/pentaho-hadoop-and-data-lakes/ (with my emphasis) Pentaho’s CTO James Dixon is credited with coining the term "data lake". He described it in his blog in 2010: "If you think of a data mart as a store of bottled water – cleansed and packaged and structured for easy consumption – the data lake is a large body of water in a more natural state. The contents of the data lake stream in from a source to fill the lake, and various users of the lake can come to examine, dive in, or take samples."
- 8. 8 Concept of a Data Lake
- 9. 9 Data Lakes Become Data Science Playgrounds… The early data scientists saw Hadoop as their personal supercomputer. Hadoop-based Data Lakes helped democratize access to state of the art supercomputing with off-the-shelf HW (and later cloud). The industry push for BI made Hadoop– based solutions the standard to bring modern analytics to any corporation.
- 10. 10 Gartner Hype Cycle – Analytics & Business Intelligence, 2020
- 11. 11 Changing the Data Lake Goals “The popular view is that a data lake will be the one destination for all the data in their enterprise and the optimal platform for all their analytics.” Nick Heudecker, Gartner
- 12. 12 …Data lakes lack semantic consistency and governed metadata. Meeting the needs of wider audiences require curated repositories with governance, semantic consistency and access controls.”
- 13. Just the Facts, Ma’am
- 14. 14 Consumers BI/Visualization “Bring Your Own Tool” reporting and visualization capabilities integrated into the Data Lake Analytics Workbench Self-service analytics workbench Data Sources Any internal or external data source that should be copied into the Data Lake Data Lakes Reference Architecture Data Sources Any internal or external data source that should be copied into the Data Lake Search and Browse Search and browse data sets, explore relationships, sample queries and export results Data Governance and Catalog Governance and Cataloging of business and technical data assets (Stewardship, Curation, Profiling, Quality) Data and Operations Management Provides a broad set of services across the ecosystem to enable security, auditing, scheduling, version management, policies, etc. Data Ingestion Physical or virtual services to ingest and integrate data rapidly across a variety of sources and data types through a common ‘ingest’ layer Data Landing Centralized location to land new data entering ecosystem separated via logical partitions, based on source, data type, characteristics, and governance requirements Raw Zone Original data received from the originating system plus tagging and typing to aid in understanding Selection & Provisioning Services to select and integrate data objects, including provisioning and prep of data ingested in Raw Zone and/or accessed via Trusted and Consumption Zones Trusted Zone Data is enhanced with business rules and identifiers added to enable integration Standardization Services to consolidate, enrich, profile and steward datasets and metadata for on-going consumption Refined Zone Data is conformed to specific uses as ‘fit for purpose’ data sets supporting common models and standards Exploratory Zone Provides a flexible and intuitive way for consumers (data stewards, data engineers, and data scientists) to research and manage data Data Delivery Services Services to connect deliver data, metadata, and insights to consumers for specific use cases Data Sources Technology Capabilities Delivery Capabilities Consumers Data Marketplace User-friendly, SSO enabled & multi tenant front-end surfacing the data lifecycle services supported by the Data Lake BI/Visualization “Bring Your Own Tool” reporting and visualization capabilities integrated into the Data Lake Analytics Workbench Self-service analytics workbench System/App/ Device Non-user consumers of data assets
- 15. 15 Real World Data Lake Example – Using AWS Trusted Zone Raw Data Zone Refined Zone Transformation Transformation Data Consumers Networking, Infrastructure & Security Data Ingestion Data Sources Data Catalog and Search – Asset Registry Workflow Orchestration, DevOps and CI/CD
- 16. 16 Real World Data Lake Example – Using Azure
- 17. 17 Data Virtualization as the Data Lake ‘Delivery Layer’ 1. As the Data Delivery Services layer 2. In the Refined Zone layer 3. As the self-service Data Catalog 4. As part of the Exploratory Zone
- 18. 18 Data Virtualization as the ‘Data Delivery Services’ Layer Data Virtualization • Delivery Services must support multiple data delivery styles and protocols • Real-time and batch • Request/response and reactive (event-driven) • Ad-hoc queries and APIs • Data Lake needs a delivery layer and Data Virtualization fits this requirement • Enables access to Data Lake and non-Data Lake sources through single, unified access layer • Data Virtualization provides data catalog for searching, finding, and understanding data available in Data Lake • Provides security and governance capabilities for Data Lake
- 19. 19 Real World Data Lake Example – Using Azure (Redux)
- 20. 20 Real World Data Lake Example – Using Azure (Revised)
- 21. The Proof is in the Pudding
- 22. 22 Customer Example - FESTO • Founded 1925 • Annual revenues (FY 2018) €3.2 B • Over 21,000 employees • Headquarters in Germany • World´s leading supplier of automation technology and technical education. BUSINESS NEED • Optimize operational efficiency, automate manufacturing processes, and deliver on- demand services to business consumers • Find smarter ways to aggregate and analyze data • An agile solution that enables the monetization of customer-facing data products • Free business users from IT reliance to become self-sufficient with reporting and analysis THE CHALLENGE: Find an agile way to integrate data from existing silos, including an analytical data lake, machine data in an IoT data lake, and traditional databases and data warehouse, that will reduce dependencies from business users on IT and provides quick turnaround and flexibility.
- 23. 23 FESTO – Digital Transformation Journey
- 24. 24 FESTO – Digital Transformation
- 25. 25 Customer - FESTO SOLUTION: • Festo developed a Big Data Analytics Framework to provide a data marketplace to better support the business • Using the Denodo Platform to integrate data from numerous on-prem and cloud systems in real-time, including Cloud- based IoT Data Lake for machine data • A unified layer for consistent data access and governance across different data silos
- 26. 26 Pilot Use Case – Energy Transparency System 2.0
- 28. 1. Large data lake projects are complex environments that will benefit from a virtual ‘consumption’ layer 2. In most cases, not all the data is going to be in the data lake, so data lake data will need integrating with non-lake data. 3. Data virtualization provides a data delivery layer that simplifies and accelerates data lake access. 4. It provides a governance, management, and security capability required for successful data lake implementation Key Takeaways
- 29. 29 Myth #3: I’m building a Data Lake. I don’t need Data Virtualization
- 30. Q&A
- 31. 31 Next Steps Access Denodo Platform 8.0 in the Cloud. Start your Free Trial today! www.denodo.com/free-trials GET STARTED TODAY
- 32. Thanks! www.denodo.com [email protected] © Copyright Denodo Technologies. All rights reserved Unless otherwise specified, no part of this PDF file may be reproduced or utilized in any for or by any means, electronic or mechanical, including photocopying and microfilm, without prior the written authorization from Denodo Technologies.