mark madsen, profile picture
Uploaded bymark madsen
PDF, PPTX732 views

Architecting a Platform for Enterprise Use - Strata London 2018

The document outlines the challenges enterprises face in data and analytics, identifying root causes like outdated infrastructure, lack of agility, and inefficient processes. It emphasizes the need for a robust data architecture rather than just technological solutions and stresses the importance of beginning analytics strategies with stakeholder needs. Additionally, it discusses the perspectives of various constituencies involved in analytics, the complexities of managing data, and the necessity of a system of record for operational analytics.

Download as PDF, PPTX
1 / 118
2 / 118
3 / 118
4 / 118
5 / 118
6 / 118
7 / 118
8 / 118
9 / 118
10 / 118
11 / 118
12 / 118
13 / 118
14 / 118
15 / 118
16 / 118
17 / 118
18 / 118
19 / 118
20 / 118
21 / 118
22 / 118
23 / 118
24 / 118
25 / 118
26 / 118
27 / 118
28 / 118
29 / 118
30 / 118
31 / 118
32 / 118
33 / 118
34 / 118
35 / 118
36 / 118
37 / 118
38 / 118
39 / 118
40 / 118
41 / 118
42 / 118
43 / 118
44 / 118
45 / 118
46 / 118
47 / 118
48 / 118
49 / 118
50 / 118
51 / 118
52 / 118
53 / 118
54 / 118
55 / 118
56 / 118
57 / 118
58 / 118
59 / 118
60 / 118
61 / 118
62 / 118
63 / 118
64 / 118
65 / 118
66 / 118
67 / 118
68 / 118
69 / 118
70 / 118
71 / 118
72 / 118
73 / 118
74 / 118
75 / 118
76 / 118
77 / 118
78 / 118
79 / 118
80 / 118
81 / 118
82 / 118
83 / 118
84 / 118
85 / 118
86 / 118
87 / 118
88 / 118
89 / 118
90 / 118
91 / 118
92 / 118
93 / 118
94 / 118
95 / 118
96 / 118
97 / 118
98 / 118
99 / 118
100 / 118
101 / 118
102 / 118
103 / 118
104 / 118
105 / 118
106 / 118
107 / 118
108 / 118
109 / 118
110 / 118
111 / 118
112 / 118
113 / 118
114 / 118
115 / 118
116 / 118
117 / 118
118 / 118
Copyright Third Nature, Inc. Architecting a Data Platform For Enterprise Use May 2018 Mark Madsen Todd Walter
Copyright Third Nature, Inc. The business complaints about data and analytics You really should fire IT.
Copyright Third Nature, Inc.Copyright Third Nature, Inc. Data deficient Takes too long Costs too much Function deficient IT root causes IT proximate causes What is said in disputes Lack of agility People & vendor cost basis Client-server infrastructure Lack of “good enough” competency 1980s-era methods Inappropriate technology Data hygiene fetishes Vendor lock 1990s-era procurement IT skills deficit Dysfunctional OLTP portfolio
Copyright Third Nature, Inc.Copyright Third Nature, Inc. Data deficient Takes too long Costs too much Function deficient IT root causes IT proximate causes What is said in disputes How business responds How IT responsds Lack of agility People & vendor cost basis Client-server infrastructure Lack of “good enough” competency SaaS / Cloud BI Consultants Self-service BI Shadow BI Workarounds & spreadmarts Hidden costs Loss of control Loss of visibility Loss of knowledge Security risks Bad data risks 1980s-era methods Inappropriate technology Data hygiene fetishes Vendor lock 1990s-era procurement IT skills deficit Dysfunctional OLTP portfolio
Copyright Third Nature, Inc.Copyright Third Nature, Inc. Data deficient Takes too long Costs too much Function deficient IT root causes IT proximate causes What is said in disputes How business responds How IT responds Lack of agility People & vendor cost basis Client-server infrastructure Lack of “good enough” competency SaaS / Cloud BI Consultants Self-service BI Shadow BI Workarounds & spreadmarts Hidden costs Loss of control Loss of visibility Loss of knowledge Security risks Bad data risks 1980s-era methods Inappropriate technology Data hygiene fetishes Vendor lock 1990s-era procurement IT skills deficit Dysfunctional OLTP portfolio This is not a technology problem – it is an architecture problem.
Copyright Third Nature, Inc.Copyright Third Nature, Inc. What do we hear in the field? "I want to do analytics, beyond what I can do with BI tools" "I need an analytics strategy" "I need an analytics roadmap" "I have a specific analytics project of type..." “We have a data lake. What should we do with it?” “Technology Y replaces technology X” "I want to modernize my DW" aka "I want to speed up my DW" “Don’t need schema, curation, ETL, governance … anymore” Good judgement is the result of experience. Experience is the result of bad judgement. —Fred Brooks
Copyright Third Nature, Inc. “Build a data lake and put all the data there first”
Copyright Third Nature, Inc. DW: Centralize, that solves all problems! Creates bottlenecks Causes scale problems Availability?
Copyright Third Nature, Inc. The data lake solution: no central authority wtf, it was fully operational!
Copyright Third Nature, Inc. The data lake solution? There’s a problem: as the lake is envisioned, it is still a centralized data architecture, but this time there is no single global model. Instead it’s files and not modeled. It can be operational while under construction. It’s still a death star.
Copyright Third Nature, Inc. Eventually we run into the same problems Seriously, wtf? It was agile and operational Rising complexity and scale break centralized models
Copyright Third Nature, Inc. “Standardize on one tech, it’s simpler for everyone” It’s called stack think. Pick your vendor. Cede all architecture.
Copyright Third Nature, Inc. Agile your way into the Analytics Shantytown
Copyright Third Nature, Inc.Copyright Third Nature, Inc. “Infrastructure one PoC at a time” • Use case driven • Leverage (any) new technology • Re-use of the technology stack • Data is captive to the application or to the technology stack • Developers tend to think “function first” • Analytics people also think “function first”, but generally require integrated data
Copyright Third Nature, Inc.Copyright Third Nature, Inc. A key aspect of operational analytics is “end-to-end” E2E does not allow you to take a random walk through technologies and BYOT because the complexity of integration leads to a mountain of technical debt. • Most data applications are organized around a process, not a task or function. • Real-time analytics systems pass their data over the network, but the dependencies can cross application boundaries, as well as requiring persistence. • Developers are being forced to think of data outside the local context.
Copyright Third Nature, Inc. “Start with the platform. The rest will follow.”
Copyright Third Nature, Inc.Copyright Third Nature, Inc. Big data reality: What users seeBig data promise: What you see
Copyright Third Nature, Inc. Persisting data is not the end of the line. If you stop here you win the battle and lose the war
Copyright Third Nature, Inc. We don’t have an analytics problem, just like we didn’t have a BI problem The origin of analytics as “business intelligence” was stated well in 1958: …the ability to apprehend the interrelationships of presented facts in such a way as to guide action towards a desired goal. ~ H. P. Luhn “A Business Intelligence System”, http://altaplana.com/ibmrd0204H.pdf ” “ Our goal is analytics as a capability, not a technology
Copyright Third Nature, Inc. Three constituencies Stakeholder Analyst Builder aka the recipient aka the data scientist aka the engineer
Copyright Third Nature, Inc. Starting points for analytics strategy Many organizations choose to start with the analysts. Create a data science team. Turn them loose to find a problem. Many more start with builders: technology solutions looking for problems, e.g. 65% of the IT driven Hadoop and Spark projects over the last five years. The right place to start? Stakeholders. The goal to achieve, the problem to solve.
Copyright Third Nature, Inc. NATURE OF THE PROBLEM FROM THE STAKEHOLDER’S PERSPECTIVE Each constituency has their own set of problems to deal with
BI and Analysis: Repeatability and Discoverability Discover Explore Analyze Model Consume Promote Focus on repeatability Application cycle time 90% of users Focus on discoverability Analyst cycle time 9% analysts, 1% data scientists 80% of data use 80% of analysis
The myth that still drives analytics – analytic gold All we need is a fat pipe and pans working in parallel…
Analytic insights that result in no action are expensive trivia. It’s not the insight, but what you do with it, that matters
Applying analytics is not an analytics problem Applying analytics is not in the analyst’s control. It’s not in the engineer’s control. It’s in the control of the people involved in the process. Failures are often in execution, not in analytics development. For example, we saw unexpectedly poor performance in a number of geographies. Was it the new analytics we tried? Was it a data problem? No, it was a simple compliance problem.
NATURE OF THE PROBLEM FROM THE ANALYST’S PERSPECTIVE
The analytics process at a high level Diagram: Kate Matsudaira
The nature of analytics problems is researching the unknown rather than accessing the known. Repeat for each new problem Diagram: Kate Matsudaira
The main hurdle: just getting the data Do you know where to find it? Because it’s may not be in the data warehouse. Do you have access to it? Is access fast enough? Because DWs are for QRD, not for moving huge piles of data. And ERP systems and SaaS apps are right out. These are why developers jump on “data lake” as the solution. But that’s a small, easy part.
Do you have the right data? Many machine learning techniques require labeled (known good) training data: Supervised learning: a person has to define the correct output for some portion of the data. Data is divided into training sets used for model building and test sets for validating the results. • What is spam and what isn’t? • What does a fraudulent transaction look like 36
Do you have enough of the right data? ML needs a lot, you may be disappointed in your efforts
Where do analysts spend their time? mostly data work Slide 38 Define the business problem Translate the problem into an analytic context Select appropriate data Learn the data Create a model set Fix problems with data Transform data Build models Assess models Deploy models Assess results % of time spent 70% 30% Source: Michael Berry, Data Miners Inc.
The analyst’s workspace in BI is relatively spare
The analyst’s workspace needs to be more like a kitchen than like BI vending machines
Copyright Third Nature, Inc.Copyright Third Nature, Inc. Schema control, space limitations, resource limitations, production lockdown Analytics work is a production workload, but…
NATURE OF THE PROBLEM FROM THE BUILDER’S PERSPECTIVE
IT and Ops people want to know “what to build?” Giant data platform? Self service tools?
Analytics has different processes and workloads None of this analytics work is the same as what IT considered “analysis” to be, which is usually equated with BI or ad-hoc query. Ad-hoc analysis = Exploratory data analysis = Batch analytics = Real-time analytics A real analytics production workflow Hatch, CIKM ‘11 Slide 44
The world changes, do the models? In BI you maintain ETL and schemas, in ML you maintain models. “Model decay” happens as the assumptions around which a model is built change, e.g. spam techniques change. When you adjust the model you need to know it is normal again ▪ Better save the data used to build the model ▪ Better save the model ▪ Baseline and measurements
THREE PERSPECTIVES, ONE SOLUTION? There are requirements from all constituents. You need to put them together to have a complete picture of what’s needed.
The missing stakeholder There is another stakeholder: analytics management - the CAO, CDO, VP of analytics, aka “your boss” if you’re a data scientist. This is the perspective and problems of the person responsible for oversight of the team and efforts is across the organization and across multiple projects
Job #1 - Repeatability
Job # 2 - Operational predictability
Job #3 - Reproducibility
You need a system of record for analytics
There is an extensive list of requirements to support Primary requirements needed by constituents S D E Data catalog and ability to search it for datasets X X Self-service access to curated data X Self-service access to uncurated (unknown, new) data X X Temporary storage for working with data X Data integration, cleaning, transformation, preparation tools and environment X X Persistent storage for source data used by production models X X Persistent storage for training, testing, production data used by models X X Storage and management of models X X Deployment, monitoring, decommissioning models X Lineage, traceability of changes made for data used by models X X Lineage, traceability for model changes X X X Managing baseline data / metrics for comparing model performance X X X Managing ongoing data / metrics for tracking ongoing model performance X X X S = stakeholder, user, D = data scientist, analyst, E = engineer, developer
© Third Nature Inc. How did we get to this point with BI & analytics? There’s a difference between having no past and actively rejecting it.
54 100 1,000 10,000 1 2 3 4 5 6 7 8 9 10 Customer Data Plateaus Add users, accumulate history, add attributes, add dimensions Entirely new data set enabled by new technology at new price point – value has to exceed cost (ROI). Earliest adopters have vision ahead of ROI
55 1 10 100 1,000 10,000 100,000 1,000,000 10,000,000 1 2 3 4 5 6 7 8 9 10 11 12 13 User Adoption of New Data Customer Data Users - Data Set 1 Users - Data Set 2 Users of Integrated Data User adoption of new data sets starts over. Very small number of experts growing to wider audience, sophisticated users moving to business analysts, then business users, B2B customers and even to consumers Greater value is derived when data sets are linked – see bigger picture (eg who buys what, when). Comes after initial extraction of easy value from standalone data
56 1 10 100 1,000 10,000 100,000 1,000,000 10,000,000 100,000,000 1,000,000,000 10,000,000,000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 Data Size Plateaus over Time Lather, Rinse, Repeat. Trend line is roughly Moore’s Law. Delay or skip a generation if new data set is two orders of magnitude instead of one.
57 1 10 100 1,000 10,000 100,000 1,000,000 10,000,000 100,000,000 1,000,000,000 10,000,000,000 100,000,000,000 1,000,000,000,000 10,000,000,000,000 100,000,000,000,000 1,000,000,000,000,000 10,000,000,000,000,000 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 Customer Data Size vs. 2x per 12mo Customer Data Moore's Law - 2x/18mo 2x/12 mo But all studies show demand/creation of data increasing significantly faster than Moore’s law If we started only 1 OOM behind, we are now 5 OOM away from user demand for data Data Exhaust
58 • <Store, Item, Week> • <Store, Item, Day> – Simple aggregations • Market Basket – Affinity – Link to person, demographics, HR • Inventory by SKU by store – Temporal, time series, forecasting – Link to product, marketing, market basket Retail Plateaus 2B records total for 9 quarters 2B records per day, keep 9 quarters
59 • Web Logs and traffic – Behavioral patterns – eg path linked to person, offers, other channels – Operations of the web site • Supply chain sensors – sampled at major event – Activity Based Costing – Link to customer, product, HR, planning • Social Media – Text analysis, Filtering, languages – Link to customer, sales, other channel interactions • Supply chain sensors – sampled at minutes or seconds – Telematics – Real time, Event detection, trending, static and dynamic rules – Link to HR, thresholds, forecasts, routing, planning Retail Plateaus 30B records per day
60 FINANCE Revenue Expenses Customers CUSTOMER CARE Customer Products Orders Case History SALES Orders Customers Products MARKETING Customers Orders Campaign History OPERATIONS Inventory Returns Manufacturing Supply Chain How many batteries are in inventory by plant? What is the trend of warranty costs? How many people made a warranty claim last week? How many sales have been made quarter to date? Which customers should get a communicatio n on extended warranties? 54 32 29 49 66
61 2954 32 49 41
62 2855 Given the rise in warranty costs, isolate the problem to be a plant, then to a battery lot. Communicate with affected customers, who have not made a warranty claim on batteries, through Marketing and Customer Service channels to recall cars with affected batteries. Inventory Returns Manufacturing Supply Chain Customer Service Orders Revenue Expenses Case History Customers Products Pipeline Customers Campaign History FINANCE SALESMARKETING OPERATIONS CUSTOMER EXPERIENCE 2855
63 Manufacturing: Data Overlap Analysis New Business Improvement Opportunities through Data Leverage Sales Force Profitability Analysis 100% 80% 66% 24% 41% 66% 0% 24% 0% 24% Transportation Planning 11% 100% 87% 28% 64% 56% 22% 34% 13% 45% Production Planning 6% 57% 100% 19% 83% 35% 17% 28% 9% 40% Vendor Managed Inventory 12% 100% 100% 100% 78% 100% 61% 100% 39% 100% Global Pricing Rationalization 4% 50% 100% 17% 100% 27% 15% 29% 11% 41% Fulfillment (Perfect Order) 16% 94% 90% 48% 58% 100% 35% 53% 19% 56% Manufacturing Quality Optimization 0% 76% 88% 60% 66% 71% 100% 79% 43% 73% Preventative Maintenance Analysis 7% 75% 93% 61% 80% 68% 50% 100% 27% 82% Warranty Claims Analysis 0% 94% 100% 83% 100% 83% 94% 93% 100% 98% Quality Life Cycle Improvement 4% 57% 75% 35% 64% 41% 26% 47% 16% 100% SalesForce ProfitabilityAnalysis ProductionPlanning VendorManaged Inventory GlobalPricing Rationalization TransportationPlanning Fulfillment (PerfectOrder) ManufacturingQuality Optimization Preventative Maintenance Analysis WarrantyClaims Analysis QualityLifeCycle Improvement If Then
64
65 PRODUCT SENSOR SOCIAL MEDIA CUSTOMER CARE AUDIO RECORDINGS DIGITAL ADVERTISING CLICKSTREAM 65 41 32 19 28 How many visitors did we have to our hybrid cars microsite yesterday? What are the temperature readings for batteries by Manufacturer? What is the sentiment towards line of hybrid vehicles? Which customers likely expressed anger with customer care? Which ad creative generated the most clicks?
66 2855 High, Well Known Quality Directional Quality Unknown/Low quality Well Defined Data Model Curated JSON, XML,… Extracted Attributes Curation Required
67 2855 SENSOR DIGITAL ADVERTISING CLICKSTREAM INTERACTIONS RATINGS & REVIEWS CUSTOMER PORTAL INTERACTIONS EXTERNAL INTERACTIONS SOCIAL MEDIA IVR Routing RFID ELECTRONIC COMMERCE FINANCE SALES MARKETING Inventory Returns Manufacturing Supply Chain Customer Service Orders Revenue Expenses Case History Customers Products Pipeline Customers Campaign History OPERATIONS CUSTOMER CARE – AUDIO RECORDINGS Maps Telemetry SERVER LOGS CUSTOMER EXPERIENCE Enterprise Data
68 2855 Schema on Write Evolving Schema Schema on Read Enterprise Data Evolving Data New Data Sources Schema?
69 2855 Out of Deviation Sensor Readings Fraud Events CUSTOMER EXPERIENCE SALES MARKETING OPERATIONS Abandoned Carts Online Price Quotes Social Media Influencers FINANCE 11234 Minimum Viable Curation Minimum Viable Data Quality
70 2855 Out of Deviation Sensor Readings Fraud Events CUSTOMER EXPERIENCE SALES MARKETING OPERATIONS Abandoned Carts Online Price Quotes Social Media Influencers FINANCE 11234Sensor data formatting Unit Normalization Vehicle/version normalization Make/model/year selection Data Comprehension, Pipelines
71 2855 Enterprise Wide Use 10s to 100s of Users 10 Users Super Users Business Analysts Action list, Report, Dashboard Users Share across Enterprise Share in department Share over cube wall User Base and Sharing
72 2855 Enterprise production analytics Departmental analytics Exploratory analytics Repeatable, auditable results Evolving Consumption Ad-Hoc Query, Self Service Data Labs
73 2855 Production tools, curated data, integrated across business areas Wide variety of tools and data forms Targeted data access, many applications, response time SLAs Evolving Consumption Requirements High CPU, moderate to low IO Bulk scans, large computation, transformation, data access, specific integration Moderate CPU, High IO, Resource management
74 2855 Relational store more appropriate Hybrid File Store more appropriate Technology, Capacity 100 TB 1 PB 10 TB
75 2855 MANUFACTURING CAMPAIGN HISTORY COSTS PRODUCTS FINANCE SALES MARKETING OPERATIONS CUSTOMER EXPERIENCE Given the rise in warranty costs, isolate the problem to be a plant and the specific lot. Exclude 2/3rd of the batteries from the lot that are fine. Communicate with affected customers, who have not made a warranty claim, through Marketing and Customer Service channels to recall cars with affected batteries. CUSTOMERS CASE HISTORY SENSOR Access Wide Variety of Data to Answer a Question
Copyright Third Nature, Inc. An event contains mainly IDs…that reference other data 2010.01.10.01:41:4468 67.189.110.179 Ser40489a07-7f6e4251801a 13ae51a6d092 301212631165031 590387153892659 DNIS5555685981- UTF8&1T4ACGW_enUS386US387&fu=0&ifi=1&dtd=204&xpc=1KoLqh374s m100109_44IOJ1-Id=105543CD1A7B8322 timestamp user ID device ID game ID event details IP adr log ID Log de-referencing and enrichment is difficult since you can’t enforce integrity like you can in a DB. What’s the glue that holds it together? It’s just keys to other data. e.g. remember that device ID 0 problem?
Copyright Third Nature, Inc.© Third Nature Inc. It’s not just the log data, it’s big data + small data Device Event logs Event logs Event logs A product has a complex set of master data and metadata and this needs to be added back to the bare events.
Copyright Third Nature, Inc.© Third Nature Inc. Where does the reference data come from? The keys come from somewhere. You don’t just make up system-wide unique identifiers in your code. The lack of local lookup data at event generation leads to development practices that lead to inconsistencies. Event logs Event logs Event logs Problem we had: product identifiers that didn’t match any known products Had to fix by analyzing each log of bad-ID events. Because developers used config files they copied from the PMS and put in the code Break here and you have nothing
Copyright Third Nature, Inc. MDM again If you want to link datasets then you must manage the keys You need canonical forms for common data (in code too) user ID IP adr 2010.01.10.01:41:4468 67.189.110.179 Ser40489a07-7f6e4251801a 13ae51a6d092 301212631165031 590387153892659 DNIS5555685981- UTF8&1T4ACGW_enUS386US387&fu=0&ifi=1&dtd=204&xpc=1KoLqh374s m100109_44IOJ1-Id=105543CD1A7B8322 UID CID Email City State Country 590387153892659 10098213 barry.dylan@odin.com Paris Île-de-FranceFrance 2010.01.10.14:26:2468 67.189.110.179 10098213 5046876319474403 MOZILLA/4.0 (COMPATIBLE; TRIDENT/4.0; GTB6; .NET CLR 1.1.4322) https://www.phisherking.com/ gifts/store/LogonForm?mmc=link-src-email_m100109 http://www.google.com/search? sourceid=navclient&aq=0h&oq=Italian&ie=UTF8&pid=1T4ACGW_13ae51a6d092&q=ita lian+rose&fu=0&ifi=1&dtd=204&xpc=1KoLqh374s game ID date IP adr customer ID Event Click Cust- user
What you think you have
Copyright Third Nature, Inc. What you really have This is called “self-service.” Analysts will love it…
Copyright Third Nature, Inc. The solution to our problems isn’t technology, it’s architecture.
Copyright Third Nature, Inc. History: This is how BI was done through the 80s First there were files and reporting programs. Application files feed through a data processing pipeline to generate an output file. The file is used by a report formatter for print/screen. Files are largely single-purpose use. Every report is a program written by a developer. Data pipeline code
Copyright Third Nature, Inc. History: This is how BI ended the 80s The inevitable situation was... Data pipeline code
Copyright Third Nature, Inc. History: This is how we started the 90s Collect data in a database. Queries replaced a LOT of application code because much was just joins. We learned about “dead code” SQL SQL SQL SQL SQL
Copyright Third Nature, Inc. Pragmatism and Data Lessons learned during the ad-hoc SQL era of the DW market: When the technology is awkward for the users, the users will stop trying to use it. Even “simple” schemas weren’t enough for anyone other than analysts and their Brio… Led to the evolution of metadata-driven SQL- generating BI tools, ETL tools.
Copyright Third Nature, Inc. BI evolved to hiding query generation for end users With more regular schema models, in particular dimensional models that didn’t contain cyclic join paths, it was possible to automate SQL generation via semantic mapping layers. We developed data pipeline building tools (ETL). Query via business terms made BI usable by non-technical people. ETL SQL Life got much easier…for a while
Copyright Third Nature, Inc. Today’s model: Lake + self-service, looks familiar… The Lake with data pipelines to files or Hive tables is exactly the same pattern as the COBOL batch.. Dataflow code We already know that people don’t scale…
Copyright Third Nature, Inc. DATA ARCHITECTURE We’re so focused on the light switch that we’re not talking about the light
Copyright Third Nature, Inc. Decouple the Architecture The core of the data warehouse isn’t the database, it’s the data architecture that the database and tools implement. We need a data architecture that is not limiting: ▪ Deals with change more easily and at scale ▪ Does not enforce requirements and models up front ▪ Does not limit the format or structure of data ▪ Assumes the range of data latencies in and out, from streaming to one-time bulk ▪ Allows both reading and writing of data from outside
Copyright Third Nature, Inc. The architecture from 1988 we SHOULD HAVE BEEN USING The general concept of a separate architecture for BI has been around longer, but this paper by Devlin and Murphy is the first formal data warehouse architecture and definition published. 91 “An architecture for a business and information system”, B. A. Devlin, P. T. Murphy, IBM Systems Journal, Vol.27, No. 1, (1988) Slide 91Copyright Third Nature, Inc.
Copyright Third Nature, Inc. The goal is to decouple: solve the application and infrastructure problems separately Platform Services Data Access Deliver & Use Data storage This separates BI from other uses of data, allowing each type of use to structure the data specific to its own requirements. Data access is already somewhat separate today. Make the separation of different access methods a formal part of the architecture. Don’t force one model.
Copyright Third Nature, Inc. The goal is to decouple: solve the application and infrastructure problems separately Platform Services Data Management Process & Integrate Data storage Data management should not be subject to the constraints of a single use Data management was blended with both data acquisition and structuring data for client tools. It should be an independent function.
Copyright Third Nature, Inc. The goal is to decouple: solve the application and infrastructure problems separately Data Acquisition Collect & Store Incremental Batch One-time copy Real time Platform Services Data storage Data arrives in many latencies, from real-time to one-time. Acquisition can’t be limited by the management or consumption layers. Data acquisition should not be directly tied to the needs of consumption. It must operate independently of data use.
Copyright Third Nature, Inc. The full analytic environment subsumes all the functions of the data warehouse, and extends them Data Acquisition Collect & Store Incremental Batch One-time copy Real time Platform Services Data Management Process & Integrate Data Access Deliver & Use Data storage The platform has to do more than serve queries; it has to be read-write.
Copyright Third Nature, Inc. The data architecture must align with system components because each of them addresses different data needs Incremental Collect Batch One-time copy Real time Manage & Integrate Data Acquisition Collect & Store Data Management Process & Integrate Data Access Deliver & Use Separating concerns is part of the mechanism for change isolation
Copyright Third Nature, Inc. Zone 1: Acquistion ▪ Focus is only on collecting and tracking data ▪ Direct recording of data from sources ▪ Each dataset has as much schema as the source provides, could be explicit or implicit or none ▪ Foundation layer for all subsequent use Zone 1 Transactions Events Objects
Copyright Third Nature, Inc. Zone 2 Zone 2: Integration - Standardized ▪ Parsed and cataloged – all structure and form are made explicit so data can be accessed ▪ Namespace (e.g. keys) managed ▪ Common elements are standardized ▪ Datasets are profiled, indexed, available Zone 1 Transactions Events Objects
Copyright Third Nature, Inc. Zone 2 Zone 2 Zone 2: Integration - Enhanced ▪ Common shared KPIs, master datasets ▪ Extracted and derived data available, e.g. NEE output ▪ Data is linkable, labeled, possibly cleaned Zone 1 Transactions Events Objects
Copyright Third Nature, Inc. Layer 1 Layer 2 Layer 0 Zone 3 Zone 3: Access ▪ Data structured to suit the workload ▪ Integrated data for a specific purpose ▪ Business rules applied, e.g. filters, controls, etc. ▪ Designed, explicit structures ▪ Generally repeatable use
Copyright Third Nature, Inc. Layer 0 Layer 1 Layer 2 Zone 3 Zone 4 Zone 4: Transient ▪ Under business / developer control ▪ The data for one-off projects of unknown value or repeatability, integrated from other layers ▪ Place for ephemeral analytics output ▪ The sandbox
Copyright Third Nature, Inc. Zone 1 Raw Zone 2 Standardized Zone 2 Enhanced Zone 3 Managed Zone 4 Transient Decoupled data architecture
Copyright Third Nature, Inc. Food supply chain: an analogy for analytic data Multiple contexts of use, differing quality levels You need to keep the original because just like baking, you can’t unmake dough once it’s mixed.
Copyright Third Nature, Inc. The design focus is different in each area Ingredients Goal: available User needs a recipe in order to make use of the data. Pre-mixed Goal: discoverable and integrateable User needs a menu to choose from the data available Meals Goal: usable User needs utensils but is given a finished meal
Copyright Third Nature, Inc. The data is in zones of management, not isolating layers Raw data in an immutable storage area Standardized or enhanced data Common or usage- specific data Transient data Relax control to enable self-service while avoiding a mess. Do not constrain access to one zone or to a single tool. Focus on visibility of data use, not control of data.
Copyright Third Nature, Inc. This data architecture resolves rate of change problems Raw data in an immutable storage area Standardized or enhanced data Common or usage- specific data Transient data New data of unknown value, simple requests for new data can land here first, with little work by IT. More effort applied to management, slower. Optimized for specific uses / workloads. Generally the slowest change. Not fast vs slow: fast vs right Not flexibility vs control: flexibility vs repeatability Agile for structure change vs agile for questions / use
Copyright Third Nature, Inc. SQLServer RDBMS Example: data environment, mid-size retailer Replication, batch ETL to collect data Web Application Standard retail / wholesale data warehouse model. Nightly batch model from stage, contains ~ 1 TB of data DW New requirement: Load all the online marketing and web activity for use in analytic models (not simple web analytics) Will add ~10 TB of data, continuous or hourly loads
Copyright Third Nature, Inc. SQLServer RDBMS Example: data environment, mid-size retailer Replication, batch ETL to collect data Hadoop Web Application Persistent store, data warehouse on same DB in different schemas DW Log file fetch & load for clickstream, summaries sent to reporting env Discovery work usually done in Hadoop, sometimes in database.
Copyright Third Nature, Inc. SQLServer This data architecture uses the 3 zone pattern Hadoop Web Application RDBMS DW Raw data in an immutable storage area Standardized or enhanced data Common or usage- specific data Transient data All reporting data and analytics output is sent to the DW where it is can be accessed. Cleaned, summarized or derived data is stored in DB Raw data is stored in two places: relational and small sets in DB, log collection in Hadoop
Copyright Third Nature, Inc. The concept of a zone is not a physical system. It’s data architecture Apps DW The biggest decision is to separate all data collection from the data integration from consumption. Physical system/technology overlays are separate, depend on the specific use cases and needs of the organization. Dist FS RDBMSs Decompress & process device logs RDBMS RDBMS RDBMS Discovery platform
Copyright Third Nature, Inc. What about the technology? Do I need an <X>?
112 Blended Architectures Are a Requirement, Not an Option Data Warehouse + Data Lake On Premise + Cloud RDBMS + S3 + HDFS Commercial + Open Source
Copyright Third Nature, Inc. Bricks are not buildings We don’t think this is equivalent to this Architecture is not technology. It’s not a product you can buy.
Copyright Third Nature, Inc.Copyright Third Nature, Inc. New tools and tool architectures are required One tool for all jobs doesn’t work any more* * it never did, we just had fewer jobs
Copyright Third Nature, Inc. Beware overengineering: do you really need special new technology infrastructure? Numberofpeople The distribution of data size is about normal, yet these guys set the tone of the market today. Bigness of data
Copyright Third Nature, Inc. Analytics: This is really raw data under storageNumberofjobs Microsoft study of 174,000 analytic jobs in their petabyte scale cluster: median size = ??? Bigness of data Copyright Third Nature, Inc.
Copyright Third Nature, Inc. Working data for analytics most often not bigNumberofjobs 14 GB Smallness of data Copyright Third Nature, Inc. The usable size distribution
Copyright Third Nature, Inc. TANSTAAFL When replacing the old with the new (or ignoring the new over the old) you always make tradeoffs, and usually you won’t see them for a long time. Technologies are not perfect replacements for one another. Often not better, only different.
© Third Nature Inc.© Third Nature Inc. 1.It’s not about storing data, it’s about using it 2.Use drives architecture. Understand the uses, what you are designing for, to drive decisions. 3.Put data at the center, not technology. Don’t let the tech define what you can do or how you do it. 4.The death star is not the answer. The data model is not a flat earth. You are not building a monolith. 5.Know your history. Avoiding wheel reinvention saves time, money, careers. Conclusion
© Third Nature Inc. Manage your data (or it will manage you) Data management is where developers are weakest. Modern engineering practices are where data management is weakest. You need to bridge these groups and practices in the organization if you want to do meaningful work with event stream data.
© Third Nature Inc. “Now is not the end. It is not even the beginning of the end. But it is, perhaps, the end of the beginning.” Winston Churchill
Mark Madsen is the global head of architecture at Teradata, Prior to that he was president of Third Nature, a research and consulting firm focused on analytics, data integration and data management. Mark is an award- winning author, architect and CTO whose work has been featured in numerous industry publications. Over the past ten years Mark received awards for his work from the American Productivity & Quality Center, TDWI, and the Smithsonian Institute. He is an international speaker, chairs several conferences, and is on the O’Reilly Strata program committee. For more information or to contact Mark, follow @markmadsen on Twitter or visit http://ThirdNature.net About the Presenter
Todd Walter Chief Technologist - Teradata • Chief Technologist for Teradata • A pragmatic visionary, Walter helps business leaders, analysts and technologists better understand all of the astonishing possibilities of big data and analytics • Works with organizations of all sizes and levels of experience at the leading edge of adopting big data, data warehouse and analytics technologies • With Teradata for more than 30 years and served for more than 10 years as CTO of Teradata Labs, contributing significantly to Teradata’s unique design features and functionality • Holds more than a dozen Teradata patents and is a Teradata Fellow in recognition of his long record of technical innovation and contribution to the company

More Related Content

PDF
The Black Box: Interpretability, Reproducibility, and Data Management
bymark madsen
 
PDF
Operationalizing Machine Learning in the Enterprise
bymark madsen
 
PDF
Pay no attention to the man behind the curtain - the unseen work behind data ...
bymark madsen
 
PDF
Solve User Problems: Data Architecture for Humans
bymark madsen
 
PDF
Architecting a Data Platform For Enterprise Use (Strata NY 2018)
bymark madsen
 
PDF
How to understand trends in the data & software market
bymark madsen
 
PDF
Building a Data Platform Strata SF 2019
bymark madsen
 
PDF
Data Architecture: OMG It’s Made of People
bymark madsen
 
The Black Box: Interpretability, Reproducibility, and Data Management
bymark madsen
 
Operationalizing Machine Learning in the Enterprise
bymark madsen
 
Pay no attention to the man behind the curtain - the unseen work behind data ...
bymark madsen
 
Solve User Problems: Data Architecture for Humans
bymark madsen
 
Architecting a Data Platform For Enterprise Use (Strata NY 2018)
bymark madsen
 
How to understand trends in the data & software market
bymark madsen
 
Building a Data Platform Strata SF 2019
bymark madsen
 
Data Architecture: OMG It’s Made of People
bymark madsen
 

What's hot

PDF
Assumptions about Data and Analysis: Briefing room webcast slides
bymark madsen
 
PPTX
Strata Data Conference 2019 : Scaling Visualization for Big Data in the Cloud
byJaipaul Agonus
 
PDF
Everything Has Changed Except Us: Modernizing the Data Warehouse
bymark madsen
 
PDF
Analytics 3.0 Measurable business impact from analytics & big data
byMicrosoft
 
PDF
How to Build Data Science Teams
byGanes Kesari
 
PDF
Building Data Science Teams
byEMC
 
PPTX
Managing Data Science | Lessons from the Field
byDomino Data Lab
 
PPTX
Machine Learning in Big Data
byDataWorks Summit/Hadoop Summit
 
PDF
Disruptive Innovation: how do you use these theories to manage your IT?
bymark madsen
 
PDF
Intro to Data Science for Non-Data Scientists
bySri Ambati
 
PDF
Embracing data science
byVipul Kalamkar
 
PDF
Wake up and smell the data
bymark madsen
 
PDF
Bi isn't big data and big data isn't BI (updated)
bymark madsen
 
PDF
Big dataplatform operationalstrategy
byHimanshu Bari
 
PDF
O'Reilly ebook: Machine Learning at Enterprise Scale | Qubole
byVasu S
 
PDF
Briefing room: An alternative for streaming data collection
bymark madsen
 
PPTX
Building Data Science Teams: A Moneyball Approach
byjoshwills
 
PPTX
Keynote Dubai
byNeil Raden
 
PDF
Everything has changed except us
bymark madsen
 
PDF
Analytics3.0 e book
byJyrki Määttä
 
Assumptions about Data and Analysis: Briefing room webcast slides
bymark madsen
 
Strata Data Conference 2019 : Scaling Visualization for Big Data in the Cloud
byJaipaul Agonus
 
Everything Has Changed Except Us: Modernizing the Data Warehouse
bymark madsen
 
Analytics 3.0 Measurable business impact from analytics & big data
byMicrosoft
 
How to Build Data Science Teams
byGanes Kesari
 
Building Data Science Teams
byEMC
 
Managing Data Science | Lessons from the Field
byDomino Data Lab
 
Machine Learning in Big Data
byDataWorks Summit/Hadoop Summit
 
Disruptive Innovation: how do you use these theories to manage your IT?
bymark madsen
 
Intro to Data Science for Non-Data Scientists
bySri Ambati
 
Embracing data science
byVipul Kalamkar
 
Wake up and smell the data
bymark madsen
 
Bi isn't big data and big data isn't BI (updated)
bymark madsen
 
Big dataplatform operationalstrategy
byHimanshu Bari
 
O'Reilly ebook: Machine Learning at Enterprise Scale | Qubole
byVasu S
 
Briefing room: An alternative for streaming data collection
bymark madsen
 
Building Data Science Teams: A Moneyball Approach
byjoshwills
 
Keynote Dubai
byNeil Raden
 
Everything has changed except us
bymark madsen
 
Analytics3.0 e book
byJyrki Määttä
 

Similar to Architecting a Platform for Enterprise Use - Strata London 2018

PDF
Harness the power of data
byHarsha MV
 
PDF
How to make your data scientists happy
byHussain Sultan
 
PDF
Where's the Brain in Analytics
byDenodo
 
PPTX
Decentralizing Analytics - A Strategy for Organizing Effective Analytics Teams
byKen Raetz
 
PPTX
ANALYTICS SOLUTIONS
byGrazitti Interactive
 
PDF
ADV Slides: What the Aspiring or New Data Scientist Needs to Know About the E...
byDATAVERSITY
 
DOCX
How Analytics Has Changed in the Last 10 Years (and How It’s Staye.docx
bypooleavelina
 
PDF
Trends in Enterprise Advanced Analytics
byDATAVERSITY
 
PDF
What makes an effective data team?
bySnowplow Analytics
 
PDF
Building the Enterprise Data Lake: A look at architecture
bymark madsen
 
PPTX
Big Data & Business Analytics: Understanding the Marketspace
byBala Iyer
 
PPTX
Creating an Enterprise AI Strategy
byAtScale
 
PPTX
Challenges in adapting predictive analytics
byPrasad Narasimhan
 
PDF
Data Analytics Initiatives Managing Analytics For Success Ondej Bothe
bymauderotto6n
 
PDF
Tips --Break Down the Barriers to Better Data Analytics
byAbhishek Sood
 
PPTX
Making advanced analytics work for you
byGirish Nookella
 
PDF
Data science and its potential to change business as we know it. The Roadmap ...
byInnoTech
 
PDF
Understanding Big Data Analytics - solutions for growing businesses - Rafał M...
byGetInData
 
PDF
Gse uk-cedrinemadera-2018-shared
bycedrinemadera
 
PPTX
MODULE 1_Introduction to Data analytics and life cycle..pptx
bynikshaikh786
 
Harness the power of data
byHarsha MV
 
How to make your data scientists happy
byHussain Sultan
 
Where's the Brain in Analytics
byDenodo
 
Decentralizing Analytics - A Strategy for Organizing Effective Analytics Teams
byKen Raetz
 
ANALYTICS SOLUTIONS
byGrazitti Interactive
 
ADV Slides: What the Aspiring or New Data Scientist Needs to Know About the E...
byDATAVERSITY
 
How Analytics Has Changed in the Last 10 Years (and How It’s Staye.docx
bypooleavelina
 
Trends in Enterprise Advanced Analytics
byDATAVERSITY
 
What makes an effective data team?
bySnowplow Analytics
 
Building the Enterprise Data Lake: A look at architecture
bymark madsen
 
Big Data & Business Analytics: Understanding the Marketspace
byBala Iyer
 
Creating an Enterprise AI Strategy
byAtScale
 
Challenges in adapting predictive analytics
byPrasad Narasimhan
 
Data Analytics Initiatives Managing Analytics For Success Ondej Bothe
bymauderotto6n
 
Tips --Break Down the Barriers to Better Data Analytics
byAbhishek Sood
 
Making advanced analytics work for you
byGirish Nookella
 
Data science and its potential to change business as we know it. The Roadmap ...
byInnoTech
 
Understanding Big Data Analytics - solutions for growing businesses - Rafał M...
byGetInData
 
Gse uk-cedrinemadera-2018-shared
bycedrinemadera
 
MODULE 1_Introduction to Data analytics and life cycle..pptx
bynikshaikh786
 

More from mark madsen

PDF
A Brief Tour through the Geology & Endemic Botany of the Klamath-Siskiyou Range
bymark madsen
 
PDF
A Pragmatic Approach to Analyzing Customers
bymark madsen
 
PDF
Briefing Room analyst comments - streaming analytics
bymark madsen
 
PDF
On the edge: analytics for the modern enterprise (analyst comments)
bymark madsen
 
PDF
Crossing the chasm with a high performance dynamically scalable open source p...
bymark madsen
 
PDF
Don't let data get in the way of a good story
bymark madsen
 
PDF
Big Data and Bad Analogies
bymark madsen
 
PDF
Don't follow the followers
bymark madsen
 
PDF
Exploring cloud for data warehousing
bymark madsen
 
PDF
Open Data: Free Data Isn't the Same as Freeing Data
bymark madsen
 
PDF
Exploring cloud for data warehousing
bymark madsen
 
PDF
Big Data Wonderland: Two Views on the Big Data Revolution
bymark madsen
 
PDF
Using Data Virtualization to Integrate With Big Data
bymark madsen
 
PDF
One Size Doesn't Fit All: The New Database Revolution
bymark madsen
 
A Brief Tour through the Geology & Endemic Botany of the Klamath-Siskiyou Range
bymark madsen
 
A Pragmatic Approach to Analyzing Customers
bymark madsen
 
Briefing Room analyst comments - streaming analytics
bymark madsen
 
On the edge: analytics for the modern enterprise (analyst comments)
bymark madsen
 
Crossing the chasm with a high performance dynamically scalable open source p...
bymark madsen
 
Don't let data get in the way of a good story
bymark madsen
 
Big Data and Bad Analogies
bymark madsen
 
Don't follow the followers
bymark madsen
 
Exploring cloud for data warehousing
bymark madsen
 
Open Data: Free Data Isn't the Same as Freeing Data
bymark madsen
 
Exploring cloud for data warehousing
bymark madsen
 
Big Data Wonderland: Two Views on the Big Data Revolution
bymark madsen
 
Using Data Virtualization to Integrate With Big Data
bymark madsen
 
One Size Doesn't Fit All: The New Database Revolution
bymark madsen
 

Recently uploaded

PPTX
A Comparative Study of Random Forest and XGBoost for Detecting Credit Card Fr...
byMohamed Riham
 
PPTX
novel_healthcare_system_innovations.pptx
byStevenHeizmannCPA
 
PDF
SQL Window Functions In Five Easy Steps
byDave Stokes
 
PDF
Black Friday Online Buying Behavior – Insights from BusinessLeague.com
byDorin Boerescu
 
PPTX
Data analytics applications by NVIDIA & Accenture
byworkwithposhita
 
PPTX
在读证明美国加利福尼亚州立大学圣伯纳迪诺分校毕业证{CSUSB offer CSUSB成绩单}
by24小时在线咨询帝国理工学院毕业证文凭【Q微:1954292140】
 
DOCX
How to Buy Verified Payoneer Account in 5 Easy Steps.docx
byusaallshop
 
PDF
Disaggregating Data for High Impact Development Policies -workshop- 29-oct-20...
byStatsCommunications
 
PDF
diabetes-mellitus.pdfgyguyuyiukhijhfjyfuygiug
byr99222608
 
DOCX
How to Buy Verified Paxful Accounts Step-By STEP.docx
byaslehobe7
 
PDF
Zalando-SE_Q1-2025_EC-presentation_CORPORATE DECK
bydeeksha2794bhardwaj
 
PPTX
lecture 4 gggggggggggggggggggggg (1).pptx
byAbdelhameedRabieaaKh
 
DOCX
Where do I get Buy a Naver account in 2026.docx
bygolam mosabber
 
PPTX
Strategic Roadmap Capturing the African Ride-Hailing Market(Festride).pptx
byPriscillaEdinamAttah
 
PDF
Master Business Accounting Fundamentals with Ankush Mukundan
byAnkush Mukundan
 
PDF
Disaggregating Data for High Impact Development Policies -workshop- 29-oct-20...
byStatsCommunications
 
PDF
Disaggregating Data for High Impact Development Policies -workshop- 29-oct-20...
byStatsCommunications
 
DOCX
Morse Code Translator_ The Timeless Language of Dots and Dashes (2).docx
byThelaptopsguide.com
 
PPTX
Week 1 Lecture Materials.pptx Petroleum Python
bykalaiselvansolairaj7
 
PPTX
BDA_UNIT_II_BDA_UNIT_II_BDA_UNIT_II_BDA_UNIT_II_
bySrikanthYadav578790
 
A Comparative Study of Random Forest and XGBoost for Detecting Credit Card Fr...
byMohamed Riham
 
novel_healthcare_system_innovations.pptx
byStevenHeizmannCPA
 
SQL Window Functions In Five Easy Steps
byDave Stokes
 
Black Friday Online Buying Behavior – Insights from BusinessLeague.com
byDorin Boerescu
 
Data analytics applications by NVIDIA & Accenture
byworkwithposhita
 
在读证明美国加利福尼亚州立大学圣伯纳迪诺分校毕业证{CSUSB offer CSUSB成绩单}
by24小时在线咨询帝国理工学院毕业证文凭【Q微:1954292140】
 
How to Buy Verified Payoneer Account in 5 Easy Steps.docx
byusaallshop
 
Disaggregating Data for High Impact Development Policies -workshop- 29-oct-20...
byStatsCommunications
 
diabetes-mellitus.pdfgyguyuyiukhijhfjyfuygiug
byr99222608
 
How to Buy Verified Paxful Accounts Step-By STEP.docx
byaslehobe7
 
Zalando-SE_Q1-2025_EC-presentation_CORPORATE DECK
bydeeksha2794bhardwaj
 
lecture 4 gggggggggggggggggggggg (1).pptx
byAbdelhameedRabieaaKh
 
Where do I get Buy a Naver account in 2026.docx
bygolam mosabber
 
Strategic Roadmap Capturing the African Ride-Hailing Market(Festride).pptx
byPriscillaEdinamAttah
 
Master Business Accounting Fundamentals with Ankush Mukundan
byAnkush Mukundan
 
Disaggregating Data for High Impact Development Policies -workshop- 29-oct-20...
byStatsCommunications
 
Disaggregating Data for High Impact Development Policies -workshop- 29-oct-20...
byStatsCommunications
 
Morse Code Translator_ The Timeless Language of Dots and Dashes (2).docx
byThelaptopsguide.com
 
Week 1 Lecture Materials.pptx Petroleum Python
bykalaiselvansolairaj7
 
BDA_UNIT_II_BDA_UNIT_II_BDA_UNIT_II_BDA_UNIT_II_
bySrikanthYadav578790
 

Architecting a Platform for Enterprise Use - Strata London 2018

  • 1.
    Copyright Third Nature,Inc. Architecting a Data Platform For Enterprise Use May 2018 Mark Madsen Todd Walter
  • 2.
    Copyright Third Nature,Inc. The business complaints about data and analytics You really should fire IT.
  • 3.
    Copyright Third Nature,Inc.Copyright Third Nature, Inc. Data deficient Takes too long Costs too much Function deficient IT root causes IT proximate causes What is said in disputes Lack of agility People & vendor cost basis Client-server infrastructure Lack of “good enough” competency 1980s-era methods Inappropriate technology Data hygiene fetishes Vendor lock 1990s-era procurement IT skills deficit Dysfunctional OLTP portfolio
  • 4.
    Copyright Third Nature,Inc.Copyright Third Nature, Inc. Data deficient Takes too long Costs too much Function deficient IT root causes IT proximate causes What is said in disputes How business responds How IT responsds Lack of agility People & vendor cost basis Client-server infrastructure Lack of “good enough” competency SaaS / Cloud BI Consultants Self-service BI Shadow BI Workarounds & spreadmarts Hidden costs Loss of control Loss of visibility Loss of knowledge Security risks Bad data risks 1980s-era methods Inappropriate technology Data hygiene fetishes Vendor lock 1990s-era procurement IT skills deficit Dysfunctional OLTP portfolio
  • 5.
    Copyright Third Nature,Inc.Copyright Third Nature, Inc. Data deficient Takes too long Costs too much Function deficient IT root causes IT proximate causes What is said in disputes How business responds How IT responds Lack of agility People & vendor cost basis Client-server infrastructure Lack of “good enough” competency SaaS / Cloud BI Consultants Self-service BI Shadow BI Workarounds & spreadmarts Hidden costs Loss of control Loss of visibility Loss of knowledge Security risks Bad data risks 1980s-era methods Inappropriate technology Data hygiene fetishes Vendor lock 1990s-era procurement IT skills deficit Dysfunctional OLTP portfolio This is not a technology problem – it is an architecture problem.
  • 6.
    Copyright Third Nature,Inc.Copyright Third Nature, Inc. What do we hear in the field? "I want to do analytics, beyond what I can do with BI tools" "I need an analytics strategy" "I need an analytics roadmap" "I have a specific analytics project of type..." “We have a data lake. What should we do with it?” “Technology Y replaces technology X” "I want to modernize my DW" aka "I want to speed up my DW" “Don’t need schema, curation, ETL, governance … anymore” Good judgement is the result of experience. Experience is the result of bad judgement. —Fred Brooks
  • 7.
    Copyright Third Nature,Inc. “Build a data lake and put all the data there first”
  • 8.
    Copyright Third Nature,Inc. DW: Centralize, that solves all problems! Creates bottlenecks Causes scale problems Availability?
  • 9.
    Copyright Third Nature,Inc. The data lake solution: no central authority wtf, it was fully operational!
  • 10.
    Copyright Third Nature,Inc. The data lake solution? There’s a problem: as the lake is envisioned, it is still a centralized data architecture, but this time there is no single global model. Instead it’s files and not modeled. It can be operational while under construction. It’s still a death star.
  • 11.
    Copyright Third Nature,Inc. Eventually we run into the same problems Seriously, wtf? It was agile and operational Rising complexity and scale break centralized models
  • 12.
    Copyright Third Nature,Inc. “Standardize on one tech, it’s simpler for everyone” It’s called stack think. Pick your vendor. Cede all architecture.
  • 13.
    Copyright Third Nature,Inc. Agile your way into the Analytics Shantytown
  • 14.
    Copyright Third Nature,Inc.Copyright Third Nature, Inc. “Infrastructure one PoC at a time” • Use case driven • Leverage (any) new technology • Re-use of the technology stack • Data is captive to the application or to the technology stack • Developers tend to think “function first” • Analytics people also think “function first”, but generally require integrated data
  • 15.
    Copyright Third Nature,Inc.Copyright Third Nature, Inc. A key aspect of operational analytics is “end-to-end” E2E does not allow you to take a random walk through technologies and BYOT because the complexity of integration leads to a mountain of technical debt. • Most data applications are organized around a process, not a task or function. • Real-time analytics systems pass their data over the network, but the dependencies can cross application boundaries, as well as requiring persistence. • Developers are being forced to think of data outside the local context.
  • 16.
    Copyright Third Nature,Inc. “Start with the platform. The rest will follow.”
  • 17.
    Copyright Third Nature,Inc.Copyright Third Nature, Inc. Big data reality: What users seeBig data promise: What you see
  • 18.
    Copyright Third Nature,Inc. Persisting data is not the end of the line. If you stop here you win the battle and lose the war
  • 19.
    Copyright Third Nature,Inc. We don’t have an analytics problem, just like we didn’t have a BI problem The origin of analytics as “business intelligence” was stated well in 1958: …the ability to apprehend the interrelationships of presented facts in such a way as to guide action towards a desired goal. ~ H. P. Luhn “A Business Intelligence System”, http://altaplana.com/ibmrd0204H.pdf ” “ Our goal is analytics as a capability, not a technology
  • 20.
    Copyright Third Nature,Inc. Three constituencies Stakeholder Analyst Builder aka the recipient aka the data scientist aka the engineer
  • 21.
    Copyright Third Nature,Inc. Starting points for analytics strategy Many organizations choose to start with the analysts. Create a data science team. Turn them loose to find a problem. Many more start with builders: technology solutions looking for problems, e.g. 65% of the IT driven Hadoop and Spark projects over the last five years. The right place to start? Stakeholders. The goal to achieve, the problem to solve.
  • 22.
    Copyright Third Nature,Inc. NATURE OF THE PROBLEM FROM THE STAKEHOLDER’S PERSPECTIVE Each constituency has their own set of problems to deal with
  • 23.
    BI and Analysis:Repeatability and Discoverability Discover Explore Analyze Model Consume Promote Focus on repeatability Application cycle time 90% of users Focus on discoverability Analyst cycle time 9% analysts, 1% data scientists 80% of data use 80% of analysis
  • 24.
    The myth thatstill drives analytics – analytic gold All we need is a fat pipe and pans working in parallel…
  • 25.
    Analytic insights thatresult in no action are expensive trivia. It’s not the insight, but what you do with it, that matters
  • 26.
    Applying analytics isnot an analytics problem Applying analytics is not in the analyst’s control. It’s not in the engineer’s control. It’s in the control of the people involved in the process. Failures are often in execution, not in analytics development. For example, we saw unexpectedly poor performance in a number of geographies. Was it the new analytics we tried? Was it a data problem? No, it was a simple compliance problem.
  • 27.
    NATURE OF THEPROBLEM FROM THE ANALYST’S PERSPECTIVE
  • 28.
    The analytics processat a high level Diagram: Kate Matsudaira
  • 29.
    The nature ofanalytics problems is researching the unknown rather than accessing the known. Repeat for each new problem Diagram: Kate Matsudaira
  • 30.
    The main hurdle:just getting the data Do you know where to find it? Because it’s may not be in the data warehouse. Do you have access to it? Is access fast enough? Because DWs are for QRD, not for moving huge piles of data. And ERP systems and SaaS apps are right out. These are why developers jump on “data lake” as the solution. But that’s a small, easy part.
  • 31.
    Do you havethe right data? Many machine learning techniques require labeled (known good) training data: Supervised learning: a person has to define the correct output for some portion of the data. Data is divided into training sets used for model building and test sets for validating the results. • What is spam and what isn’t? • What does a fraudulent transaction look like 36
  • 32.
    Do you haveenough of the right data? ML needs a lot, you may be disappointed in your efforts
  • 33.
    Where do analystsspend their time? mostly data work Slide 38 Define the business problem Translate the problem into an analytic context Select appropriate data Learn the data Create a model set Fix problems with data Transform data Build models Assess models Deploy models Assess results % of time spent 70% 30% Source: Michael Berry, Data Miners Inc.
  • 34.
    The analyst’s workspacein BI is relatively spare
  • 35.
    The analyst’s workspaceneeds to be more like a kitchen than like BI vending machines
  • 36.
    Copyright Third Nature,Inc.Copyright Third Nature, Inc. Schema control, space limitations, resource limitations, production lockdown Analytics work is a production workload, but…
  • 37.
    NATURE OF THEPROBLEM FROM THE BUILDER’S PERSPECTIVE
  • 38.
    IT and Opspeople want to know “what to build?” Giant data platform? Self service tools?
  • 39.
    Analytics has differentprocesses and workloads None of this analytics work is the same as what IT considered “analysis” to be, which is usually equated with BI or ad-hoc query. Ad-hoc analysis = Exploratory data analysis = Batch analytics = Real-time analytics A real analytics production workflow Hatch, CIKM ‘11 Slide 44
  • 40.
    The world changes,do the models? In BI you maintain ETL and schemas, in ML you maintain models. “Model decay” happens as the assumptions around which a model is built change, e.g. spam techniques change. When you adjust the model you need to know it is normal again ▪ Better save the data used to build the model ▪ Better save the model ▪ Baseline and measurements
  • 41.
    THREE PERSPECTIVES, ONESOLUTION? There are requirements from all constituents. You need to put them together to have a complete picture of what’s needed.
  • 42.
    The missing stakeholder Thereis another stakeholder: analytics management - the CAO, CDO, VP of analytics, aka “your boss” if you’re a data scientist. This is the perspective and problems of the person responsible for oversight of the team and efforts is across the organization and across multiple projects
  • 43.
    Job #1 -Repeatability
  • 44.
    Job # 2- Operational predictability
  • 45.
    Job #3 -Reproducibility
  • 46.
    You need asystem of record for analytics
  • 47.
    There is anextensive list of requirements to support Primary requirements needed by constituents S D E Data catalog and ability to search it for datasets X X Self-service access to curated data X Self-service access to uncurated (unknown, new) data X X Temporary storage for working with data X Data integration, cleaning, transformation, preparation tools and environment X X Persistent storage for source data used by production models X X Persistent storage for training, testing, production data used by models X X Storage and management of models X X Deployment, monitoring, decommissioning models X Lineage, traceability of changes made for data used by models X X Lineage, traceability for model changes X X X Managing baseline data / metrics for comparing model performance X X X Managing ongoing data / metrics for tracking ongoing model performance X X X S = stakeholder, user, D = data scientist, analyst, E = engineer, developer
  • 48.
    © Third NatureInc. How did we get to this point with BI & analytics? There’s a difference between having no past and actively rejecting it.
  • 49.
    54 100 1,000 10,000 1 2 34 5 6 7 8 9 10 Customer Data Plateaus Add users, accumulate history, add attributes, add dimensions Entirely new data set enabled by new technology at new price point – value has to exceed cost (ROI). Earliest adopters have vision ahead of ROI
  • 50.
    55 1 10 100 1,000 10,000 100,000 1,000,000 10,000,000 1 2 34 5 6 7 8 9 10 11 12 13 User Adoption of New Data Customer Data Users - Data Set 1 Users - Data Set 2 Users of Integrated Data User adoption of new data sets starts over. Very small number of experts growing to wider audience, sophisticated users moving to business analysts, then business users, B2B customers and even to consumers Greater value is derived when data sets are linked – see bigger picture (eg who buys what, when). Comes after initial extraction of easy value from standalone data
  • 51.
    56 1 10 100 1,000 10,000 100,000 1,000,000 10,000,000 100,000,000 1,000,000,000 10,000,000,000 1 2 34 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 Data Size Plateaus over Time Lather, Rinse, Repeat. Trend line is roughly Moore’s Law. Delay or skip a generation if new data set is two orders of magnitude instead of one.
  • 52.
    57 1 10 100 1,000 10,000 100,000 1,000,000 10,000,000 100,000,000 1,000,000,000 10,000,000,000 100,000,000,000 1,000,000,000,000 10,000,000,000,000 100,000,000,000,000 1,000,000,000,000,000 10,000,000,000,000,000 1 3 57 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 Customer Data Size vs. 2x per 12mo Customer Data Moore's Law - 2x/18mo 2x/12 mo But all studies show demand/creation of data increasing significantly faster than Moore’s law If we started only 1 OOM behind, we are now 5 OOM away from user demand for data Data Exhaust
  • 53.
    58 • <Store, Item,Week> • <Store, Item, Day> – Simple aggregations • Market Basket – Affinity – Link to person, demographics, HR • Inventory by SKU by store – Temporal, time series, forecasting – Link to product, marketing, market basket Retail Plateaus 2B records total for 9 quarters 2B records per day, keep 9 quarters
  • 54.
    59 • Web Logsand traffic – Behavioral patterns – eg path linked to person, offers, other channels – Operations of the web site • Supply chain sensors – sampled at major event – Activity Based Costing – Link to customer, product, HR, planning • Social Media – Text analysis, Filtering, languages – Link to customer, sales, other channel interactions • Supply chain sensors – sampled at minutes or seconds – Telematics – Real time, Event detection, trending, static and dynamic rules – Link to HR, thresholds, forecasts, routing, planning Retail Plateaus 30B records per day
  • 55.
    60 FINANCE Revenue Expenses Customers CUSTOMER CARE Customer Products Orders Case History SALES Orders Customers Products MARKETING Customers Orders Campaign History OPERATIONS Inventory Returns Manufacturing SupplyChain How many batteries are in inventory by plant? What is the trend of warranty costs? How many people made a warranty claim last week? How many sales have been made quarter to date? Which customers should get a communicatio n on extended warranties? 54 32 29 49 66
  • 56.
  • 57.
    62 2855 Given the risein warranty costs, isolate the problem to be a plant, then to a battery lot. Communicate with affected customers, who have not made a warranty claim on batteries, through Marketing and Customer Service channels to recall cars with affected batteries. Inventory Returns Manufacturing Supply Chain Customer Service Orders Revenue Expenses Case History Customers Products Pipeline Customers Campaign History FINANCE SALESMARKETING OPERATIONS CUSTOMER EXPERIENCE 2855
  • 58.
    63 Manufacturing: Data OverlapAnalysis New Business Improvement Opportunities through Data Leverage Sales Force Profitability Analysis 100% 80% 66% 24% 41% 66% 0% 24% 0% 24% Transportation Planning 11% 100% 87% 28% 64% 56% 22% 34% 13% 45% Production Planning 6% 57% 100% 19% 83% 35% 17% 28% 9% 40% Vendor Managed Inventory 12% 100% 100% 100% 78% 100% 61% 100% 39% 100% Global Pricing Rationalization 4% 50% 100% 17% 100% 27% 15% 29% 11% 41% Fulfillment (Perfect Order) 16% 94% 90% 48% 58% 100% 35% 53% 19% 56% Manufacturing Quality Optimization 0% 76% 88% 60% 66% 71% 100% 79% 43% 73% Preventative Maintenance Analysis 7% 75% 93% 61% 80% 68% 50% 100% 27% 82% Warranty Claims Analysis 0% 94% 100% 83% 100% 83% 94% 93% 100% 98% Quality Life Cycle Improvement 4% 57% 75% 35% 64% 41% 26% 47% 16% 100% SalesForce ProfitabilityAnalysis ProductionPlanning VendorManaged Inventory GlobalPricing Rationalization TransportationPlanning Fulfillment (PerfectOrder) ManufacturingQuality Optimization Preventative Maintenance Analysis WarrantyClaims Analysis QualityLifeCycle Improvement If Then
  • 59.
  • 60.
    65 PRODUCT SENSOR SOCIALMEDIA CUSTOMER CARE AUDIO RECORDINGS DIGITAL ADVERTISING CLICKSTREAM 65 41 32 19 28 How many visitors did we have to our hybrid cars microsite yesterday? What are the temperature readings for batteries by Manufacturer? What is the sentiment towards line of hybrid vehicles? Which customers likely expressed anger with customer care? Which ad creative generated the most clicks?
  • 61.
    66 2855 High, Well Known Quality Directional Quality Unknown/Low quality WellDefined Data Model Curated JSON, XML,… Extracted Attributes Curation Required
  • 62.
    67 2855 SENSOR DIGITAL ADVERTISING CLICKSTREAM INTERACTIONS RATINGS& REVIEWS CUSTOMER PORTAL INTERACTIONS EXTERNAL INTERACTIONS SOCIAL MEDIA IVR Routing RFID ELECTRONIC COMMERCE FINANCE SALES MARKETING Inventory Returns Manufacturing Supply Chain Customer Service Orders Revenue Expenses Case History Customers Products Pipeline Customers Campaign History OPERATIONS CUSTOMER CARE – AUDIO RECORDINGS Maps Telemetry SERVER LOGS CUSTOMER EXPERIENCE Enterprise Data
  • 63.
  • 64.
    69 2855 Out of Deviation SensorReadings Fraud Events CUSTOMER EXPERIENCE SALES MARKETING OPERATIONS Abandoned Carts Online Price Quotes Social Media Influencers FINANCE 11234 Minimum Viable Curation Minimum Viable Data Quality
  • 65.
    70 2855 Out of Deviation SensorReadings Fraud Events CUSTOMER EXPERIENCE SALES MARKETING OPERATIONS Abandoned Carts Online Price Quotes Social Media Influencers FINANCE 11234Sensor data formatting Unit Normalization Vehicle/version normalization Make/model/year selection Data Comprehension, Pipelines
  • 66.
    71 2855 Enterprise Wide Use 10s to100s of Users 10 Users Super Users Business Analysts Action list, Report, Dashboard Users Share across Enterprise Share in department Share over cube wall User Base and Sharing
  • 67.
  • 68.
    73 2855 Production tools, curated data, integratedacross business areas Wide variety of tools and data forms Targeted data access, many applications, response time SLAs Evolving Consumption Requirements High CPU, moderate to low IO Bulk scans, large computation, transformation, data access, specific integration Moderate CPU, High IO, Resource management
  • 69.
    74 2855 Relational store more appropriate Hybrid FileStore more appropriate Technology, Capacity 100 TB 1 PB 10 TB
  • 70.
    75 2855 MANUFACTURING CAMPAIGN HISTORY COSTS PRODUCTS FINANCE SALES MARKETING OPERATIONS CUSTOMER EXPERIENCE Giventhe rise in warranty costs, isolate the problem to be a plant and the specific lot. Exclude 2/3rd of the batteries from the lot that are fine. Communicate with affected customers, who have not made a warranty claim, through Marketing and Customer Service channels to recall cars with affected batteries. CUSTOMERS CASE HISTORY SENSOR Access Wide Variety of Data to Answer a Question
  • 71.
    Copyright Third Nature,Inc. An event contains mainly IDs…that reference other data 2010.01.10.01:41:4468 67.189.110.179 Ser40489a07-7f6e4251801a 13ae51a6d092 301212631165031 590387153892659 DNIS5555685981- UTF8&1T4ACGW_enUS386US387&fu=0&ifi=1&dtd=204&xpc=1KoLqh374s m100109_44IOJ1-Id=105543CD1A7B8322 timestamp user ID device ID game ID event details IP adr log ID Log de-referencing and enrichment is difficult since you can’t enforce integrity like you can in a DB. What’s the glue that holds it together? It’s just keys to other data. e.g. remember that device ID 0 problem?
  • 72.
    Copyright Third Nature,Inc.© Third Nature Inc. It’s not just the log data, it’s big data + small data Device Event logs Event logs Event logs A product has a complex set of master data and metadata and this needs to be added back to the bare events.
  • 73.
    Copyright Third Nature,Inc.© Third Nature Inc. Where does the reference data come from? The keys come from somewhere. You don’t just make up system-wide unique identifiers in your code. The lack of local lookup data at event generation leads to development practices that lead to inconsistencies. Event logs Event logs Event logs Problem we had: product identifiers that didn’t match any known products Had to fix by analyzing each log of bad-ID events. Because developers used config files they copied from the PMS and put in the code Break here and you have nothing
  • 74.
    Copyright Third Nature,Inc. MDM again If you want to link datasets then you must manage the keys You need canonical forms for common data (in code too) user ID IP adr 2010.01.10.01:41:4468 67.189.110.179 Ser40489a07-7f6e4251801a 13ae51a6d092 301212631165031 590387153892659 DNIS5555685981- UTF8&1T4ACGW_enUS386US387&fu=0&ifi=1&dtd=204&xpc=1KoLqh374s m100109_44IOJ1-Id=105543CD1A7B8322 UID CID Email City State Country 590387153892659 10098213 [email protected] Paris Île-de-FranceFrance 2010.01.10.14:26:2468 67.189.110.179 10098213 5046876319474403 MOZILLA/4.0 (COMPATIBLE; TRIDENT/4.0; GTB6; .NET CLR 1.1.4322) https://www.phisherking.com/ gifts/store/LogonForm?mmc=link-src-email_m100109 http://www.google.com/search? sourceid=navclient&aq=0h&oq=Italian&ie=UTF8&pid=1T4ACGW_13ae51a6d092&q=ita lian+rose&fu=0&ifi=1&dtd=204&xpc=1KoLqh374s game ID date IP adr customer ID Event Click Cust- user
  • 75.
  • 76.
    Copyright Third Nature,Inc. What you really have This is called “self-service.” Analysts will love it…
  • 77.
    Copyright Third Nature,Inc. The solution to our problems isn’t technology, it’s architecture.
  • 78.
    Copyright Third Nature,Inc. History: This is how BI was done through the 80s First there were files and reporting programs. Application files feed through a data processing pipeline to generate an output file. The file is used by a report formatter for print/screen. Files are largely single-purpose use. Every report is a program written by a developer. Data pipeline code
  • 79.
    Copyright Third Nature,Inc. History: This is how BI ended the 80s The inevitable situation was... Data pipeline code
  • 80.
    Copyright Third Nature,Inc. History: This is how we started the 90s Collect data in a database. Queries replaced a LOT of application code because much was just joins. We learned about “dead code” SQL SQL SQL SQL SQL
  • 81.
    Copyright Third Nature,Inc. Pragmatism and Data Lessons learned during the ad-hoc SQL era of the DW market: When the technology is awkward for the users, the users will stop trying to use it. Even “simple” schemas weren’t enough for anyone other than analysts and their Brio… Led to the evolution of metadata-driven SQL- generating BI tools, ETL tools.
  • 82.
    Copyright Third Nature,Inc. BI evolved to hiding query generation for end users With more regular schema models, in particular dimensional models that didn’t contain cyclic join paths, it was possible to automate SQL generation via semantic mapping layers. We developed data pipeline building tools (ETL). Query via business terms made BI usable by non-technical people. ETL SQL Life got much easier…for a while
  • 83.
    Copyright Third Nature,Inc. Today’s model: Lake + self-service, looks familiar… The Lake with data pipelines to files or Hive tables is exactly the same pattern as the COBOL batch.. Dataflow code We already know that people don’t scale…
  • 84.
    Copyright Third Nature,Inc. DATA ARCHITECTURE We’re so focused on the light switch that we’re not talking about the light
  • 85.
    Copyright Third Nature,Inc. Decouple the Architecture The core of the data warehouse isn’t the database, it’s the data architecture that the database and tools implement. We need a data architecture that is not limiting: ▪ Deals with change more easily and at scale ▪ Does not enforce requirements and models up front ▪ Does not limit the format or structure of data ▪ Assumes the range of data latencies in and out, from streaming to one-time bulk ▪ Allows both reading and writing of data from outside
  • 86.
    Copyright Third Nature,Inc. The architecture from 1988 we SHOULD HAVE BEEN USING The general concept of a separate architecture for BI has been around longer, but this paper by Devlin and Murphy is the first formal data warehouse architecture and definition published. 91 “An architecture for a business and information system”, B. A. Devlin, P. T. Murphy, IBM Systems Journal, Vol.27, No. 1, (1988) Slide 91Copyright Third Nature, Inc.
  • 87.
    Copyright Third Nature,Inc. The goal is to decouple: solve the application and infrastructure problems separately Platform Services Data Access Deliver & Use Data storage This separates BI from other uses of data, allowing each type of use to structure the data specific to its own requirements. Data access is already somewhat separate today. Make the separation of different access methods a formal part of the architecture. Don’t force one model.
  • 88.
    Copyright Third Nature,Inc. The goal is to decouple: solve the application and infrastructure problems separately Platform Services Data Management Process & Integrate Data storage Data management should not be subject to the constraints of a single use Data management was blended with both data acquisition and structuring data for client tools. It should be an independent function.
  • 89.
    Copyright Third Nature,Inc. The goal is to decouple: solve the application and infrastructure problems separately Data Acquisition Collect & Store Incremental Batch One-time copy Real time Platform Services Data storage Data arrives in many latencies, from real-time to one-time. Acquisition can’t be limited by the management or consumption layers. Data acquisition should not be directly tied to the needs of consumption. It must operate independently of data use.
  • 90.
    Copyright Third Nature,Inc. The full analytic environment subsumes all the functions of the data warehouse, and extends them Data Acquisition Collect & Store Incremental Batch One-time copy Real time Platform Services Data Management Process & Integrate Data Access Deliver & Use Data storage The platform has to do more than serve queries; it has to be read-write.
  • 91.
    Copyright Third Nature,Inc. The data architecture must align with system components because each of them addresses different data needs Incremental Collect Batch One-time copy Real time Manage & Integrate Data Acquisition Collect & Store Data Management Process & Integrate Data Access Deliver & Use Separating concerns is part of the mechanism for change isolation
  • 92.
    Copyright Third Nature,Inc. Zone 1: Acquistion ▪ Focus is only on collecting and tracking data ▪ Direct recording of data from sources ▪ Each dataset has as much schema as the source provides, could be explicit or implicit or none ▪ Foundation layer for all subsequent use Zone 1 Transactions Events Objects
  • 93.
    Copyright Third Nature,Inc. Zone 2 Zone 2: Integration - Standardized ▪ Parsed and cataloged – all structure and form are made explicit so data can be accessed ▪ Namespace (e.g. keys) managed ▪ Common elements are standardized ▪ Datasets are profiled, indexed, available Zone 1 Transactions Events Objects
  • 94.
    Copyright Third Nature,Inc. Zone 2 Zone 2 Zone 2: Integration - Enhanced ▪ Common shared KPIs, master datasets ▪ Extracted and derived data available, e.g. NEE output ▪ Data is linkable, labeled, possibly cleaned Zone 1 Transactions Events Objects
  • 95.
    Copyright Third Nature,Inc. Layer 1 Layer 2 Layer 0 Zone 3 Zone 3: Access ▪ Data structured to suit the workload ▪ Integrated data for a specific purpose ▪ Business rules applied, e.g. filters, controls, etc. ▪ Designed, explicit structures ▪ Generally repeatable use
  • 96.
    Copyright Third Nature,Inc. Layer 0 Layer 1 Layer 2 Zone 3 Zone 4 Zone 4: Transient ▪ Under business / developer control ▪ The data for one-off projects of unknown value or repeatability, integrated from other layers ▪ Place for ephemeral analytics output ▪ The sandbox
  • 97.
    Copyright Third Nature,Inc. Zone 1 Raw Zone 2 Standardized Zone 2 Enhanced Zone 3 Managed Zone 4 Transient Decoupled data architecture
  • 98.
    Copyright Third Nature,Inc. Food supply chain: an analogy for analytic data Multiple contexts of use, differing quality levels You need to keep the original because just like baking, you can’t unmake dough once it’s mixed.
  • 99.
    Copyright Third Nature,Inc. The design focus is different in each area Ingredients Goal: available User needs a recipe in order to make use of the data. Pre-mixed Goal: discoverable and integrateable User needs a menu to choose from the data available Meals Goal: usable User needs utensils but is given a finished meal
  • 100.
    Copyright Third Nature,Inc. The data is in zones of management, not isolating layers Raw data in an immutable storage area Standardized or enhanced data Common or usage- specific data Transient data Relax control to enable self-service while avoiding a mess. Do not constrain access to one zone or to a single tool. Focus on visibility of data use, not control of data.
  • 101.
    Copyright Third Nature,Inc. This data architecture resolves rate of change problems Raw data in an immutable storage area Standardized or enhanced data Common or usage- specific data Transient data New data of unknown value, simple requests for new data can land here first, with little work by IT. More effort applied to management, slower. Optimized for specific uses / workloads. Generally the slowest change. Not fast vs slow: fast vs right Not flexibility vs control: flexibility vs repeatability Agile for structure change vs agile for questions / use
  • 102.
    Copyright Third Nature,Inc. SQLServer RDBMS Example: data environment, mid-size retailer Replication, batch ETL to collect data Web Application Standard retail / wholesale data warehouse model. Nightly batch model from stage, contains ~ 1 TB of data DW New requirement: Load all the online marketing and web activity for use in analytic models (not simple web analytics) Will add ~10 TB of data, continuous or hourly loads
  • 103.
    Copyright Third Nature,Inc. SQLServer RDBMS Example: data environment, mid-size retailer Replication, batch ETL to collect data Hadoop Web Application Persistent store, data warehouse on same DB in different schemas DW Log file fetch & load for clickstream, summaries sent to reporting env Discovery work usually done in Hadoop, sometimes in database.
  • 104.
    Copyright Third Nature,Inc. SQLServer This data architecture uses the 3 zone pattern Hadoop Web Application RDBMS DW Raw data in an immutable storage area Standardized or enhanced data Common or usage- specific data Transient data All reporting data and analytics output is sent to the DW where it is can be accessed. Cleaned, summarized or derived data is stored in DB Raw data is stored in two places: relational and small sets in DB, log collection in Hadoop
  • 105.
    Copyright Third Nature,Inc. The concept of a zone is not a physical system. It’s data architecture Apps DW The biggest decision is to separate all data collection from the data integration from consumption. Physical system/technology overlays are separate, depend on the specific use cases and needs of the organization. Dist FS RDBMSs Decompress & process device logs RDBMS RDBMS RDBMS Discovery platform
  • 106.
    Copyright Third Nature,Inc. What about the technology? Do I need an <X>?
  • 107.
    112 Blended Architectures Area Requirement, Not an Option Data Warehouse + Data Lake On Premise + Cloud RDBMS + S3 + HDFS Commercial + Open Source
  • 108.
    Copyright Third Nature,Inc. Bricks are not buildings We don’t think this is equivalent to this Architecture is not technology. It’s not a product you can buy.
  • 109.
    Copyright Third Nature,Inc.Copyright Third Nature, Inc. New tools and tool architectures are required One tool for all jobs doesn’t work any more* * it never did, we just had fewer jobs
  • 110.
    Copyright Third Nature,Inc. Beware overengineering: do you really need special new technology infrastructure? Numberofpeople The distribution of data size is about normal, yet these guys set the tone of the market today. Bigness of data
  • 111.
    Copyright Third Nature,Inc. Analytics: This is really raw data under storageNumberofjobs Microsoft study of 174,000 analytic jobs in their petabyte scale cluster: median size = ??? Bigness of data Copyright Third Nature, Inc.
  • 112.
    Copyright Third Nature,Inc. Working data for analytics most often not bigNumberofjobs 14 GB Smallness of data Copyright Third Nature, Inc. The usable size distribution
  • 113.
    Copyright Third Nature,Inc. TANSTAAFL When replacing the old with the new (or ignoring the new over the old) you always make tradeoffs, and usually you won’t see them for a long time. Technologies are not perfect replacements for one another. Often not better, only different.
  • 114.
    © Third NatureInc.© Third Nature Inc. 1.It’s not about storing data, it’s about using it 2.Use drives architecture. Understand the uses, what you are designing for, to drive decisions. 3.Put data at the center, not technology. Don’t let the tech define what you can do or how you do it. 4.The death star is not the answer. The data model is not a flat earth. You are not building a monolith. 5.Know your history. Avoiding wheel reinvention saves time, money, careers. Conclusion
  • 115.
    © Third NatureInc. Manage your data (or it will manage you) Data management is where developers are weakest. Modern engineering practices are where data management is weakest. You need to bridge these groups and practices in the organization if you want to do meaningful work with event stream data.
  • 116.
    © Third NatureInc. “Now is not the end. It is not even the beginning of the end. But it is, perhaps, the end of the beginning.” Winston Churchill
  • 117.
    Mark Madsen isthe global head of architecture at Teradata, Prior to that he was president of Third Nature, a research and consulting firm focused on analytics, data integration and data management. Mark is an award- winning author, architect and CTO whose work has been featured in numerous industry publications. Over the past ten years Mark received awards for his work from the American Productivity & Quality Center, TDWI, and the Smithsonian Institute. He is an international speaker, chairs several conferences, and is on the O’Reilly Strata program committee. For more information or to contact Mark, follow @markmadsen on Twitter or visit http://ThirdNature.net About the Presenter
  • 118.
    Todd Walter Chief Technologist- Teradata • Chief Technologist for Teradata • A pragmatic visionary, Walter helps business leaders, analysts and technologists better understand all of the astonishing possibilities of big data and analytics • Works with organizations of all sizes and levels of experience at the leading edge of adopting big data, data warehouse and analytics technologies • With Teradata for more than 30 years and served for more than 10 years as CTO of Teradata Labs, contributing significantly to Teradata’s unique design features and functionality • Holds more than a dozen Teradata patents and is a Teradata Fellow in recognition of his long record of technical innovation and contribution to the company