Data Warehouse Development Approaches
1
1
Fundamental Questions
Before deciding to build a data warehouse for your organization, you need to ask the following basic and fundamental questions and address the relevant issues:
Top-down or bottom-up approach? Enterprise-wide or departmental? Which firstdata warehouse or data mart? Build pilot or go with a full-fledged implementation? Dependent or independent data marts?
2
Data Warehouse Development Approaches
Data warehouse development approaches
Inmon Model: EDW approach Kimball Model: Data mart approach
Which model is better?
There is no one-size-fits-all strategy to data warehousing One alternative is the hosted warehouse
General Data Warehouse Development Approaches
Big bang approach
Incremental approach: Top-down incremental approach Bottom-up incremental approach
ISQS 6339, Data Mgmt & BI, Zhangxi Lin
Big Bang Approach
Analyze enterprise requirements
Build enterprise data warehouse
Report in subsets or store in data marts
ISQS 6339, Data Mgmt & BI, Zhangxi Lin
Incremental Approach to Warehouse Development
Multiple iterations Shorter implementations Validation of each phase
Increment 1 Strategy Definition Analysis Design
Iterative
Build
Production
ISQS 6339, Data Mgmt & BI, Zhangxi Lin
Top-Down Approach
Analyze requirements at the enterprise level Develop conceptual information model Identify and prioritize subject areas Complete a model of selected subject area Map to available data Perform a source system analysis Implement base technical architecture Establish metadata, extraction, and load
processes for the initial subject area
Create and populate the initial subject area
data mart within the overall warehouse framework
ISQS 6339, Data Mgmt & BI, Zhangxi Lin 7
Top down
The advantages of this approach are:
A truly corporate effort, an enterprise view of data
Inherently architectednot a union of disparate data marts Single, central storage of data about the content Centralized rules and control May see quick results if implemented with iterations
The disadvantages are:
Takes longer to build even with an iterative method High exposure/risk to failure
Needs high level of cross-functional skills
High outlay without proof of concept
Bottom-Up Approach
Define the scope and coverage of the
data warehouse and analyze the source systems within this scope
Define the initial increment based on the
political pressure, assumed business benefit and data volume
Implement base technical architecture
and establish metadata, extraction, and load processes as required by increment
Create and populate the initial subject
areas within the overall warehouse framework
ISQS 6339, Data Mgmt & BI, Zhangxi Lin
Bottom-Up
The advantages of this approach are:
Faster and easier implementation of manageable pieces
Favorable return on investment and proof of concept Less risk of failure Inherently incremental; can schedule important data marts first Allows project team to learn and grow
The disadvantages are:
Each data mart has its own narrow view of data Permeates redundant data in every data mart
Perpetuates inconsistent and irreconcilable data
Proliferates unmanageable interfaces
10
Dimensional Modeling Process
High level dimensional model design
Choosing business model Declaring the grain Choosing dimensions Identifying the facts
Detailed dimensional model development Dimensional model review and validation
IS Core users Business community
Final design iteration
ISQS 6339, Data Mgmt & BI, Zhangxi Lin
11
Supplemental Slides : Data Warehouse Design Phases
12
Defining the Business Requirements
The concept of business dimensions is fundamental to the requirements definition for a data warehouse.
13
Information package
Your primary goal in the requirements definition phase is to compile information packages
Once you have firmed up the information packages, youll be able to proceed to the other phases. Essentially, information packages enable you to:
Define the common subject areas Design key business metrics Decide how data must be presented Determine how users will aggregate or roll up Decide the data quantity for user analysis or query Decide how data will be accessed
14
15
16
Supplemental Slides : The Others
17
Snowflake Schema Model
Country
Direct use by some tools More flexible to change Provides for speedier data loading Can become large and unmanageable Degrades query performance More complex metadata
State County City
18
18
Degenerate Dimensions
order_number and order_line in the fact table
For example, you may be looking for average number of products per order. Then you will have to relate the products to the order number to calculate the average. Attributes such as order_number and order_line in the example are called degenerate dimensions and these are kept as attributes of the fact table.
19
Storage and Performance Considerations
Database sizing Data partitioning Indexing Star query optimization
20
20
Database Sizing - Test Load Sampling
Analyze a representative sample of the data chosen using proven statistical methods. Ensure that the sample reflects: Test loads for different periods Day-to-day operations Seasonal data and worst-case scenarios Indexes and summaries
21
21
Data Partitioning
Breaking up of data into separate physical units that can be handled independently Types of data partitioning Horizontal partitioning. Vertical partitioning
22
22
Indexing
Indexing is used for the following reasons: It is a huge cost saving, greatly improving performance and scalability. It can replace a full table scan by a quick read of the index followed by a read of only those disk blocks that contain the rows needed.
23
23
Parallelism
Sales table P1 P2 P3
Customers table
P1
P2
P3
Parallel Execution Servers
24
24
Using Summary Data
Designing summary tables offers the following benefits: Provides fast access to precomputed data Reduces use of I/O, CPU, and memory
25
25
