Data Warehouse Architecture -Processes-
Overview
Architecture a technical blueprint stage Must support 3 major driving forces:
Populating the warehouse
Data extraction, cleaning and loading
Day-to-day management of the warehouse
Large volumes of data, create/delete summaries
The ability to cope with requirement evolution
Cope with future changes with query profiles
Typical Process Flow
Extract and load the data Clean and transform data into a form that provides good query performance Backup and archive data Manage queries, and direct them to appropriate data sources
Extract & Load Process
Extract
Takes data from source systems and make it available to the data warehouse
Load
Takes extracted data and loads it into the data warehouse
Data in operational systems is held in a from suitable for that system Before loading the data into the DW, information content must be reconstructed Data must become value added business information
Extract & load process must take data and add context and meaning
Issues with ELP
When to start extracting the data, run transformation and consistency checks and so on?
A controlling mechanism is essential to fire each module when appropriate
When to extract?
Data must be in consistent Start extracting data from data sources when it represents the same snapshot of time as all other data sources
Eg. Customer database
Issues
Loading the data
Extracted data are loaded into temporary data store to perform clean up and check for consistency Do not execute consistency checks until all the data sources have been loaded into the temporary data store
Eg. Customer canceling subscriptions
Error recovery must be an integral part of the design The effort required to clean up the source systems increases exponentially with the number of overlapping data sources
Issues
Copy Management tools and clean up
Eg. IBMs Information Warehouse Framework
Data Refresher & Data Hub
Most copy management tools do not have the capability of performing consistency check directly (user must write the logic & code it) Make cost-benefit analysis before purchasing copy management tool
If source systems do not overlap, then consistency checks are very simple
Clean and Transform Data
Steps involved are:
Clean and transform the loaded data into a structure that speeds up queries Partition the data to speed up queries, optimize hardware performance and simplify the DW management Create aggregations to speed up the common queries
Clean and Transform Data
Data needs to be cleaned and checked in the following ways:
Make sure data is consistent with itself Make sure data is consistent with other data within the same source Make sure data is consistent with data in the other source systems Make sure data is consistent with the information already in the DW
Once data is cleaned, convert source data into a structure that is designed to balance query performance and operational cost
The structure must be suitable for long term storage
Backup & archive
Regular backup is essential to recover data from loss Archiving
Older data is removed from the system in a format that allows it to be quickly restored if required Issue
As DW evolves, all information may change Hence to ensure that a restored archive is valid, it becomes necessary to extract all related data and structures as well
Query Management Process
It is a system process
Manages the queries Speeds them up by directing queries to the most effective data source Ensure that all system resources are used effectively Monitor query profiles manage which aggregations to generate This process operates at all times
