Best Practices DB2 BLU Acceleration Für SAP - Olaf Depper
Best Practices DB2 BLU Acceleration Für SAP - Olaf Depper
Applications
ISICC Walldorf, Mai 2015
Performance is based on measurements and projections using standard IBM benchmarks in a controlled
environment. The actual throughput or performance that any user will experience will vary depending
upon many factors, including considerations such as the amount of multiprogramming in the user's job
stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no
assurance can be given that an individual user will achieve results similar to those stated here.
All customer examples described are presented as illustrations of how those customers have used IBM
products and the results they may have achieved. Actual environmental costs and performance
characteristics may vary by customer. Nothing contained in these materials is intended to, nor shall
have the effect of, stating or implying that any activities undertaken by you will result in any specific
sales, revenue growth or other results.
Trademarks
© Copyright IBM Corporation 2015. All rights reserved.
• U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA
ADP Schedule Contract with IBM Corp.
IBM, the IBM logo, ibm.com, AIX and DB2 are trademarks or registered trademarks of International
Business Machines Corporation in the United States, other countries, or both. If these and other IBM
trademarked terms are marked on their first occurrence in this information with a trademark symbol (®
or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time
this information was published. Such trademarks may also be registered or common law trademarks in
other countries. A current list of IBM trademarks is available on the Web at “Copyright and trademark
information” at www.ibm.com/legal/copytrade.shtml
Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.
Windows is a trademark of Microsoft Corporation in the United States, other countries, or both.
UNIX is a registered trademark of The Open Group in the United States and other countries.
SAP, R/3 and all other SAP product and service names mentioned herein are trademarks or registered
trademarks of SAP AG in Germany and several other countries.
Other company, product, or service names may be trademarks or service marks of others.
Table of contents
Dynamic In-Memory
– In-memory columnar processing with dynamic
movement of unused data to storage
Actionable Compression
– Industry’s first data compression that preserves order
so that the data can be used without decompressing
Parallel Vector Processing
– Multi-core and SIMD parallelism
(Single Instruction Multiple Data)
Data Skipping
DB2 BLU – Dynamic In-memory
SAP BW Support Prior DB2 10.5
Prior BLU Acceleration: All SAP BW and NLS objects are created on row-organized tables
Reporting
DB2 optimizations
Aggre
• Database Partitioning
Aggr Aggre
egate
Aggre
gate gate gate Feature (DPF)
Reporting • Multi-Dimensional
Layer Clustering (MDC)
Maste
Info
• Compression
Info Info r
Cub Cub Cub
Data
e e e
Enterprise
Data
Warehouse DSO DSO
Layer
Data BW Near-Line
Acquisition PSA PSA PSA Storage (NLS)
Layer
Source Systems
SAP BW Support on DB2 10.5 <=FP3
First Roll-out phase: Selected SAP BW and NLS objects can be implemented on column-organized (BLU) tables
Enterprise • Advantages
Data
Warehouse DSO DSO DSO • Faster reporting on InfoCubes
Layer • Faster reporting on DSOs
• Faster ETL processing
BW Near-Line
Data
Acquisition PSA PSA PSA PSA Storage (NLS) • Significant space reduction
Layer • Less ETL processing steps
• Performance improvements
Source Systems with Cancun
• No index tuning required
Tables that can be converted to BLU
Reporting
InfoCube fact and dimension tables
– /BI%/F%, /BI%/E%, /BI%/D%,
Aggregate
– /BI%/A%00 Inf
Mast
er
Fla
t Inf
o Inf o
Data
Cu o Cu
PSA Tables be Cu
be
be
– /BI%/B% Enterprise
Data
Warehouse
Layer DSO DSO DSO
Before you start using column-organized tables, ensure that you have a system
and database configuration that is supported:
Supported Platform and Operating Systems (check also
http://www-01.ibm.com/support/docview.wss?uid=swg27038033):
– Linux 64-bit on Intel/AMD
RHEL 6 or higher (recommended: RHEL 7.0 or higher)
SLES 10 SP4, SLES 11 SP2 or higher (recommended: SLES 11 SP3 or higher)
– AIX on POWER7 or higher hardware
AIX 6.1 TL7 SP6, AIX 7.1 TL1 SP6 or higher (recommended: AIX 7.1 TL3 or higher)
– Windows (since 10.5 FP 5)
Windows 7, Windows Server 2008 R2 (recommended: Windows Server 2012)
System Requirements (II)
For productive usage of BLU Acceleration your database server should at least have the
following resources:
– Minimum of 8 CPU cores and minimum of 64 GB main memory
– For test systems, this can be lower. However, keep in mind that BLU Acceleration
performance depends heavily on parallel processing and the possibility to keep the
active data in main memory.
IBM Best Practice:
– 16 cores for every 3TB of raw (i.e. uncompressed) data, and a minimum of 8-16 cores in
the system
– A useful quick cross-check: take your raw data size, and ensure you have at least 5% of
that number in main memory.
DB2 Requirements (I)
Adapt your database and database manager configuration for the use of BLU Acceleration
(details in SAP note 1851832).
– Instance (DBM) Configuration:
INSTANCE_MEMORY >= 16,000,000 (4 KB pages)
INTRA_PARALLEL = YES
MAX_QUERYDEGREE = ANY
SHEAPTHRES = 0
– Database Configuration:
SHEAPTHRES_SHR >= 6,400,000 (4 KB pages)
40% of INSTANCE_MEMORY
SORTHEAP >= 320,000 (4 KB pages)
1/20 of SHEAPTHRES_SHR
CATALOGCACHE_SZ >= 8,000 (4 KB pages)
DB2 Configuration Settings (II)
Note that the sort parameters must be set manually for DB2 BLU Acceleration. Even if you
use STMM the sort parameters cannot be tuned automatically by DB2.
Depending on your available memory the following best practices apply:
– Use 40% of the available instance memory for the buffer pool
(ALTER BUFFERPOOL IBMDEFAULTBP SIZE <40% of instance memory>
AUTOMATIC)
– Use 40% of the available instance memory for sort memory threshold
(SHEAPTHRES_SHR).
– Set SORTHEAP to 1/20 of SHEAPTHRES_SHR (to allow 20 parallel queries)
Keep the default SAP / DB2 10.5 settings for automatic maintenance (incl. AUTO_REORG)
also for BLU environments.
– Will take care automatically of space reclamation (REORG TABLE … RECLAIM
EXTENTS)
DB2 Workload Manager (I)
Set a DB2 Workload Manager (WLM) threshold for the number of expensive BLU
queries that can run in parallel in the database.
– The WLM threshold prevents that too many expensive queries can run in
parallel in the database to avoid CPU overload.
Proceed as follows:
– Create one column-organized table in the database (either manually or by
creating a dummy column-organized InfoCube or DSO in BW transaction
RSA1)
For example, run the following in the DB2 command line:
CREATE TABLE MYBLUTABLE(i int) ORGANIZE BY COLUMN
– Run db6_update_db.sh from your DB2 10.5 DVD to generate the SQL script
db6_update_db_out.sql
DB2 Workload Manager (II)
The DB6 update script db6_update_db_out.sql should contain the following code
at the end:
[…]
alter threshold SYSDEFAULTCONCURRENT enable;
alter threshold SYSDEFAULTCONCURRENT when CONCURRENTDBCOORDACTIVITIES >
<number> AND QUEUEDACTIVITIES UNBOUNDED stop execution;
alter work action set SYSDEFAULTUSERWAS enable;
alter work action set SYSDEFAULTUSERWAS alter work action
SYSMAPMANAGEDQUERIES enable;
alter work class set SYSDEFAULTUSERWCS alter work class
SYSMANAGEDQUERIES FOR TIMERONCOST FROM <number> TO UNBOUNDED;
– Note that the concurrency threshold is not compatible with the standard SAP
WLM settings. Therefore, the db6_update_db script will drop the SAP WLM
setup when creating the concurrency threshold.
DB2 Workload Manager (III)
Install the latest available version of the SAP Kernel and the DBSL library
– Make sure to at least use the version that contains the patch from SAP Note
1911087 with the following patch text: "DB6: DROP INDEX to DROP
CONSTRAINT“
Add the following SAP profile parameters to your profile and restart the
application servers (mandatory beginning with 10.5.4):
– dbs/db6/dbsl_ur_to_cc=*
Changes the default isolation level for SAP queries from UR (uncommitted
read) to CC (currently committed)
– dbs/db6/deny_cde_field_extension=1
Prevents the extension of a VARCHAR column in column-organized tables
SAP Requirements (III)
Install the required SAP BASIS and SAP BW Support
Packages in your system
For converting tables from row to column organized we use the SAP report
DB6CONV in SAP environments.
– For non-SAP environments, the db2convert tool is available – do not use this
tool in SAP systems.
– Both DB6CONV and db2convert are based on the DB2
ADMIN_MOVE_TABLE stored procedure.
For more information see the DB6CONV documentation attached to SAP note
1513862
– Make sure to always use the newest version of DB6CONV Version 6.
Before you use DB6CONV to convert BW objects to BLU make sure that your
system fulfills all the prerequisites for the usage of DB2 BLU Acceleration.
– Apply SAP Note 1825340
DB6CONV (II)
You can convert existing InfoCubes and DSOs to column-organized tables with
DB6CONV V6.0 or higher.
When you install DB6CONV >= V6.0 and your system fulfills all prerequisites for
DB2 BLU Acceleration then a new tab “BW Conversions” is shown when you run
the DB6CONV report.
You can select InfoCubes, DSOs, InfoObjects, and PSA tables and create a job
that converts these tables from row-organized to BLU (or vice versa).
Conversion of row-organized tables to column-organized tables can be done
online. A conversion from column-organized tables to row-organized tables,
however, can only be done in read-only mode.
Using DB6CONV V6 (I)
3) Click on
“Conversion Queue”
2) Choose all BW
Objects
Optional: Keep
Aggregates
Using DB6CONV V6 (III)
The conversion queue is now shown: Mark & delete unwanted objects, if
applicable
Using DB6CONV V6 (IV)
Back in the start screen select the new „Conversion Job“ and start it
You can use DB6CONV to monitor the progress of the conversion job:
Using DB6CONV V6 (VII)
When you run BW queries on column-organized InfoCubes and DSOs the SQL
queries that are executed may contain references to BW temporary tables that
contain hierarchies, lists of SIDs, or the result of materializing dimension/master
data branches.
For optimal query performance these temporary tables should be column-
organized.
After you have made your system ready for DB2 BLU Acceleration run report
SAP_DROP_TMPTABLES to drop existing row-organized BW temporary tables.
Then set the RSADMIN parameter DB6_TMP_USE_CDE = YES with report
SAP_RSADMIN_MAINTAIN
– With this setting, when new BW temporary tables are needed, these tables will
be created column-organized.
BLU Acceleration for New Tables
You can create new InfoCubes and DataStore objects (DSOs) with column-
organized tables.
If you also want to create new InfoObjects and new PSA tables as column-
organized tables you must set the following BW RSADMIN parameters with report
SAP_RSADMIN_MAINTAIN:
– DB6_IOBJ_USE_CDE = YES
Create new InfoObjects as column-organized tables
– DB6_PSA_USE_CDE = YES
Create new PSA tables as column-organized tables
Screen Shot: SAP_RSADMIN_MAINTAIN
Support for NLS (I)
Upgrade your BW system to the required Support Packages for using DB2 BLU
Acceleration on DB2 NearLine Storage systems:
7.30 11
7.31 10
7.40 5
Support for NLS (II)
Implement the following SAP notes for BLU on top of the Support Packages:
– SAP note 1834310 - DB6: Use of DB2 BLU Acceleration with DB2 Near-Line Storage
– SAP note 2030925 - DB6: NLS Impl. - Structure changes for DSOs with column-
organized NLS tables
– SAP note 2063589 - DB6: NLS Impl. - Mandatory corrections for NLS with DB2 10.5
FP4 release
Support for NLS (III)
Run report SMIGR_CREATE_DDL in the source system and choose one of the
three following options:
– Keep Source Settings
– Use BW Basic Table Layout
– Use Multi-Dimensional Clustering (MDC) for BW
Perform the BW migration as usual, including execution of the BW post migration
report RS_BW_POST_MIGRATION.
You can then use DB6CONV to convert tables.
Migration to Column-organized Tables (I)
Upgrade the source system to the required Support Package level for DB2 BLU
Acceleration:
SAP BASIS Support SAP BW Support
SAP BW Release Technical Name
Package Package
7.00 31 33 SAPKW70033
7.01 (SAP BW 7.0 EHP 1) 16 16 SAPKW70116
7.02 (SAP BW 7.0 EHP 2) 16 16 SAPKW70216
7.11 14 14 SAPKW71114
7.30 12 12 SAPKW73012
7.31 (SAP BW 7.3 EHP 1) 13 13 SAPKW73113
7.40 7 7 SAPKW74007
Migration to Column-organized Tables (II)
SAP BW Fact Table: organized by row SAP BW Fact Table: organized by column
CREATE TABLE "/BIC/EZSD_C15" ( CREATE TABLE "/BIC/EZSD_C15" (
"KEY_ZSD_C15P" INTEGER NOT NULL WITH DEFAULT 0 , "KEY_ZSD_C15P" INTEGER NOT NULL WITH DEFAULT 0 ,
"KEY_ZSD_C15T" INTEGER NOT NULL WITH DEFAULT 0 , "KEY_ZSD_C15T" INTEGER NOT NULL WITH DEFAULT 0 ,
"KEY_ZSD_C15U" INTEGER NOT NULL WITH DEFAULT 0 , "KEY_ZSD_C15U" INTEGER NOT NULL WITH DEFAULT 0 ,
… …
"KEY_ZSD_C15C" INTEGER NOT NULL WITH DEFAULT 0 , "KEY_ZSD_C15C" INTEGER NOT NULL WITH DEFAULT 0 ,
"KNVAL" DECIMAL(17,2) NOT NULL WITH DEFAULT 0 , "KNVAL" DECIMAL(17,2) NOT NULL WITH DEFAULT 0 ,
"KPRICE" DECIMAL(17,2) NOT NULL WITH DEFAULT 0 , "KPRICE" DECIMAL(17,2) NOT NULL WITH DEFAULT 0 ,
…) …)
COMPRESS YES ADAPTIVE
DISTRIBUTE BY HASH("KEY_ZSD_C151", DISTRIBUTE BY HASH("KEY_ZSD_C151",
"KEY_ZSD_C152", "KEY_ZSD_C152",
… …
"KEY_ZSD_C15U") "KEY_ZSD_C15U")
IN "BWP#FACTD" INDEX IN "BWP#FACTI" IN "BWP#FACTD" INDEX IN "BWP#FACTI"
ORGANIZE BY ROW; ORGANIZE BY COLUMN;
CREATE INDEX "/BIC/EZSD_C15~020" ON "/BIC/EZSD_C15“
CREATE INDEX "/BIC/EZSD_C15~040" ON "/BIC/EZSD_C15“ No indexes created on BLU Table
CREATE INDEX "/BIC/EZSD_C15~050" ON "/BIC/EZSD_C15“
...
CREATE UNIQUE INDEX "/BIC/EZSD_C15~P" ON "/BIC/EZSD_C15" ALTER TABLE SAPBWP."/BIC/EZSD_C15“
("KEY_ZSD_C15T" ASC, ADD CONSTRAINT "/BIC/EZSD_C15~P" UNIQUE
"KEY_ZSD_C151" ASC, ("KEY_ZSD_C15T" ASC,
… "KEY_ZSD_C151" ASC,
"KEY_ZSD_C15P" ASC) …
PCTFREE 0 CLUSTER "KEY_ZSD_C15P" ASC);
COMPRESS YES
INCLUDE NULL KEYS ALLOW REVERSE SCANS;
DBA Cockpit – Time Spent Analysis
DBA Cockpit – Single Table Analysis
DBA Cockpit – New Performance Metrics for DB2 BLU
OPTGUIDELINE: Degree Value = ANY
– For non-BW statements (for example, an ABAP SELECT SINGLE) it is usually better
not to use intra-partition parallelism – therefore the query degree for these statements
remains 1 (no parallelism).
Sort Monitoring
If you followed the previous best practices, you are off to a great start.
To keep your installation running smoothly here are some important hints & tips to
be aware of:
– Sort memory: The SORTHEAP is used in BLU not only for sorting, but also for
important run-time operations (such as hash join and group by).
– You can access several monitoring elements to understand if your
SORTHEAP configuration is healthy. An “Overflow” indicates a query
performed a run-time operation such as sort, hash join, group by, and required
more memory than what was available in the sort heap.
– “Post threshold” operations indicate operations that did not receive all
requested SORTHEAP memory, due to consumption by other concurrent
operations.
Sort Monitoring – Monitor Elements
Sort ACTIVE_SORTS,
TOTAL_SORTS,
SORT_OVERFLOWS,
POST_SHRTHRESHOLD_SORTS
Join ACTIVE_HASH_JOINS,
TOTAL_HASH_JOINS,
HASH_JOIN_OVERFLOWS,
POST_SHRTHRESHOLD_HASH_JOINS
Group By ACTIVE_HASH_GRPBYS,
TOTAL_HASH_GRPBYS,
HASH_GRPBY_OVERFLOWS,
POST_THRESHOLD_HASH_GRPBYS
Monitoring Space Consumption
Upgrade your SAP BW 7.40 system to SAP Basis and SAP BW Support Package
8 or higher.
Install the following additional SAP notes on top of the Support Packages: SAP
Note 1969500, SAP Note 2020190, SAP Note 2035816, SAP Note 2061394, SAP
Note 2038632, SAP Note 2075280.
Flat InfoCubes (III)
Since SAP BW 7.40 SP10 you can create flat InfoCubes directly in transaction
RSA1.
– With SAP BW 7.40 SP8 and SP9, you cannot create flat InfoCubes directly in
transaction RSA1. Instead you must create a standard star schema InfoCube
and convert it to flat with report RSDU_REPART_UI.
If you want to create a semantically partitioned flat InfoCube you first create a
standard star schema InfoCube and then convert all the part providers, including
part 0, to flat with report RSDU_REPART_UI.
You also use report RSDU_REPART_UI to convert existing standard star schema
InfoCubes to flat or existing flat InfoCubes back to star schema.
Flat InfoCubes (IV)
A new version of SAP BW on DB2 for LUW Administration Guide with detailed
information about how to deploy and use BLU is available on the SAP service
marketplace.
A new version of SAP DB2 NearLine Storage Administration Guide is also
available.
Two IBM Redbooks available for DB2 BLU
with SAP:
– Architecting and Deploying DB2 with
BLU Acceleration
– Implementation Best Practices for DB2 BLU
Acceleration with SAP BW on
IBM POWER Systems (draft)
Thank You