DB2 LUW V11.
1 CERTIFICATION TRAINING PART #2
Kent Collins @rcollins963
Mohankumar Saraswatipura @mohankumarsp
www.EnterpriseDB2.com
Agenda
BLU Acceleration Best Practices
New Monitoring Table Function
DB2 pureScale Enhancements
Sample Questions
DB2 BLU Acceleration
Data Load Best Practices
Column-Organized Table Row-Organized Table
CREATE TABLE "MS"."T3" ( CREATE TABLE "MS"."T4" (
"ID" INTEGER NOT NULL, "ID" INTEGER NOT NULL,
"NAME" VARCHAR(20 OCTETS) , "NAME" VARCHAR(20 OCTETS) ,
"ADDRESS" VARCHAR(50 OCTETS) , "ADDRESS" VARCHAR(50 OCTETS) ,
"SALARY" INT ) "SALARY" INT )
DISTRIBUTE BY HASH("ID") COMPRESS YES ADAPTIVE
IN "USERSPACE1" DISTRIBUTE BY HASH("ID")
ORGANIZE BY COLUMN; IN "USERSPACE1"
ORGANIZE BY ROW;
Records Loaded: 62,914,560 Records Loaded: 62,914,560
DB2 BLU Acceleration… Cont.
Synopsis Table
db2 "describe table SYSIBM.SYN170323092546645613_T3"
Data type Column
Column name schema Data type name Length Scale Nulls
------------------------------- --------- ------------------- ---------- ----- -----
IDMIN SYSIBM INTEGER 4 0 NO
IDMAX SYSIBM INTEGER 4 0 NO
NAMEMIN SYSIBM VARCHAR 20 0 Yes
NAMEMAX SYSIBM VARCHAR 20 0 Yes
ADDRESSMIN SYSIBM VARCHAR 50 0 Yes
ADDRESSMAX SYSIBM VARCHAR 50 0 Yes
SALARYMIN SYSIBM INT 8 0 Yes
SALARYMAX SYSIBM INT 8 0 Yes
TSNMIN SYSIBM BIGINT 8 0 No
TSNMAX SYSIBM BIGINT 8 0 No
10 record(s) selected.
DB2 BLU Acceleration… Cont.
Synopsis Table
This is a meta-data describing which ranges of values exist in which portion of the user table.
• This enables data skipping when scanning data to fulfill a query request
• Pre-sorted data
• It is managed and maintained by DB2 automatically and transparently to the users
• It is a column-organized table and is compressed
• Generally 0.1 or 0.2% of the base table
For a table having 62,914,560 records, what will the synopsis table records be around?
61K ~ (62M/1024)
db2 "SELECT COUNT(*) FROM SYSIBM.SYN170323092546645613_T3"
1
-----------
61512
DB2 BLU Acceleration… Cont.
ALTER TABLE MS.T3 ADD CONSTRAINT PK_T3 PRIMARY KEY (ID) ENFORCED;
db2 "SELECT SUBSTR(TABNAME,1,30) as TABNAME, SUBSTR(INDNAME,1,30) as INDNAME,
INDEXTYPE, COMPRESSION from SYSCAT.INDEXES where TABNAME like '%T3%'"
TABNAME INDNAME INDEXTYPE COMPRESSION
------------------------------ ------------------------------ --------- ----
T3 SQL170324061319724733 CPMA Y
SYN170324061319264040_T3 SQL170324061319951469 CPMA Y
T3 PK_T3 REG Y
3 record(s) selected.
CPMA: Column Page Map Index
Data Load Best Practices for BLU
• Batch up the data at least 1024 records per insert
• Set UTIL_HEAP_SZ to AUTOMATIC and over 1M pages
• Consider presorting data on column(s) that will frequently appear in query predicates to
improve compression ratios and synopsis effectiveness
• Reduce total time required for the first load of the table by reducing the duration of the
ANALYZE phase. This can be achieved by cdeanalyzefrequency and/or maxanalyzesize file
type modifier in the LOAD command.
• cdeanalyzefrequency: How frequently sample needs to be collected during ANALYZE
phase?
• maxanalyzesize: How much data is sampled in the ANALYZE phase? Default is 128 GB
and can be modified. Value ZERO means unlimited.
Data Load Best Practices for BLU… Cont.
LOAD INGEST IMPORT
Offline, parallel, bulk and non- Online, parallel, transactional Online, single threaded logged
logged operation logged operation operation
SETUP, ANALYZE, LOAD and Process a continuous data Serialized INSERT operation and
BUILD stream and perform an array pretty slow
insert
Fast and yields best Fast, however it is Generally not a recommended
compression recommended to use LOAD for way to load the data for BLU
the initial data load to have best
compression results and
subsequent data load can use
INGEST
Data Load Best Practices for BLU
LOAD Phases for BLU
Source ANALYZE LOAD BUILD BLU
• Analyze: This occurs only if dictionary creation is necessary. When necessary it builds
histogram to track value frequency and column compression dictionaries.
• Load: Compress data pages, update synopsis table
• Build: Builds key for UNIQUE indexes
Data Load Comparison
Column-Organized LOAD INGEST IMPORT
1ST Batch (12 M) 27 seconds (Analyze 16 21 seconds 14 minutes
sec, Load 10 sec, Build 1
sec)
2nd Batch (12 M) 14 seconds (Load 13 21 seconds 14 minutes
sec, build 1 sec)
Row-Organized LOAD INGEST IMPORT
1ST Batch (12 M) 45 seconds 22 seconds 14 minutes
2nd Batch (12 M) 43 seconds 20 seconds 14 minutes
Table Size Comparison
SELECT
SUBSTR(A.TABNAME,1,10 ) as TABNAME,
ROWCOMPMODE,
(DATA_OBJECT_P_SIZE + INDEX_OBJECT_P_SIZE +
COL_OBJECT_P_SIZE)/1024 as TOTAL_SIZE_MB
from SYSCAT.TABLES a, SYSIBMADM.ADMINTABINFO b
where
A.TABNAME='T3' AND Table Records Size
A.TABSCHEMA='MS' AND
MS.T3 (Column-organized) 62,914,560 47 MB
A.TABSCHEMA = B.TABSCHEMA AND
A.TABNAME=B.TABNAME WITH UR; MS.T4 (Row-organized) 62,914,560 843 MB
TABNAME ROWCOMPMODE TOTAL_SIZE_MB
---------- ----------- --------------------
T3 47
T4 A 843.375
Execution Time Comparison
Table SQL Statement Time
MS.T3 (Column-organized) SELECT COUNT (*) 0.033793 sec
MS.T4 (Row-organized) SELECT COUNT (*) 16.973138 sec
Table SQL Statement Time
MS.T3 (Column-organized) SELECT * 174.598066 sec
MS.T4 (Row-organized) SELECT * 166.198395 sec
Analytical SQL Statement Performance Assessment
BLU In-Memory
Overall Time: 155.60 Seconds 170.68 Seconds
13
New Monitoring Table Function
Lock Latch
Locks assure logical consistency of the data in a Latch is a lock at the internal memory structure
table via record level or table level locks. consistency of the object in memory
Held till the logical completion of the transaction. This will only be held at the page level for a short
term.
Monitoring via SYSPROC.MON_GET_LOCKS Monitoring via SYSPROC.MON_GET_LATCH
SYSPROC.MON_GET_LATCH
SELECT SUBSTR(LATCH_NAME,1,40) LATCH_NAME,
SUBSTR(MEMORY_ADDRESS,1,20) MEMORY_ADDRESS,
EDU_ID,
SUBSTR(EDU_NAME,1,20) EDU_NAME,
APPLICATION_HANDLE,
LATCH_STATUS,
LATCH_WAIT_TIME FROM TABLE ( MON_GET_LATCH( NULL, -2 ) )
ORDER BY LATCH_NAME, LATCH_STATUS;
SYSPROC.MON_GET_LATCH
MON_GET_LATCH (<search_args>,<member>)
• application_handle
• latch_name
• edu_id
• latch_status
PureScale Enhancements
Packaging
New Direct Advanced Edition
Passive/Active Licensing
Improvements in Installation and Deployment
Install Complexity Reduced
Smarter Defaults
GPFS Replication step reduction from 30 to 4
db2cluster command enhancements
Install recovery/restart logic
Improved Documentation
Improved Buffer Management
Truncate, Drop Table, Load/Import and Ingest speed increases
PureScale Enhancements - Continues
Enhanced pre-checking script
Improved Member Set Alternate Member Failover Priority
Linux Virtual Enhancements
Single-Root I/O Virtualization
Shared Single Adapter across VMWare Partitions
HADR support for Sync and NearSync Mode
HADR Streamline Upgrade Support
Crash Recovery Time improvements
Allow logs written during crash recovery to be replayed
GDPC Enhancements
Multiple adapter ports per member
Dual Switches
BSSL – (Bold, Simple, Stable, Limited) - HADR
No SPOF: DB2 Version Upgrade Outage
Power8 DB2INST1:DB1:55000 Power8 DB2INST1:DB1:55000
CFs CFs
40G Ethernet with RDMA 40G Ethernet with RDMA
P HADR S
10G Ethernet IN/OUT 10G Ethernet IN/OUT
Members Members
Flash900 TSM Flash900 TSM
Physical Servers Physical Servers
TCNL – (Tight, Complex, Network, Limited) - GDPC
CF Members
TieBreaker Site 1 Storage
Site 3 Replication
Site 2
CF Members
SAAU(Simple, Always Available, Unlimited)- CDC/QRep
No SPOF: No Scheduled Outage Requirement
Power8 DB2INST1:DB1:55000 Linux DB2INST2:DB2:55000
CFs CFs
40G Ethernet with RDMA 40G Ethernet with RDMA
SQL Replication
10G Ethernet IN/OUT 10G Ethernet IN/OUT
Members Members
Flash900 TSM HDD
Physical Servers Virtual Servers
PureScale Monitoring
Three Ways to get Information
Table Functions and Administrative Views
DB2_GET_CLUSTER_HOST_STATE Table Function
DB2_CLUSTER_HOST_STATE View
DB2_GET_INSTANCE_INFO Table Function
DB2_MEMBER View
DB2_CF View
DB2_INSTANCE_ALERTS View
CLP Commands
LIST INSTANCE
LIST INSTANCE SHOW DETAIL
LIST APPLICATIONS AT MEMBER <node>
LIST APPLICATIONS GLOBAL
PureScale Monitoring
Three Ways to get Information
Shell Commands
db2instance –list
db2cluster -list options
db2nps <node> db2_ps executes db2nps on each member
db2nkill <node> db2_kill Kills all engine processes on all member drastically
db2_all <command> Executes <command> on all logical nodes
db2cluster -list –alert
db2cluster -cm -verify –resources
db2cluster -cfs -verify -configuration
PureScale Index Space Reclaim
List reclaimable space for all indexes in schema XX
SELECT TRIM (indname) AS index_name,
TRIM (tabname) AS TABLE_NAME,
iid,
index_object_l_size,
index_object_p_size,
reclaimable_space
FROM TABLE (admin_get_index_info ('', ‘XX', NULL)) AS T1
ORDER BY reclaimable_space DESC
WITH UR@
INDEX_NAME TABLE_NAME INDEX_OBJECT_L_SIZE INDEX_OBJECT_P_SIZE RECLAIMABLE_SPACE
XPK TB1 1262208 1262208 901824
XD1 TB1 266880 266880 179776
PureScale Index Space Reclaim
Reorg Index ME.XPK with CLEANUP ALL AND Reclaim Extents
REORG index XX.XPK for table TB1 allow write access
CLEANUP ALL RECLAIM EXTENTS @
INDEX_NAME TABLE_NAME INDEX_OBJECT_L_SIZE INDEX_OBJECT_P_SIZE RECLAIMABLE_SPACE
XPK TB1 285184 285184 704
PureScale Index Space Reclaim
Run All Indexes for table TB6 CLEANUP ALL with Reclaim Extents
reorg indexes all for table TB6 allow write access
CLEANUP ALL RECLAIM EXTENTS @
INDEX_NAME TABLE_NAME INDEX_OBJECT_L_SIZE INDEX_OBJECT_P_SIZE RECLAIMABLE_SPACE
XD2 TB6 174464 174464 150272
XPK TB6 174464 174464 150272
INDEX_NAME TABLE_NAME INDEX_OBJECT_L_SIZE INDEX_OBJECT_P_SIZE RECLAIMABLE_SPACE
XD2 TB6 16384 16384 768
XPK TB6 16384 16384 768
Get all the space for a single index across data partitions
SELECT TRIM (indname) AS index_name,
TRIM (tabname) AS TABLE_NAME,
SUM(index_object_l_size) AS index_object_l_size,
SUM(index_object_p_size) AS index_object_p_size,
SUM(reclaimable_space) AS reclaimable_space
FROM TABLE (admin_get_index_info ('', ‘XX', '')) AS T1
GROUP BY indname, tabname
HAVING SUM(reclaimable_space) > 0
ORDER BY reclaimable_space DESC
WITH UR@
INDEX_NAME TABLE_NAME INDEX_OBJECT_L_SIZE INDEX_OBJECT_P_SIZE RECLAIMABLE_SPACE
XD1 TB7 149504 149504 103296
XPK TB7 149504 149504 103296
INDEX_NAME TABLE_NAME INDEX_OBJECT_L_SIZE INDEX_OBJECT_P_SIZE RECLAIMABLE_SPACE
XD1 TB7 41472 41472 128
XPK TB7 41472 41472 128
PureScale Tablespace Reclaimable Storage
SELECT varchar(tbsp_name, 30) as tbsp_name,
reclaimable_space_enabled,
tbsp_free_pages,
tbsp_page_top,
tbsp_usable_pages,
( tbsp_free_pages - (tbsp_usable_pages - tbsp_page_top)) as
freeSpaceBelowHWMinPages,
tbsp_page_size * ( tbsp_free_pages - (tbsp_usable_pages - tbsp_page_top))
as freeSpaceBelowHWMinBytes
FROM TABLE(MON_GET_TABLESPACE('',-2)) AS t
where ( tbsp_free_pages - (tbsp_usable_pages - tbsp_page_top)) > 0
ORDER BY freeSpaceBelowHWMinPages desc @
TBSPNM RECLAIMABLE_SPACE TBSP_FREE_PAGES TBSP_PAGE_TOP TBSP_USABLE_PAGES FREESPACEBELOWHWM FREESPACEBELOWHWM
TS1 1 2195424 1644160 2555776 1283808 5258477568
TS2 1 2195424 1644160 2555776 1283808 5258477568
PureScale Table CF Potential Conflict Report
select substr(tabname,1,55) as name
, substr(objtype,1,5) as type
, page_reclaims_x as PGRCX
, page_reclaims_s as PGRCS
, spacemappage_page_reclaims_x as SMPPGRCX
, spacemappage_page_reclaims_s as SMPPGRCS
from table( mon_get_page_access_info(‘XX', NULL,
NULL ) ) AS WAITMETRICS
order by page_reclaims_x DESC, page_reclaims_s DESC
with ur @
NAME TYPE PGRCX PGRCS SMPPGRCX SMPPGRCS
TB1 INDEX 60244 17 69 0
TB1 INDEX 38477 4 44 0
TB1 TABLE 12945 258 26 1
Thank you!
Robert (Kent) Collins Mohankumar Saraswatipura
@rollins963 @mohankumarsp
Kent, founder of Shiloh Consulting, Inc., is currently a Data Solutions
Mohan works as a Database Solutions Architect focusing on IBM DB2,
Architect with BNSF Railway. He is an IBM Champion (2010–2017)
Linux, UNIX, and Windows solutions. Prior to his current position, he
and a frequent speaker at the DB2Night Show and IDUG and IOD
worked as a Database Solutions Architect at Reckitt Benckiser Group,
(IBM Insight) conferences. Kent has worked continually with DB2
plc (UK) focusing on IBM Smart Analytics System 5600, Siebel, SQL
from its introduction to the market in 1984, amassing a wealth of
Server, and SAP HANA solutions. He is an IBM Champion (2010–2017)
knowledge and experience. He graduated from the University of
and a DB2's Got Talent 2013 winner. Mohan has written dozens of
Texas at Dallas with majors in mathematics and computer science. He
technical papers for IBM developerWorks and IBM Data Magazine.
is an IBM Certified Solutions Expert and also holds certifications in
He is an IBM Certified DB2 Advanced Database Administrator, DB2
DB2, AIX, Linux, .NET, Oracle, SQL Server, Windows, and z/OS. Kent is
Application Developer, and DB2 Problem Determination Master.
proficient in many programming languages and, as a Java Architect,
Mohan holds a Master's of Technology (M Tech) degree in computer
specializes in Enterprise HA/Performance systems. He lives in Dallas
science and an Executive MBA (IT).
with his wife, Vicki; together, they have three children and one
grandchild.
Please contact us if you have any questions @ www.EnterpriseDB2.com
