CROSS-PLATFORM TRANSPORTABLE TABLESPACES UTILIZING INCREMENTAL BACKUPS Centric Consulting

Enterprise Application Solutions

(EAS) National Practice

How To Migrate A 200+ TB Database

In Under A Day Using Oracle’s Cross-Platform
Transportable Tablespaces Utilizing RMAN
Incremental Roll-Forward Backups

AUTHOR: Jay Barnhart

Published: September 11, 2017 | Centric Consulting - EAS

CROSS-PLATFORM TRANSPORTABLE TABLESPACES UTILIZING INCREMENTAL BACKUPS Centric Consulting
Enterprise Application Solutions
(EAS) National Practice

Contents

AUDIENCE ---------------------------- 1

BACKGROUND ---------------------------- 1

OBJECTIVE ---------------------------- 2

THE PROCESS ---------------------------- 3

SETUP ---------------------------- 4
Software ----------------- 4
Initial Configuration ----------------- 5
Network ----------------- 5
NFS Shares ----------------- 6
AIX Source Side ----------------- 6
ZFS Appliance ----------------- 7
Exadata Destination Side ----------------- 7
Configuration ----------------- 7
XTTDRIVER.PL Modification ----------------- 8
Directory Setup ----------------- 8
Custom Scripts Created ----------------- 8

THE EXECUTION ---------------------------- 13

THE ISSUES FOUND ---------------------------- 15

SUMMARY ---------------------------- 16

REFERENCES ---------------------------- 16

ABOUT CENTRIC CONSULTING ---------------------------- 17

Centric at a Glance ---------------- 17
EAS Practice Overview ---------------- 17
Contact Information ---------------- 17
 We will be migrating this to an Oracle Exadata
AUDIENCE
X5 four node cluster environment that is
This document is intended for review and as running Oracle Enterprise Linux (OEL).
reference material for Oracle DBA’s and
Oracle subject-matter experts on Oracle Because AIX uses a big-endian and OEL uses a
database migrations. It is intended to help little-endian format there are limited options
organizations learn more about how to migrate for a smooth migration without writing a bunch
very large databases utilizing Oracles Cross of custom code. The first, would be utilizing
Platform Transportable Tablespace (XTTS) Oracle Goldengate (OGG). This options allows
utility and how to customize it. for almost real-time cut-over capabilities.

BACKGROUND However, during our proto-typing of this

option we found that the amount of redo
Migrating a very large database that is mission being generated was too extensive for OGG to
critical to your clients can be a daunting and keep up with. In our discussions with Oracle’s
risky endeavor. Many clients cannot afford for product development team on this issue, they
these mission critical applications to be offline stated that 800 gigabytes per hour is their
for an extended period of time. current maximum throughput. In our client’s
environment, we were routinely seeing over
Therefore, developing a solution that allows for one terabyte per hour redo log generation.
minimum downtime is critical for these type of
applications. The next option, would be to utilize Oracle’s
Cross Platform Transportable Tablespaces
The issue is further complicated when (XTTS). This allows for only a READ-ONLY
migrating between two different OS outage that is dependent on the number of
environments, and/or database versions. objects migrating and amount of data required
In our scenario, we are going to be migrating an to migrate in the final backups and conversions.
11.2.0.3 Oracle Enterprise Edition (EE) In our migration, this become the approach of
database to Oracle EE 12.1.0.2. choice, and we will talk later about how quickly
The database currently resides on an AIX we were able to do this migration.
Power 8 Series machine running in an LPAR.

Cross-Platform Backup & Restore

Uses Data Pump for metadata, backup sets for data files

Source Database Destination Database

(AIX) (Linux)
EXADATA

Extreme Performance for Database Clouds

Backup to Disk / Tape Restore Backup

1
 Other options would include things like data We were utilizing 18 cores in each of the four
pump, ETL tools, etc, but these have a much compute nodes, which provided 36 vcpu’s per
larger downtime requirement and are much compute node for a total of 144 vcpu’s for the
more customized for each implementation. entire RAC environment. In addition, our
We wanted to develop an approach that was Exadata was connected to an Oracle ZFS ZS4
much more universal and could be re-used at appliance which stored all RMAN backups.
other clients in the future.
This appliance has usable storage of 1.2 PBs.
Therefore, we settled on using Oracle’s XTTS The two appliances are connected using
utility with RMAN incremental roll-forward standard Infini-band network connectivity.
backups as our migration approach.
Oracle ZFS Storage ZS4-4
We chose to use the RMAN transfer option as
our DBAs were more familiar with that ZS4-4 Increase

approach. However, the dbms_file transfer Generation 4th

option is another method and according to the CPU cores 120 50%
documentation has less steps required. We DRAM 3TB 50%
plan to attempt to try this method in the near Read & Write SSD Cache 2x IOPS 100%
future to compare and contrast the benefits Infiniband PCIe Gen3 >70%
and drawbacks of each. PCIe Gen 3 2x Bandwidth

System Performance 2x Double vs ZS3-4

The current database is in excess of 200 TBs in
size and has a very large user community Check out our architectural diagram, later in
(Over ten thousand user). This database is this paper.
active 24x7 365 days a year and supports
critical decision making activities within our OBJECTIVE
client’s business. This database is extremely
important for the day to day operations of their The objective was to be able to migrate our
business to function properly. client’s existing enterprise data warehouse
from AIX to Exadata over a weekend, while
Our source hardware was an AIX Power 8 leaving the existing AIX environment in
running an LPAR that had 200 vcpu’s allocated READ-ONLY mode the entire time, so that the
to that LPAR. In addition, this LPAR has application was not totally down.
physical access to two 10 gb/s NIC cards for
network traffic. This, also, is a single instance In addition, we would be upgrading the
database environment. database from its existing 11.2.0.3 version to
12.1.0.2. Additionally, the objective was to
Our destination hardware was an Oracle increase the overall performance and
Exadata X5 four node cluster configuration. availability of their enterprise data warehouse.

2
 You will then run subsequent RMAN
THE PROCESS
incremental backups on the source system,
Since the process that we ended up selecting until you are ready for a READ-ONLY window
is Oracle’s XTTS utility, let’s take a look at how to occur. Each RMAN incremental is also
Since the process that we ended up selecting is converted and then rolled-forward into each
Oracle’s XTTS utility, let’s take a look at how this data file on the destination.
process works.
Once you are ready for your READ-ONLY
The XTTS process allows for you to migrate the window, all tablespaces that are being
data that you are moving ahead of the actual migrated must be placed in READ-ONLY mode.
“Cut-Over” process. This is accomplished by Then a final RMAN incremental backup is
utilizing RMAN incremental and roll-forward taken on the source system. This incremental
processing that will “roll forward” your backup is converted and rolled-forward on the
transferred data files when each incremental destination, bringing the data files up to date
backup. Changes are tracked through the with the source side. At this point, you have
use of the database SCN process. With each the same data files on the source as on the
incremental being backed up until a particular destination.
SCN. We will reference that aspect later on in
the document.
However, you still need to “hook” everything up.
In a nutshell, we are using the Oracle provided That is where the Oracle Data Pump
xttdriver.pl, which is a Perl script, to execute
Transportable Tablespace process comes into play.
RMAN incremental backup scripts based on
where you are at in the process. The initial
RMAN backup is really an RMAN level 0.
This makes an image copy of the data files from There are a couple of ways you can do this.
your source system. These data files will then You need the meta-data about the tables and
be converted using the RMAN convert process tablespaces that make up the structures that
on your destination system. are currently in your data files. You could
export this meta-data and output it to a dump
This makes the data files compatible with the file, and then import that meta-data into your
OEL operating system that is used on the destination system. Or, you can use a database
Oracle Exadata platform, after an RMAN link and read the information directly from the
convert process is run. If you are using another source system and import it into the
type of Operating system, they would be destination system. In our case, we choose the
converted to that type. We will discuss how to latter approach, because we had issues getting
set that up later in this document. the former to consistently work. This is
covered in “THE ISSUES FOUND” section
below.

3
 Once the meta-data is imported into the On this project, we utilized version 2 of the
destination system, the “hook-up” is complete, utility, since that was the latest version at the
and you can now view all of the table time our project began. However, a newer
structures, definitions, etc of your database. version is now available, version 3. This newer
The XTTS process only brings over the objects version of the utility has many additional
that are relevant to the tablespaces and tables features developed by Oracle in conjunction
in the tablespaces migrated. The SYSTEM with our project. We recommend that you
tablespace cannot be migrated as part of the check for additional releases as new
XTTS process. Therefore, roles, grants, functionality may be released at any time.
profiles, views, pl/sql, sequences, triggers, etc
are not brought over. In the version 2, you could not add a tablespace
or data file once the initial RMAN level 0
Due to some meta-data information not being backup image copy started. Therefore, if you
brought over, you will need to do a full went a week or more before finally entering
meta-data export on the source side and into the ‘READ-ONLY’ window, you might have
import that data into the destination. This some issues with space becoming a problem.
ensures that all objects are transferred to the In the new version, you can add tablespaces
destination system. and/or data files any time during the process.

SETUP Once you unzip the XTTS utility, you will find
the following files:

Software

As part of MOS note DOC ID 2005729.1,

there is an attachment that includes the zipped
scripts necessary to run the XTTS process. You
will need to download these zip files and unzip
on the source system and into a directory that
you plan to use for the migration.

The xttdriver.pl script is the master script that

is called to execute the RMAN level 0, RMAN
incremental, convert process, and roll-forward
processing.

It will also create the par file that can be used

for importing into your destination system.
We ended up creating our own, but more on
that later.

4
 The xtt.properties file holds all of the Once all data files have had an image copy
configuration information about your made for that tablespace, it will move onto the
environment: what data you want to migrate, next tablespace in the list.
where the files will be placed, and what RMAN
configuration options you want to utilize. In our specific case, the delivered process was not
fast enough to meet our outage timeframe.
Once the initial steps have been run on the
source, you will need to copy this directory to Our client had over 530 tablespaces with over
the destination, since much of this information four thousand data files that needed to have an
will be used there as well. Again, more on that image copy made. Within our client’s network,
later in this document. we were getting less than 100 MBs per second
throughput with a single xttdriver Perl script
Initial Configuration running. That would equate to over 23 days to
create image copies of a 200 TBs database.

Normally, if you were going to use the

Obviously, this timeframe far exceeded our
xttdriver.pl and xtt.properties files exactly as
client’s requirements. To bring the timeframe
they are delivered, you would simply include
down, we modified how the delivered code
your tablespaces in the xtt.properties file and
process would make image copies. More de-
you would be ready to begin the RMAN level 0
tails on these changes can be found in section
process.
“The Process” below.

However, the xttdriver Perl script works by

reading the tablespaces section of the config Network
file. It then will process one tablespace at a
time. Depending on how many parallel In order to ensure that the data can be
processes you defined in the config file, this will transferred as easily as possible, we utilized an
determine how many data files can be copied Oracle ZFS appliance as a NAS device.
concurrently. In other words, if you have ten
tablespaces listed in the TABLESPACES This allowed us to NFS mount shares from the
section. The xttdriver will read the first ZFS appliance onto the AIX LPAR so the files
tablespace name. could be written directly to the ZFS appliance.
With this configuration, the Exadata appliance
It will then look at the PARALLEL section of the could read from these same shares and process
config file to determine how many channels the files without having to copy files from one
to open. If you have PARALLEL=2, then it will server to another.
open two channels for the PREPARE PHASE
(ie – RMAN level 0). So two data files will be
read concurrently.

5
 Below is a diagram of our network environment.

We asked that the Exadata and ZFS appliance

NFS Shares
be plugged directly into the same Cisco
Enterprise switch as the AIX Power 8 machine.
This ensures that no network hops are AIX Source Side
necessary. In addition, we ensured that the
two NICs configured in the AIX machines were We mounted the NFS shares to the AIX
bonded, so that we could get up to 20 gbs environment utilizing the following mount
throughput. commands:

We did not enable Jumbo frames, but if all ldcgrp01 /export/PROD/NFSAIX1B /export/
of the network components in your path are PROD/NFSAIX1B nfs3 dio,bg,hard,intr,r-
rated for Jumbo frames capabilities, this could size=65536,wsize=65536,timeo=600
potentially add additional network bandwidth.
The DIO option was critical for us. Without
We wanted to ensure the fastest throughput it, we had significant performance issues with
possible due to the size of the database we these mounts on the AIX box.
were attempting to migrate. Having the ability
to re-run the process was very important to In addition, with a ZFS appliance, you need to
us. If it took a very long time to get the data ensure that the write bias parameter is set to
pushed, the ability to re-run the process would LATENCY for writing data from the AIX side to
be delayed. the ZFS appliance. Again, we saw significant
performance issues if that option was not set.

6
 ZFS Appliance Although during the XTTS process we do not
write from Exadata back to the ZFS appliance,
We followed Oracles best practices for setting this setting is critical if you use the ZFS
up our ZFS shares and projects, see the appliance for RMAN backups from Exadata.
“REFERENCE” section below.
There are various dNFS views that you can
We created two ZFS pools as destinations for query to ensure that your database is actually
RMAN backup pieces. These would initially be using the dNFS service when processing
used for the XTTS processing, but later would backup pieces. A couple of options is to
be utilized as the destination for the Exadata monitor the gv$dnfs_stats view and/or setup
RMAN backups. These pools were created a worksheets under the ZFS Analytics tab.
on each of the array controller’s setup on the Either will show you the throughput being
ZFS appliance. This ensured that we can read generated by ZFS across the dNFS utilized
and write utilizing both array controllers for shares.
maximum performance. In addition, you need
to ensure that you have sufficient number of Configuration
disks allocated to the pool for maximum i/o
operations (IOPS).
As we stated above, in order to get the
required performance for our migration, we
As is standard practice, our ZFS controllers are
modified the way that the XTTS process works.
set to cross-failover so if one controller goes
down, the shares are automatically mounted
In our case, we decided to create forty (40)
on the other controller.
individual XTT directories. Each would hold
a complete copy of the XTTS utility that we
Exadata Destination Side
unzipped from Oracle. This would allow us to
break up our 530+ tablespaces into forty
We mounted these NFS shares utilizing Oracle processing groups. We would have
best practices for an Exadata environment. PARALLEL=2 set in the xtt.properties file for
We are utilizing Direct NFS on the Exadata, each directory, so we would get eighty data
but we still setup the fstab and the following files being written concurrently.
mount options:

In our original testing, we were only able to

rw,hard,rsize=1048576,w- achieve about 100 MBs per second with the
size=1048576,tcp,nfsvers=4,timeo=600 delivered code running a single xttdriver.
pl script. By creating forty directories, we
When writing from the Exadata to the ZFS, achieved throughput or roughly 800 MBs per
you need to ensure that the ZFS share has the second. This equates to around 2.9 TBs per hr
write bias set to THROUGHPUT. throughput.

7
 To complete the modified process, the Next, we created a custom Perl script that
following modifications and enhancements to would read this spool file and pull out the
the existing process were required: tablespace string and put it in the
corresponding xtt.properties files for each
directory.
XXTDRIVER.PL Modification

By scripting this process, we could dynamically

The Perl script attempts to shut down and
update the xtt.properties files before each run
startup the database in NOMOUNT mode,
in case new tablespaces were added or they
utilizing the xttdbopen.sql script. Since we will
changed in size dramatically from the previous
be running forty xttdriver.pl scripts the
run. Overall, the process allowed us to have a
database cannot be shut down and started up
load balanced forty directories so that we got
as it will kill processes that are already running.
maximum throughput for the entire RMAN
To solve the problem, we commented out this
level 0 run.
step in the code. We changed this in the XTT1
directory so that when all directories were
copied, this change was replicated as part of As a safety precaution, we then created a
that process. number of cross-check scripts that ensured
that all of the tablespaces in the source
database had a corresponding tablespace entry
Directory Setup
in one of the forty xtt.properties files.

Initially, we created a single directory called

Finally, we ensured that each odd number XTT
/home/oracle/xtt1. We then copied this
directory was writing to the NFS share
directory to /home/oracle/xtt2 thru xtt40.
pointing to array controller 1 on our ZFS
appliance. All even number XTT
Custom Scripts Created directories were written to the NFS share
pointing to array controller 2 on our ZFS
Next, in order to dynamically create the appliance. This further ensured we could
necessary tablespaces for each of the forty maximize our throughput by utilizing both
XTT directories, a number of custom scripts control heads.
and processes had to be created.
Here is an example of the key sections in the
First, we created a PL/SQL package that would XTT.PROPERTIES files for XTT1 and XTT2.
create forty groups of tablespaces that were
of about equal size. These tablespace groups
were spooled to a text file. This process would
be run on the source side.

8
 XTT1 – XTT.PROPERTIES Key Values

Section Value Notes

Tablespace Abc_tblspc1, xyz_tablspc2

Platformid 6 Indicates AIX OS

This is the source side location that the Prepare Phase

Dfcopydir /zfssa/PROD/backup1/xtt
(RMAN Level 0) will write the data file image copies to

This is the source side location that the incremental backups

Backupformat /zfssa/PROD/backup1/xtt
(RMAN level 1) will write the backup pieces to

This is the destination location that will be used by RMAN

Stageondest /zfssa/PROD/backup1/xtt
convert and the roll-forward processing

This is the location of the storage to be used on the

Storageondest +DATAC1/EDWSTG/DATAFILE destination for the RMAN convert process.
This would be our SAN luns or ASM.

This is the location of the storage on the destination server

Backupondest +DATAC1/EDWSTG/DATAFILE that the incremental backup pieces will be applied to the
data file images. This would be your SAN luns or ASM.

This is the number of channels that are opened by RMAN

Parallel 2
during the Prepare Phase. The default value is 8.

For the Incremental processing on the source side, you must

set the parallel value to be used inside of RMAN.
(ie – DEVICE TYPE DISK PARALLELISM 2)

This is the number of channels that are opened during the

Rollparallel 2 roll-forward phase on the destination side.
The default value is 0.

9
 XTT2 – XTT.PROPERTIES Key Values

Section Value Notes

Tablespace CDE_tblspc1, RST_tablspc2

Platformid 6 Indicates AIX OS

This is the source side location that the Prepare Phase

Dfcopydir /zfssa/PROD/backup2/xtt
(RMAN Level 0) will write the data file image copies to

This is the source side location that the incremental backups

Backupformat /zfssa/PROD/backup2/xtt
(RMAN level 1) will write the backup pieces to

This is the destination location that will be used by RMAN

Stageondest /zfssa/PROD/backup2/xtt
convert and the roll-forward processing

This is the location of the storage to be used on the

Storageondest +DATAC1/EDWSTG/DATAFILE destination for the RMAN convert process.
This would be our SAN luns or ASM.

This is the location of the storage on the destination server

Backupondest +DATAC1/EDWSTG/DATAFILE that the incremental backup pieces will be applied to the
data file images. This would be your SAN luns or ASM.

This is the number of channels that are opened by RMAN

Parallel 2
during the Prepare Phase. The default value is 8.

For the Incremental processing on the source side, you must

set the parallel value to be used inside of RMAN.
(ie – DEVICE TYPE DISK PARALLELISM 2)

This is the number of channels that are opened during the

Rollparallel 2 roll-forward phase on the destination side.
The default value is 0.

10
 We also created custom scripts to run the Xttbi.sh » This script sets up the env variables
various steps of the XTTS process. Since we and runs the xttdriver.pl for the RMAN
would be running forty concurrent jobs, Incremental step
instead of just one.

Export ORACLE_HOME=/u01/app/oracle/11.2/
Below are all of the source side scripts that 11.2.0.3
were used to run the xttdriver.pl script options. Export ORACLE_SID=EDWPRO
Export TMPDIR=/home/edwpro/$1
RUNALL_B0.sh » This script runs the forty Cd $TMPDIR
RMAN Prepare Phase jobs $ORACLE_HOME/Perl/bin/Perl xttdriver.pl –i -L

Date > runall.log RUNALL_SCN.sh » This script runs the forty

Nohup xttb0.sh xtt1 1>run1 2>&1 & RMAN SCN increment jobs
---
Nohup xttb0.sh xtt40 1>run40 2>&1 & Date > runall.log
Nohup xttscn.sh xtt1 1>run1 2>&1 &
Xttb0.sh » This script sets up the env variables ---
and runs the xttdriver.pl for the Prepare Phase Nohup xttscn.sh xtt40 1>run40 2>&1 &
step
Xttscn.sh » This script sets up the env variables
Export ORACLE_HOME=/u01/app/oracle/11.2/ and runs the xttdriver.pl for the RMAN SCN
11.2.0.3 Increment Step
Export ORACLE_SID=EDWPRO
Export TMPDIR=/home/edwpro/$1 Export ORACLE_HOME=/u01/app/oracle/11.2/
Cd $TMPDIR 11.2.0.3
$ORACLE_HOME/Perl/bin/Perl xttdriver.pl –p–L Export ORACLE_SID=EDWPRO
Export TMPDIR=/home/edwpro/$1
RUNALL_BI.sh » This script runs the forty RMAN Cd $TMPDIR
Incremental jobs $ORACLE_HOME/Perl/bin/Perl xttdriver.pl –s -L

Date > runall.log

Nohup xttbi.sh xtt1 1>run1 2>&1 &
---
Nohup xttbi.sh xtt40 1>run40 2>&1 &

11
 Below are all of the destination side scripts The roll-forward processing we found was
that were used to run the xttdriver.pl script done just fine by running on a single Exadata
options. node. So we did not break up that process.

On the Exadata side, we found that running the RUNALL_FORWARD.sh » This script runs the
RMAN convert on all four nodes was extremely forty RMAN convert and roll-forward jobs on a
efficient. To take advantage of this, we broke single node on target
up the jobs so that certain directories where
run on each node. We actually kicked off four Date > runall.log
separate shell scripts on each node: 16 RMAN Nohup xttforward.sh xtt1 1>run1 2>&1 &
converts running concurrently across the RAC. ---
This division allowed for us to convert over Nohup xttforward.sh xtt40 1>run40 2>&1 &
200 TBs of data in about 8 hours.
Xttforward.sh » This script sets up the env
Here is a sample of the shell script on the variables and runs the xttdriver.pl script to
destination side that we used for one of the convert and roll-forward each data file to make
sixteen convert jobs. There would be RMAN_ them up-to-date with the current incremental
CONV_PART1.sh thru RMAN_CONV_PART4. backup
sh on each compute node.
Export ORACLE_HOME=/u01/app/oracle/
RUNALL_CONV_PART1.sh » This script runs the product/12.1.0.2/dbhome_1
forty RMAN convert jobs Export ORACLE_SID=EDWSTG1
Export TMPDIR=/home/oracle/$1
nohup xttconv.sh xtt1 1>run1 2>&1 & Cd $TMPDIR
wait $ORACLE_HOME/Perl/bin/Perl xttdriver.pl –r
nohup xttconv.sh xtt5 1>run5 2>&1 & –L –d

Xttconv.sh » This set sets up the env variables

and runs the xttdriver.pl script to convert all of
the data files to the target OS file type

Export ORACLE_HOME=/u01/app/oracle/
product/12.1.0.2/dbhome_1
Export ORACLE_SID=EDWSTG1
Export TMPDIR=/home/oracle/$1
Cd $TMPDIR
$ORACLE_HOME/Perl/bin/Perl xttdriver.pl –c–L

12
 THE EXECUTION
Once everything was configured, we were ready to actually do the RMAN copy. The following is a
list of the steps that were executed to migrate the database utilizing the XTTS process.

Step Name Location Description

Kick off Prepare This kicks of 40 xttdriver.pl scripts that copy two
Source data files at a time. So we are getting 80 data files
Phase copied concurrently.

In our environment, this run for about 6 days and

transfers about 230 TBs of data

Copy Directories from Copy the contents of XTT1 thru XTT40 to Node 1
Source / Target on Exadata on Target
source to Node 1 Target

Copy Directories from Copy key directories from node 1 to node 2,3, and 4
Target as needed. This will allow for all four nodes to be
Node 1 to Nodes 2,3, and 4 used to do RMAN convert.

Kick off the RMAN convert jobs on all four nodes.

Kick off Convert Target Run for about 8 hours

REPEAT These steps are repeated until READ-ONLY window

Kick off Incremental Source This runs the incremental process on the source.

Copy key files from each XTT directory on source

Copy key files Source / Target to corresponding XTT directory on target

Kick off Convert / Kick off the RMAN convert and roll-forward on
Target node 1 for all 40 XTT directories
Roll-Forward

Kick off job to increment the SCN for next

Kick off SCN update Source incremental RMAN job.

13
 These steps occur when source tablespaces are
READ-ONLY in READ-ONLY mode

Place Tablespaces in Execute SQL script to place all tablespaces being

Source migrated in READ-ONLY mode
READ-ONLY mode

Kick off final

Source This runs the final incremental RMAN process
Incremental

Kick off full meta-data This run a data pump export job to get all meta-data
Source in the database, excludes tablespaces.
export

Run SQL script to dynamically create parameter

Create dynamic
Source needed for data pump import job for the
parameter file “Plug-In” step

Create users Target Load the roles, profiles, and users in target system.

Copy key files from each XTT directory on source

Copy Key Files Source / Target to corresponding XTT directory on target

Kick off Convert / Kick off the RMAN convert and roll-forward on node
Target 1 for all 40 XTT directories
Roll-Forward

Execute the dynamically created data pump import

parameter file. This plugs-in the tablespaces into the target.
Plug-In Tables Target We use a database link to accomplish this step. This runs
for about 9 hours.

Import missing meta-data information into

Import Meta-Data Target Target database to get all missing objects.

COMPLETE

The initial Prepare phase takes the longest to execute. Using our modified version of the XTTS
process, we are unable to add tablespaces or data files during this entire process. Therefore,
getting the data migrated as quickly as possible is critical.

14
 The modified process took about six days to It is important to remember that your target
complete and migrated close to 230 TBs of database will be almost a mirror copy of your
data. Using the existing XTTS process, out-of- source. You target system will have the same
the-box, would have taken a month or more to block size, the same national character set,
complete. the same encryption options, etc. In our case,
we wanted to change a lot of things about our
Once the initial process is complete, the RMAN database structure that cannot be done as part
convert process runs for about 8 – 10 hours on of the XTTS process. Therefore, we had to do
the Exadata system. By running the convert jobs another migration, but that will be covered in
on all four Exadata compute nodes, we were able another white paper.
to process about 25 TBs of data per hour.
THE ISSUES FOUND
Next are the repetitive steps to roll the
destination database forward. These are the So during our migration utilizing Oracle’s
incremental backups, the copying of files, the version 2 of the XTTS process, we did find a
convert / roll-forward processing, and then number of issues.
setting the new SCN. These steps are repeated
over and over again until we are ready for the I’ve listed these here in order to help clients out in
READ-ONLY window to occur. Basically, the the future, if they run into similar problems.
first time the incremental backup is run, it will
execute for the longest period of time, due to 1. Ensuring that the DIO option was set on
six days’ worth of archive logs that have been the NFS mounting options for the AIX
generated to-date. server. Otherwise, performance was
terrible and AIX server started having
Each subsequent run, there will be less and less memory issues.
archive logs to process, so the time to run will
2. Ensuring that the Write Bias is set to
go down. By the time the READ-ONLY window
Latency on the ZFS appliance, when writing
occurs, you should have less than 12 hours
from AIX to the ZFS NFS file shares. Again,
of archive logs to process. In our case, we
otherwise, performance for the transfers
had about 8 hours’ worth, and it took about 2
were seriously degraded.
hours for the incremental backup to run in the
READ-ONLY window. 3. An issue with utilizing the Import Plug-In
Step via a database link required two
The entire time of the READ-ONLY window patches to be applied. One to the
is around 16 hours to complete. At the end of Exadata target side, and one to the source
this process, all of the data will have been AIX side. We were getting ORA- errors
successfully migrated to the new target when attempting to use the plug-in after
system. the July 2016 Bundle Patch was applied to
our Exadata system.

15
 4. We were unable to get the Export
SUMMARY
Transportable Tablespace dump files to
work successfully when attempting to Hopefully, you found some insights from this
Plug-In these tablespaces on the target paper. We learned quite bit during this project
side. We consistently got tablespaces were and look forward to utilizing many of these
not self-contained, even though, we knew techniques again on future projects.
they were. This was not an issue if we used This project has shown that the Cross
the database link option. Platform Transportable Tablespace (XTTS)
utility delivered by Oracle does work and
5. A number of xtf extension files that were
gives you a repeatable process for the future.
created in ASM as the new data files came
In order to move very large databases, some
over in the wrong format. The format
modifications will need to be made. However,
should be: <tablespace_name>_<data file
once in place, it is very possible to “cut-over” a
number>.xtf . Instead, we had about thirty
very large database within a day.
data files that consistently did not follow
this format. We had to manually fix our
There is no guarantee that you might not run
dynamically created Import Plug-In par file
into bugs and require patches, so plan for that
to match these anomalies in ASM.
in your project estimates. Also, network
6. Obviously, the speed of a single xttdriver.pl capacity and the ability to move the backup
script was an issue that we had to pieces easily are also key considerations.
overcome. So if you have a significantly Although you don’t need a ZFS appliance,
large database or your network is fairly having some type of mountable NFS share was
slow, running a single xttdriver.pl script quite helpful.
may take too long for your migration.
Thanks for reading our paper. If you have any
7. The amount of time it takes to actually
questions or would like to discuss how Centric
Plug-In the meta-data seems significantly
can help with your database migration, please
longer that it should. It took nine hours for
reach out to me at :
our meta-data about the tablespaces to get
loaded. If you have a very large number of [email protected]
tables and indexes, something like Oracle
EBS, this might take even significantly REFERENCES
longer to Plug-In.
Refer to the following MOS note for additional details:

• 12C - Reduce Transportable Tablespace Downtime using

Cross Platform Incremental Backup (Doc ID 2005729.1)

• Guidelines When Using ZFS Storage in an Exadata

Environment (Doc ID 2087231.1)

• Guidelines When Using ZFS Storage in an Exadata

Environment (Doc ID 2087231.1)

16
CROSS-PLATFORM TRANSPORTABLE TABLESPACES UTILIZING INCREMENTAL BACKUPS
About Centric Consulting

CENTRIC AT A GLANCE
Centric is a management and technology consulting company. We have over 700 consultants with
extensive experience delivering high-profile projects for clients of all shapes and sizes, including
Fortune 500 companies.

• Highly capable consultants

• Passion for delivering results
• Long term relationship focused
• Easy to work with

Please visit our web site for a complete description of our ERP offerings. www.centricconsulting.com

100% of Centric’s Clients are willing to reference our work. A sampling of our Clients includes:

• Ameritas • Ethicon Endo-Surgery, Inc. • LION Apparel

• Aramark • Luxottica / EyeMed • Nationwide Insurance
• Cincinnati Children’s • Honda • Ohio National Financial
Hospital • Humana • Vitamix
• Dayton Power & Light • Kroger

EAS PRACTICE OVERVIEW

Successful deployment and support of an ERP solution requires synergy between the business and
IT. Our holistic solutions combine deep industry knowledge with business process and technology
expertise to create unmatched value and customer experiences. We are transparent, flexible and
easy to work with. We help businesses connect with the global economy and enable “game
changer” capabilities. Whether it is a first-time implementation, managed services, application or
database upgrade, Centric can help. Clients consistently compliment us on our ability to adapt our
approach and style to their culture, working directly with their team in a collaborative partnership.
We are able to accomplish this through our “do the right thing” approach to client engagements.

CONTACT INFORMATION
Andy Park, VP and Partner, EAS Chris Szaz, VP and Partner, EAS Jay Barnhart, Sr. Tech Manager, EAS
[email protected] [email protected] [email protected]
(513) 382-3011 mobile (513) 235-2648 mobile (740) 501-2551 mobile