Cloning 10gR2

RAC /ASM Db Tier

with 11i EBS on
Linux Servers
A phased Procedure

Original procedure was written by A.Theodoridis, ORACLE Consulting, Sep 2009

Document revised by Nikos Plevris, ORACLE Advanced Customer Services, Nov
2009
 CLONE 10gR2 RAC with 11i EBS on Linux Server
PREPARE BACKUP DIRECTORY ON THE PRODUCTION ENVIRONMENT....................................................5

BACKUP PRODUCTION.............................................................................................................................5

PREPARE BACKUP DIRECTORY ON THE DEVELOPMENT ENVIRONMENT.................................................5

CLEANUP ASM DIRECTORIES ON THE DEVELOPMENT ENVIRONMENT...................................................5

PHASE A: SINGLE-INSTANCE DATABASE..................................................................................................6

PREPARE THE initialization parameters ON THE DEVELOPMENT ENVIRONMENT...............................6

RESTORE CONTROLFILES AND MOUNT ERPDEV1 INSTANCE...............................................................7

RECOVER AND RESTORE ERPDEV1 INSTANCE......................................................................................8

RENAME ONLINE REDO LOGFILES (SINGLE-INSTANCE DATABASE PHASE)..........................................8

OPEN DATABASE WITH RESETLOGS (SINGLE-INSTANCE DATABASE PHASE)........................................9

CHANGE THE DB_NAME OF THE DATABASE (SINGLE-INSTANCE DATABASE PHASE)...........................9

PHASE B: RAC DATABASE.......................................................................................................................10

TRANSFORM SINGLE-INSTANCE TO RAC DATABASE (RAC DATABASE PHASE)...................................10

CREATE SPFILE FOR RAC DATABASE (RAC DATABASE PHASE)............................................................11

STARTUP RAC DATABASE USING CLUSTERWARE UTILITIES (RAC DATABASE PHASE)........................11

APPLICATIONS SPECIFIC TASKS (RAC DATABASE PHASE)...................................................................12

APPLICATIONS Configuration:....................................................................................................................13

RUN THE AUTOCONFIG ON appdev1 APPLICATION NODE................................................................13

RUN THE AUTOCONFIG ON appdev2 APPLICATION NODE................................................................14

ADD NODE appdev2 IN THE APPLICATIONS NODES ENVIRONMENT.................................................14

SETUP CONCURRENT MANAGERS PRIMARY AND SECONDARY NODES (THROUGH Oracle Application
Manager)...........................................................................................................................................14

CHANGE THE APPS AND SYSADMIN PASSWORD...............................................................................15

STEPS FOR CLONING THE MIDDLE TIER ENVIRONMENT....................................................................15

 System Configuration:
1. 2-Node RAC database 10.2.0.4/ASM on Linux x86_64
2. eBS farm on 11.5.10.2 (with ATG delta 6) on Linux x86
3. Four identical application tier servers hosting all services (Concurrent Processing, Web, Forms,
Administration)
4. Application Farm serviced by HTTP Hardware Load Balancer

PRODUCTION environment exists on hosts prdb01 & prdb02.

ERPDEV & ERPTEST environments

Summary Configuration
Both ERPDEV & ERPTEST environments have a RAC database which exists on devdb01 & devdb02.
Together with the REPOS database (Production’s RMAN repository), we have a total of three RAC
databases on the same 2-nodes

All environments have been deployed on the common storage SAN.

 Database Configuration:
STEPS ACTION COMMAND
The /orabkp mount point is used as a common backup area across environments being mount and
unmounted from the production environment to the target development system.
A PREPARE BACKUP DIRECTORY ON THE PRODUCTION ENVIRONMENT
 1
Make sure that /orabkp is unmounted unmount /orabkp
on devdb01
 On prdb01 mount the following directory # mount /dev/mapper/backup1p1 /oraclone -t ext3
2
 On prdb01 as root change permissions # chown –R oracle /oraclone
3 on /oraclone

B BACKUP PRODUCTION
 rman target / nocatalog
1 Backup Source Production database
@/oraclone/RMAN/scripts/11i_10grac_clone.rman
configure device type disk parallelism 8 backup type to backupset;
configure maxsetsize to 31000m;
backup as backupset tag 'CLONE_DB' database format '/oraclone/RMAN/CLONE/%U';
backup as backupset tag 'CLONE_DB' archivelog all format
'/oraclone/RMAN/CLONE/%U';
sql ‘ALTER SYSTEM ARCHIVE LOG CURRENT’;
backup tag 'CLONE_DB' as copy current controlfile format
'/oraclone/RMAN/CLONE/backup_controlfile.ctl';
exit

After successful completion do not use crosscheck or report obsolete commands in rman
C PREPARE BACKUP DIRECTORY ON THE DEVELOPMENT ENVIRONMENT
 Unmount the /oraclone from prdb01 # umount /oraclone
1
 Mount /orabkp at devdb01: # mount /dev/mapper/bckp1p1 /orabkp -t ext3
2
 On devdb01 as root change permissions # chown -R oracle /orabkp
3
on /orabkp
D CLEANUP ASM DIRECTORIES ON THE DEVELOPMENT ENVIRONMENT
 On devdb01 as the oracle user, set the Run asm bash profile function
1 ASM-related environment variables

 2 SignOn to ASMCMD tool asmcmd -p

 ASMCMD [+] > ls
DEVDATA/
3 List the ASM directories TESTDATA/
ASMCMD [+] >

ASMCMD [+] > cd devdata

ASMCMD [+devdata] > ls
ERPDEV/
ASMCMD [+devdata] > cd erpdev
ASMCMD [+devdata/erpdev] > ls
ARCHIVELOG/
CONTROLFILE/
DATAFILE/
Delete the ASM-related directories for
ONLINELOG/
4 the ERPDEV environment
 PARAMETERFILE/
TEMPFILE/
spfile.ora
ASMCMD [+devdata/erpdev] > rm DATAFILE/*
ASMCMD [+devdata/erpdev] > rm CONTROLFILE/*
ASMCMD [+devdata/erpdev] > rm ONLINELOG/*
ASMCMD [+devdata/erpdev] > rm spfile.ora
ASMCMD [+devdata/erpdev] > rm PARAMETERFILE/*
ASMCMD [+devdata/erpdev] > rm TEMPFILE/*
 During the delete operation you might encounter an error when you try to navigate inside the ASM
directories. The solution is to exit the ASM command prompt and login again. After completion close the
terminal session and open a new one.
PHASE A: SINGLE-INSTANCE DATABASE
PREPARE THE initialization parameters ON THE DEVELOPMENT ENVIRONMENT
D
(SINGLE-INSTANCE DATABASE PHASE)
 On devdb01 as the oracle user, set the Run db bash profile function
1
environment variables
 In this directory we will find two directories: ERPDEV and
ERPTEST. Depending on which environment you are working on,
Navigate to directory
2 change directory accordingly. We assume that we are currently
$ORACLE_HOME/appsutil/admin/ERPDEV
working on ERPDEV.

 Edit the temporary initialization file Beware that the db_name in the temporary init.ora is still set to
3 initERPDEV.ora.tmp PROD11I until step …

## Source System Global Database Name...NOT A SID NAME

db_name=PROD11I
## New Target System Global Database Name...NOT A SID NAME
db_unique_name=ERPDEV
## The number of DB files available to RMAN, set to appropriate value
db_files = 4096
## The location to which the shared dbf (datafiles) will be written. This
## location should be the mount point to which both target RAC nodes can
## read and write file to. +ASM_DISK_GROUP, NFS or OCFS disks
## Note that RMAN will create a sub-directory under this location with the
## NEW GLOBAL Database name in UPPER case.
db_create_file_dest=+DEVDATA
## RMAN will place the archive logs in this location and it should be
## located on shared storage, accessible from both RAC nodes
db_recovery_file_dest=+DEVDATA
## The amount of space available for RMAN to use to write archive logs
## Set to a more appropriate value as needed
db_recovery_file_dest_size=100g
## The location & name to which the "new" Target System control file
## will be restored to...Set to desired location...must be on shared
## storage location access by both Target System RAC nodes
control_files='+DEVDATA/ERPDEV/CONTROLFILE/controlfile_ERPDEV.ctl'
## Tempoary SGA TARGET size, set to appropriate value as needed
sga_target=4g
## Tempoary SGA MAX size, set to appropriate value as needed
sga_max_size=8g
## Tempoary number for OPEN CURSORS, set to appropriate value as needed
open_cursors=1000
background_dump_dest='/opt/app/oracle/admin/ERPDEV/bdump'

RESTORE CONTROLFILES AND MOUNT ERPDEV1 INSTANCE

E
(SINGLE-INSTANCE DATABASE PHASE)
 1 Set the ORACLE_SID Export ORACLE_SID=ERPDEV1

WARNING: In every session opened during the cloning process, be aware of setting the ORACLE_SID variable

 Use RMAN script rman target / nocatalog @restore-rac1.rman >

2 $ORACLE_HOME/appsutil/admin/ERPDEV/re restore-rac1.log
store-rac1.rman

startup nomount
pfile=/opt/app/oracle/product/10.2.0/db_1/appsutil/admin/ERPDEV/initERPDEV.ora.tmp
restore controlfile from '/orabkp/RMAN/CLONE/backup_controlfilec.ctl';
alter database mount;
exit

 Check the logifle restore-rac1.log if there

3
are any errors reported
 Delete restore-rac2.log, and run the sqlplus "/as sysdba" @restore-rac2.sql
4 script restore-rac2.sql to create RMAN
script restore-rac2.rman

connect / as sysdba
set echo off pages 0 feed off sqlp #
spool restore-rac2.rman
select 'catalog start with ''/orabkp/RMAN/CLONE'' NOPROMPT;' from dual;
select 'RUN { ' from dual;
select 'set newname for datafile '||file#||' to NEW;' from v$datafile;
select 'set newname for tempfile '||file#||' to NEW;' from v$tempfile;
select 'restore database;' from dual;
select 'switch datafile all;' from dual;
select 'switch tempfile all;' from dual;
select 'recover database;' from dual;
select '} ' from dual;
select 'exit' from dual;
spool off
exit

RECOVER AND RESTORE ERPDEV1 INSTANCE

F (SINGLE-INSTANCE DATABASE PHASE)

 Crosscheck and delete any obsolete and RMAN>crosscheck

1
expired files RMAN>delete obsolete;
WARNING: If flash recover area is 100% full then recover will fail . So the step F1 will be needed
Run the RMAN script restore-rac2.rman rman target / nocatalog @restore-rac2.rman >
 2 created at step E4 to restore the restore-rac2.log
database
 Check the progress of the restore using Sqlplus / as sysdba
3 SQL>select * from v$session_longops;
sqlplus and the view v$session_longops
 4 Use ASMCMD command tool to check if asmcmd –p
datafiles are being created ASMCMD [+] > cd devdata
 ASMCMD [+devdata] > cd erpdev
ASMCMD [+devdata] > cd erpdev
ASMCMD [+devdata/erpdev] > ls
ARCHIVELOG/
CONTROLFILE/
DATAFILE/
ONLINELOG/
PARAMETERFILE/
TEMPFILE/
spfile.ora
G RENAME ONLINE REDO LOGFILES (SINGLE-INSTANCE DATABASE PHASE)
 sqlplus / as sysdba @restore-rac21.sql
1 Run the script restore-rac21.sql

set echo off pages 0 feed off linesize 5000 TRIM on TRIMS on sqlp #
spool restore-rac-logfile.sql
select 'connect /as sysdba' from dual;
select 'alter database rename file '''||member||''' to
'||'''+DEVDATA/erpdev/onlinelog/'|| REGEXP_REPLACE(''||member||'','.*/') ||''';'
from v$logfile;
select 'exit' from dual;
spool off
exit

The above step G1 will create the script sqlplus "/ as sysdba" @restore-rac-logfile.sql
 2
restore-rac-logfile.sql which you should
run it as sysdba in order to rename redo
files
H OPEN DATABASE WITH RESETLOGS (SINGLE-INSTANCE DATABASE PHASE)
sqlplus / as sysdba
 1
Open the database with resetlogs. This
will recreate the onlinelogs. SQL> alter database open resetlogs;

Shutdown database. Exit the sqlplus SQL> shutdown immediate;

 2 session and the terminal session.

I CHANGE THE DB_NAME OF THE DATABASE (SINGLE-INSTANCE DATABASE PHASE)

asmcmd –p
Edit the temporary initialization file ASMCMD [+] > cd devdata
ASMCMD [+devdata] > cd erpdev
initERPDEV.ora.tmp with the name of
 ASMCMD [+devdata/erpdev] > cd controlfile
1 the controlfile restored. SignOn to
ASMCMD [+devdata/erpdev/controlfile] > ls
ASMCMD tool to get the controlfile controlfile_erpdev.ctl
name. current.353.689167275

 2 In the above example the correct

controfile is current.353.689167275
which must be included in our
 temporary init.ora under
$ORACLE_HOME/appsutil/admin/ERPDEV
before we continue. So edit
initERPDEV.ora.tmp and modify the line:
control_files=…
Save and exit.

…
…
## The location & name to which the "new" Target System control file
## will be restored to...Set to desired location...must be on shared
## storage location access by both Target System RAC nodes
control_files='+DEVDATA/ERPDEV/CONTROLFILE/current.353.689167275’
…
…
Sqlplus / as sysdba
 3 Startup mount database SQL> startup mount pfile=’
$ORACLE_HOME/appsutil/admin/ERPDEV/initERPDEV.ora.tmp’
Change the database name using nid nid target=/ dbname=ERPDEV
 4 utility. The database will shutdown after
the run.
Set the new db_name in the temporary
init.ora under directory
$ORACLE_HOME/appsutil/admin/ERPDEV
 5 before we continue. So edit
initERPDEV.ora.tmp and modify the line:
Db_name=…

…
…
## Source System Global Database Name...NOT A SID NAME
db_name=ERPDEV
## New Target System Global Database Name...NOT A SID NAME
db_unique_name=ERPDEV
## The number of DB files available to RMAN, set to appropriate value
db_files = 4096
…
…
Sqlplus / as sysdba
 Startup mount the database and open it SQL> startup mount pfile=’
6 $ORACLE_HOME/appsutil/admin/ERPDEV/initERPDEV.ora.tmp’
with the resetlogs option
SQL>ALTER DATABASE OPEN RESETLOGS;
 7 Shutdown the database
SQL> shutdown immediate;

PHASE B: RAC DATABASE

J TRANSFORM SINGLE-INSTANCE TO RAC DATABASE (RAC DATABASE PHASE)
Inside the directory /opt/app/oracle/product/10.2.0/db_1/appsutil/admin/ERPDEV there are two more init.ora files:
initIMP.ora and initRAC.ora. We will use the first one to open the mount the database for the first time with the correct
parameters and disable archivelog mode. The second will be used to transform the database from single instance to RAC.
Edit initIMP.ora under
$ORACLE_HOME/appsutil/admin/ERPDEV
before we continue. So edit
 1 initERPDEV.ora.tmp and modify the line:
control_files=…
Save and exit.

…
…
## The location & name to which the "new" Target System control file
## will be restored to...Set to desired location...must be on shared
## storage location access by both Target System RAC nodes
control_files='+DEVDATA/ERPDEV/CONTROLFILE/current.353.689167275’
…
…
SQL> startup mount
pfile='/opt/app/oracle/product/10.2.0/db_1/a
 Startup mount the database using this ppsutil/admin/ERPDEV/initIMP.ora';
2
new init.ora and disable archivelog mode SQL> alter database noarchivelog;
SQL> alter database open

 SQL> shutdown immediate;

3 Shutdown instance
Edit initRAC.ora under
$ORACLE_HOME/appsutil/admin/ERPDEV
before we continue. So edit
 4 initERPDEV.ora.tmp and modify the line:
control_files=…
Save and exit.

…
…
## The location & name to which the "new" Target System control file
## will be restored to...Set to desired location...must be on shared
## storage location access by both Target System RAC nodes
control_files='+DEVDATA/ERPDEV/CONTROLFILE/current.353.689167275’
…
…
K CREATE SPFILE FOR RAC DATABASE (RAC DATABASE PHASE)
SQL> startup
 pfile='/opt/app/oracle/product/10.2.0/db_1/a
1 Startup the database using initRAC.ora ppsutil/admin/ERPDEV/initRAC.ora';

 2 Create spfile for the RAC database SQL> create

spfile='+DEVDATA/ERPDEV/spfile.ora' from
pfile='/opt/app/oracle/product/10.2.0/db_1/a
 ppsutil/admin/ERPDEV/initRAC.ora';

 3 Shutdown instance
SQL> shutdown immediate;

L STARTUP RAC DATABASE USING CLUSTERWARE UTILITIES (RAC DATABASE PHASE)

 Use srvctl utility to startup RAC database srvctl start database –d ERPDEV
1
on the RAC environment
 Use shell script check_crs.sh to view check_crs.sh
2
cluster services status
HA Resource Target State
----------- ------ -----
ora.ERPDEV.ERPDEV1.inst ONLINE ONLINE on devdb01
ora.ERPDEV.ERPDEV2.inst ONLINE ONLINE on devdb02
ora.ERPDEV.db ONLINE ONLINE on devdb01
ora.ERPTEST.ERPTEST1.inst ONLINE ONLINE on devdb01
ora.ERPTEST.ERPTEST2.inst ONLINE ONLINE on devdb02
ora.ERPTEST.db ONLINE ONLINE on devdb01
ora.REPOS.REPOS1.inst ONLINE ONLINE on devdb01
ora.REPOS.REPOS2.inst ONLINE ONLINE on devdb02
ora.REPOS.REPOS_TAF.REPOS1.srv ONLINE ONLINE on devdb01
ora.REPOS.REPOS_TAF.REPOS2.srv ONLINE ONLINE on devdb02
ora.REPOS.REPOS_TAF.cs ONLINE ONLINE on devdb01
ora.REPOS.db ONLINE ONLINE on devdb02
ora.devdb01.ASM1.asm ONLINE ONLINE on devdb01
ora.devdb01.LISTENER_DEVDB01.lsnr ONLINE ONLINE on devdb01
ora.devdb01.gsd ONLINE ONLINE on devdb01
ora.devdb01.ons ONLINE ONLINE on devdb01
ora.devdb01.vip ONLINE ONLINE on devdb01
ora.devdb02.ASM2.asm ONLINE ONLINE on devdb02
ora.devdb02.LISTENER_DEVDB02.lsnr ONLINE ONLINE on devdb02
ora.devdb02.gsd ONLINE ONLINE on devdb02
ora.devdb02.ons ONLINE ONLINE on devdb02
ora.devdb02.vip ONLINE ONLINE on devdb02
All services must be up and running.

M APPLICATIONS SPECIFIC TASKS (RAC DATABASE PHASE)

sqlplus apps/<apps_pass>
 1
Clean Nonexistent Nodes or IP Addresses SQL>exec FND_CONC_CLONE.SETUP_CLEAN
from FND_NODES SQL> commit;

Node devdb01:
export ORACLE_SID=ERPDEV1
 2 Run autoconfig on both database nodes ./$ORACLE_HOME/appsutil/scripts/ERPDEV1_devdb01/
adautocfg.sh

Node devdb02:
export ORACLE_SID=ERPDEV2
./$ORACLE_HOME/appsutil/scripts/ERPDEV2_devdb02/
adautocfg.sh
 APPLICATIONS Configuration:
STEPS ACTION COMMAND

A RUN THE AUTOCONFIG ON appdev1 APPLICATION NODE

On appdev1 source the application scripts
 1
environment by running the profile shell
function script scripts

 2 On appdev1 run adautocfg.sh adautocfg.sh

Source the application environment by oracle

 3
running running the profile shell function
script oracle

Applications Steps A4-C3 do not apply if you are configuring the clone of ERPTEST since it has only one application server.
So you only need to run autoconfig on appdev3 and nothing else. Also run step 33.

 On appdev1 run adadmin to maintain adadmin

4
snapshot information

AD Administration Main Menu

--------------------------------------------------

1. Generate Applications Files menu

2. Maintain Applications Files menu

3. Compile/Reload Applications Database Entities menu

4. Maintain Applications Database Entities menu

5. Change Maintenance Mode

6. Exit AD Administration

Enter your choice [6] : 2

 Maintain Applications Files
----------------------------------------

1. Relink Applications programs

2. Create Applications environment file

3. Copy files to destinations

4. Convert character set

5. Maintain snapshot information

6. Check for missing files

7. Return to Main Menu

Select 5 and update the current view snapshot.

B RUN THE AUTOCONFIG ON appdev2 APPLICATION NODE

Source the application environment by scripts
 1
running the profile shell function script
scripts

 2 On appdev2 run adautocfg.sh adautocfg.sh

C ADD NODE appdev2 IN THE APPLICATIONS NODES ENVIRONMENT

Check if a row with the node we are SQL> select APPL_TOP_ID, NAME from ad_appl_tops;
adding is returned (ie. appdev2), then
 1
delete that row (OR you can follow the
instructions described in Metalink Note
ID: 394615.1).

Copy the file

 2
$APPL_TOP/admin/adconfig.txt.pre to
$APPL_TOP/admin/ adconfig.txt

Run the adaddclone.pl script under perl adaddclone.pl

 3 directory /opt/erp/comn/clone/bin/ to
add node
SETUP CONCURRENT MANAGERS PRIMARY AND SECONDARY NODES (THROUGH Oracle
D
Application Manager)
Start apache on one of the two $scripts
 1 application servers (appdev1 or $adapcctl.sh start
appdev2)
 2 Navigate to OAM Application Dashboard
Application Dashboard => Application Services => Select each
concurrent manager => edit primary and secondary nodes
 3 adstrtal.sh apps/<apps_pass>
Start the remaining apps services
If you are working on the Development environment on appdev1 & appdev2 make a note that the script adcmctl.sh runs
only on the appdev1 and not on appdev2.

E CHANGE THE APPS AND SYSADMIN PASSWORD

FNDCPASS apps/<apps_pass> 0 Y system/<system_pass>
 1 Change APPS password SYSTEM APPLSYS <new_pass>

 2 $scripts
Run autoconfig on all apps tiers
$adautocfg.sh
FNDCPASS apps/<apps_pass> 0 Y system/<system_pass>
 3 Change SYSADMIN password USER SYSADMIN <system_pass>

$scripts
 4 Bounce all application services $adstpall.sh apps/<apps_pass>
$adstrtal.sh apps/<apps_pass>
F STEPS FOR CLONING THE MIDDLE TIER ENVIRONMENT
If the application tier is also cloned from the Production Environment to any of the Development or Test Environments
then you should follow the instructions in Metalink Note: 230672.1 and tasks: Section 2 (task 1b & 2a & 3b).
Because the target environment has If working on the DEVELOPMENT:
many customizations, after successfully under appdev1:/erp/BACKUP copy ERPDEV_appdev1.xml in
1 running all the cloning scripts, copy the appdev1:/opt/erp/appl/admin
following XML files depending on which and copy ERPDEV_appdev2.xml in appdev2:/opt/erp/appl/admin
environment you are working on
If working on the TEST:
under appdev3:/erp/BACKUP copy ERPTEST_appdev3.xml in
appdev3:/opt/erp/appl/admin
When the files are copied for the working environment, run
autoconfig:

2 Run AutoConfig on applications nodes $scripts

appdev1 (or appdev2 or appdev3) $adautocfg.sh