A Practical Guide to Oracle 11g

RAC

Oracle 11g R2
1
Agenda
Introduction
Availability
Scalability
Manageability
Total Cost of Ownership
Conclusion

2
What Is a RAC Cluster?
What Is a RAC Cluster? Interconnect

Nodes
Interconnect Node 1
Node 2
Shared disk subsystem
Instances
Database

Shared Disks

3
What is RAC?
Multiple instances running on separate servers (nodes)
Single database on shared storage accessible to all nodes
Instances exchange information over an interconnect
network

Instance 1 Interconnect Instance 2

Node 1 Node 2

Local Shared Local

Disk Storage Disk
4
Why RAC?
High Availability – survive node and instance failures

Scalability – Add or remove nodes when needed

Pay as you grow – harness the power of multiple low-cost

computers

Enable Grid Computing

DBA’s have their own vested interests!

5
Instances versus Databases
A RAC cluster includes
one database
one or more instances

A database is a set of files

Located on shared storage
Contains all persistent resources

An instance is a set of memory structures and processes

Contain all temporal resources
Can be started and stopped independently

6
Instances versus Databases
Public
Private Network Network
(Interconnect)

Instance 1 Instance 2 Instance 3 Instance 4

Node 1 Node 2 Node 3 Node 4

Storage
Network
Database

7
A RAC Database –what’s different?
Contents similar to single instance database except …

Create and enable one redo thread per instance

If using Automatic Undo Management also require one UNDO

tablespace per instance

Additional cluster specific data dictionary views created by

running the script $ORACLE_HOME/rdbms/admin/catclust.sql

New background processes

Cluster specific init.ora parameters

8
What is the Interconnect?
Instances communicate with each other over the
interconnect (network)

Information transferred between instances includes

data blocks
locks
SCNs

Typically 1GB Ethernet Adapter.

9
Why Use Shared Storage?
Mandatory for
Database files
Control files
Online redo logs
Server Parameter file (if used)
Optional for
Archived redo logs (recommended)
Executables (Binaries)
 Password files
 Parameter files

 Network configuration files

Administrative directories
 Alert Log
 Dump Files

10
What Shared Storage is Supported?
Oracle supplied options
Oracle Cluster File System (OCFS)
 Version 1
 Windows and Linux

 Supports database and archived redo logs

 No executables

 Version 2 - August 2005

 Linux, Windows and Solaris

 As OCFS1 plus executables

Automatic Storage Management (ASM)

 Oracle 10.1 and above
 More transparent in Oracle 10.2 and above

Both require underlying SAN (Storage Area Network) or NAS (Network

Attached Storage)
 Do not require LVM
11
What is a Shared Oracle Home?
Can install multiple copies of Oracle executables on
local disks on each node
Can also install Shared Oracle Home
single copy of Oracle executables on shared storage

12
Internal Structures and Services
Global Resource Directory (GRD)
Records current state and owner of each resource
Contains convert and write queues
Distributed across all instances in cluster

Global Cache Services (GCS)

Implements cache coherency for database
Coordinates access to database blocks for instances
Maintains GRD

Global Enqueue Services (GES)

Controls access to other resources (locks) including
 library cache
 dictionary cache

13
Background Processes
Each RAC instance has set of standard background processes e.g.
PMON
SMON
LGWR
DBWn
ARCn
RAC instances use additional background processes to support GCS and
GES including
LMON
LCK0
LMDn
LMSn
DIAG

14
RAC init.ora Parameters
*.db_cache_size=113246208
*.java_pool_size=4194304
*.db_name='racdb‘
racdb2.instance_number=2
racdb1.instance_number=1
*.log_archive_dest_1='LOCATION=USE_DB_RECOVERY
_FILE_DEST'
racdb2.thread=2
racdb1.thread=1
*.undo_management='AUTO'
racdb2.undo_tablespace='UNDOTBS2'
racdb1.undo_tablespace='UNDOTBS1'

15
Why Do Users Deploy RAC?
Users may deploy RAC to achieve

Increasing availability

Increasing scalability

Improving maintainability

Reduction in total cost of ownership

16
Availability

17
What is Failover?
If one node or instance fails
Node detecting failure will
 Read redo log of failed instance from last checkpoint
 Apply redo to datafiles including undo segments (roll
forward)
 Rollback uncommitted transactions

Cluster is frozen during part of this process

Instance 1 Interconnect Instance 2

Node 1 Node 2

18
What are Database Services?
Database Services are logical groups of sessions

Can be configured using

DBCA
Enterprise Manager (10.2 and above)

Can also be configured using

SRVCTL (Oracle Cluster Registry only)
SQL*Plus (Data Dictionary only)
Text editor (Network Configuration)

In Oracle 10.1 and above, each service has

Preferred Nodes (used by default)
Available Nodes (used if preferred node fails)

19
What are Database Services?
Can be used with Resource Manager to control resource usage e.g.
 CPU
 Parallel execution

Can be used for monitoring

V$SERVICE_STATS

Can be used for diagnostics

DBMS_MONITOR
 trace
 statistics

20
What is Oracle Clusterware?
Oracle Clusterware is the technology that transforms a server farm into a
cluster. A cluster in general is a group of independent servers that cooperate
as a single system. Oracle Clusterware is the intelligence in this system that
ensures that required cooperation.

Oracle Clusterware was introduced with Oracle Database 10g as the general
underlying clusterware that is required to run Oracle Real Application
Clusters 10g. At this time Oracle Clusterware was known as Oracle CRS.

Cluster Manager providing

Node membership services
Global resource management
High availability functions

21
What is the OCR?
Oracle Cluster Registry (OCR)
Configuration information for Oracle Clusterware / CRS

Introduced in Oracle 10.1

Replaced Server Management (SRVM) disk/file

Similar to Windows Registry

Located on shared storage, May be on ASM Disks also

In Oracle 10.2 and above can be mirrored

Maximum two copies

22
What is the OCR?
Defines cluster resources including:
Databases
Instances
 RDBMS
 ASM

Services
Node Applications
 VIP
 ONS

 GSD

Listener Process

23
What is a Voting Disk?
Known as Quorum Disk / File in Oracle 9i

Located on shared storage accessible to all instances

Used to determine RAC instance membership

In the event of node failure voting disk is used to determine which
instance takes control of cluster
Avoids split brain

In Oracle 10.2 and above can be mirrored

Odd number of copies (1, 3, 5 etc)

24
Network Requirements
Each node must have at least two network adapters; one
for the public network interface and one for the private
network interface (the interconnect).

The public network adapter must support TCP/IP

For the private network, the interconnect must preferably

be a Gigabit Ethernet switch that supports UDP. This is
used for Cache Fusion inter-node connection

Host name and IP addresses associated with the public

interface should be registered in DNS and /etc/hosts

25
What is VIP?
VIPs are used in order to facilitate faster failover in the event of a
node failure

Each node not only has its own statically assigned IP address as
well as also a virtual IP address that is assigned to the node

The listener on each node will be listening on the Virtual IP and

client connections will also come via this Virtual IP.

Without VIP, clients will have to wait for long TCP/IP timeout
before getting an error message or TCP reset from nodes that have
died

26
What is SCAN?
SCAN was first introduced in Oracle RAC 11g Release 2
Single Client Access Name (SCAN) provides a single
name for clients to access any Oracle Database running
in a cluster
You can think of SCAN as a cluster alias for databases
in the cluster
Client can user Ezconnect Method to connect to server
like Sqlplus hr/manager@ol5-112-scan/oltp

27
What is TAF?
TAF is Transparent Application Failover

Sessions connected to a failed instance will be terminated

Uncommitted transactions will be rolled back

Sessions can be reconnected to another instance automatically if using TAF

Can optionally re-execute in-progress SELECT statements
 Statement re-executed with same SCN
 Fetches resume at point of failure

Session state is lost including

 Session parameters
 Package variables
 Class and ADT instantiations

28
What is TAF?
TAF defined by FAILOVER_MODE parameter
TYPE=SESSION
- User does not need to reconnect
- Session failed over to another available instance in the list
- But SQL statements in progress will have to be
reissued
 TYPE=SELECT
- Query will be restarted after failover
- Rows not fetched before failover will be retrieved

29
What is TAF?
Connection modes METHOD=BASIC or PRECONNECT

BASIC
- After failover connection must reconnect to next address in
the list
- Additional time to failover

PRECONNECT
- Session is opened against all addresses in the list
- Only one is used – others remain connected
- Faster failover with preconnected sessions
- More memory resources consumed by preconnected sessions
on other nodes

30
What is TAF?
TAF is Transparent Application Failover

Requires additional coding in client

Requires configuration in TNSNAMES.ORA

RAC_FAILOVER =
(DESCRIPTION =
(ADDRESS_LIST =
(FAILOVER = ON)
(ADDRESS = (PROTOCOL = TCP)(HOST = node1)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = node2)(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = RAC)
(SERVER = DEDICATED)
(FAILOVER_MODE =(TYPE=SELECT)(METHOD=BASIC)
(RETRIES=30)(DELAY=5))
)
) 31
What is FAN?
Fast Application Notification (FAN)
The Oracle RAC FAN APIs enable application code to receive and
respond to FAN event notifications sent by Oracle RAC. This is
achieved by enabling the code to respond to FAN events in the
following way:
 Listening for Oracle RAC service down, service up, and node down events
 Listening for load balancing advisory events and responding to them
This feature exposes a subset of FAN events, which are the
notifications sent by a cluster running Oracle RAC, to inform the
subscribers about the configuration changes within the cluster. The
supported FAN events are the following:
Service UP
Service Down
Load Balancing Advisory
Node Down 32
What is ONS?
Oracle Notification Service (ONS)

Introduced in Oracle 10.1

Allows out-of-band messages to be sent to

Nodes in cluster
Middle-tier application servers
Clients

Underlying mechanism for Fast Application Notification (FAN)

33
Does RAC Increase Availability?
Depends on definition of availability
May achieve less unplanned downtime
May have more time to respond to failures

Instance failover means any node can fail without total loss of
service

Must provide have overcapacity in cluster to survive failover

Additional Oracle and RAC licenses
Load can be distributed over all running nodes
Can use Grid to provision additional nodes

34
Does RAC Increase Availability?
Can still get data corruptions
Human errors / software errors
Only one logical copy of data
Only one logical copy of application / Oracle software

Lots of possibility for human errors

Power / network cabling / storage configuration

Upgrades and patches are more complex

Can upgrade software on subset of nodes
If database is affected then still need downtime

35
Scalability

36
What is Scalability?
Scalability is the relationship between increments of resources and
workloads
Can be any resource but with RAC normally refers to adding instances
Scalability can be
 linear - optimal but rare
non-linear - suboptimal but normal
Workload

Workload
Linear Non-
Linear

Resource Resource
37
What is Workload Balancing?
Balancing of workload across available instances
Can have
Client-side connection balancing
Server-side connection balancing

Client-side connection balancing

Workload distributed randomly across nodes

RAC =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = node1)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = node2)(PORT = 1521))
(LOAD_BALANCE = ON)
(FAILOVER = ON)
(CONNECT_DATA =
(SERVICE_NAME = RAC)
(FAILOVER_MODE = (TYPE = SELECT)(METHOD = BASIC))
)
)

38
Increasing Scalability
If application scales well on a single-instance then it should scale well on
RAC

Eliminate contention
Use sequences

Use locally partitioned tables and indexes

 Attempt to achieve node affinity

Avoid contention for single blocks

 Distribute rows for hot blocks
 Small block size e.g. 2048 or 4096

 ALTER TABLE MINIMIZE RECORDS PER BLOCK

 High PCTFREE / Low PCTUSED
 Filler columns e.g. CHAR (2000)

39
Increasing Scalability
Use Automatic Segment Space Management
Default in Oracle 11.2

Use larger block size for read-only objects

Reduce number of GCS messages required

Minimize lock usage

Eliminate unnecessary parsing
 Increase size of shared pool
 Bind variables
 Cursor sharing

Use optimistic locking

Eliminate unnecessary SELECT FOR UPDATE statements

40
Manageability

41
Server Parameter File
Introduced in Oracle 9.0.1
Must reside on shared storage
Shared by all RAC instances
Binary (not text) files
Parameters can be changed using ALTER SYSTEM
Can be backed up using the Recovery Manager (RMAN)
Created using
CREATE SPFILE [ = ‘SPFILE_NAME’ ]
FROM PFILE [ = ‘PFILE_NAME’ ];
 init.ora file on each node must contain SPFILE
parameter
SPFILE = <pathname>
42
Parameters
RAC uses same parameters as single-instance
Some must be different on each instance
Some must be same on each instance

Can be global or local

[*.]<parameter_name> = <value>
[<sid>]<parameter_name> = <value>

 Must be set using ALTER SYSTEM statement

ALTER SYSTEM SET parameter = value

[ SCOPE = MEMORY | SPFILE | BOTH ]
[ SID = <sid>]
ALTER SYSTEM RESET parameter = value
[ SCOPE = MEMORY | SPFILE | BOTH ]
[ SID = <sid>]

43
Parameters
Some parameters must be same on each instance including :

ARCHIVE_LAG_TARGET
CLUSTER_DATABASE
CLUSTER_DATABASE_INSTANCES
CONTROL_FILES
COMPATIBLE
DB_BLOCK_SIZE
DB_DOMAIN
DB_FILES
DB_NAME
DB_RECOVERY_FILE_DEST
DB_RECOVERY_FILE_DEST_SIZE
DB_UNIQUE_NAME
INSTANCE_TYPE (RDBMS Or ASM)
PARALLEL_EXECUTION_MESSAGE_SIZE
UNDO_MANAGEMENT
44
Parameters
Some parameters, if used, must be different on each
instance including
THREAD
INSTANCE_NUMBER
INSTANCE_NAME
SPFILE
SERVICE_NAMES
UNDO_TABLESPACE
ROLLBACK_SEGMENTS

DML_LOCKS must be identical on each instance if

set to zero
45
What is SRVCTL?
Utility used to manage cluster database
Configured in Oracle Cluster Registry (OCR)
Controls
Database
Instance
ASM
Listener
Node Applications
Services
Options include
Start / Stop
Enable / Disable
Add / Delete
Show current configuration
Show current status

46
SRVCTL - Examples
Starting and Stopping a Database
srvctl start database -d RAC
srvctl stop database -d RAC

 Starting and Stopping an Instance

srvctl start instance -d RAC -i RAC1

srvctl stop instance -d RAC -i RAC1

 Starting and Stopping a Service

srvctl start service -d RAC -s SERVICE1
srvctl stop service -d RAC -s SERVICE1

 Starting and Stopping ASM on a specified node

srvctl start asm -n node1
srvctl stop asm -n node1

47
What is CLUVFY?
Introduced in Oracle 10.2

Supplied with Oracle Clusterware

Can be downloaded from OTN (Linux and Windows)

Written in Java - requires JRE (supplied)

Checks cluster configuration

stages - verifies all steps for specified stage have been completed
components - verifies specified component has been correctly
installed

48
CLUVFY
Stages include

-post hwos post check for hardware and operating system

-pre cfs pre-check for CFS setup
-post cfs post-check for CFS setup
-pre crsinst pre-check for Oracle Clusterware installation
-post crsinst post-check for Oracle Clusterware installation
-pre dbinst pre-check for database installation
-pre dbcfg pre-check for database configuration

49
CLUVFY
Components include
nodereach Checks reachability between nodes
nodecon Checks node connectivity
cfs Checks CFS integrity
ssa Checks shared storage accessibility
space Checks space availability
sys Checks minimum system requirements
clu Checks cluster integrity
clumgr Checks cluster manager integrity
ocr Checks OCR integrity
crs Checks Oracle Clusterware (CRS) integrity
nodeapp Checks node applications exist
admprv Checks administrative privileges
peer Compares properties with peers

50
CLUVFY
For example, to check configuration before installing Oracle
Clusterware on node1 and node2 use:
sh runcluvfy.sh stage -pre crsinst -n node1,node2

 Checks:
 node reachability
 user equivalence
 administrative privileges
 node connectivity
 shared stored accessibility

 If any checks fail append -verbose to display more information

51
Does RAC Improve Manageability?
Advantages
Fewer databases to manage
Easier to monitor
Easier to upgrade
Easier to control resource allocation
Resources can be shared between applications

Disadvantages
Upgrades potentially more complex
Downtime may affect more applications
Requires more experienced operational staff
 Higher cost / harder to replace

52
Total Cost of
Ownership

53
Reduction in TCO?
Possible for sites with legacy systems
Mainframes / Minicomputers
Applications / Packages

RAC option adds 50% to licence costs except for

Users with site licences
Standard edition (10.1+, max 4 CPU with ASM)

Retrain existing staff or use dedicated staff

Consolidation may bring economies of scale

Monitoring
Backups
Disaster Recovery

54
Reduction in TCO?
Additional resources required
Redundant hardware
 Nodes
 Network switches

 SAN fabric
 Hardware e.g. fibre channel cards

Reduction in hardware support costs

May not require 24 hour support
Viable to hold stock of spare components

55
What are the Alternatives to RAC?
Data Guard
Physical Standby
 Introduced in Oracle 7.3.4
 Stable, well proven technology

 Requires redundant hardware

 Implemented by many sites

 Can be used with RAC

Logical Standby
 Introduced in Oracle 9.2
 Still not widely adopted

Streams
 Introduced in Oracle 9.2
 Implemented by increasing number of sites

Advanced Replication

56
What are the Alternatives to RAC?
Symmetric Multiprocessing (SMP) Systems
Single Point of Failure
Simplified configuration
Eliminate RAC overhead

Parallel systems
For systems with deterministic input
Messaging
Data Warehouses

Other Clustering Technologies

SAN
Operating System
etc

57
Conclusion
Success of RAC deployments dependent on
Application design and implementation
Failover requirements
IT infrastructure
Flexibility and commitment of IT department(s)

Before deploying RAC

Investigate and reject alternatives
Perform proof of concept
 Test application
 Evaluate benefits and costs

 Learn RAC concepts and administration

Buy a good book :)

58
Thank You

59