Chapter 4: Advanced Database Management

G2CS&IT
Database Recovery Techniques
DBMS is highly complex system with hundreds of transactions being executed every second.
Availability of DBMS depends on its complex architecture and underlying hardware or system
software. If it fails or crashes amid transactions being executed, it is expected that the system
would follow some sort of algorithm or techniques to recover from crashes or failures.

Recovery from transaction failures means that the database is restored to the most recent
consistent state just before the failure. It is a service provided by the DBMS to ensure that the
database is reliable and remains in a consistent state in the presence of failure.

Failure Classification
- System crash: hardware, software, network error occurs during transaction execution.
- Transaction error: some operation [e.g. Div by 0, overflow] in the transaction may cause
it to fail)
- Exception conditions, detected by the transaction: Data not found.
- Concurrency control enforcement: serializability, deadlock.
- Disk failure (read/write phase), physical problems (theft).

Database Recovery Techniques

Recovery algorithms are techniques to ensure transaction atomicity and durability despite
failures.
- The recovery subsystem, using recovery algorithm, ensures atomicity by undoing the
actions of transactions that do not commit and durability by making sure that all actions
of committed transactions survive even if failures occur.
Two main approaches in recovery process
- Log-based recovery using WAL protocol.
- Shadow-paging.

A Typical Strategy for Recovery

- If there is a catastrophic failure –i.e. a disk crash-, restores a past copy of the database
that was backed up to archival storage –typically tape- and reconstructs a more current
state by reapplying the operations of committed transactions from the backed up log up to
the time of failure.
- If there is an inconsistency due to non-catastrophic failure, reverse any changes that
caused the inconsistency and if necessary, reapply some operations in order to restore a
consistent state of the database.

Concepts Used in Recovery

The System Log (Audit trail or DBMS journal)

- A log to keep track of all transaction operations that affect the values of database items,
as well as other transaction information that may be needed to permit recovery from
failures.
- The log is a sequential, append-only file that is kept on disk, so it is not affected by any
type of failure except for disk or catastrophic failure.

Prepared by : Mrs.Pravicha.M.T Page 1

 Chapter 4: Advanced Database Management
G2CS&IT
- Main memory buffers hold the last part of the log file, so that log entries are first added to
the main memory buffer. When the log buffer is filled, or when certain other conditions
occur, the log buffer is appended to the end of the log file on disk.
- In addition, the log file from disk is periodically backed up to archival storage (tape) to
guard against catastrophic failures.

Types of Entries (Log Records)

[start_transaction, T] Indicates that transaction T has started execution.
[write_item, T, X, old_value, new_value] Indicates that transaction T has changed the value of
database item X from old_value to new_value.
[read_item, T, X] Indicates that transaction T has read the value of
database item X.
[commit, T] Indicates that transaction T has completed
successfully, and affirms that its effect can be
committed (recorded permanently) to the database.
[abort, T] Indicates that transaction T has been aborted.

- Recovery from a transaction failure amounts to either undoing or redoing transaction

operations individually from the log.
- Undo the effect of these WRITE operations of a transaction T by tracing backward
through the log and resetting all items changed by a WRITE operation of T to their
old_values.
- Redo of an operation may also be necessary if a transaction has its updates recorded in
the log but a failure occurs before the system can be sure that all these new_values have
been written to the actual database on disk from the main memory buffers.

Commit Point of a Transaction

- A transaction T reaches its commit point when all its operations that access the database have
been executed successfully and the effect of all the transaction operations on the database
have been recorded in the log.
- If a system failure occurs, we can search back in the
Updating a disk file involves
log for all transactions T that have written a
copying the appropriate block of
[start_transaction, T] record into the log but have
the file from disk to a buffer in
not written their [commit, T] record yet; these
main memory, updating the buffer
transactions may have to be rolled back to undo
in main memory, and copying the
their effect on the database during the recovery buffer to disk.
process.
- Transactions that have written their commit record
in the log must also have recorded all their WRITE operations in the log, so their effect on
the database can be redone from the log records.
- One or more blocks of the log file is kept in main memory buffers, called the log buffer,
until they are filled with log entries and then to write them back to disk only once, rather
than writing to disk every time a log entry is added saving the overhead of multiple disk
writes.

Prepared by : Mrs.Pravicha.M.T Page 2

 Chapter 4: Advanced Database Management
G2CS&IT
- At the time of a system crash, only the log entries that have been written back to disk are
considered in the recovery process. Hence, before a transaction reaches its commit point,
any portion of the log that has not been written to the disk yet must now be written to the
disk. This process is called force-writing the log buffer before committing a transaction.

Caching (Buffering) of Disk Blocks

- Cache the disk pages (blocks)–containing database items to be updated- into main
memory buffers.
- Then update the main memory buffers before being written back to disk.
- For the efficiency of recovery purpose, the caching of disk pages is handled by the
DBMS instead of the OS. Typically, a collection of in-memory buffers, called DBMS
cache kept under the control of the DBMS.
- A directory for the cache is used to keep track of which DB items are in the buffers.
• A table of <disk page address, buffer location> entries.
• The DBMS cache holds the database disk blocks including
o Data blocks o Log blocks
o Index blocks
When DBMS requests action on some item
• First checks the cache directory to determine if the corresponding disk page is in
the cache.
• If no, the item must be located on disk and the appropriate disk pages are copied
into the cache.
• It may be necessary to replace (flush) some of the cache buffers to make space
available for the new item.
- Dirty bit
• Associate with each buffer in the cache a dirty bit.
• The dirty bit can be included in the directory entry.
• It indicates whether or not the buffer has been modified.
• Set dirty bit=0 when the page is first read from disk to the buffer cache.
• Set dirty bit=1 as soon as the corresponding buffer is modified.
• When the buffer content is replaced –flushed- from the cache, write it back to the
corresponding disk page only if dirty bit=1
- Pin-unpin bit.
• A page is pinned –i.e. pin-unpin bit value=1-, if it cannot be written back to disk
as yet.
- Strategies that can be used when flushing occurs.
• In-place updating
• Writes the buffer back to the same original disk location overwriting the
old value on disk.
• Shadowing
o Writes the updated buffer at a different disk location.
o Multiple versions of data items can be maintained.
o The old value called BFIM –before image, & the new value AFIM –after
image are kept on disk, so no need of log for recovery.
Prepared by : Mrs.Pravicha.M.T Page 3
 Chapter 4: Advanced Database Management
G2CS&IT
Checkpoints
- In case of failure, the recovery manager requires that the entire log be examined to
process recovery which is a time consuming one.
- A quick way to limit the amount of log to scan on recovery can be achieved using
checkpoints.
- A [checkpoint] record is written into the log periodically at that point when the
system writes out to the database on disk all DBMS buffers that have been modified.
- Hence, all transactions with [commit, T] entry in the log before [checkpoint] entry do not
need to have their WRITE operations redone in case of crash.
- Since all their update have been recorded in the DB on disk during check pointing.
- The recovery manager must decide at what intervals (measured in time) to take a
checkpoint.
- Take a checkpoint consists of the following:
o Suspend execution of transactions temporarily.
o Force-write all main memory buffers that have been modified to disk.
o Write a [checkpoint] record to the log, and force-write the log to disk.
o Resume executing transactions.
- The time needed to force-write all modified memory buffers may delay transaction
processing.
- Use fuzzy check pointing to reduce this delay.
- The system can resume transaction processing after [checkpoint] record written to the log
without having to wait for step 2 to finish. However, until step 2 is completed, the
previous [checkpoint] record should remain to be valid. The system maintains a pointer to
the valid checkpoint which points to the previous [checkpoint] recording the log. Once
step2 is conducted, that pointer is changed to point to the new checkpoint in the log.

Write-Ahead Logging (WAL)

Two types of log entry information for a write command.
- The information needed for UNDO: A UNDO-type log entries include the old values
(BFIM), since this is needed to undo the effect of the operations from the log.
- The information needed for REDO: A REDO-type log entries including the new values
(AFIM), since it is needed to redo the effect of the operations from the log.
- In UNDO/REDO algorithms, both types of log entries are combined.
- The log includes read commands only when cascading rollback is possible.
Write-Ahead Logging (WAL) protocol ensures that a record entry- of every change to the DB
is available while attempting to recover from a crash. WAL ensures that the BFIM of the data
item is recorded in the appropriate log entry and that the log entry is flushed to disk before the
BFIM is overwritten with the AFIM in the database on disk.
- Suppose that that the BFIM of a data item on disk has been overwritten by the AFIM on
disk and a crash occurs.
o Without ensuring that this BFIM is recorded in the appropriate log entry and the
log is flushed to disk before the BFIM is overwritten with the AFIM in the DB on
disk, the recovery will not be possible.
Prepared by : Mrs.Pravicha.M.T Page 4
 Chapter 4: Advanced Database Management
G2CS&IT
- Suppose a transaction made a change and committed with some of its changes not yet
written to disk.
o Without a record of these changes written to disk, there would be no way to
ensure that the changes of the committed transaction survive crashes
- When a log record is written, it is stored in the current log in the DBMS cache and after
written to disk as soon as is feasible.
- With Write-Ahead Logging, the log blocks containing the associated log records for a
particular data block update, must be first written to disk before the data block itself can
be written back to disk.
- IBM DB2, Informix, Microsoft SQL Server, Oracle 8, and Sybase ASE all use a WAL
scheme for recovery.
- To facilitate the recovery process, the DBMS recovery subsystem may need to maintain a
number of lists.
o List of active transactions: transactions started but not committed yet.
o List of committed transactions since last checkpoint.
o List of aborted transactions since last checkpoint.

Steal/no-steal-- Force/no-force Approaches

No-steal approach A cache page updated by a transaction cannot be written to disk before the
transaction commits. Deferred update follows this approach. The pin-unpin
bit indicates if a page cannot be written back to disk
Steal approach An updated buffer can be written before the transaction commits. Used
when the buffer manager replaces an existing page in the cache, that has been
updated by a transaction not yet committed, by another page requested by
another transaction. Advantage: avoid the need for a very large buffer space.
Force approach All pages updated by a transaction are immediately written to disk when the
transaction commits.
No-Force All pages updated by a transaction are not immediately written to disk when
approaches the transaction commits. Advantage: an updated page of a committed
transaction may be still in the buffer when another transaction needs to update
it; saving I/O cost.
Typical database systems follow a steal/no-force strategy.

Main Recovery Techniques

1. Deferred Update Techniques.

- A transaction cannot change the database on disk until it reaches its commit point.
- A transaction does not reach its commit point until all its update operations are recorded
in the log and the log is force-written to disk –i.e. WAL.
- During commit, the updates are first recorded persistently in the log and then written to
the DB.
- If a transaction fails before reaching its commit point, no UNDO is needed because the
transaction has not affected the database on disk in any way.
- If there is a crash, it may be necessary to REDO the effects of committed transactions
from the Log because their effect may not have been recorded in the database.

Prepared by : Mrs.Pravicha.M.T Page 5

 Chapter 4: Advanced Database Management
G2CS&IT
- Deferred update also known as NO-UNDO/REDO algorithm.
- In multiuser systems, the method of recovery from failure is usually closely related to the
concurrency control method.

Recovery Using Deferred update on Multiuser Environment (RDU_M)

-(NO-UNDO/REDO with checkpoints)
- The concurrency control and recovery process are interrelated. The recovery process may be
more complex depending on the protocols used for concurrency control.
- Assume a transaction processing system that uses Strict 2PL as concurrency control protocol
and [checkpoint] entries are included in the log
1. Identify two lists for a transaction.
2. A list of committed transaction since last checkpoint and a list of active transactions.
Apply the REDO operation to all the write_item operations of the committed transaction
from the log in the order in which they were written to the log.
o REDO(write_opr)
 Examine the log entry [write_item,T,X,new_value] and setting the value
of item X in the database to new_value, which is the after image (AFIM).
3. For uncommitted active transaction T, a log entry is made as [abort,T] & restart the active
transactions either automatically by the recovery process or manually by the user.
4. Make the NO-UNDO/REDO algorithm more efficient by only REDO the last update of
X. Start from the end of the log and only REDO the first occurrence of X in the log.
- Advantages
 A transaction does not record any changes in the DB on disk until it commits & so never
rollback because of transaction failure during transaction execution.
 A transaction will never read the value of an item that is written by an uncommitted
transaction; hence no cascading rollback will occur.
- Drawbacks
 Limits the concurrent execution of transactions because all items remain locked until the
transaction reaches its commit point –due to 2PL.
 Require excessive buffer space to hold all updated items until the transactions commit

2. Immediate Update Techniques.

- In these techniques, the DB on disk can be updated immediately without any need to wait
for the transaction to reach its commit point.

Prepared by : Mrs.Pravicha.M.T Page 6

 Chapter 4: Advanced Database Management
G2CS&IT
- However, the update operation must still be recorded in the log (on disk) before it is
applied to the database -using WAL protocol- so that we can recover in case of failure.
- If a transaction fails before reaching commit point, it must be rolled back by undoing the
effect of its operations on the DB.
- If the recovery technique ensures that all updates of a transaction are recorded in the DB
on disk before the transaction commits, there is never a need to REDO any operations of
committed transactions – UNDO/NO-REDO recovery algorithm.
- If the transaction is allowed to commit before all its changes are written to the DB,
REDO all the operations of committed transactions. – UNDO/REDO recovery
algorithm.

Recovery Using Immediate Update in a Multiuser User Environment (RIU_M)

- (UNDO/REDO)
Assume
- Log includes checkpoints.
- Strict 2PL concurrency control protocol is used.

1. Identify two lists for a transaction.

2. A list of committed transaction since last checkpoint and a list of active transactions.
3. Undo all write_item operations of the active transaction from the log, in the reverse of
the order in which they were written into the log & writes a log record [abort,T].
4. Redo the write_item operations of the committed transactions from the log, in the order
in which they were written in the log.
a. REDO(write_op).
b. Examine the log entry [write_item,T,X,old_value,new_value] and setting the
value of item X in the DB to old_value which is the before image (BFIM).
5. REDO is more efficiently done by starting from the end of the log and redoing only the
last update of each item X.
6. Whenever an item is redone, it is added to a list of redone items and is not redone again.

Shadow Paging
A recovery scheme
 In a single-user environment, doesn’t require the use of log.
 In multi-user environment, the log may be needed for concurrency control method.
- The DB is made up of ‘n’ fixed-size disk pages –blocks.
- A directory with ‘n’ entries where the ith entry points to the ith DB page on disk.
o All references –reads or writes- to the DB pages on disk go through the directory.
o The directory is kept in main memory if not too large.
o The current directory entries point to the most recent or current DB pages on disk.
- When a transaction begins executing, the current directory is copied into a shadow directory
and the shadow directory is saved on disk.
- During transaction execution, all updates are performed using the current directory and the
shadow directory is never modified.
Prepared by : Mrs.Pravicha.M.T Page 7
 Chapter 4: Advanced Database Management
G2CS&IT
- When a write_item operation is performed
o A new copy of the modified DB page is created and the old copy is not overwritten. Two
versions, of the page updated by the transaction, are kept.
o The new page is written elsewhere
on some unused disk block.
o The current directory entry is
modified to point to the new disk
block.
o The shadow directory is not
modified.
- To recover from a failure:
o Delete the modified database
pages & discard the current
directory.
o The state of the database before
transaction execution is available through the shadow directory and is recovered by
reinstating the shadow directory.
o Committing a transaction corresponds to discarding the previous shadow directory.
o NO-UNDO/NO-REDO technique since neither undoing or redoing of data items
In a multiuser environment, logs and checkpoints must be incorporated.
Drawbacks
- The updated pages change location on disk.
- Difficult to keep related DB pages close together on disk without complex storage
management strategies.
- The overhead of writing shadow directories to disk as transactions start is significant.
- A complicated garbage collection when a transaction commits.
- The old pages referenced by the shadow directory that have been updated must be released
and added to a list of free pages for future use.
- The migration between current and shadow directories must be implemented as an atomic
operation
ARIES ALGORITHM - An Example of a recovery algorithm used in database systems.

- Steal/No-Force approach for writing.

- Based on the following three concepts
1. Write-ahead logging.
2. Repeating history during redo.
a. Retrace all actions of the database system prior to the crash to reconstruct the
database state when the crash occurred.
b. Transactions that were uncommitted at the time of the crash are undone.
3. Logging changes during redo.
a. Prevents from repeating the completed undo operations if a failure occurs during
recovery, which causes a restart of the recovery process.

Prepared by : Mrs.Pravicha.M.T Page 8

 Chapter 4: Advanced Database Management
G2CS&IT
 Log Sequence Number: ARIES log record has an associated log sequence number (LSN)
that is monotonically increasing and indicates the address of the log record on disk. Each
LSN corresponds to a specific change (action) of some transaction.
o Common fields in all log records include the previous LSN for that transaction, the
transaction ID, and the type of log record. The previous LSN is important because it
links the log records (in reverse order) for each transaction. For an update (write)
action, additional fields in the log record include the page ID for the page that
contains the item, the length of the updated item, its offset from the beginning of the
page, the before image of the item, and its after image.
 Checkpointing: Log consists of the following: writing a begin_checkpoint record to the log,
writing an end_checkpoint record to the log, and writing the LSN of the begin_checkpoint
record to a special file. This special file is accessed during recovery to locate the last
checkpoint information. With the end_checkpoint record, the contents of both the
Transaction Table and Dirty Page Table are appended to the end of the log. To reduce the
cost, fuzzy checkpointing is used so that the DBMS can continue to execute transactions
during checkpointing.
o Additionally, the contents of the DBMS cache do not have to be flushed to disk
during checkpoint, since the Transaction Table and Dirty Page Table which are
appended to the log on disk contain the information needed for recovery. If a crash
occurs during checkpointing, the special file will refer to the previous checkpoint,
which is used for recovery.
 Transaction Table: Contains an entry for each active transaction, with information such as
the transaction ID, transaction status, and the LSN of the most recent log record for the
transaction.
 Dirty Page Table: Contains an entry for each dirty page in the buffer, which includes the
page ID and the LSN corresponding to the earliest update to that page.

Consists of mainly 3 steps:

1. Analysis
a. Identifies the dirty (updated) pages in the buffer.
b.Identifies the set of transactions at the time of crash.
c. Identify the appropriate point in the log, where the REDO operation should start.
2. Redo
a. Reapplies updates from the log to the database, generally applied to the committed
transactions. ARIES log will provide the start point for REDO, from which REDO
operations are applied until the end of the log is reached. Also ARIES determine whether
the operation to be redone has actually been applied to the database or not. Only those
operations are redone, which is not applied to the database.
3. Undo
a. The log is scanned backward and the operations of transactions that were active at the
time of the crash are undone in reverse order. The information required for undo
operation is stored in log, the Transaction Table and the Dirty page table. These tables are
maintained by the transactions manager and written to the log during check pointing.

Prepared by : Mrs.Pravicha.M.T Page 9

 Chapter 4: Advanced Database Management
G2CS&IT
Recovery Process: After a crash, the ARIES recovery manager takes over.
Analysis Phase
 Information about the last begin_checkpoint is first accessed through the special file.
 The analysis phase starts at the begin_checkpoint record and proceeds to the end of the log.
 When the end_checkpoint record is encountered, the Transaction Table and Dirty Page Table
are accessed.(these tables were written in the log during check pointing).
 Analysis phase will further reconstruct Transaction Table and Dirty Page Table.
 When the analysis phase encounters log record after end_checkpoint, a new entry for
transaction is made in the Transaction Table and a new entry for the related page is made in
the Dirty Page Table.
 During analysis, the log records being analyzed may cause modifications to these two tables.
o If an end log record was encountered for a transaction T in the Transaction Table, then
the entry for T is deleted from that table.
o If some other type of log record is encountered for a transaction T, then an entry for T is
inserted into the Transaction Table, if not already present, and the last LSN field is
modified.
o If the log record corresponds to a change for page P, then an entry would be made for
page P (if not present in the table) and the associated LSN field would be modified.
o When the analysis phase is complete, the necessary information for REDO and UNDO
has been compiled in the tables.

REDO Phase
ARIES starts redoing at a point in the log where it knows (for sure) that previous changes to dirty
pages have already been applied to the database on disk.
 Suppose the smallest LSN of all the dirty pages in the Dirty Page Table is M.
 REDO starts at the log record with LSN = M and scans forward to the end of the log.
 For any changes corresponding to an LSN < M, transactions, must have already been
propagated to disk or already been overwritten in the buffer; otherwise, those dirty pages
with that LSN would be in the buffer (and the Dirty Page Table).
For each change recorded in the log, the REDO algorithm would verify whether or not the
change has to be reapplied.
 If a change recorded in the log pertains to page P that is not in the Dirty Page Table, then
this change is already on disk and does not need to be reapplied.
 If a change recorded in the log (with LSN = N, say) pertains to page P and the Dirty Page
Table contains an entry for P with LSN greater than N, then the change is already present.
 If neither of these two conditions hold, page P is read from disk and the LSN stored on
that page, LSN(P), is compared with N. If N < LSN(P), then the change has been applied
and the page does not need to be rewritten to disk.

UNDO Phase
 The set of active transactions called the undo_set has been identified in the Transaction
Table during the analysis phase.

Prepared by : Mrs.Pravicha.M.T Page 10

 Chapter 4: Advanced Database Management
G2CS&IT
 The UNDO phase proceeds by scanning backward from the end of the log and undoing
the appropriate actions.
 A compensating log record is written for each action that is undone.
 When this is completed, the recovery process is finished and normal processing can begin
again.

Consider the recovery example shown in Figure 23.5. There are three transactions:
T1, T2, and T3. T1 updates page C, T2 updates pages B and C, and T3 updates page A.

Fig a) – A log at point of crash.

Fig b) –Transaction & Dirty page table at the time of
checkpoint.
Fig c) –Transaction & Dirty page table after the
analysis phase.

Review Questions
1. Discuss the different types of transaction failures. What is meant by catastrophic failure?
2. What is the system log used for? What are the typical kinds of entries in a system log?
What are checkpoints, and why are they important? What are transaction commit points,
and why are they important?
3. How are buffering and caching techniques used by the recovery subsystem?
4. Describe the write-ahead logging protocol.
5. What is the difference between the UNDO/REDO and the UNDO/NO-REDO algorithms
for recovery with immediate update?
6. Describe the shadow paging recovery technique. Under what circumstances does it not
require a log?
7. What are log sequence numbers (LSNs) in ARIES? How are they used? What
information do the Dirty Page Table and Transaction Table contain?
8. Describe how fuzzy checkpointing is used in ARIES.
9. Describe the three phases of the ARIES recovery method.
10. What do the terms steal/no-steal and force/no-force mean with regard to buffer
management for transaction processing?

Prepared by : Mrs.Pravicha.M.T Page 11