Oracle Trace File Analyzer

Overview
OR AC LE WHI TE P AP E R | M AY 2017
Table of Contents

Introduction 1

Install Oracle Trace File Analyzer 2

Supported Environments 2

Install on Linux / Unix as root User in Daemon Mode 2

Install on Linux / Unix as Non root User in Non Daemon Mode 2

Install on Microsoft Windows 3

Important Directories 3

How Oracle Trace File Analyzer Works 4

Command Interfaces 5

Configure Oracle Trace File Analyzer 5

Mask Sensitive Data 5

Change Default User Access 6

Automatic Diagnostic Collections 6

Configure Email Notification 7

On-demand Analysis and Collection 8

View System & Cluster Summary 9

Investigate Logs & Look for Errors 10

Perform Analysis Using the Included Tools 10

Collect Diagnostic Data & Use One Command SRDCs 12

Change Clusterware Trace Levels 15

Maintain Oracle Trace File Analyzer to the Latest Version 15

Conclusion 16

0 | ORACLE TRACE FILE ANALYZER – OVERVIEW GUIDE

Introduction
As a DBA, you're expected to do more work, with fewer resources all the time. You're under pressure to keep mission-
critical apps up and running. When something goes wrong, everyone looks to you to understand what went wrong and
how to fix it.

It's not always easy. You've got to run the right tools, at the right time. If you're using Oracle Clusterware, then you have
to also collect from all the database nodes. You might need to use lots of different tools that you only ever use now and
again, and they each have their own syntax.

Once you've managed to get all the data, it can be huge. Only a fraction of what you've collected is useful, but who
knows which bit is relevant? That's if you were able to get everything. If you were quick enough before it was
overwritten.

In the meantime, you’ve still got a problem. This is costing your company money and you still need to get it fixed.

Oracle Trace File Analyzer helps you analyze and collect diagnostic data. This is a crucial step to resolving problems
that may occur with your Oracle Database.

It watches your logs for significant problems that may impact your service. If desired it can also automatically collect the
relevant diagnostics, when it sees these problems.

Oracle Trace File Analyzer knows what is relevant in log files. This allows it to trim them to the smallest size, yet still
gather everything necessary. It also collects data across cluster nodes and consolidates everything in one place.

Using important database diagnostic tools is easy with Oracle Trace File Analyzer. It hides the complexity by providing
a single interface and syntax to them all.

This all combines to ensure you can get exactly what you need, when you need it and save your business money.

1 | ORACLE TRACE FILE ANALYZER – OVERVIEW GUIDE

Install Oracle Trace File Analyzer

Supported Environments
You can use Oracle Trace File Analyzer with all supported versions of:

» Oracle Database
» Oracle Clusterware

It works on the following operating systems:

» Linux (OEL, RedHat, SUSE, Itanium & zLinux)

» Oracle Solaris (SPARC & x86-64)
» AIX
» HPUX (Itanium & PA-RISC)
» Microsoft Windows
Operating system versions supported are the same as those supported by the Oracle Database. You need to use a Java Runtime
Edition of version 1.8.

You may already have Oracle Trace File Analyzer installed. The Oracle Clusterware install shipped with Oracle Trace File Analyzer
since versions 11.2.0.4 and 12.1.0.2. However, this install does not include many of the database tools. Oracle releases new versions
of Oracle Trace File Analyzer several times a year. These new releases include new features and bug fixes. Ensure you get the latest
Oracle Trace File Analyzer with database support tools bundle, from Document 1513912.1

Install on Linux / Unix as root User in Daemon Mode

If possible you should install Oracle Trace File Analyzer as root. This will give you the richest capabilities. If Oracle Trace File Analyzer
is already installed, reinstalling will perform an upgrade to the existing location. If it is not already installed, the recommended location is
/opt/oracle.tfa

To install as root:

1. Download the appropriate Oracle Trace File Analyzer zip, copy to required machine and unzip.
2. Run the installTFA command:

$ ./installTFA<platform>

The installation will prompt if you want to do a local or cluster install.

Cluster install requires passwordless ssh user equivalency for root to all cluster nodes. If not already configured the installation can set
this up and then remove again at the end. If you do not wish to use ssh, you can install on each host using a local install. Then use
tfactl syncnodes to generate and deploy the relevant SSL certificates.

The Cluster Ready Services (CRS) do not manage Oracle Trace File Analyzer. This is because it needs to be available should CR S go
down. The installation configures Oracle Trace File Analyzer for auto start. The implementation of auto start is platform dependent.
Linux uses init or an init replacement such as upstart or systemd. Microsoft Windows uses a Windows Service.

Install on Linux / Unix as Non root User in Non Daemon Mode

If you are unable to install as root, then you can install as the ORACLE_HOME owner. In this installation mode, Oracle Trace File
Analyzer will have reduced capabilities. You will not be able to use the following functionality:

» Automatic collections
» Collections from remote hosts
» Collecting of files which are not readable by the ORACLE_HOME owner, e.g. /var/log/messages or certain clusterware daemon logs
To install as an ORACLE_HOME owner use the –extractto option. This tells Oracle Trace File Analyzer where to install to. Also use the
–javahome option to instruct which JRE to use. Use the JRE already available in the RDBMS home unless you have a later version
available.

./installTFA<platform> -extractto <install_dir> -javahome <jre_home>

Install on Microsoft Windows

1. Download the appropriate Oracle Trace File Analyzer zip, copy to one of the required machines and unzip.
2. Open a command prompt as administrator and run installation script specifying a perl_home, e.g.
D:\oracle\product\12.2.0\dbhome_1\perl:

install.bat -perlhome <perl_home>

The installation will prompt if you want to do a local or cluster install. If you select cluster install, it will be installed remotely on nodes of
the cluster. Alternatively you can perform a local install on each host. Then use tfactl syncnodes to generate and deploy the relevant
SSL certificates.

Important Directories
The location of the tfa_home and bin directory differs based on installation type. When installed with Oracle Clusterware the tfa_home
will be within the GRID_HOME directory.

KEY ORACLE TRACE FILE ANALYZER DIRECTORIES

Directory Description

tfa/bin Contains the command line interface tfactl ( tfactl is also in GRID_HOME/bin
directory if Oracle Clusterware is installed)

tfa/repository Where Oracle Trace File Analyzer stores collections

tfa/<node>/tfa_home/database Contains Berkeley database that stores data about the system

tfa/<node>/tfa_home/diag Tools for troubleshooting Oracle Trace File Analyzer

tfa/<node>/tfa_home/diagnostics_to_collect Place files in here to include them in the next collection, then have them
deleted afterwards

tfa/<node>/tfa_home/log Contains logs about Oracle Trace File Analyzer operation

tfa/<node>/tfa_home/resources Contains resource files, for example the log masking control file

tfa/<node>/tfa_home/output Contains extra meta data about the environment

How Oracle Trace File Analyzer Works

Oracle Trace File Analyzer runs a daemon on each cluster node, or single instance if you're not using Oracle Clusterware.

The tfactl command communicates with the local daemon. The local daemon then coordinates with all nodes.
The daemon on each node handles:
» Script execution
» Collection of diagnostics
» Trimming of log contents
The local daemon consolidates cluster wide collection output on the originating node
Automatic collection will perform a local only collection.

Oracle Trace File Analyzer discovers databases and cluster nodes on its own. Yet you can still manually add or remove them from the
configuration, if needed. See the user guide on Document 1513912.1 for a full list of command options.

The resource footprint is small. You will not usually be aware it is running. The only times Oracle Trace File Analyzer will consume
noticeable CPU are:
» When performing an inventory of diagnostic files
» During diagnostic collection

Command Interfaces
The tfactl tool functions as:
» Command line interface
» Shell interface
» Menu interface

INTERFACE TYPES & USAGE

Interface Type Command How to use

Command line $ tfactl <command> Specify all command options at the command line

Shell interface $ tfactl Set and change context. Then run commands from within the shell

Menu Interface $ tfactl menu Select menu navigation options then choose the command you want to run

Configure Oracle Trace File Analyzer

Mask Sensitive Data

Oracle Trace File Analyzer can mask sensitive data such as hostnames or IP addresses. To configure masking, edit or create the file
tfa_home/resources/mask_strings.xml, then copy to each node. The mask_strings.xml should use the following format to define data
replacements:

<mask_strings>
<mask_string>
<original>WidgetNode1</original>
<replacement>MyReplacementName</replacement>
</mask_string>
<mask_string>
<original>192.168.5.1</original>
<replacement>Node1-IP</replacement>
</mask_string>
</mask_strings>

Change Default User Access

By default the GRID_HOME owner and all ORACLE_HOME owners will have access to their respective information via Oracle Trace
File Analyzer. No one else will be able to perform collections. User access is only applicable if installed as root on Linux / Unix. User
access is not applicable if installed as non root or on Microsoft Windows. Enable or disable user access via:

$ tfactl access enable

or

$ tfactl access disable

All commands will apply cluster wide unless you specify -local to restrict to only the local node.

USEFUL USER ACCESS COMMANDS

Command Description

$ tfactl access lsusers List user access

$ tfactl access add –user <user> [-local] Add users

$ tfactl access remove –user <user> [-local] Remove users

$ tfactl access removeall [-local] Remove everyone

$ tfactl access reset Rest to default access

Automatic Diagnostic Collections

Oracle Trace File Analyzer will watch your logs for significant problems, such as internal errors like ORA-00600 or node evictions. If
detected it will:
 Invoke any necessary diagnostics and collect all relevant log data at the time of a problem
Trim log files around the time of the problem, so it only collects what is necessary for diagnosis
Collect and package all trimmed diagnostics. From all nodes in the cluster, consolidating everything on a single node
Store the collection in the Oracle Trace File Analyzer repository
Send you email notification of the problem and details of diagnostic collection, ready for upload to Oracle Support

Oracle Trace File Analyzer uses a flood control mechanism. Repeated errors do not flood the system with automatic collections.

When an event is first identified, this triggers the start point for a collection. 5 minutes later diagnostic gathering starts. This is to capture
any other relevant events together. If after 5 minutes events are still occurring, diagnostic collection continues to wait. It will wait for a
period of 30 seconds with no events occurring, up to a further 5 minutes.

If events are still occurring after 10 minutes, a diagnostic collection happens. A new collection point starts.

Once collection is complete TFA will send email notification to relevant people, including details of where the collection results are.

Automatic collections are ON by default. You can turn them OFF or back ON again with:

$ tfactl set autodiagcollect=<ON|OFF>

LOG ENTRIES THAT TRIGGER AUTOMATIC COLLECTIONS

String pattern Logs monitored

ORA-297(01|02|03|08|09|10|40) » Alert Log - DB

ORA-00600 » Alert Log – ASM
ORA-07445 » Alert Log – ASM Proxy
ORA--04(69|([7-8][0-9]|9([0-3]|[5-8]))) » Alert Log – ASM IO Server
ORA-32701
ORA-00494
System State dumped

CRS-016(07|10|11|12) » Alert Log - CRS

Configure Email Notification

You can provide Oracle Trace File Analyzer with one or more comma separated email addresses to send notification of problems.
To set the notification email address to use for a specific ORACLE_HOME, include the OS owner in the command:

$ tfactl set notificationAddress=oracle:[email protected]

To set the notification email to use for any ORACLE_HOME use:

$ tfactl set [email protected]

After receiving notification of a problem you should:

Inspect the referenced collection details to determine if you know the root cause
Resolve the underlying cause of the problem if you know how
If you do not know the root cause of the problem then log an SR with Oracle Support and upload the relevant collection

On-demand Analysis and Collection

You can run Oracle Trace File Analyzer on-demand via the command line tool tfactl.
The tfactl command can perform analysis using a combination of different database tools. The tfactl command allows you to acc ess all
tools using a common syntax. This hides the complexity of the syntax differences between the tools.

Use the Oracle Trace File Analyzer tools to perform analysis and resolve problems. If you need more help, use the tfactl command to
collect diagnostics for Oracle Support.
» Oracle Trace File Analyzer will collect all relevant log data from a time of your choosing
» It will trim logs files around the time, collecting only what is necessary for diagnosis
» Oracle Trace File Analyzer will package all diagnostics on the node where tfactl was run from

View System & Cluster Summary

The summary command can be used to quickly understand the viability of systems and clusters. It shows a fast, easy to read summary
of the status including any potential problems with important elements such as:

» Clusterware
» ASM
» Patch
» Listener
» OS
» Network
Summary is currently only supported on Linux

Usage:

$ tfactl summary [options]

The summary command provides an interface of its own allowing you to navigate the summary output.
Run with no options for default summary.

SUMMARY OPTIONS

Option Description

-overview Complete summary collection - overview

-crs CRS status summary

-asm ASM status summary

-acfs ACFS status summary

-database Database status summary

-exadata Exadata status summary

-patch Patch details

-listener Listener status summary

-network Network status summary

-os OS status summary

-tfa Oracle Trace File Analyzer status summary

-summary Summary tool metadata

-json Outputs summary in json format as well as via the summary interface

-html Outputs summary in html format as well as via the summary interface

-print Display [html or json] report at console

-silent Interactive console by default

-node < local | node_name > Local or comma separated node name(s)

-help Display help

Investigate Logs & Look for Errors

You can use Oracle Trace File Analyzer to analyze all your logs across your cluster and tell you about any recent errors.

For example:

$ tfactl analyze –last 1d

or

$ tfactl analyze –last 18h

This will report all errors it finds over the specified duration.

You can also use Oracle Trace File Analyzer to find all occurrences of a specific error on any node. For example, this command will
search for ORA-00600 errors:

$ tfactl analyze -search “ora-00600" -last 8h

Perform Analysis Using the Included Tools

Oracle Trace File Analyzer with database support tools bundle includes the following tools. These tools are only available when Oracle
Trace File Analyzer is downloaded from Document 1513912.1.

TOOLS INCLUDED ON LINUX / UNIX

Tool Description

orachk or exachk Provides health checks for the Oracle stack.

Oracle Trace File Analyzer will install either
» Oracle EXAchk for Engineered Systems, see document 1070954.1 for more details
or
» Oracle ORAchk for all non-Engineered Systems, see document 1268927.2 for more details

oswatcher Collects and archives OS metrics. These are useful for instance or node evictions & performance Issues. See document 301137.1 for more
details

procwatcher Automates & captures database performance diagnostics and session level hang information. See document 459694.1 for more details

oratop Provides near real-time database monitoring. See document 1500864.1 for more details.

sqlt Captures SQL trace data useful for tuning. See document 215187.1 for more details.

alertsummary Provides summary of events for one or more database or ASM alert files from all nodes

ls Lists all files Oracle Trace File Analyzer knows about for a given file name pattern, across all nodes

pstack Generates the process stack for the specified processes, across all nodes

grep Searches for a given string in the alert or trace files with a specified database

summary Provides high level summary of the configuration

vi Opens alert or trace files for viewing a given database and file name pattern in the vi editor

tail Runs a tail on an alert or trace files for a given database and file name pattern

param Shows all database and OS parameters that match a specified pattern

dbglevel Sets and unsets multiple CRS trace levels with one command

history Shows the shell history for the tfactl shell

changes Reports changes in the system setup over a given time period. This includes database parameters, OS parameters and patches applied

calog Reports major events from the Cluster Event log

events Reports warnings and errors seen in the logs

managelogs Shows disk space usage and purges ADR log and trace files

ps Finds processes

triage Summarize oswatcher/exawatcher data

TOOLS INCLUDED ON WINDOWS

Tool Description
calog Reports major events from the Cluster Event log

changes Reports changes in the system setup over a given time period. This includes database parameters, OS parameters and patches applied

dir Lists all files Oracle Trace File Analyzer knows about for a given file name pattern, across all nodes

events Reports warnings and errors seen in the logs

findstr Searches for a given string in the alert or trace files with a specified database

history Shows the shell history for the tfactl shell

managelogs Shows disk space usage and purges ADR log and trace files

notepad Opens alert or trace files for viewing a given database and file name pattern in the notepad editor

param Shows all database and OS parameters that match a specified pattern

summary Provides high level summary of the configuration

tasklist Finds processes

To verify which tools you have installed run:

$ tfactl toolstatus

Each tool can be run using tfactl either in command line or shell mode.

To run a tool from the command line use:

$ tfactl run <tool>

The following example shows how to use tfactl in shell mode. It will start tfactl, connect to the database MyDB and run oratop:

$ tfactl

tfaclt > database MyDB

MyDB tfactl > oratop

Collect Diagnostic Data & Use One Command SRDCs

To perform an on-demand diagnostic collection use:
$ tfactl diagcollect

This will trim & collect all important log files updated in the past 12 hours across the whole cluster. Oracle Trace File Analyzer will store
collections in the repository directory. You can change the diagcollect timeframe with the option –last <n>h|d.

Oracle Support will often ask you to run a Service Request Data Collection (SRDC). The SRDC is dependent on the type of probl em
you experienced. It is a series of many data gathering instructions aimed at diagnosing your problem. Collecting the SRDC manually
can be difficult, with many different steps required.

Oracle Trace File Analyzer can run SRDC collections for you with a single command:

$ tfactl diagcollcet -srdc <srdc_type>

To run SRDCs use one of the Oracle privileged user accounts:

» ORACLE_HOME owner
» GRID_HOME owner

These are the different SRDC types available:

Available Collection Scope

Type of Problem
SRDCs

ORA Errors » ORA-00600 » ORA-27300 Local only

» ORA-00700 » ORA-27301
» ORA-04030 » ORA-27302
» ORA-04031
» ORA-07445

Other internal database errors » internalerror Local only

Database performance problems » dbperf Cluster wide

Database patching problems » dbpatchinstall Local only

» dbpatchconflict

Database install / upgrade problems » dbinstall Local only

» dbupgrade

Enterprise Manager tablespace usage metric problems » emtbsmetrics Local only (on EM Agent target)

Enterprise Manager general metrics page or threshold problems » emmetricalert Local only (on EM Agent target & Repository DB)

Enterprise Manager debug log collection » emdebugon Local only (on EM Agent target & OMS)
Run emdebugon, reproduce the problem then run emdebugoff, » emdebugoff
which disables debug again and collects debug logs

What the SRDCs collect varies for each type, here are two examples:
ONE COMMAND SRDC AND COLLECTIONS

Command What gets collected

$ tfactl diagcollect –srdc ORA-04031 » IPS Package

» Patch Listing
» AWR report
» Memory information
» RDA HCVE output

$ tfactl diagcollect –srdc dbperf » ADDM report

» AWR for good period and problem period
» AWR Compare Period report
» ASH report for good and problem period
» OS Watcher
» IPS Package (if errors during problem period)
» ORAchk (performance related checks)

Oracle Trace File Analyzer will prompt you to enter the information it needs based on the SRDC type.

For example, this is what happens when you run the ora4031 SRDC:

$ tfactl diagcollect –srdc ORA-04031

You are prompted to enter event date/time and database name

Oracle Trace File Analyzer scans the system to identify recent events in the system (up to 10)
Once the relevant event is chosen, it proceeds with diagnostic collection
All required files are identified
All files are trimmed where applicable
All data is package in a zip ready to provide to support
You can also run an SRDC collection in non-interactive silent mode. Just provide all the required parameters up front like this:

$ tfactl diagcollect –srdc <srdc_type> -database <db> -from "<date> <time>" -to "<date> <time>"
Change Clusterware Trace Levels
Oracle Support may ask you to enable certain trace levels when reproducing a problem. This is to allow you to collect enough
diagnostics to diagnose the cause of the problem.

Oracle Trace File Analyzer makes it easy to enable and then disable the correct trace levels. Use dbglevel.

You can find the required trace level settings grouped by problem trace profiles. To set a trace profile use:

$ tfactl dgblevel –set <profile >

For a list of the available profiles run:

$ tfactl dgblevel –help

Maintain Oracle Trace File Analyzer to the Latest Version

Oracle releases a new version of Oracle Trace File Analyzer approximately every 3 months.

Applying standard PSUs will automatically update Oracle Trace File Analyzer. However the PSU version does not contain the rest of
the Database Support Tools Bundle updates. So if you want the latest and greatest version of Oracle Trace File Analyzer with the
database support tools bundle download it from Document 1513912.1

Upgrading is similar to first time install. As root use the installTFA<platform> script. If Oracle Trace File Analyzer is already installed, it
will update the existing installation. When already installed, a cluster upgrade does not need ssh. The cluster upgrade uses the existing
daemon secure socket communication between hosts.

$ ./installTFA<platform>

If you're not able to install as root then install as the ORACLE_HOME owner. Use the -extractto and –javahome options:

$ ./installTFA<platform> -extractto <dir> -javahome <jre_home>

For upgrades on Windows follow the Windows installation steps.

Conclusion
Oracle Trace File Analyzer helps you keep mission-critical apps up and running.

The key features are:

» A single interface to all database diagnostic tools
» Monitoring of logs to detect significant problems
» Logs analysis to determine which diagnostic information is relevant to a problem
» Simple collection of data across cluster nodes, consolidated in one place
» Reduction in time required to obtain diagnostic data, which saves your business money

If something does go wrong, Oracle Trace File Analyzer helps you analyze and collect what you need, to fix it.

For more information and to download Oracle Trace File Analyzer see Document 1513912.1.

Oracle Corporation, World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone: +1.650.506.7000
Redwood Shores, CA 94065, USA Fax: +1.650.506.7200

C ON N E C T W I TH U S

blogs.oracle.com/oracle
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only, and the
contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other
facebook.com/oracle warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or
fitness for a particular purpose. We specifically disclaim any liability with respect to this document, and no contractual obligations are
formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any
twitter.com/oracle means, electronic or mechanical, for any purpose, without our prior written permission.

oracle.com Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and
are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are
trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group. 0517

Oracle Trace File Analyzer – Overview Guide

May 2017
Author: Gareth Chapman & Bob Caldwell
Contributing Authors: Bill Burton & Sandesh Rao