ExaWatcher Utility On Exadata Compute and Storage Nodes

Applies to: Oracle Exadata Storage Server Software - Version 11.2.3.3.0 and later
This document describes a newly designed system data collection and diagnostic tool called
ExaWatcher. ExaWatcher replaces OSWatcher in Exadata engineered systems.

For 19.x Exadata Images:

The preferred method is to use the systemctl command to start and stop ExaWatcher.

To stop ExaWatcher:

# systemctl stop exawatcher

To start ExaWatcher:

# systemctl start exawatcher

However, the command for other image versions will also work.

There is also a cron job in 19.x images, /etc/cron.daily/exawatcher, that restarts the
ExaWatcher utility daily.

To verify ExaWatcher is running:

systemctl status exawatcher

Examples of Manual Collection:

1. To collect from/to a certain date and time:

# ./GetExaWatcherResults.sh --from 12/25/2023_13:00:00 --to 12/25/2023_14:00:00

[2023-12-27 09:13:38 -
0500][INFO][/opt/oracle.ExaWatcher/GetExaWatcherResults.sh][main][] Result files which
are modified after 2023-12-2513:00:00 and 2023-12-2514:00:00 are extracted in a zipped
package in /opt/oracle.ExaWatcher/archive/ExtractedResults.

Results are written to :

/opt/oracle.ExaWatcher/archive/ExtractedResults/Searched_2023_12_27_09_13_37.tar.bz
2

2. To collect for a time range. In this case, we are collecting for 4 hrs before and after 1300:
# ./GetExaWatcherResults.sh --at 12/25/2023_13:00:00 --range 4

[2023-12-27 08:57:27 -
0500][INFO][/opt/oracle.ExaWatcher/GetExaWatcherResults.sh][main][] Result files which
are modified at around (~4 hours before and after) time point 2023-12-25 13:00:00 are
extracted in a zipped package in /opt/oracle.ExaWatcher/archive/ExtractedResults.

Results are written to

/opt/oracle.ExaWatcher/archive/ExtractedResults/Searched_2023_12_27_08_57_22.tar.bz
2

3. The default archive directory is /opt/oracle.ExaWatcher/archive/ExtractedResults;

however, you can change this using [-d|--archivedir] flag:

Example of changed default archive location to /tmp/ExaWatcherArchive:

# ./GetExaWatcherResults.sh --from 12/25/2023_13:00:00 --to 12/25/2023_14:00:00 --

archivedir /tmp/ExaWatcherArchive

[2023-12-27 09:20:49 -
0500][INFO][/opt/oracle.ExaWatcher/GetExaWatcherResults.sh][main][] Result files which
are modified after 2023-12-2513:00:00 and 2023-12-2514:00:00 are extracted in a zipped
package in /tmp/ExaWatcherArchive/ExtractedResults.

Results are now written to /tmp/ExaWatcherArchive/ExtractedResults instead of

/opt/oracle.ExaWatcher/archive/ExtractedResults

Additional information about ExaWatcher may be found in the help menu:

# ExaWatcher.sh --help

# ExaWatcherResults.sh --help

EXAWATCHER COLLECTION MODULE DETAILS:

- Diskinfo.ExaWatcher
# Collection Module: DiskinfoExaWatcher
# Collection Command: /bin/cat /proc/diskstats

- IBCardInfo.ExaWatcher
# Collection Module: IBCardInfoExaWatcher
# Collection Command: /opt/oracle.ExaWatcher/IBCardInfoExaWatcher.sh 2>/dev/null

- IBprocs.ExaWatcher
# Collection Module: IBprocsExaWatcher
# Collection Command: /bin/cat /sys/class/net/bondib0/bonding/active_slave
# Collection Command: /bin/cat /sys/class/net/ib0/carrier
# Collection Command: /bin/cat /sys/class/net/ib1/carrier
# Collection Command: /usr/sbin/ibstatus
# Collection Command: /usr/bin/ibv_devinfo

- Iostat.ExaWatcher
# Collection Module: IostatExaWatcher
# Collection Command: /usr/bin/iostat -t -x 5 720

- LGWR.ExaWatcher
# Collection Module: LGWRExaWatcher
# Collection Command: /opt/oracle.ExaWatcher/LGWRExaWatcher.sh

- Lsof.ExaWatcher
# Collection Module: LsofExaWatcher
# Collection Command: /usr/sbin/lsof +c0 -w +L -b -R -i
# Collection Command: /usr/sbin/lsof +c0 -w +L -b -R -U
# Collection Command: /usr/sbin/lsof +c0 -w +L1 -b -R

- MegaRaidFW.ExaWatcher
# Collection Module: MegaRaidFWExaWatcher
# Collection Command: /opt/MegaRAID/MegaCli/MegaCli64 fwtermlog dsply -a0

- Meminfo.ExaWatcher
# Collection Module: MeminfoExaWatcher
# Collection Command: /bin/cat /proc/meminfo

- Mpstat.ExaWatcher
# Collection Module: MpstatExaWatcher
# Collection Command: /usr/bin/mpstat -P ALL 5 720

- Netstat.ExaWatcher
# Collection Module: NetstatExaWatcher
# Collection Command: /bin/netstat -a -i -n
# Collection Command: /bin/netstat -s
# Collection Command: /bin/netstat -n -p -l

- Ps.ExaWatcher
# Collection Module: PsExaWatcher
# Collection Command: /bin/ps -eo
flags,s,ruser,pid,ppid,c,psr,pri,ni,addr,sz,wchan,stime,tty,time,cmd

- RDSinfo.ExaWatcher
# Collection Module: RDSinfoExaWatcher
# Collection Command: /opt/oracle.ExaWatcher/RDSinfoExaWatcher.sh 2>/dev/null

- Slabinfo.ExaWatcher
# Collection Module: SlabinfoExaWatcher
# Collection Command: /bin/cat /proc/slabinfo
- Top.ExaWatcher
# Collection Module: TopExaWatcher
# Collection Command: /usr/bin/top -b -d 5 -n 720

- Vmstat.ExaWatcher
# Collection Module: VmstatExaWatcher
# Collection Command: /usr/bin/vmstat 5 2

Collect a sundiag on Linux compute nodes and cell servers

Run as root on the compute node or cell server:
# /opt/oracle.SupportTools/sundiag.sh
Execution will create a date stamped tar.bz2 file in /tmp/sundiag_/tar.bz2

This list of the files created by version 12.1.2.2.0_150917 of sundiag.sh:

asr
cell
disk
etc_configs
etc_sysconfig_net
fru-print_ipmitool.out
ilom
imagehistory-all.out
imageinfo-all.out
messages
mrdiag
net
osw
RackMasterSN
raid
SerialNumbers
stderr.txt
sysconfig
var_log_cellos
.version_sundiag

For gathering sundiag data across a whole rack:

For gathering sundiag.sh outputs on versions of sundiag.sh, where the filename is unique for
each node (v1.4 and later), use the following from DB01:
1. [root@exadb01 ~]# cd /opt/oracle.SupportTools/onecommand (or wherever the all_group
file is with the list of the rack hostnames)

2. [root@exadb01 onecommand]# dcli -g all_group -l root

/opt/oracle.SupportTools/sundiag.sh 2>&1
<this will take up to several minutes while each node runs sundiag.sh>

3. Verify there is output in /tmp or /var/log/exadatatmp/ on each node:

[root@exadb01 onecommand]# dcli -g all_group -l root --serial 'ls -l /tmp/sundiag* ' (v.1.4-
1.5.1)

[root@exadb01 onecommand]# dcli -g all_group -l root --serial 'ls -l

/var/log/exadatatmp/sundiag* ' (v12.1.2.2.0_150917)

4. Make a temporary directory to copy for zipping:

[root@exadb01 onecommand]# mkdir dbm01_sundiags_date

It is recommended the date be of the format YYMMDD year, month, day for SR's where
multiple days of analysis may be required.

5. Copy the generated sundiag files from the nodes to the temporary directory (/tmp on
v1.4-1.5.1, /var/log/exadatatmp on v12.1.2.2.0_150917):

[root@exadb01 onecommand]# for H in `cat all_group`; do scp -p $H:/tmp/sundiag*.tar.bz2

dbm01_sundiags_date ; done

[root@exadb01 onecommand]# for H in `cat all_group`; do scp -p

$H:/var/log/exadatatmp/sundiag*.tar.bz2 dbm01_sundiags_date ; done

6. Bundle them into a single file for upload to Oracle:

[root@exadb01 ~]# tar jcvf exa_rack_sundiag_date.tar.bz2 dbm01_sundiags_date

For gathering sundiag.sh outputs on older versions of sundiag.sh (prior to v1.4) where the
filename generated does not include hostname.

When gathering sundiag.sh outputs across a whole rack using dcli, the outputs may end up
with the same tarball name which will overwrite each other upon unzipping. To avoid this,
use the following from DB01:

1. [root@exadb01 ~]# cd /opt/oracle.SupportTools/onecommand (or wherever the all_group

file is with the list of the rack hostnames)
2. [root@exadb01 onecommand]# dcli -g all_group -l root
/opt/oracle.SupportTools/sundiag.sh 2>&1
<this will take up to about 2 minutes>

3. Verify there is output in /tmp on each node:

[root@exadb01 onecommand]# dcli -g all_group -l root --serial 'ls -l /tmp/sundiag* '

4. Sort them by hostname into directories, as they will likely mostly have the same filename
with the same date stamp:
[root@exadb01 onecommand]# for H in `cat all_group`; do mkdir /root/rack-sundiag/$H ;
scp -p $H:/tmp/sundiag*.tar.bz2 /root/rack-sundiag/$H ; done

5. [root@exadb01 onecommand]# cd /root/rack-sundiag

6. [root@exadb01 ~]# ls exa*

exacel01:
sundiag_2011_05_24_10_11.tar.bz2

exacel02:
sundiag_2011_05_24_10_11.tar.bz2
...
exadb08:
sundiag_2011_05_24_10_11.tar.bz2

7. [root@exadb01 ~]# tar jcvf exa_rack_sundiag_oracle.tar.bz2 exa*

exacel01/
exacel01/sundiag_2011_05_24_10_11.tar.bz2
exacel02/
exacel02/sundiag_2011_05_24_10_11.tar.bz2
...
exadb08/
exadb08/sundiag_2011_05_24_10_11.tar.bz2

8. [root@exadb01 ~]# ls -l exa_rack_sundiag_oracle.tar.bz2

-rw-r--r-- 1 root root 3636112 May 24 10:21 exa_rack_oracle.tar.bz2