Best practices for large oracle apps r12 implementations apps14
3 likes2,906 views
This document provides an agenda and overview for a presentation on best practices for large Oracle Applications R12 implementations. The agenda covers application tier topics like Forms, response time, concurrent processing, and workflow. It also covers database tier topics such as AWR and ADDR reports, ORACHK health checks, and cluster callout scripts. The goal is to review strategies for ensuring performance, scalability, and proactive issue identification.
![Who Am I?
Ajith Narayanan
Technical Lead – Oracle ERP Configuration Management
GE Healthcare, Bangalore, India
• 10 years of Oracle [APPS] DBA/Technical Architect experience.
• Oracle ACE Associate
• Blogger :- http://oracledbascriptsfromajith.blogspot.com
• Member:-OAUG, AIOUG
• Website Chair (2011-2013) – Oracle RACSIG http://www.oracleracsig.org
• AIOUG Real Application Clusters SIG Leader
11/14/2014
4](https://image.slidesharecdn.com/bestpracticesforlargeoracleappsr12implementationsapps14-151214140349/85/Best-practices-for-large-oracle-apps-r12-implementations-apps14-4-320.jpg)
![Response Time/CPU Issues
• When users complain about response time, configure Apache to log the time it
takes to service a request.
• Access logs monitoring should be good way to keep a tab on the response times of
user requests.
– Configure Apache to log the time it takes to service a request
– Logs: $LOG_HOME/ora/10.1.3/Apache/access_log*
– Edit: $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.conf
LogFormat "%h %T
(Or)
– Enable Execution Content ID's (ECID) for the access_log in Release 12. ECID's allow correlation of
requests across log files so that one can map the flow of a given request across the various
components of iAS.
LogFormat "%h %l %u %t [ecid: %{Oracle-ECID}i] "%r" %>s %b [%T (secs)]" combined
Refer to http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html for getting http return code definitions
Note - R12.2 replaces Oracle 10gAS OC4J containers with Weblogic managed servers.
Application Tier
11/14/2014
12](https://image.slidesharecdn.com/bestpracticesforlargeoracleappsr12implementationsapps14-151214140349/85/Best-practices-for-large-oracle-apps-r12-implementations-apps14-12-320.jpg)
![EBS Concurrent Processing Analyzer
• The EBS Concurrent Processing Analyzer is a self-service health-check script
• Reviews the overall Concurrent Processing
• Analyzes the current configurations & settings for the environment.
• Providing feedback and recommendations on best practices.
• APPS Workload (Concurrent, FORM, ICX) observations.
• Mandatory Concurrent Programs fro purging.
Sample Output
Useful MOS Articles
Application Tier
2DEMO_cp_analyzer.html
11/14/2014
14
Best Practices for Performance for Concurrent Managers in E-Business Suite [Doc ID- 1057802.1]
Concurrent Processing - CP Analyzer for E-Business Suite (Doc ID 1411723.1)](https://image.slidesharecdn.com/bestpracticesforlargeoracleappsr12implementationsapps14-151214140349/85/Best-practices-for-large-oracle-apps-r12-implementations-apps14-14-320.jpg)
![EBS Workflow Analyzer
• The EBS Workflow Analyzer is a script that reviews the current Workflow Footprint.
• Analyzes the workflow configurations & Provides feedback and makes
recommendations on best practices and areas of concern.
• Provides immediate analysis and output of the EBS Workflow environment.
• Workflow Process Stuck & Work Flow Checks
Sample Output
Useful MOS Articles
Application Tier
2DEMO_wf_analyzer.html
11/14/2014
15
How to run EBS Workflow Analyzer Tool as a Concurrent Request (Doc ID 1425053.1)
11i-12 Workflow Analyzer script for E-Business Suite Workflow Monitoring and Maintenance [Video] (Doc
ID 1369938.1)
How To Use Concurrent Program "Purge Order Management Workflow" (Doc ID 878032.1)](https://image.slidesharecdn.com/bestpracticesforlargeoracleappsr12implementationsapps14-151214140349/85/Best-practices-for-large-oracle-apps-r12-implementations-apps14-15-320.jpg)
![Database Tier
#!/bin/ksh
# # Author: Ajith Narayanan
## http://oracledbascriptsfromajith.blogspot.com
## Version 1.0
## This callout script is extended to report/mail the affected weblogic services when any Oracle cluster event occurs.
##
umask 022
FAN_LOGFILE=$ORACLE_HOME/racg/usrco/`hostname`_uptime.log
EVENTLINE=$ORACLE_HOME/racg/usrco/`hostname`_eventline.log
EVENTLINE_MID=$ORACLE_HOME/racg/usrco/`hostname`_eventline_mid.log
MAIL_CONT=$ORACLE_HOME/racg/usrco/`hostname`_mail.log
WEBLOGIC_DS=$ORACLE_HOME/racg/usrco/weblogic_ds
echo $* "reported="`date` >> $FAN_LOGFILE &
tail -1 $FAN_LOGFILE > $EVENTLINE
awk '{
for (f = 1; f <= NF; f++) { a[NR, f] = $f } } NF > nf { nf = NF } END {
for (f = 1; f <= nf; f++) { for (r = 1; r <= NR; r++) { printf a[r, f] (r==NR ? RS : FS) } }
}' $EVENTLINE > $EVENTLINE_MID
SER=`grep "service=" $EVENTLINE_MID|awk -F= '{print $2}'`
DB=`grep "database=" $EVENTLINE_MID|awk -F= '{print $2}'`
INST=`grep "instance=" $EVENTLINE_MID|awk -F= '{print $2}'`
HOST=`grep "host=" $EVENTLINE_MID|awk -F= '{print $2}'`
STAT=`grep "status=" $EVENTLINE_MID|awk -F= '{print $2}'`
if [ "$SER" != " " | "$DB" != " " | "$INST" != " " | "$HOST" != " " | "$STAT" != " " ]; then
if [ $STAT = nodedown ]; then
cat $EVENTLINE_MID > $MAIL_CONT
11/14/2014
26
Cluster callout scripts. – Quickly get alerted on any cluster events.](https://image.slidesharecdn.com/bestpracticesforlargeoracleappsr12implementationsapps14-151214140349/85/Best-practices-for-large-oracle-apps-r12-implementations-apps14-26-320.jpg)
![echo "**============================SERVICES AFFECTED===============================**" >> $MAIL_CONT
grep -i "$DB_" $WEBLOGIC_DS >> $MAIL_CONT
elif [ $STAT = up ]; then
cat $EVENTLINE_MID > $MAIL_CONT
echo "**============================SERVICES RESTORED===============================**" >> $MAIL_CONT
grep -i "$DB_" $WEBLOGIC_DS|grep "SERVICE_NAME=$SER" >> $MAIL_CONT
else
cat $EVENTLINE_MID > $MAIL_CONT
echo "**============================SERVICES AFFECTED===============================**" >> $MAIL_CONT
grep -i "$DB_" $WEBLOGIC_DS|grep "SERVICE_NAME=$SER" >> $MAIL_CONT
#fi
cat $MAIL_CONT| /bin/mail -s "Cluster $STAT event: $DB $INST $SER $HOST" ajithpathiyil@gmail.com
fi
#cat $MAIL_CONT| /bin/mail -s "Cluster $STAT event: $DB $INST $SER $HOST" ajithpathiyil@gmail.com
fi
rm $EVENTLINE $EVENTLINE_MID $MAIL_CONT
Database Tier
11/14/2014
27](https://image.slidesharecdn.com/bestpracticesforlargeoracleappsr12implementationsapps14-151214140349/85/Best-practices-for-large-oracle-apps-r12-implementations-apps14-27-320.jpg)
Best practices for large oracle apps r12 implementations apps14
- 1. Imagination at work. Ajith Narayanan Technical Lead-Oracle ERP Configuration Management GE Healthcare, Bangalore, India 8th Dec-2014, Liverpool, U.K. Best Practices For Large Oracle Apps R12 Implementations
- 2. Disclaimer
- 3. The views/contents in this slides are those of the author and do not necessarily reflect that of General Electric Company or Oracle Corporation and/or their respective affiliates/subsidiaries. The material in this document is for informational purposes only, and is provided on an AS-IS basis. 11/14/2014 3
- 4. Who Am I? Ajith Narayanan Technical Lead – Oracle ERP Configuration Management GE Healthcare, Bangalore, India • 10 years of Oracle [APPS] DBA/Technical Architect experience. • Oracle ACE Associate • Blogger :- http://oracledbascriptsfromajith.blogspot.com • Member:-OAUG, AIOUG • Website Chair (2011-2013) – Oracle RACSIG http://www.oracleracsig.org • AIOUG Real Application Clusters SIG Leader 11/14/2014 4
- 5. Agenda Why do we need to implement best practices? • Application Tier • R12 Tech Stack • Forms • Response Time • Web Applications • Concurrent Manager • Workflow Engine • Taking Diagnostics Help • CBO Statistics • Application Patching • Network & Latency Check 11/14/2014 5
- 6. Agenda • Database Tier • CPU Utilization Observations and analysis • High DB_IO observation and correlation with CPU usage. • Identify Top SQLs • Effective usage of AWR, ADDM Reports • ORACHK Check (From Oracle RAC assurance team) • ADR Command Interpreter (ADRCI) • Cluster Callout scripts example. • Looking At Issues Proactively & From Greater Heights • References • Q & A 11/14/2014 6
- 7. Why do we need to implement best practices? 11/14/2014 7
- 8. Oracle Applications R12 • Huge infrastructure – Oracle Applications R12 – 8 Node middle tiers – 16 Node RAC database tier. – Database Size 40+ TB – Annual DB growth of 2 TB – Daily Orders: 45 K –3 M order lines – 1500 online users – 1M+ XML messages/day Best Practices • Benefits – Enables us being proactive – simple in terms of manageability – Helps in support cost reduction. – Time & effort saving Why do we need to implement best practices? 11/14/2014 8
- 9. Upgrade Tech Stack • R12 Tech Stack • OC4J: Oracle AS 10g 10.1.3./ Weblogic Managed Servers (FMW) • Forms: Oracle AS 10.1.2 • JAVA, JRE Plugin Useful MOS Articles Oracle E-Business Suite Technology Stack Release Notes for Release 12.1.3 (MOS Note - 1098650.1) Oracle E-Business Suite Technology Stack Release Notes for Release 12.2* Oracle E-Business Suite - Technology Stack : Navigation (landing) Page (MOS Note - 1480550.1) Troubleshooting Assistant: EBS Technology Stack (Doc ID 1607365.2) Application Tier 11/14/2014 9
- 11. Forms • Socket Mode for Internal users (MOS 384241.1) • Servlet mode is default in R12 and recommended for external access. – cat $FORMS_WEB_CONFIG_FILE|grep serverURL= If the serverURL parameter has no value then Forms is implemented in socket mode else it is servlet • Forms dead client detection (DCD) and Abnormal termination handler. – Value specified in minutes : FORMS_TIMEOUT=10 env | grep -i FORMS_TIMEOUT – Terminates fwebmx processes for dead clients. – Enable Forms Abnormal Termination Handle – Do not set FORMS_CATCHTERM – SQLNET.EXPIRE_TIME – Probes client connection at regular interval. Useful MOS Articles Application Tier 11/14/2014 11 Forms Process (FRMWEB) Consumes 100% of CPU in Oracle Applications R12 (Doc ID 745711.1)
- 12. Response Time/CPU Issues • When users complain about response time, configure Apache to log the time it takes to service a request. • Access logs monitoring should be good way to keep a tab on the response times of user requests. – Configure Apache to log the time it takes to service a request – Logs: $LOG_HOME/ora/10.1.3/Apache/access_log* – Edit: $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.conf LogFormat "%h %T (Or) – Enable Execution Content ID's (ECID) for the access_log in Release 12. ECID's allow correlation of requests across log files so that one can map the flow of a given request across the various components of iAS. LogFormat "%h %l %u %t [ecid: %{Oracle-ECID}i] "%r" %>s %b [%T (secs)]" combined Refer to http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html for getting http return code definitions Note - R12.2 replaces Oracle 10gAS OC4J containers with Weblogic managed servers. Application Tier 11/14/2014 12
- 13. Web Applications • Train users to logout when transactions are complete. – Logging out gracefully releases the memory and corresponding resources (i.e. connections, etc.) Releases process resources sooner • JVM/OC4J Sizing + Heap Size – OACORE, OAFORMS, OAFM etc.. – Start with no more than 100 active users per JVM/OC4J instance. – Start with 1 active JVM/OC4J instance per CPU core Note:- Each 'core' in a multi-core system is considered to be a separate CPU. For example, a dual quad- core chip would be the same as stating 2 x 4 x CPU or 8 CPUs. Useful MOS Articles JVM: Guidelines to setup the Java Virtual Machine in Apps Ebusiness Suite 11i and R12 (Doc ID 362851.1) MOS Note 357597.1 on how to enable SQL trace for OA Framework applications Application Tier 11/14/2014 13
- 14. EBS Concurrent Processing Analyzer • The EBS Concurrent Processing Analyzer is a self-service health-check script • Reviews the overall Concurrent Processing • Analyzes the current configurations & settings for the environment. • Providing feedback and recommendations on best practices. • APPS Workload (Concurrent, FORM, ICX) observations. • Mandatory Concurrent Programs fro purging. Sample Output Useful MOS Articles Application Tier 2DEMO_cp_analyzer.html 11/14/2014 14 Best Practices for Performance for Concurrent Managers in E-Business Suite [Doc ID- 1057802.1] Concurrent Processing - CP Analyzer for E-Business Suite (Doc ID 1411723.1)
- 15. EBS Workflow Analyzer • The EBS Workflow Analyzer is a script that reviews the current Workflow Footprint. • Analyzes the workflow configurations & Provides feedback and makes recommendations on best practices and areas of concern. • Provides immediate analysis and output of the EBS Workflow environment. • Workflow Process Stuck & Work Flow Checks Sample Output Useful MOS Articles Application Tier 2DEMO_wf_analyzer.html 11/14/2014 15 How to run EBS Workflow Analyzer Tool as a Concurrent Request (Doc ID 1425053.1) 11i-12 Workflow Analyzer script for E-Business Suite Workflow Monitoring and Maintenance [Video] (Doc ID 1369938.1) How To Use Concurrent Program "Purge Order Management Workflow" (Doc ID 878032.1)
- 16. OAM Diagnostics Help • Oracle Application Object Library Best Practices: Database Security Tests Health Check Test • Oracle Application Object Library Best Practices: E-Business Suite Security Tests Health Check Test • Oracle Application Object Library Best Practices: E-Business Suite Diagnostic Tests Health Check Test • And many more…… Application Tier 11/14/2014 16
- 17. CBO Statistics • FND_STATS uses the standard the DBMS_STATS package, but enhances the functionality as it provides several additional PL/SQL procedures for gathering statistics at the table, schema, or database level. • The FND_STATS package also supports histograms, table exclusions, extended stats, incremental stats gathering for partitioned tables, and concurrent statistics gathering. Useful MOS Articles Application Tier 11/14/2014 17 Best Practices for Gathering Statistics with Oracle E-Business Suite (Doc ID - 1586374.1) bde_last_analyzed.sql - Verifies CBO Statistics (Doc ID - 163208.1)
- 18. Application Patching • Thorough patch analysis with timing reports on sand box & non-prods environments before proceeding with the Prod patching. • In Release 12, you register your customized files in the Register Flagged Files tool in Oracle Applications Manager (OAM), which replaces the $APPL_TOP/admin/applcust.txt file. Useful MOS Articles Application Tier 11/14/2014 18 See Note 9766881 "My server does not have internet connection Can I Still use Patch Wizard?“ Patching Best Practices And Reducing Downtime (Doc ID 225165.1)
- 19. • There are two diagnostic tools available in the Oracle E-Business Suite: The Network test Form – This form can be added to user’s menu , So that they can run the tests when they feel a performance degradation Note:- Network latency should be always low (<3ms) and bandwidth should be high. Application Tier Network & Latency 11/14/2014 19
- 20. Useful MOS Articles The Client Analyzer Application Tier Network & Latency 11/14/2014 20 Oracle E-Business Suite Network Utilities: Best Practices (Doc ID 556738.1)
- 21. CPU Utilization Observations and analysis • CPU spikes for more than 30 minutes continuously is a candidate for RCA. • High DB_IO observation and correlation with CPU usage. • I/O Thresholds: Logical RDS > 300K/sec, Physical RDS> 3000/Sec are candidate for RCA. – How many of you think RAM access is 10,000 times faster than Physical disk access? – In real world, LIO is only 25-100 times cheaper than PIO – Internal locks & latch serialization mechanisms involved. – Targeting only PIO counts(or high cache hit ratios) important pitfall to avoid. Even with no PIOs, a query can still be outrageously inefficient – LIO are a critical component of query cost Database Tier 11/14/2014 21
- 22. AWR, ASH, ADDM Reports • Identify Top SQL, Latch Spin Counts etc., Memory Advisors • Identify top wait events (System, User IO, Cluster, Concurrency, Configuration, Applications, Network) • Fragmented Interface tables • Object to be pinned – (Frequently used database objects) • RAC Check (From Oracle RAC assurance team) RACcheck - RAC Configuration Audit Tool (Doc ID 1268927.1) • ORAchk - Health Checks for the Oracle Stack (Doc ID 1268927.2) • Sample ORACHK report • ORAchk-Collection Manager – ORAchk daemon auto-start mode after node reboot (init integration) – Merge multiple ORAchk collection reports – Exclude checks based on profile – Upload of installed patches to database – Collection Manager for ORAchk, RACcheck and Exachk (Doc ID 1602329.1) – ORAchk signature file in /tmp on all nodes to verify last ORAchk run – New checks and bug fixes, including o 30 Oracle Ebusiness AP module data integrity checks – 12 new Database checks – 8 new Solaris system checks Database Tier ORAchk-Report-Example.html 11/14/2014 22
- 23. • ADRCI has multiple features such as – Analyze diagnostic data within the Automatic Diagnostic Repository (ADR). – Create a package with incident / problem information for Oracle Support. – Diagnostic data includes incident and problem descriptions, trace files, dumps, health monitor reports, alert log entries, and more. – Run the Health Checks – Helps in purging old diagnostics trace & dump files – Simple to use a rich command set Set correct ORACLE_HOME and just enter adrci The utility starts and displays the following prompt: adrci> Database Tier 11/14/2014 23 ADR Command Interpreter (ADRCI)
- 24. • At the ADRCI prompt, enter the following command: adrci> HELP To get help for a specific ADRCI command: adrci> HELP command • For example, to get help on the SHOW TRACEFILE command, enter the following: adrci> HELP SHOW TRACEFILE adrci> set homepath diag/rdbms/orclbi/orclbi2 adrci> show homes ADR Homes: diag/rdbms/orclbi/orclbi2 • To view the alert log: 1. Start ADRCI in interactive mode, and ensure that the homepath points to the correct directory within the ADR base directory hierarchy. Database Tier 11/14/2014 24
- 25. Useful MOS Articles 2. At the ADRCI prompt, enter the following command: SHOW ALERT The following are variations on the SHOW ALERT command: SHOW ALERT -TAIL SHOW ALERT -TAIL 50 SPOOL /tmp/MYALERT.LOG SHOW ALERT SPOOL OFF This outputs the alert log, without XML tags, to the file /tmp/MYALERT.LOG. SHOW ALERT -P "MESSAGE_TEXT LIKE '%ORA-600%'" Database Tier 11/14/2014 25 11G ADR DOCUMENTATION (Doc ID 445022.1) How to Invoke ADR Command Interpreter (adrci) in FMW home? (Doc ID 1669923.1)
- 26. Database Tier #!/bin/ksh # # Author: Ajith Narayanan ## http://oracledbascriptsfromajith.blogspot.com ## Version 1.0 ## This callout script is extended to report/mail the affected weblogic services when any Oracle cluster event occurs. ## umask 022 FAN_LOGFILE=$ORACLE_HOME/racg/usrco/`hostname`_uptime.log EVENTLINE=$ORACLE_HOME/racg/usrco/`hostname`_eventline.log EVENTLINE_MID=$ORACLE_HOME/racg/usrco/`hostname`_eventline_mid.log MAIL_CONT=$ORACLE_HOME/racg/usrco/`hostname`_mail.log WEBLOGIC_DS=$ORACLE_HOME/racg/usrco/weblogic_ds echo $* "reported="`date` >> $FAN_LOGFILE & tail -1 $FAN_LOGFILE > $EVENTLINE awk '{ for (f = 1; f <= NF; f++) { a[NR, f] = $f } } NF > nf { nf = NF } END { for (f = 1; f <= nf; f++) { for (r = 1; r <= NR; r++) { printf a[r, f] (r==NR ? RS : FS) } } }' $EVENTLINE > $EVENTLINE_MID SER=`grep "service=" $EVENTLINE_MID|awk -F= '{print $2}'` DB=`grep "database=" $EVENTLINE_MID|awk -F= '{print $2}'` INST=`grep "instance=" $EVENTLINE_MID|awk -F= '{print $2}'` HOST=`grep "host=" $EVENTLINE_MID|awk -F= '{print $2}'` STAT=`grep "status=" $EVENTLINE_MID|awk -F= '{print $2}'` if [ "$SER" != " " | "$DB" != " " | "$INST" != " " | "$HOST" != " " | "$STAT" != " " ]; then if [ $STAT = nodedown ]; then cat $EVENTLINE_MID > $MAIL_CONT 11/14/2014 26 Cluster callout scripts. – Quickly get alerted on any cluster events.
- 27. echo "**============================SERVICES AFFECTED===============================**" >> $MAIL_CONT grep -i "$DB_" $WEBLOGIC_DS >> $MAIL_CONT elif [ $STAT = up ]; then cat $EVENTLINE_MID > $MAIL_CONT echo "**============================SERVICES RESTORED===============================**" >> $MAIL_CONT grep -i "$DB_" $WEBLOGIC_DS|grep "SERVICE_NAME=$SER" >> $MAIL_CONT else cat $EVENTLINE_MID > $MAIL_CONT echo "**============================SERVICES AFFECTED===============================**" >> $MAIL_CONT grep -i "$DB_" $WEBLOGIC_DS|grep "SERVICE_NAME=$SER" >> $MAIL_CONT #fi cat $MAIL_CONT| /bin/mail -s "Cluster $STAT event: $DB $INST $SER $HOST" [email protected] fi #cat $MAIL_CONT| /bin/mail -s "Cluster $STAT event: $DB $INST $SER $HOST" [email protected] fi rm $EVENTLINE $EVENTLINE_MID $MAIL_CONT Database Tier 11/14/2014 27
- 28. Looking At Issues Proactively & From Greater Heights 11/14/2014 28
- 29. References • My Oracle Support • Steven Chan’s blog (https://blogs.oracle.com/stevenChan) 11/14/2014 29