Java Performance: 
Biggest Mistakes 
- More on http://blog.dynatrace.com 
- Dynatrace Free Trial: http://bit.ly/dttrial 
Andreas Grabner - @grabnerandi 
1 @Dynatrace
And this is where I am from 
2 @Dynatrace
3 @Dynatrace
4 @Dynatrace
Can you tell my age by looking at my first 
computer? 
5 @Dynatrace
That’s why I ended up talking about performance 
6 @Dynatrace
Nobody wants this … 
7 @Dynatrace
8 @Dynatrace
Nor this … 
9 @Dynatrace
10 @Dynatrace
As it leads to this … 
11 @Dynatrace
The “War Room” 
Facebook – December 2012 
12 @Dynatrace
And potentially to this … 
13 @Dynatrace
14 @Dynatrace
And this … 
15 @Dynatrace
16 @Dynatrace
And that’s why Business 
doesn’t like it either … 
17 @Dynatrace
YES we know this 
80%Dev Time in Bug Fixing 
$60B Defect Costs 
BUT 
~80% of problems 
caused by ~20% patterns 
18 @Dynatrace
6 Situations on 
WHY this happened, 
HOW to avoid it 
19 @Dynatrace
20 @Dynatrace
“Blindly” (Re)use 
Existing 
Components 
21 @Dynatrace
Requirement: We need a report 
22 @Dynatrace
Using Hibernate results in 4k+ SQL Statements to 
display 3 items! 
Hibernate Executes 
4k+ Statements 
Individual Execution 
VERY FAST 
But Total SUM 
takes 6s 
23 @Dynatrace
24 @Dynatrace
Not every 
Architect makes 
good decisions 
25 @Dynatrace
Project: Online Room Reservation System 
• Symptoms 
• HTML takes between 60 and 120s to render 
• High GC Time 
• Assumptions 
• Bad GC Tuning 
• Probably bad Database Performance as rendering was simple 
26 @Dynatrace
Developers built own monitoring 
void roomreservationReport(int roomid) 
{ 
long startTime = System.currentTimeMillis(); 
Object data = loadDataForRoom(roomid); 
long dataLoadTime = System.currentTimeMillis() - startTime; 
generateReport(data, roomid); 
27 @Dynatrace 
} 
Result: 
Avg. Data Load Time: 45s! 
DB Tool says: 
Avg. SQL Query: <1ms!
#1: Loading too much data 
24889! Calls to the 
Database API! 
High CPU and High 
Memory Usage to keep all 
data in Memory 
28 @Dynatrace
#2: On individual connections 12444! 
individual 
connections 
Classical N+1 
Query Problem 
Individual SQL 
really <1ms 
29 @Dynatrace
#3: Putting all data in temp Hashtable 
Lots of time 
spent in 
Hashtable.get 
Called from their 
Entity Objects 
30 @Dynatrace
Lesson Learned 
• Custom Measuring 
• Was impacted by Garbage Collection 
• Just measured overall time but not # SQL Executions 
void roomreservationReport(int roomid) 
{ 
long startTime = System.currentTimeMillis(); 
Object data = loadDataForRoom(roomid); 
long dataLoadTime = System.currentTimeMillis() - startTime; 
generateReport(data, roomid); 
} 
• Learn SQL and don’t use Hashtables as Workaround 
31 @Dynatrace
32 @Dynatrace
Implementation 
Flaws 
33 @Dynatrace
Business Impact requires Action! 
34 @Dynatrace
Solution: Cache to the RESCUE!! 
35 @Dynatrace
Implementation and Rollout 
Implemented 
InMemory Cache 
Worked well in 
Load Testing 
36 @Dynatrace
Result: Out of Memory Crashes!! 
Still crashes 
Fixed Version Deployed Problem fixed! 
37 @Dynatrace
38 @Dynatrace
Disconnected 
Teams 
39 @Dynatrace
“Teamwork” between Dev and Ops 
SEV1 Problem in Production 
Need access to log files 
Where are they? Can’t get them 
Need to increase log level 
Can’t do! Can’t change config 
files in prod! 
40 @Dynatrace
Solution: Implement a Custom “On Demand” 
Remote Logger 
41 @Dynatrace
Implementation and Rollout 
Implemented 
Custom Logger 
Worked well in 
Load Testing 
42 @Dynatrace
What happened? 
~ 1Mio Lock Exceptions 
in 30 mins 
43 @Dynatrace
Root Cause: A special WebSphere Setting! 
Log Service provides a 
synchronized log file across 
Log Service provides a synchronized 
ALL JVMs 
log file across ALL JVMs 
44 @Dynatrace
45 @Dynatrace
“Deployment” 
Gone Bad! 
46 @Dynatrace
Test Environment 
Production Environment 
8x slower 
3x more SQL 
47 @Dynatrace
Test Environment Production Environment 
Hibernate, 
Hibernate, 
Classloading, 
XML – The Key 
Classloading, XML 
– The Key Hotspots 
I/O for Web 
Requests Hotspots 
doesn’t 
even show up! 
That’s Normal: 
Having I/O for Web 
Request as main 
contributor 
48 @Dynatrace
Top Contributor 
Class.getInterfaces 
Called from Hibernates 
FieldInterceptionHelper 
These calls all originate 
form thousands of calls to 
find item by code 
49 @Dynatrace
Top Methods related to XML 
Processing 
Classloading is triggered through 
CustomMonnkey and the Xalan 
CustomMonkey and the Xalan 
Parser 
50 @Dynatrace
51 @Dynatrace
#Push 
without a Plan 
52 @Dynatrace
Mobile Landing Page of Super Bowl Ad 
434 Resources in total on that page: 
230 JPEGs, 75 PNGs, 50 GIFs, … 
Total size of ~ 
20MB 
53 @Dynatrace
Fifa.com during Worldcup 
http://apmblog.compuware.com/2014/05/21/is-the-fifa-world-cup-website-ready-for-the-tournament/ 
54 @Dynatrace
Waiting for 
Bug Reports 
55 @Dynatrace
Look behind the scenes 
56 @Dynatrace
57 @Dynatrace
Performance Clinic: Bring Your Own App 
When: Thursday, December 18th 
Lunch Session: NOON – 3PM 
After Work Session: 4PM – 7PM 
Where: Dynatrace Office 
404 Wyman Street, Waltham 
Register: http://bit.ly/onlineperfclinic 
58 @Dynatrace
Quick Demo 
59 @Dynatrace
30 Days Dynatrace Free Trial 
• Get it here: http://bit.ly/dttrial 
• YouTube Tutorials: http://bit.ly/dttutorials 
• Live Q&A Sessions: http://bit.ly/onlineperfclinic 
• Contact me: agrabner@dynatrace.com - @grabnerandi 
•Special Offer: 
• SaaS VIP Program: http://bit.ly/dtaasvip 
60 @Dynatrace
Andreas Grabner 
Dynatrace Developer Advocate 
@grabnerandi 
http://blog.dynatrace.com 
61 @Dynatrace

Java Performance Mistakes

  • 1.
    Java Performance: BiggestMistakes - More on http://blog.dynatrace.com - Dynatrace Free Trial: http://bit.ly/dttrial Andreas Grabner - @grabnerandi 1 @Dynatrace
  • 2.
    And this iswhere I am from 2 @Dynatrace
  • 3.
  • 4.
  • 5.
    Can you tellmy age by looking at my first computer? 5 @Dynatrace
  • 6.
    That’s why Iended up talking about performance 6 @Dynatrace
  • 7.
    Nobody wants this… 7 @Dynatrace
  • 8.
  • 9.
    Nor this … 9 @Dynatrace
  • 10.
  • 11.
    As it leadsto this … 11 @Dynatrace
  • 12.
    The “War Room” Facebook – December 2012 12 @Dynatrace
  • 13.
    And potentially tothis … 13 @Dynatrace
  • 14.
  • 15.
    And this … 15 @Dynatrace
  • 16.
  • 17.
    And that’s whyBusiness doesn’t like it either … 17 @Dynatrace
  • 18.
    YES we knowthis 80%Dev Time in Bug Fixing $60B Defect Costs BUT ~80% of problems caused by ~20% patterns 18 @Dynatrace
  • 19.
    6 Situations on WHY this happened, HOW to avoid it 19 @Dynatrace
  • 20.
  • 21.
    “Blindly” (Re)use Existing Components 21 @Dynatrace
  • 22.
    Requirement: We needa report 22 @Dynatrace
  • 23.
    Using Hibernate resultsin 4k+ SQL Statements to display 3 items! Hibernate Executes 4k+ Statements Individual Execution VERY FAST But Total SUM takes 6s 23 @Dynatrace
  • 24.
  • 25.
    Not every Architectmakes good decisions 25 @Dynatrace
  • 26.
    Project: Online RoomReservation System • Symptoms • HTML takes between 60 and 120s to render • High GC Time • Assumptions • Bad GC Tuning • Probably bad Database Performance as rendering was simple 26 @Dynatrace
  • 27.
    Developers built ownmonitoring void roomreservationReport(int roomid) { long startTime = System.currentTimeMillis(); Object data = loadDataForRoom(roomid); long dataLoadTime = System.currentTimeMillis() - startTime; generateReport(data, roomid); 27 @Dynatrace } Result: Avg. Data Load Time: 45s! DB Tool says: Avg. SQL Query: <1ms!
  • 28.
    #1: Loading toomuch data 24889! Calls to the Database API! High CPU and High Memory Usage to keep all data in Memory 28 @Dynatrace
  • 29.
    #2: On individualconnections 12444! individual connections Classical N+1 Query Problem Individual SQL really <1ms 29 @Dynatrace
  • 30.
    #3: Putting alldata in temp Hashtable Lots of time spent in Hashtable.get Called from their Entity Objects 30 @Dynatrace
  • 31.
    Lesson Learned •Custom Measuring • Was impacted by Garbage Collection • Just measured overall time but not # SQL Executions void roomreservationReport(int roomid) { long startTime = System.currentTimeMillis(); Object data = loadDataForRoom(roomid); long dataLoadTime = System.currentTimeMillis() - startTime; generateReport(data, roomid); } • Learn SQL and don’t use Hashtables as Workaround 31 @Dynatrace
  • 32.
  • 33.
  • 34.
    Business Impact requiresAction! 34 @Dynatrace
  • 35.
    Solution: Cache tothe RESCUE!! 35 @Dynatrace
  • 36.
    Implementation and Rollout Implemented InMemory Cache Worked well in Load Testing 36 @Dynatrace
  • 37.
    Result: Out ofMemory Crashes!! Still crashes Fixed Version Deployed Problem fixed! 37 @Dynatrace
  • 38.
  • 39.
  • 40.
    “Teamwork” between Devand Ops SEV1 Problem in Production Need access to log files Where are they? Can’t get them Need to increase log level Can’t do! Can’t change config files in prod! 40 @Dynatrace
  • 41.
    Solution: Implement aCustom “On Demand” Remote Logger 41 @Dynatrace
  • 42.
    Implementation and Rollout Implemented Custom Logger Worked well in Load Testing 42 @Dynatrace
  • 43.
    What happened? ~1Mio Lock Exceptions in 30 mins 43 @Dynatrace
  • 44.
    Root Cause: Aspecial WebSphere Setting! Log Service provides a synchronized log file across Log Service provides a synchronized ALL JVMs log file across ALL JVMs 44 @Dynatrace
  • 45.
  • 46.
  • 47.
    Test Environment ProductionEnvironment 8x slower 3x more SQL 47 @Dynatrace
  • 48.
    Test Environment ProductionEnvironment Hibernate, Hibernate, Classloading, XML – The Key Classloading, XML – The Key Hotspots I/O for Web Requests Hotspots doesn’t even show up! That’s Normal: Having I/O for Web Request as main contributor 48 @Dynatrace
  • 49.
    Top Contributor Class.getInterfaces Called from Hibernates FieldInterceptionHelper These calls all originate form thousands of calls to find item by code 49 @Dynatrace
  • 50.
    Top Methods relatedto XML Processing Classloading is triggered through CustomMonnkey and the Xalan CustomMonkey and the Xalan Parser 50 @Dynatrace
  • 51.
  • 52.
    #Push without aPlan 52 @Dynatrace
  • 53.
    Mobile Landing Pageof Super Bowl Ad 434 Resources in total on that page: 230 JPEGs, 75 PNGs, 50 GIFs, … Total size of ~ 20MB 53 @Dynatrace
  • 54.
    Fifa.com during Worldcup http://apmblog.compuware.com/2014/05/21/is-the-fifa-world-cup-website-ready-for-the-tournament/ 54 @Dynatrace
  • 55.
    Waiting for BugReports 55 @Dynatrace
  • 56.
    Look behind thescenes 56 @Dynatrace
  • 57.
  • 58.
    Performance Clinic: BringYour Own App When: Thursday, December 18th Lunch Session: NOON – 3PM After Work Session: 4PM – 7PM Where: Dynatrace Office 404 Wyman Street, Waltham Register: http://bit.ly/onlineperfclinic 58 @Dynatrace
  • 59.
    Quick Demo 59@Dynatrace
  • 60.
    30 Days DynatraceFree Trial • Get it here: http://bit.ly/dttrial • YouTube Tutorials: http://bit.ly/dttutorials • Live Q&A Sessions: http://bit.ly/onlineperfclinic • Contact me: [email protected] - @grabnerandi •Special Offer: • SaaS VIP Program: http://bit.ly/dtaasvip 60 @Dynatrace
  • 61.
    Andreas Grabner DynatraceDeveloper Advocate @grabnerandi http://blog.dynatrace.com 61 @Dynatrace

Editor's Notes

  • #2 More detailed stories can also be found on our blog: http://blog.dynatrace.com All examples have been found using Dynatrace Free Trial – http://bit.ly/dttrial
  • #3 Just a bit about myself: That’s where I am from: Austria!
  • #4 Not the country with the kangaroos!
  • #6 That was my first Amiga 500 which I got when I was 9 years old!
  • #7 And that’s my professional background
  • #8 But now lets get started …
  • #19 Nobody likes to spend time in bugfixing
  • #56 Stop waiting for problems to happen
  • #57 Start looking behind the scenes