Efficient Memory and Thread Management in Highly Parallel Java Applications

JavaOne

Efficient Memory and Thread
Management in Highly Parallel
Applications

Phillip Koza
IBM

JavaOne
IBM CONFIDENTIAL

Agenda

 Java memory overview
 Causes of excessive java heap memory usage
 Estimating memory usage of objects
 Runtime memory usage estimation
 Track and control memory usage
 Minimize usage of memory
 Efficiently manage threads
 Summary

2

JavaOne
IBM CONFIDENTIAL

Java memory usage overview

 Many different forms of memory usage by the Java Virtual
Machine (JVM)
 OS and C runtime
 Native heap
 Method area
 Storage for objects that describe classes and methods

3

JavaOne
IBM CONFIDENTIAL


 JVM stack
 Java heap
 This is where instances of java classes and arrays are
allocated
 Excessive use of java heap memory will cause performance
issues and ultimately, java.lang.OutOfMemoryError
 This presentation covers monitoring and controlling java
heap memory usage:
 Better performance
 Higher application availability

4

JavaOne
IBM CONFIDENTIAL


 The memory usage of a Java object is composed of:
 Overhead: This stores the reference to the class object and
various flags. Can range from 8 – 24 bytes depending on JVM
and 32 bit vs. 64 bit
 Memory for primitive fields
 Memory for reference fields
 Alignment bytes. In some (most) JVMs, the size of all objects
must be a multiple of 8. Because of this, an Integer object
uses as much memory as a Long
 Arrays use an extra 4-8 bytes to store the size of the array

5

JavaOne
IBM CONFIDENTIAL

Causes of excessive memory usage

 Lack of insight into the memory usage of common and
custom objects
 Overuse of delegation in class design
 i.e. too many objects!
 Too many threads
 This applies to the java heap usage of a thread. Most threads
will use some amount of java heap memory.
 Incorrect use of collection classes

6

JavaOne
IBM CONFIDENTIAL

Causes of excessive memory usage

 Memory leaks
 “Forgotten” or “lost” reference: application is holding a
reference, but has forgotten about it or doesn’t know where it is
 Infrequent or delayed cleanup of no longer needed objects
 Excessive use of finalizers
 There is no guarantee when a finalizer will be run or that it will
be run at all. An object that has a finalizer will not be garbage
collected until its finalizer is run

7

JavaOne
IBM CONFIDENTIAL

Excessive memory usage symptoms

 Degradation in performance
 High Garbage Collection (GC) costs
 GC must run more frequently when large amounts of memory
are being used
 GC must sweep over and analyze more memory to find that
which can be garbage collected
 JVM can thrash when memory utilization exceeds 85-90% of
max memory

8

JavaOne
IBM CONFIDENTIAL

Excessive memory usage symptoms

 Java.lang.OutOfMemoryError (OOM)
 In most JVMs, by default this only kills the thread attempting to
allocate the memory that triggered the error
 Can lead to threads hanging or behaving strangely: “Zombie
Threads”
 JVM may crash – but perhaps not immediately!

9

JavaOne
IBM CONFIDENTIAL

Long term object memory usage

 Most JVM memory allocation and GC is generational
 Default in Oracle HotSpot
 Default in IBM J9 1.6.0 build 2.6 and later, and in WepSphere App
Server V8 and later
 In generational GC, objects are initially allocated in the “young”
generation
 If the object survives enough garbage collections, it is moved to
the “old” or “tenured” generation
 GC’ing the tenured generation is more expensive
 Reducing the amount of data that is put in the tenured generation
can increase performance dramatically!

10

JavaOne
IBM CONFIDENTIAL


 Hard to say for sure what objects will become tenured, but
likely candidates are objects:
 Stored in static structures
 Passed between threads
 Stored in member variable collections
 By definition, long term objects tie up memory for longer
periods of time, increasing the odds of OOM
 It is still possible to run out memory with no or very few long
term objects – but this is unlikely!

11

JavaOne
IBM CONFIDENTIAL


 How to minimize the amount of long term memory:
 Delay allocating objects until you need them
 Immediately delete objects that are no longer needed or
unlikely to be needed again
 Convert objects to a more memory-efficient form if they won’t
be needed for a while
 Design your objects to be memory efficient from the start!

12

JavaOne
IBM CONFIDENTIAL

Estimating memory usage

 Memory usage will depend on JVM vendor and 32 bit vs. 64
bit
 For 64 bit, must determine if references will be compressed
 Oracle HotSpot VM:
 Enabled by default in 6.0.23 & later
 Enabled by default in Java 7 if –Xmx is less than 32 GB
 Can be disabled via –XX:-UseCompressedOops
 IBM: Enabled via –Xcompressedrefs if –Xmx is less than 30GB

13

JavaOne
IBM CONFIDENTIAL


 Two ways to estimate memory usage:
 Runtime.totalMemory() – Runtime.freeMemory()
 java.lang.instrument.Instrumentation.getObjectSize()

14

JavaOne
IBM CONFIDENTIAL


 The classic way to determine the memory use of an object
(and the only way prior to Java 5) is to use the Runtime
memory methods
 First determine how much memory is used at the start of the
test via Runtime.totalMemory() – Runtime.freeMemory()
 Then allocate objects of a given type and see how much
memory is now being used
 The difference between the before and after memory usage
should be the memory used by the allocated objects

15

JavaOne
IBM CONFIDENTIAL


 Unfortunately, this can be inaccurate at times. To be accurate, the
garbage collector must run before we measure the before memory
use
 It’s not possible to guarantee the garbage collector is run, and even
if it is, it might not collect all the garbage objects
 So, call System.gc() at the start of the test many times to increase
the odds it will run
 gc() problems can be minimized by allocating a large number of
objects of the same type and then taking the average
 Note: Escape analysis may decide to allocate your objects on the
stack if they don’t “escape” the method you are allocating them in

16

JavaOne
IBM CONFIDENTIAL

Code sample
public interface ObjectFactoryInterface {
public Object makeObject( int index);
}
public class ObjectFactory implements ObjectFactoryInterface {
public Object makeObject( int index) {
return new Object();
}
}
public class IntegerFactory implements ObjectFactoryInterface {
return new Integer(123456);
}
}
public class LongFactory implements ObjectFactoryInterface {
return new Long(123456L);
}
}

17

JavaOne
IBM CONFIDENTIAL

Code sample

public class FloatFactory implements ObjectFactoryInterface {
return new Float(123.456);
}
}
public class DoubleFactory implements ObjectFactoryInterface {
return new Double(123.456);
}
}
public class BigDecimalFactory implements ObjectFactoryInterface {
return new BigDecimal(“123.456”);
}
}

18

JavaOne
IBM CONFIDENTIAL

Code sample
public class StringFactory implements ObjectFactoryInterface {
// Make a string from the provided index. This is necessary so that
// each String object will be new and will create a new character array
// (Strings can be interred and String character arrays can be shared
// if they represent the same value)
return Integer.valueOf(index).toString();
}
}
public class HashMapDefaultFactory implements ObjectFactoryInterface {
public Object makeObject(int index) {
return new HashMap();
}
}
public class IntArrayEmptyFactory implements ObjectFactoryInterface {
public Object makeObject(int index) {
return new int[0];
}
}

19

JavaOne
IBM CONFIDENTIAL

Code sample
public class MeasureMemoryUsage {

public static void main(String[] argv) {
System.out.println("Measuring Memory Usage: ");
printMemUsage( new ObjectFactory());
printMemUsage( new IntegerFactory());
printMemUsage( new LongFactory());
printMemUsage( new FloatFactory());
printMemUsage( new DoubleFactory());
printMemUsage( new StringFactory());
printMemUsage( new BigDecimalFactory());
printMemUsage( new HashMapDefaultFactory());
printMemUsage( new IntArrayEmptyFactory());
}
private static void printMemUsage(ObjectFactoryInterface objFactory) {
System.out.println(objFactory.getClass().getName() + " used " + estimateMemUsage(objFactory) + " bytes per
Object");
}

20

JavaOne
IBM CONFIDENTIAL

Code sample

private static long estimateMemUsage(ObjectFactoryInterface objFactory) {
final int numObjects = 5000;
Object[] objArray = new Object[numObjects];
runGc();
long beforeMemUsed = getMemUsed();
System.out.println("Before memory used: " + beforeMemUsed);
for ( int i = 0; i < numObjects; i++) {
objArray[i] = objFactory.makeObject(10000000+i);
}
long afterMemUsed = getMemUsed();
System.out.println("After memory used: " + afterMemUsed);
return Math.round(((double)(afterMemUsed - beforeMemUsed)) / (double)numObjects);
}

21

JavaOne
IBM CONFIDENTIAL

Code sample
private static long getMemUsed() {
Runtime rTime = Runtime.getRuntime();
return rTime.totalMemory() - rTime.freeMemory();
}
private static void runGc() {
// Run GC 25 times to increase the odds it is really done
for ( int i=0; i < 25; i++) {
System.gc();
try {
Thread.sleep(200);
}
catch (InterruptedException e) {
}
}
}
}

22

JavaOne
IBM CONFIDENTIAL


Using java.lang.instrument.Instrumentation.getObjectSize()
 Create an instrument agent class:
 Must have a method called “premain” with the following
signature: public static void premain(String args,
Instrumentation inst)
 The JVM will pass an object that implements the
Instrumentation interface to the premain method
 We can then use this object to invoke the Instrumentation
interface method getObjectSize(object) , passing the object
we want to estimate the memory for

23

JavaOne
IBM CONFIDENTIAL


 Must package the agent class into a jar with a Manifest file
(for example, manifestFile.txt) with the following line:
Premain-class: <packagename>.<agentclassname>
 To run a program that can use the getObjectSize method,
must specify the agent on the command line via
-javaagent:<agentjarname>
 Note that getObjectSize doesn’t include the memory of
objects referenced by the specified object. Only the
memory for the reference will be included.
 To get a deep estimate, reflection would need to be used

24

JavaOne
IBM CONFIDENTIAL

Code sample
import java.lang.instrument.*;
public class MemInstrumentAgent {
private static volatile Instrumentation instrObj;
public static void premain(String args, Instrumentation instrParam) {
instrObj = instrParam;
}
public static long getObjectSize(Object obj) {
if (instrObj != null)
return instrObj.getObjectSize(obj);
else
throw new IllegalStateException(“Instrumentation agent not initialized!”);
}
}

25

JavaOne
IBM CONFIDENTIAL

Code sample

jar –cmf manifestFile.txt instrAgent.jar
MemInstrumentAgent.class
public class testMemoryUse {
public static void main(String args[]) {
Integer intObj = new Integer(100);
System.out.println(“Memory estimate for Integer: “ +
MemInstrumentAgent.getObjectSize(intObj));
}
java –javaagent:instrAgent.jar –cp . testMemoryUse

26

JavaOne
IBM CONFIDENTIAL


Following results are for:
HotSpot 6.0_29 –XX-UseTLAB

IBM J9, 1.6.0 build 2.4
Note: -Xgcpolicy:gencon is the default as of 1.6.0
build 2.6. Also the default in WebSphere AppServer V8
and later.

27

JavaOne
IBM CONFIDENTIAL

HotSpot32 HotSpot64 HotSpot IBM32 IBM64 IBM64
CompRefs
CompOops
Object 8 16 16 12 24 16

Integer 16 24 16 16 32 16

Long 16 24 24 24 32 24

Float 16 24 16 16 32 16

Double 16 24 24 24 32 24

String 56 82 65 64 90 65
(8 chars)
BigDecimal 96 148 123 104 156 115
(123.456)
HashMap 120 216 128 128 224 128
(16,0.75)
int[0] 16 24 16 16 24 16

int[100] 416 424 416 416 424 416

28

JavaOne
IBM CONFIDENTIAL


 Which is better?
 Using the Runtime methods is the simplest
 Instrumentation is more accurate, so it is preferred if you
don’t need a deep memory estimate
 For deep estimates, Instrumentation + Reflection is the
ultimate solution, but is more complex and slower

29

JavaOne
IBM CONFIDENTIAL

Runtime memory usage estimation

 Your goal is to write a method to estimate the memory
usage for objects of each class for which you will tracking
the memory usage
 Any referenced classes will need a separate method
 Since each class needs its own estimation method, you
don’t need deep memory estimates when testing to
determine how much your objects will use
 Have utility classes that have pre-calculated the memory
usage of all primitives and standard objects

30

JavaOne
IBM CONFIDENTIAL


 Need methods/constants for the following:
 Object overhead
 Alignment policy
 Size of a reference
 Size of a primitive (boolean, char, byte, short, int, long,
float, double)
 Size of primitive arrays
 Size of any basic common objects used (String, Integer,
Long, BigDecimal, Timestamp, etc)

31

JavaOne
IBM CONFIDENTIAL


 For collections, measure the size of an empty collection
 Then determine the overhead for an entry added to the
collection
 It is better to overestimate then underestimate!
 Better to use a little less than is available then to run out!
 You can always refine the estimates to make them better
 And anyway, the JVM is probably using more memory than
you think!

32

JavaOne
IBM CONFIDENTIAL


 Have a MemoryUsage interface with one method:
long getMemoryUsage()
 Have each class whose memory you are going to track
implement this interface
 Each implementation estimates the memory usage of its
primitives and object references
 Primitives, object overhead, and object reference memory use can
be a static final for that class
 For any non-null object references, invoke the appropriate
method to get the memory estimate for that object
 This includes arrays

33

JavaOne
IBM CONFIDENTIAL


 Round the final result up to a multiple of 8
 If recursive references are possible, you need to remember if
you’ve already visited an object when computing the estimate
 Shared objects should only be counted once, or not at all
 For potential “flyweight” objects (interred strings, cached
Integers, etc.) :
 Assume not a flyweight unless you know you’ve used the
value before or will probably use it again
 For example, a String that stores a frequently used file name

34

JavaOne
IBM CONFIDENTIAL


 Most custom objects quickly devolve down to objects
containing primitives and basic common objects
 You don’t need to do this for every class – just for those
that you are going to track the memory use
 You can easily check the accuracy of your estimates
against reality using the techniques described earlier
 The memory estimation is quick

35

JavaOne
IBM CONFIDENTIAL

Runtime memory usage estimation :
Code Sample
public class WorkUnit implements MemoryUsage
{
private static final int primitiveUsage = 16;
private static final int thisOhead = 4;
private static final int refsUsage = 2 * 4;
private static final int fixedOhead = primitiveUsage + thisOhead + refsUsage;
long userid;
String originatingSystem;
Request requestObj;
long memUsage = -1;

36

JavaOne
IBM CONFIDENTIAL

Code Sample
{
if (memUsage == -1) {
int estimate = fixedOhead;
estimate +=
MemUtility.getStringMemUsage(originatingSystem.length());
estimate += requestObj.getMemoryUsage();
memUsage = MemUtility.roundUsage(estimate);
}
}
} // WorkUnit

37

JavaOne
IBM CONFIDENTIAL

Code Sample
public class Request implements MemoryUsage
{
private static final int primitiveUsage = 16;
private static final int thisOhead = 4;
private static final int refsUsage = 2 * 4;
private static final int fixedOhead = primitiveUsage + thisOhead + refsUsage;
long requestId;
String request;
String discountCode;
long memUsage = -1;

38

JavaOne
IBM CONFIDENTIAL

Code Sample
{
if (memUsage == -1) {
int estimate = fixedOhead;
estimate +=
MemUtility.getStringMemUsage(request.length());
if (discountCode != null) {
estimate +=
MemUtility.getStringMemUsage(discountCode.length());
}
memUsage = MemUtility.roundUsage(estimate);
}
}
} // Request

39

JavaOne
IBM CONFIDENTIAL

Code Sample
public class MemUtility
{
static long getStringMemoryUsage( int length)
{
return 40 + (length << 1);
}
static long roundUsage( long usage)
{
long usageDiv8 = usage >> 3;
if ((usageDiv8 << 3) == usage)
return usage;
else
return (usageDiv8 + 1) << 3;
}

}

40

JavaOne
IBM CONFIDENTIAL

Track and Control Memory Usage

 Have one logical pool of memory which consists of all the
heap memory that is used by the objects whose memory
usage will be controlled
 Avoid separate pools!
 Memory usage can be controlled “statically” or “dynamically”
 Controlling memory usage statically means we reserve the
memory up front, when the application starts, or when a new
thread starts
 Note that you don’t need to actually allocate the memory up front,
you just reserve it

41

JavaOne
IBM CONFIDENTIAL


 Dynamic tracking and control:
 Keep track of the total used by all threads
 Get the estimated memory usage of an object from its getMemoryUsage
method and increment the total amount used
 When the object is no longer needed, the memory use is decremented
from the total amount used
 Only track long term memory usage
 Don’t need to “pay” for the memory dynamically if it was
accounted for statically
 Each thread maintains a local total that tracks its
memory usage

42

JavaOne
IBM CONFIDENTIAL


 The global total amount used is a globally
accessible long
 All increments and decrements of the global total
are synchronized
 Updates of the global total are buffered to minimize
synchronization costs

43

JavaOne
IBM CONFIDENTIAL


 The absolute limit on the memory that can be used is the
JVM max memory specified with the –Xmx parameter
 But dynamically tracked memory can’t have all of that
 Memory should be reserved statically for objects or buffers if:
 The application must have them
 There are not very many
 They will be used for a long time
 Static size, or a static upper limit (reserve the upper limit)

44

JavaOne
IBM CONFIDENTIAL


 Examples of buffers to reserve memory for statically:
 I/O buffers
 LIFO page caches
 Need to reserve a certain amount of memory for “everything
else” – i.e. what we are not accounting for statically or
dynamically
 This can be one sum, a per-thread value * # of threads, or both
 Also need to reserve memory to prevent the JVM from
thrashing
 Don’t want to allow the heap to grow past 85-90% of the JVM max
memory
 Recommend reserve 15% of the JVM max memory

45

JavaOne
IBM CONFIDENTIAL


 So, the amount of memory available for dynamically
tracked memory is: max memory – (staticBuffers +
thrashingOverhead + everythingElse)
 Let’s call this “globalMaxDynamicMemory”
 Every time a thread increments the global total, the
global total is compared against
globalMaxDynamicMemory
 If the global max is exceeded, the increment request is
denied

46

JavaOne
IBM CONFIDENTIAL


 If an increment of the global total is denied, the
requesting thread must handle it.
 A thread can handle this in a number of ways:
 Wait and periodically retry until more memory becomes
available
 Reduce some of its existing registered memory usage –
throw away unneeded objects, etc
 Write some of its data to disk
 Throw an exception, i.e. give up
 Get another thread to give up some memory (possible, but
more complicated!)

47

JavaOne
IBM CONFIDENTIAL


 If a thread reduces its registered memory usage by the
amount of the denied request, it doesn’t need to re-
submit the request
 To prevent one thread from “hogging” all the memory,
can have heuristics, such as one thread cannot have
more than 90% of the dynamically tracked memory if
there is more than one thread
 The decrement of a thread’s memory use should be in
finally blocks to ensure the memory is released if the
thread gets an uncaught exception

48

JavaOne
IBM CONFIDENTIAL

Soft references

 Use of soft references to control your memory use really
only works for objects you can get again or don’t really
need
 This may be acceptable for something like a web page
image – if it becomes reclaimed by the GC, you just
read the image from disk again
 Doesn’t work for objects where the only copy is in
memory!
 No way to control the order in which softly reachable
objects are reclaimed

49

JavaOne
IBM CONFIDENTIAL

Minimize Memory Usage

 Prefer arrays over more complex structures
 Collections:
 Avoid empty collections
 Size collections appropriately
 Choose correct collection
 Class structure impact on memory usage
 Avoid storing data in a separate class if all instances of your
class need that data, and it is not needed outside of that
class
 Store info specific to only a subset of class instances in a
derived class or in a separate class

50

JavaOne
IBM CONFIDENTIAL


 Convert objects to byte arrays
 Objects that won’t be needed for a while
 Objects that are going to be written to a stream soon
 Example: objects in a hash map when most won’t be
used for a while
 Convert to byte array
 Convert back when fetched
 Can have a mix of converted/unconverted data: convert a
byte array once it has been fetched once
 However, the hash map won’t be able to use generics if it
has a mix of converted/unconverted data

51

JavaOne
IBM CONFIDENTIAL


 Cleanup unneeded objects ASAP
 Optimize for mainline processing, not exceptional processing
 For example, if possible, delete requested info as soon as send it
back to the user – don’t wait for the confirmation response
 If response is negative, read the requested info back in
 Of course, this is only worthwhile if failures are rare!
 Use finally blocks liberally!
 Make sure any allocated memory is freed
 And if it’s dynamically tracked, be sure to decrement the memory
usage from the global total!

52

JavaOne
IBM CONFIDENTIAL


 Object pooling
 Doesn’t reduce your memory usage, but reduces the cost to
allocate and GC it
 Only worth it for simple, multi-purpose objects, such as primitive
arrays
 Byte arrays are the best candidate as they can be used to store any kind
of data, and don’t have to be an exact match
 Relatively easy to have a pool of byte arrays of various sizes that can be
used to store other objects
 Store large objects (i.e. lobs) as a series of smaller byte array buffers
instead of one large one
 If you find your pooling is getting complicated – stop! Leave
complex memory mgmt to the JVM

53

JavaOne
IBM CONFIDENTIAL

Balance and control threads

 Feed work to a thread via an ArrayBlockingQueue or your own
synchronized buffering scheme, with a limited size
 When a thread input queue is empty, or an output queue is full, the
thread will have to wait
 This keeps threads from getting too far ahead of each other
 The capacity of the queue/buffer determines how far ahead threads
are allowed to get relative to each other
 To balance, keep track of the times producers and consumers must
wait to put/take data from the queue
 Lots of consumer waits = need more producer threads
 Lots of producer waits = need more consumer threads

54

JavaOne
IBM CONFIDENTIAL


 The capacity of a java blocking queue is specified as a
number of objects
 Unless all objects placed in the queue have the same size,
the memory usage of the queue can vary widely
 Can run out of memory if don’t limit or keep track of the
memory usage of blocking queues

55

JavaOne
IBM CONFIDENTIAL


 The solution is decorate the blocking queue with a logical
queue
 The logical queue buffers objects in lists and adds these list
buffers to the queue when the buffers are full
 The lists are considered full when they reach a certain
amount of estimated memory usage
 So the queue is a queue of lists, each of which will use a
similar amount of memory
 The maximum amount of memory the queue can use is then
bufferSize * (queueCapacity)

56

JavaOne
IBM CONFIDENTIAL


 The maximum amount of memory usage for each inter-
thread queue is statically reserved when each queue is
created, and released when those threads stop
 This has the additional benefit of reducing the number of
puts/gets to the underlying queue, which reduces the
synchronization expense
 Also bases the amount of queued-up work for a thread on
the amount of memory that work consumes, instead of a
number of requests
 This is probably a better metric, since requests that
consume more memory probably take longer to process

57

JavaOne
IBM CONFIDENTIAL

Waiting/Flushing

 A thread needs to flush a buffered output queue if it is idle
or has to wait
 Waiting indefinitely is dangerous!
 Try to NEVER wait on anything indefinitely
 If you are waiting for another thread to do something, and it
dies, you will be waiting forever
 So, use loops+timeouts whenever possible
 When wake up, check if anyone wants you to stop, process a more
urgent request, etc.
 If not, re-issue request

58

JavaOne
IBM CONFIDENTIAL

Handle OutOfMemoryError

 Despite your best efforts, your app may still run out of
memory
 There are many reasons for this:
 Tracked memory usage estimates are too low
 Untracked memory usage was larger or was utilized longer
than expected
 Short term memory usage burst overwhelmed the GC
 Memory Leaks
 Too many threads, etc

59

JavaOne
IBM CONFIDENTIAL


 If an OOM error occurs, you will probably need a heap dump
to determine why!
 IBM JVM enables heap dumps on OOM by default
 To disable, set environment variable
IBM_HEAPDUMP_OUTOFMEMORY=FALSE
 Oracle HotSpot disables heap dumps on OOM by default
 To enable: -XX+HeapDumpOnOutOfMemoryError
 To see the value of all flags, specify: -XX:+PrintFlagsFinal

60

JavaOne
IBM CONFIDENTIAL


 When an OutOfMemoryError occurs, this is only guaranteed
to stop the thread that triggered it
 This can lead to “zombie threads”
 Threads that were dependent on the thread that died can hang
 The death of the thread getting the OOM might be sufficient to
avoid another OOM, but not enough to keep the JVM from
thrashing, thus preventing remaining threads from doing
anything
 Need a mechanism to bring down related threads

61

JavaOne
IBM CONFIDENTIAL


 Bring down related threads with stop request objects
 Bring down the JVM if the thread that dies is a “critical”
thread – i.e. it is essential for your application
 Or if zombie threads are a big problem, can have your
uncaught exception handler bring down the entire JVM if the
exception is OOM

62

JavaOne
IBM CONFIDENTIAL

Stopping

 Requesting stop via flag objects
 Have a stop request object that indicates thread should stop
 Classes check regularly, and always before sleeping, waiting,
I/O, putting/getting from a blocking queue, etc.
 Works the same regardless of whether the class is running as
a thread or not
 Stop request object can indicate different types of stop: stop
immediate, stop gracefully (finish work in progress), etc.

63

JavaOne
IBM CONFIDENTIAL

Stopping

 Have a stop request object that indicates thread should stop
 Can have a single stop request object for all related threads
 But this only allows all of them to be stopped at once
 For finer granularity of stop control, have a higher level stop
request object with references to thread stop request objects
 Setting of stop request object must be synchronized – but this
does not occur very often

64

JavaOne
IBM CONFIDENTIAL

Stopping

Have a request stop object that indicates thread
should stop
Reading of the stop request object will occur
frequently
Reading stop request object does not need to be
synchronized for most applications
 It is not critical to see a stop request instantaneously, and
synchronization doesn’t guarantee this anyway
 The stop request should be a boolean or enum, so updates
of it are atomic

65

JavaOne
IBM CONFIDENTIAL

Stopping

Detecting a thread has actually stopped
Could use Thread.isAlive() to detect if a thread has
stopped
 This requires a handle to the thread
 Presumes the class is running as a thread
 Won’t work for higher abstractions

66

JavaOne
IBM CONFIDENTIAL

Stopping

 Detecting a thread has actually stopped
 Better to make this abstract
 Separate Runnable class from class that does the real work
 Runnable class invokes the work class
 The Runnable class has an “isRunning” object that tracks the
run status
 All threads use this common Runnable class

67

JavaOne
IBM CONFIDENTIAL

Stopping

Detecting a thread has actually stopped
If a graceful stop is requested, threads must wait for
all of their producer threads to stop before they can
stop
This allows in-flight data to be processed
 Unexpected thread termination
 Just set the stop request object to bring down all related
threads

68

JavaOne
IBM CONFIDENTIAL

Stopping

 Returning control to the user
 Controller or highest level thread must make sure all threads
are truly stopped before returning
 Strange things can happen if you try to restart and some
threads from a previous incarnation are still running!
 Could call thread.join()
 But must have handle to all threads, and can’t be responsive to
other requests
 Better for the controller to query the isRunning flag of all thread
classes and wait for all to be stopped.

69

JavaOne
IBM CONFIDENTIAL

Example: buffering, flushing, stopping

work()
{
while (!checkForStopRequests()) {
int maxBufSize = 102400;
int curBufMemUsage = 0;
int outBufCapacity = 10;
ArrayList<WorkUnit> outBuf = new ArrayList<WorkUnit>(outBufCapacity);
ArrayList<WorkUnit> inBuf = inQueue.poll(200,TimeUnit.MILLISECONDS);
// If have to wait for input, flush output
if (inBuf == null) {
if (outBuf.size() > 0) {
flushOutput();
}
}

70

JavaOne
IBM CONFIDENTIAL


else {
for (WorkUnit workUnit : inBuf) {
// process workUnit
…
int unitMemUsage = workUnit.getMemoryUsage();
if (curBufMemUsage + unitMemUsage > maxBufSize) {
flushOutput(outBuf);
curBufMemUsage = 0;
}
else {
curBufMemUsage += unitMemUsage;
outBuf.add(workUnit);
}
}
}
}

71

JavaOne
IBM CONFIDENTIAL


flushOutput(ArrayList<WorkUnit> outBuf) {
boolean success = false;
while (!success && !checkForStopRequests()) {
success = outQueue.offer(outBuf, 200,TimeUnit.MILLISECONDS);
}
}

Boolean checkForStopRequests() {
// check for stop requests. Basic handling is:
if (stop requested) {
threadRunnableObj.isRunning = false;
return true; // stop this thread
}
return false;
}

72

JavaOne
IBM CONFIDENTIAL

Uncaught exception handling

 Proper handling for uncaught exceptions is absolutely
critical !
 Uncaught exceptions are the leading cause of unexpected
thread termination
 If any threads are dependent on others, if one goes down, the
remaining threads can hang!
 Hung threads can be hard to detect
 The best way to handle this is to prevent it!

73

JavaOne
IBM CONFIDENTIAL


 By default, uncaught exceptions are handled by the default
uncaught exception handler
 This just writes the stack trace to standard error
 This is rarely acceptable!
 Stack trace may be lost
 If class tracks its run status, this will be inaccurate after an uncaught
exception: run status will still be ‘running’
 What about threads that are dependent on this thread?

74

JavaOne
IBM CONFIDENTIAL

 Ensure you have handling for all uncaught exceptions
 Keep it simple: do what must be done, but no more
 Complex handling for uncaught exceptions could trigger other uncaught
exceptions, which can be extremely difficult to debug!
 And since uncaught exceptions don’t occur often, the handler probably has not
been tested as much as other modules
 All handlers, at a minimum, should:
 Log the error and stack trace somewhere durable and secure
 Alert the user so the error will be noticed
 If the thread class has a run status, the run status should be set to stopped,
error, or some equivalent
 Requires access to the thread class state

75

JavaOne
IBM CONFIDENTIAL


 Ideally, the handler should also alert dependent threads
 Can notify controller or parent, which can:
 Just restart the thread that got the uncaught exception
 Not recommended! Any related threads could be in an indeterminate state
 Stop all related threads (safest)
 Set the stopping request object for the related threads
 Since all threads are monitoring, this will bring them all down
 Threads won’t hang!

 Controller/parent doesn’t need an explicit message to know a
thread has died, if it is monitoring the run status of the threads
 In fact, the controller doesn’t even need to get involved –
 Just have the uncaught exception handler set the stop request object to stop

76

JavaOne
IBM CONFIDENTIAL


 Where to handle?
 Three primary possibilities:
 Custom uncaught exception handler
 Catch Throwable() in run() method
 Finally block in run() method

77

JavaOne
IBM CONFIDENTIAL


 Custom uncaught exception handler
 If a uncaught exception handler is declared and is in scope,
the default uncaught exception handler is NOT invoked
 Determine scope:
 Per JVM
 Per Thread group
 Per Runnable class

78

JavaOne
IBM CONFIDENTIAL


 Custom uncaught exception handler – per JVM scope
 Logging the error and alerting the user is about all that can
be done
 Since this is per-JVM, it does not have access to any specific
thread classes or thread groups
 Set via the static Thread method
setDefaultUncaughtExceptionHandler
 Thread.setDefaultUncaughtExceptionHandler(
Thread.UncaughtExceptionHandler ueh)
 ueh is a class that must implement uncaughtException(Thread thr,
Throwable the)

79

JavaOne
IBM CONFIDENTIAL


 Custom uncaught exception handler – per Thread group
 Can guarantee standard handling for a group of threads by having
their Thread instance use the same custom ThreadGroup class
 Custom ThreadGroup class extends ThreadGroup and overrides the
ThreadGroup uncaughtException method
 Allows for common code for the thread group, but doesn’t have access
to individual Runnable classes
 Could have the stop object for the threads in the group registered here
 set to true if an uncaught exception occurs in any thread in the group

80

JavaOne
IBM CONFIDENTIAL


 Custom uncaught exception handler – per Thread class
 Can have access to the Runnable class, so the handler can log
details about the class structure, set run status, etc. before
exiting
 Guaranteed to be invoked
 Set via the Thread method setUncaughtExceptionHandler
 setUncaughtExceptionHandler(
Thread.UncaughtExceptionHandler ueh)
 ueh is a class that must implement uncaughtException(Thread thr,
Throwable the)

81

JavaOne
IBM CONFIDENTIAL


 Catch Throwable() in run() method
 Not recommended as the sole mechanism for handling
uncaught exceptions
 If you do this, do it outside any loops in the thread run method, i.e.
at the highest level possible.
 Otherwise, the exception will not trigger the thread to exit, which can
cause very strange thread conditions!
 Thread can be “alive”, but constantly cycling through errors, or hanging

82

JavaOne
IBM CONFIDENTIAL


 Finally block in run() method
 Have handled flag that is set to true at end of try block and every catch
block
 !handled = uncaught exception
 Advantage: Can have handling customized to the invoked class
 Can set run status and stop request
 Disadvantages
 Every catch block must set “handled” flag or the checked exception will be
treated as unhandled
 An uncaught exception handler that logs the exception is still necessary,
since the finally block doesn’t have access to the exception

83

JavaOne
IBM CONFIDENTIAL


Critical threads
Some threads may be critical to your application
If they stop unexpectedly, the application will not
function
Best to stop the entire application when this occurs
The uncaught exception handling for the thread
should call System.exit(<non-zero value>)

84

JavaOne
IBM CONFIDENTIAL

Example using finally block
public class bulletproofRunnable implements Runnable {
boolean isRunning;
Worker worker;
boolean handled;
bulletproofRunnable(Worker worker) {
this.worker = worker;
handled = false;
}

85

JavaOne
IBM CONFIDENTIAL

Example using finally block
run() {
try {
isRunning = true;
worker.work();
handled = true;
}
finally {
if (!handled) {
// uncaught exception!
isRunning = false;
worker.setStopRequest(true);
// have uncaught exception handler log error and stack trace to error file
}
}
}

}

86

JavaOne
IBM CONFIDENTIAL

Example using custom handler

public class bulletproofRunnable implements Runnable,
Thread.uncaughtExceptionHandler {
boolean isRunning;
Worker worker;
bulletproofRunnable(Worker worker) {
this.worker = worker;
}

87

JavaOne
IBM CONFIDENTIAL

Example using custom handler

public void uncaughtException {Thread thr, Throwable the)
isRunning = false;
worker.setStopRequest(true);
// log error and stack trace to error file
}
}
Add to where the actual thread is created:
bulletproofThread.setUncaughtExceptionHandler(bulletproofRunnable);

88

JavaOne
IBM CONFIDENTIAL

Summary

 Excessive memory usage can have a large adverse affect on
application performance
 Estimating memory usage of the objects used by your application
is the first step to managing the application memory usage
 Tracking and controlling long-term memory usage is essential to
avoid OOM
 Many techniques exist to minimize memory usage
 Limiting thread input/output queue sizes by their memory usage
balances thread memory use and helps avoid OOM
 A framework for stopping threads and consistent handling of
uncaught exceptions is essential to avoid hanging when OOM
errors or other unexpected exceptions occur

89

JavaOne Thank You

Phillip Koza
pkoza@us.ibm.com

Efficient Memory and Thread Management in Highly Parallel Java Applications

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Efficient Memory and Thread Management in Highly Parallel Java Applications (20)

Recently uploaded (20)

Efficient Memory and Thread Management in Highly Parallel Java Applications

Editor's Notes