Spring Interview Questions

java.lang.Out OfMemoryError

Close Window Print Story

Unveiling the java.lang.Out OfMemoryError

When we encounter a java.lang.OutOfMemoryError, we often find that Java heap dumps, along with other artifacts, are generated by the Java Virtual Machine. If you feel like jumping right into a Java heap dump when you get a java.lang.OutOfMemoryError, don't worry, it's a normal thought. You may be able to discover something serendipitously, but it's not always the best idea to analyze Java heap dumps, depending on the situation you are facing. We first need to investigate the root cause of the java.lang.OutOfMemoryError.
Only after the root cause is identified can we decide whether or not to analyze Java heap dumps. What is a java.lang.OutOfMemoryError? Why in the world does it occur? Let's find out.
What Is a java.lang.OutOfMemoryError?
A java.lang.OutOfMemoryError is a subclass of java.lang.VirtualMachineError that is thrown when the Java Virtual Machine is broken or has run out of resources that are necessary to continue the operation of the Java Virtual Machine. Obviously, memory is the exhausted resource for a java.lang.OutOfMemoryError, which is thrown when the Java Virtual Machine cannot allocate an object due to memory constraints. Unfortunately, the Java specification of java.lang.OutOfMemoryError does not elaborate further on what kind of memory it's talking about.
There are six different types of runtime data areas, or memory areas, in the Java Virtual Machine (see Figure 1).

1. Program Counter Register
2. Java Virtual Machine Stack
3. Heap
4. Method Area
5. Runtime Constant Pool
6. Native Method Stack

The Program Counter Register, also known as the pc register, stores the address of the Java byte code instruction that is currently being executed (just like the processor register in your central processing unit of the device from which you are reading or printing this article). You will not see a java.lang.OutOfMemoryError from the pc register since a program counter is not conventionally considered as a memory.
Java Virtual Machine Stacks contain frames where data, return values, and partial execution results are stored. Java Virtual Machine Stacks can be expanded during runtime. If there's not enough memory for the expansion of an existing Java Virtual Machine stack, or for the creation of a new Java Virtual Machine stack for a new thread, the Java Virtual Machine will throw a java.lang.OutOfMemoryError.
The Heap is where instances of Java classes and arrays are allocated. A java.lang.OutOfMemoryError will be thrown when there is not enough memory available for instances of Java classes or arrays.
The Method Area stores class-related information, the runtime constant pool, for instances, the code for methods and constructors, and field/method data. If there's not enough memory in the method area, you will encounter java.lang.OutOfMemoryError.
The Runtime Constant Pool contains constants such as field references and literals. A java.lang.OutOfMemoryError will be thrown when not enough memory is available for the construction of the runtime constant pool area.
Native Memory Stacks store conventional stacks, also known as C stacks, to support native methods that are written in a non-Java language such as C/C++. Native memory stacks can be expanded during runtime. If there's not enough memory for the expansion of an existing native memory stack or for the creation of a new native memory stack for a new thread, you would see a java.lang.OutOfMemoryError.
You may have seen a java.lang.StackOverflowError, which is completely different from a java.lang.OutOfMemoryError. A java.lang.StackOverflowError is thrown when native memory stacks or Java Virtual Machine stacks need more memory than is configured. In most IBM Java Virtual Machine implementations, the -Xmso command-line option controls the stack size for operation system threads or native thread, and the -Xss command-line option controls the stack size for Java threads. In some implementations, such as Sun Microsystems HotSpot Java Virtual Machine, the Java methods share stack frames with C/C++ native code. The maximum stack size for a thread can be configured with the -Xss Java command-line option. The default sizes of these options vary by platform and implementation, but are usually between 256 Kbytes-1024 Kbytes. Please refer to the documentation of your Java virtual machine for more specific information. We will cover more about java.lang.StackOverflowError in a separate article.
Now that we understand which memory areas could cause a java.lang.OutOfMemoryError, let's take a look at actual error messages. What does a java.lang.OutOfMemoryError look like and how can I address each symptom? Have you ever seen a java.lang.OutOfMemoryError similar to the following?
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
This error message indicates that there is a memory request for an array but that's too large for a predefined limit of a virtual machine. What do we do if we encounter this kind of java.lang.OutOfMemoryError? We need to check the source code to make sure that there's no huge array created dynamically or statically. Fortunately, latest virtual machines usually do not have this limit.
java.lang.OutOfMemoryError: PermGen space
You will see an OutOfMemoryError when the Permanent Generation area of the Java heap is full, like the above message.
On some Java Virtual Machines, such as Sun Microsystems' HotSpot Java Virtual Machine, a dedicated memory area called permanent generation (or permanent region) stores objects that describe classes and methods. We can visualize the usage of a permanent generation with the IBM Pattern Modeling and Analysis Tool for the Java Garbage Collector.
In Figure 2 we enabled the "Max Perm" button and  the "Used Tenured" button to visualize permanent generation usage and its maximum size. We can see that the used amount of permanent generation reaches its maximum limit. That's why we're getting the java.lang.OutOfMemoryError: PermGen space message. If there's no memory leak, we can just use the -XX:MaxPermSize command-line option to increase the maximum limit of the permanent generation. For example,
-XX:MaxPermSize=128m
will set the maximum size of the permanent generation to 128 Mbytes.
So far we've seen a Java.lang.OutOfMemoryError due to exhaustion in the Java heap or an area in the Java heap such as permanent generation. Surprisingly, a Java.lang.OutOfMemoryError can be thrown when the Java Virtual Machine cannot find any more memory in the native memory as well as in the Java heap. How can we tell whether it's caused by the Java heap or native memory?
In the following message, there's no information in the message whether java.lang.OutOfMemoryError is caused by the Java heap or native memory:
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
In the following case, the Java virtual machine is kind enough to tell us that there's native memory exhaustion. In the message, the Java virtual machine says "allocateMemory failed" which means a native memory allocation failed:
java.lang.OutOfMemoryError: JVMCI046: allocateMemory failed
In the following message, there's no clue as to whether it's native memory or a Java heap. Fortunately we have a line number, 20, and the source code file name, HeapExhaustionSimulator.java. This might be Java heap related.
JVMDG274: Dump Handler has Processed OutOfMemory.
Exception in thread "main" java.lang.OutOfMemoryError
at HeapExhaustionSimulator.main(HeapExhaustionSimulator.java:20)
In the following message, there's no clue whether it's native memory or a Java heap. But "sun.misc.Unsafe.allocateMemory(Native Method)" indicates that it might be native memory related.
Exception in thread "main" java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:99)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
at NativeMemorySimulator.main(NativeMemorySimulator.java:11)
In the following message, the  Java Virtual Machine indicates that the Java heap space is related to the java.lang.OutOfMemoryError.
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid6280.hprof ...
Heap dump file created [50549348 bytes in 1.444 secs]
You may have seen a java.lang.OutOfMemoryError similar to the following:
java.lang.OutOfMemoryError: requested NNN bytes for MMMM. Out of swap space?
Literally you could check the operating system configuration for swap space. It seems that the Java Virtual Machine is not sure if the swap space is the root cause of the problem (?).We can check whether this Java Virtual Machine is consuming too much native memory .We also need to make sure there's enough memory for this JVM and no other processes are consuming most of memory resource. The last thing we can try is to find any known defects related to the module, MMMM.
java.lang.OutOfMemoryError: unable to create new native thread
This kind of message is seen when you have an excessive number of threads or if the native memory is exhausted and a thread is attempting to be created.
What Is a Java Heap Dump?
We've learned that a Java heap is a runtime data area where all class instances and arrays are allocated and shared among all Java Virtual Machine threads during execution of the JVM. A Java heap dump is a snapshot of a Java heap at a specific time. It's like taking a picture of a busy warehouse at a given time. If we look at the picture, we can identify what items were available at that time. Some items may be shipped to Canada a few minutes later, but you can see them in the picture because they were there at the time of the snapshot.
Because the Java specification does not mention the Java heap dump, there are different forms of Java heap dump implementations from different Java Virtual Machines. The IBM Java heap dump provides information mostly about the Java heap.
The Sun Microsystems hprof Java heap dump provides information about the Java Virtual Machine stacks, the runtime constant pool as well as the Java heap.
How Can I Generate Java Heap Dumps?
A Java heap dump is usually automatically generated by the Java Virtual Machine, but you can also force Java heap dump generation. On most IBM Java Virtual Machines, Java heap dumps are generated automatically when the Java heap becomes exhausted. On most Sun Microsystems JVMs, you need to configure the virtual machine to generate Java heap dumps. If you want to generate a Java heap dump when a java.lang.OutOfMemoryError occurs, you need to set the -XX:+HeapDumpOnOutOfMemoryError command-line option on certain releases of Sun's JVM. You could also use a HPROF profiler by using the -agentlib:hprof=heap=dump command-line option. You could also use jmap if your Sun JVM provides the utility. For example, jmap -dump 1234 will generate the Java heap dump from the process whose identifier is 1234. You could utilize JConsole by calling the HotSpotDiagnostic MBean and the dumpHeap operation if it's available from your Sun JVM.
If you want to generate Java heap dumps for Java virtual machine crashes (an unexpected termination of process) or user signals on IBM JVMs, you can set the environment variable IBM_HEAPDUMP or IBM_HEAP_DUMP to TRUE. For example, you can send the IBM Java virtual machine the signal SIGQUIT for the Linux operating systems and AIX operating systems or SIGINT(Control-Break key combination) for Windows to generate Java heap dumps. The IBM JVM provides an API, com.ibm.jvm.Dump.HeapDump(), that you can invoke from application code to generate Java heap dumps programmatically.
Please refer to the documentation of your JVM for detailed information since these options vary by platform and implementation.
Where Can I Find Java Heap Dumps?
You can find Java heap dumps in the current working directory of the Java Virtual Machine process, unless you specify a different location. You can specify the location with the environment variable IBM_HEAPDUMPDIR or _CEE_DMPTARG on IBM JVMs. If there's not enough space available for Java heap dumps or the JVM cannot acquire write-permission in the location, Java heap dumps are generated to the operating system's temporary directory on the IBM JVM. Please refer to your operating system manual for the location of the system's temporary directory and its configuration.
What Do Java Heap Dumps Look Like and How Can I read Them?
Nowadays, the majority of Java heap dump formats are generated in binary. Thus you might want to use a tool unless your brain can interpret hexadecimal codes without any headaches.
The IBM HeapAnalyzer is one of the most popular Java heap analysis tools. It can analyze all Java heap dump formats provided by Sun, HP and most of the Java heap dump formats provided by IBM. It's powered by object query engines and patent-pending heuristic analysis engines. I've been developing the IBM HeapAnalyzer from scratch since 2003, spending my vacation, weekends and weeknights on it. The IBM HeapAnalyzer was so successful that IBM decided to make the IBM HeapAnalyzer an official IBM software product and bundle it with existing products. So I donated all the source code of IBM HeapAnalyzer to IBM to run it on an Eclipse-based interface. Now the IBM HeapAnalyzer has a daughter, MDD4J, which always reminds me of my late daughter lost while I was working on the MDD4J project. The IBM HeapAnalyzer has been the top technology at IBM alphaWorks for six consecutive years since its inception as of March 2009.
Whether your Java heap dump is in binary or text/ascii format, the heap dump contains information about all the live objects that are on the Java heap such as address, object size, and referenced addresses. Let's take a look at the text/ascii format Java heap dumps since binary heap dumps have similar information but are in hexadecimal format to save disk space.
Dissecting IBM Text(Classic) Heap Dump File Structure
The following is a sample Java heap dump taken from an IBM Java Virtual Machine on Windows:
// Version: J2RE 6.0 IBM J9 2.4 Windows x86-32 build 20080209
0x10000000 [1000] OBJ ObjectA
0x10000100
0x10000100 [2000] OBJ ObjectB
0x10000200 0x10000300
0x10000200 [3000] OBJ ObjectC
0x10000500
0x10000300 [4000] CLS ClassD
0x10000400
0x10000400 [5000] OBJ byte[]
0x10000500 [6000] OBJ char[]
// Breakdown - Classes: 1, Objects: 3, ObjectArrays: 0, PrimitiveArrays: 2
// EOF:
Each entry has information about a live object. For instance, the first entry:
0x10000000 [1000] OBJ ObjectA
0x10000100
shows that there's an object named ObjectA at the address of 0x10000000. Its size is 1,000 bytes. In an older IBM JVM, you might not be able to see the "OBJ" tag. The object, ObjectA, has a reference to another object that is located at the address, 0x10000100, which is the addresss of the object, ObjectB, as you can see in the next entry:
0x10000100 [2000] OBJ ObjectB
0x10000200 0x10000300
This entry shows that the object, ObjectB, is located at the address of 0x10000100. The size of the object ObjectB is 2,000 bytes. The object, ObjectB, has reference to an object located at the address, 0x10000200, which is the address of the object, ObjectC, and to another object located at the address 0x10000300, which is the address of the class, ClassD, as you can see in the next entries:
0x10000200 [3000] OBJ ObjectC
0x10000500
0x10000300 [4000] CLS ClassD
0x10000400
The object, ObjectC, is located at the address of 0x10000200. The size of ObjectC is 3,000 bytes. It has references to the address, 0x10000500, which is the address of the array of character(char[]).
The class, ClassD, is located at the address of 0x10000300. The size of classC is 4,000 bytes. In an older JVM, you might not be able to see the "CLS" tag. It has references to the address, 0x10000400, which is the address of an array of bytes(byte[]). The primitive arrays, byte[] and char[], do not have any reference to other objects.
Now we can draw the directional graph shown in Figure 3 to visualize the references from each object. The char[] and byte[] do not have any references, so they are shown as leaf nodes in the graph.
We need to assess the weight of each object, class or primitive type. It's called TotalSize. The TotalSize of an object is the sum of an object's size and sum of its children's TotalSizes. If an object does not have any children, TotalSize is the same as object size of the object.
If you put the graph shown in Figure 3 upside down, it may look like an apple tree with fruit. The TotalSize is like the sum of all the weights of all the apples hanging above a specific branch. If you cut off a branch, pick all the apples from the branch, and put them on a scale, you will get the TotalSize of that branch. By looking at the TotalSize of each object, we can tell which object has references to objects using larger amounts of the Java heap, which is most likely where a memory leak might occur.
The graph shown in Figure 4 has total sizes calculated from each branch:
The TotalSize of byte[] is just the size of itself, 5,000 bytes, because it does not have any descendents. The TotalSize of classD (4,000 + 5,000) is the sum of the size of itself (4,000 bytes) and the sum of all its descendents, byte[] (5,000 bytes).
We could easily spend a night or two interpreting millions of objects in heap dumps and drawing directional graphs. That's why we need the IBM HeapAnalyzer as it will do all the work for you.
Figure 5 is a screenshot of the IBM HeapAnalyzer with the sample Java heap dump loaded. The directional graph is represented as a tree in the IBM HeapAnalyzer. There are an analysis view and a tree view. We can select any object in the tree view and get detailed information about the object such as address, name, and number of parent objects.
Dissecting the HPROF ASCII Java Heap Dump File Structure
Let's take a look at the hprof ASCII output.
The Java heap dump shown in Listing 1 was taken from Sun's JVM. The first line indicates a version of the Java heap dump format, JAVA PROFILE 1.0.1 followed by a timestamp of the file creation. (The listings for this article can be downloaded here.)
JAVA PROFILE 1.0.1, created Sat Feb 14 07:07:45 2009
The next record is thread information. Each thread record has a reference to an object, an identifier, a thread name and a thread group.
THREAD START (obj=5ecd1b8, id = 1, name="Signal dispatcher", group="system")
We don't see this information in IBM Java heap dumps. Next to Java thread records, we can see Java stack trace records. Each Java stack trace has a series of Java stack frames. Java stack traces are referenced from Java heap dump entries for example object record.  We do not see Java stack trace records in IBM Java heap dumps.
Finally we can find Java heap dump records. They are located between the following the two lines:
HEAP DUMP BEGIN (333 objects, 97716 bytes) Sat Feb 14 07:08:15 2009
and
HEAP DUMP END
This section has list of all live instances in the JVM. There are four types of instances:

• ROOT : Root instance
• CLS : Class
• OBJ : Object
• ARR : Array

The first entry in the heap dump record has the following:
ROOT 5ef2ee0 (kind=<thread>, id=3, trace=6)
This is a ROOT. The address of this instance is 5ef2ee0. We can find what's at the address 5ef2ee0 by looking further into the heap dump entries. It's an instance of java/lang/ref/Finalizer$FinalizerThread:
OBJ 5ef2ee0 (sz=76, trace=0, class=java/lang/ref/Finalizer$FinalizerThread@5ef2ea0)
This instance is referenced from <thread>. The thread ID is 3. We can find the thread from the THREAD section:
THREAD START (obj=5ef2ee0, id = 3, name="Finalizer", group="system")
Java stack trace ID is 6.We saw the  Java stack trace 6 in the TRACE section:
TRACE 6:
java/lang/Object.wait(Object.java:Compiled method)
java/lang/ref/ReferenceQueue.remove(ReferenceQueue.java:144)
java/lang/ref/ReferenceQueue.remove(ReferenceQueue.java:160)
java/lang/ref/Finalizer$FinalizerThread.run(Finalizer.java:225)
This is very useful information. It says java/lang/ref/Finalizer$FinalizerThread is a root object referenced from a thread. It provides even the Java stack trace of the thread, thread name, and thread group name. Unfortunately, we don't see this kind of information from an IBM Java heap dump.
Let's move on to a class entry:
CLS 607fd58 (name=HeapExhaustionSimulator, trace=0)
super          5e2aba8
loader          60a2490
domain        60a9590
constant[3]   5f5f2f8
This is a class(CLS). It's located at 607fd58. The name of this class is HeapExhaustionSimulator, and then we have a list of references. The super class is located at 5e2aba8.
Have you noticed anything that's not found in the IBM Java heap dump? We have the names of variables: super, loader and domain. In the IBM Java heap dump, we have only addresses. The names of variables provide critical information in lots of cases where you want to track down the reference chains. The following is an array entry.
ARR 60a9a30 (sz=44, trace=0, nelems=20, elem type=char)
This array is located at the address 60a9a30. The size of the array is 44 bytes. There are 20 elements in this array. This is a character array. The last entry is an object:
OBJ 5ef2ee0 (sz=76, trace=0, class=java/lang/ref/Finalizer$FinalizerThread@5ef2ea0)
This object is located at 5ef2ee0. The size of this object is 76 bytes. The Java stack trace identifier is 0. The class of this object is located at 5ef2ea0. The name of the class is java/lang/ref/Finalizer$FinalizerThread.
Generally hprof heap dumps are larger than IBM Java heap dumps. The following is the list of Java heap dump files:
02/14/2009  07:07 AM          9,087,076 heapdump.20090214.070746.6092.phd
02/14/2009  07:08 AM        49,468,469 heapdump.20090214.070747.6092.txt
02/14/2009  07:10 AM        94,548,647 java.hprof.bin
02/14/2009  07:08 AM      199,752,135 java.hprof.txt
IBM binary (PHD) format takes up the least space. The IBM Text (Classic) file takes about five times more space than the PHD format does. The Hprof binary takes almost 10 times the space as the IBM PHD file does. The Hprof ASCII heap dump is 20 times larger than the IBM PHD file. Generally hprof heap dumps are larger than IBM heap dumps because the hprof format carries much more information that the IBM heap dumps do not have. Text/ASCII heap dumps are larger than binary heap dumps because the binary format carries record tables to avoid duplicate records that are often written multiple times (or millions of times in worst case) in Text/ASCII heap dumps. Text/ASCII Java heap dumps are rarely used unless there's a problem generating binary Java heap dumps.
If you are running a JVM based on Sun's implementation, you don't have many choices: hprof binary or hprof ascii format. Since the two formats contain same information, you would probably pick the binary format that is chosen by default.
If you're running the IBM JVM, you have two more choices in addition to the hprof formats. PHD (binary) is the preferable format and the default format. PHD is the smallest in size in a majority of cases. If you need to read the contents of the objects, the Java thread name, the Java stack traces associated with the objects, or names of the variables, the hprof binary format is your choice.
Table 1 provides a list of the available hprof options from the IBM Java 2 Runtime Environment, Standard Edition Service Release 11 (build 1.4.2, J2RE 1.4.2 IBM Windows 32 build cn142-20080515) :
Hprof usage: -Xrunhprof[:help]|[:<option>=<value>, ...]
Table 2 shows a list of the available hprof options from the IBM Java SE Runtime Environment Service Release 2 (build pwa6460sr2-20080818_01)
hprof usage: java -agentlib:hprof=[help]|[<option>=<value>, ...]
Please be aware that IBM's hprof is demonstration code for the Java Virtual Machine Tool Interface (JVMTI); it's not an official product or a formal part of the Java Virtual Machine.
Actually for more than a decade, Sun has stated that the hprof format is still considered highly experimental and is subject to change without notice. If you are tired of reading now, let's have some fun with experimenting and simulation.
Experiment and Simulation of java.lang.OutOfMemoryError
First, let's simulate the native memory exhaustion and analyze the different artifacts generated by the JVM (see Listing 2a). In this experiment, we are creating a list of direct byte buffers. Because we're using native memory excessively, we expect to see native memory exhaustion. Please don't run this application on your production system since it could cause system hang, abnormal termination, or unexpected failures.
IBM's Java Virtual Machine 1.4.2 throws the message shown in Listing 2. It says the JVM was unable to allocate 1,000,000 bytes of direct memory after five unsuccessful retries. IBM's Java Virtual Machine 1.6 throws the following message:
Exception in thread "main" java.lang.OutOfMemoryError: native memory exhausted
at sun.misc.Unsafe.allocateMemory(Native Method)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:111)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:300)
at NativeMemorySimulator.main(NativeMemorySimulator.java:11)
It definitely says native memory is exhausted.
Sun's JVM 1.4.2 and 1.6 throws the following message:
Exception in thread "main" java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:633)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:95)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
at NativeMemorySimulator.main(NativeMemorySimulator.java:11)
It indicates it's out of memory while the Java Virtual Machine is reserving the direct buffer memory. We can take a look at line 11 in the NativeMemorySimulator.java to see if we are using too much direct buffer memory.
Even though the Java heap is not exhausted, let's make sure that we're not missing anything. We can bring up the IBM Pattern Modeling and Analysis Tool for the Java Garbage Collector (http://www.alphaworks.ibm.com/tech/pmat) and analyze the garbage collection trace as shown in Figure 6.
Used heap size does not even reach the total heap size but we encounter java.lang.OutOfMemoryError, which is indicated by red vertical line. Let's experiment with the Java heap exhaustion (see Listing 2b). In this simulation, we keep creating a list of strings, an array of characters, and an array of integers. The following is the command line I used to generate the hprof ASCII Java heap dump on the IBM 1.4.2 JVM service release 11:
E:\IBMSDK14211\bin\java -Xrunhprof:heap=dump,format=a,dooom=y -Xmx50m -classpath e:\heapdump -verbosegc HeapExhaustionSimulator 2> verbosegc.txt
The following is the command line I used to generate THE hprof binary Java heap dump on THE IBM 1.4.2 Java Virtual Machine service release 11:
E:\IBMSDK14211\bin\java -Xrunhprof:heap=dump,dooom=y -Xmx50m -classpath e:\heapdump -verbosegc HeapExhaustionSimulator 2> verbosegc2.txt
If you want to generate the IBM PHD binary Java heap dump and IBM Text/ASCII Java heap dumps at the same time, you can set the environment variable, IBM_JAVA_HEAPDUMP_TEST, to true, for example, on Windows.
set IBM_JAVA_HEAPDUMP_TEST=true
The following is the command line to generate the hprof binary Java heap dump on Sun Microsystem's Java Virtual Machine 1.6.0 update 12:
E:\sun1.6.0_12\bin\java -Xmx50m -classpath e:\heapdump -verbosegc -XX:+HeapDumpOnOutOfMemoryError HeapExhaustionSimulator > verbosegc3.log
Do we expect to see the Java heap exhaustion? Yes, the Java heap will be exhausted.
IBM's Java Virtual Machine throws the following message:
JVMDG274: Dump Handler has Processed OutOfMemory.
Exception in thread "main" java.lang.OutOfMemoryError
at HeapExhaustionSimulator.main(HeapExhaustionSimulator.java:20)
It doesn't indicate whether it's caused by native memory or Java heap.
Sun's Java Virtual Machine throws the following message:
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid6280.hprof ...
Heap dump file created [50549348 bytes in 1.444 secs]
It says the JVM ran out of Java heap space. Let's take a look at the Java heap usage from the garbage collection trace.
Analysis of Garbage Collection Trace from Java Heap Exhaustion
We can bring up the IBM Pattern Modeling and Analysis Tool for the Java Garbage Collector and analyze garbage collection trace as shown in Figure 7. The analysis indicates that the Java heap was completely exhausted. The tool also provides recommendations.
Assess required Java heap and increase maximum Java heap size using -Xmx option.
If it does not help, analyze Java heap dump with IBM HeapAnalyzer.
If we are still in doubt, we can take a look at the chart view in Figure 8. The red solid line indicates the used Java heap size. The yellow line indicates the total Java heap size. The blue line indicates the amount of the requested Java heap. The vertical red-dotted line indicates java.lang.OutOfMemoryError. When java.lang.OutOfMemoryError occurred, even though the Java heap usage was smaller than the total Java heap size, more Java heap than was available was requested. If increasing the maximum Java heap size does not alleviate the situation, we definitely need to take a look at theJava heap dumps.
What if you don't have a garbage collection trace? If you have an IBM Java Virtual Machine, it probably would generate the Java thread dump or javacore.
Analysis of Java Thread Dump from Java Heap Exhaustion
Let's bring up the IBM Thread and Monitor Dump Analyzer for Java and open a javacore generated during java.lang.OutOfMemoryError (see Figure 9). The tool alerts you with a warning: Java heap is almost exhausted (6% free space) as well as a recommendation. Okay, all the tools recommend the Java heap dump analysis.
Analysis of Java Heap Dump from Java Heap Exhaustion
Let's bring up the IBM HeapAnalyzer and open the Java heap dump. If you've already downloaded a copy of the IBM HeapAnalyzer several months ago, it's a good time to get the latest version. IBM HeapAnalyzer is updated frequently in order to accommodate recently discovered memory leak patterns. Figure 10 is the first screen after the Java heap dump is loaded.
Java heap usage size is 52,404,992 bytes (approximately 49.77 Mbytes), which is pretty close to the maximum Java heap size of 50 Mbytes. This tells us that the Java heap is almost exhausted. Let's find out which objects are using most of the Java heap. We can take a look at the Java heap from a reference tree perspective in the lower part of the screen. If there are no memory leak suspects, we need to open the Tree View manually by selecting Tree View from the Analysis Menu.
Here's list of columns in the Reference Tree Window:
TotalSize (TotalSize/HeapSize%) [ObjectSize] NumberOfChildObject(501) ObjectName Address

[ObjectSize] is the amount of space used by a specific object, class, or array.

TotalSize is the sum of all [ObjectSize] of objects referenced from an object and its decendants.

(TotalSize/HeapSize%) is the ratio of TotalSize to total Java heap usage.

NumberOfChildObject is the number of children under this level or branch.

ObjectName is the name of an object, a class, or an array.

Address is the address of an object, a class, or an array.

Here are the first couple of entries:
TotalSize (TotalSize/HeapSize%) [ObjectSize] NumberOfChildObject(501) ObjectName Address
50,001,176 (95%) [32] 1 java/util/ArrayList 0x1abb6d8
50,001,144 (95%) [152] 26 array of [Ljava/lang/Object; 0x29fdd70
2,000,016 (3%) [2,000,016] 0 char[] 0x1502760
The object, java/util/ArrayList, has references to other objects and the sum of the sizes of all the objects referenced from the object and its descendants is 50,001,176 bytes.
The next largest TotalSize is 2,000,016 bytes from char[]. If there's any memory issue, the root cause should be under the object java/util/ArrayList not from long[] because the object java/util/ArrayList has a much larger TotalSize, which is as much as 95% of the total Java heap usage. We could say that most of apples are under this branch.
How can we find a memory leak? It's already been highlighted. There's also a menu called "Subpoena Leak Suspect(s)" in the Reference Tree. Leak suspects are displayed in blue. By default the most suspected area is displayed (see Figure 11).
It's very straightforward. We have a Java heap leak suspect, an array of java/lang/Object that is responsible for 50,001,144 bytes (95.41293 %) of the Java heap. The parent of the array of java/lang/Object is java/util/ArrayList, which means java/util/ArrayList contains the array of java/lang/Object. What objects do we have in the array? We can take a look at the children of the array of java/lang/Object. There are lots of arrays of integer (int[]) and arrays of character(char[]). Interestingly we cannot find who created java/util/ArrayList from the Java heap dump since it's marked as a root, which is not referenced from any object. We'll revisit this later in this article.
In summary, we have java/util/ArrayList that contains an array of int[] and char[]. It's responsible for 95.4 % of the Java heap based on the IBM HeapAnalyzer's analysis. We need to find out why java/util/ArrayList consumes so much Java heap. If there's any memory leak, we should address it. If there's no memory leak and we just need more memory, we could increase maximum Java heap size by adjusting -Xmx command-line option.
We've use very simple application. What if there's any memory leak in static variable? Can we find which class is responsible for memory leak from a static variable?
Simulation of a Memory Leak in a Static Variable
We can build a class with a static variable (see Listing 2c). We can build ClassVariableTest to instantiate ClassVariableExample (see Listing 2d)
Let's run ClassVariableTest with 1024 Mbytes of THE maximum Java heap size. No wonder, we got java.lang.OutOfMemoryError and a Java heap dump. We can open the Java heap dump with the IBM HeapAnalyzer, which would lead you to a leak suspect right away (see Figure 12).
This heap dump was taken from a JVM with a Java maximum heap size of 1024 Mbytes. The used heap size is 1,072,617,312 bytes (1023MBytes), which is pretty close to 1024 Mbytes. So the Java heap got almost exhausted. Let's find out which objects are using most of the Java heap.
The first class, com/ibm/jvm/ExtendedSystem, has references to other objects and the sum of the sizes of all the objects referenced from the class and its descendants is 1,070,471,976 bytes. 757 is the number of root objects.
The array of java/lang/Object is a leak suspect. 1,070,442,368 is the TotalSize. [10,256] is the Size of the object. 2,007 is the number of children. array of java/lang/Object is the name of the object. 0x101f8ed0 is the address of the object.
Let's climb up the tree. The java/util/Vector contains the array of the java/lang/Object. The class ClassVariableExample has a reference to the java/lang/Vector. With this information, we can tell the java/util/Vector came from the class ClassVariableExample.
TotalSize (TotalSize/HeapSize%) [ObjectSize] NumberOfChildObject(757) ObjectName Address
1,070,471,976 (99%) [312] 11 class com/ibm/jvm/ExtendedSystem 0x2d8d68
1,070,449,016 (99%) [312] 6 class java/lang/ClassLoader 0x2d29a8
1,070,446,664 (99%) [128] 15 sun/misc/Launcher$AppClassLoader 0x101eb7b0
1,070,443,112 (99%) [32] 1 java/util/Vector 0x102537b8
1,070,443,080 (99%) [56] 4 array of java/lang/Object 0x10253780
1,070,442,712 (99%) [312] 1 class ClassVariableExample 0x101defb0
1,070,442,400 (99%) [32] 1 java/util/Vector 0x10270d10
1,070,442,368 (99%) [10,256] 2,007 array of java/lang/Object 0x101f8ed0
800,016 (0%) [800,016] 0 long[] 0x11cefae8
800,016 (0%) [800,016] 0 long[] 0x2c831f68
...
Let's expand an array of java/lang/Object to find what are referenced from the array of java/lang/Object (see Figure 13).
We can see there are arrays of long[]. There are 2,007 entries in the array. The array of java/lang/Object is referenced from java/util/Vector because java/util/Vector is implemented with an array of java/lang/Object. java/util/Vector is referenced from the class ClassVariableExample. We found a leak suspect and class where the suspect is located.
We've accomplished our mission but it wouldn't hurt to explore the Java heap dump further. Let's find out how many long[] we have in this Java heap dump by selecting "List same type" menu from the context popup menu (see Figure 14). There are 671 of long[] in the Java heap dump and there are 2,007 objects referenced from an array of java/lang/Object. So there are other types of objects referenced from the array of java/lang/Object (see Figure 15). Let's display all the objects referenced from the array of java/lang/Object by selecting the "List children" menu from the context popup menu (see Figure 16).
We can see lots of long[] in Figure 17. If we scroll down a little, we can find float[] and int[] in Figure 18. How many of float[], int[] and long[] do we have in the Java heap dump? We can find it out by selecting "Types List" menu (see Figure 19).
Types List View provides an aggregated list of all classes and arrays sorted by size. We can see that long[], int[], and float[] are responsible for most of the Java heap space in the Java heap dump (see Table 3). We've shown that we can find memory leaks in static variables (see Figure 20). Can we also find memory leaks from instance variables and which object is responsible for it?
Simulation of Memory Leak in a Java Instance Variable
We can write an application to simulate a memory leak in an instance variable (see Listing 2e). After running InstanceVariableTest we can bring up the Java heap dump:
TotalSize (TotalSize/HeapSize%) [ObjectSize] NumberOfChildObject(502) ObjectName Address
51,202,240 (98%) [16] 1 InstanceVariableExample 0x14e2278
51,202,224 (98%) [32] 1 java/util/Vector 0x14e2288
51,202,192 (98%) [656] 96 array of [Ljava/lang/Object; 0x3db47a8
800,016 (1%) [800,016] 0 long[] 0x2300568
800,016 (1%) [800,016] 0 long[] 0x354ffa8
...
An array of the object is a leak suspect. It's referenced from java/util/Vector, which is referenced from an object, InstanceVariableExample. We can definitely find out which object is responsible for instance variables from the Java heap dump (see Figure 21).
Simulation of a Memory Leak in a Java Local Variable
We can chase down static variable and instance variable. What about local variables? Let's build an application to simulate memory leak in local variable (see Listing 3). We can run LocalVariableTest and analyze the Java heap dump.
TotalSize (TotalSize/HeapSize%) [ObjectSize] NumberOfChildObject(503) ObjectName Address
51,202,224 (98%) [32] 1 java/util/Vector 0x14b2208
51,202,192 (98%) [656] 96 array of [Ljava/lang/Object; 0x3d84728
800,016 (1%) [800,016] 0 long[] 0x22d04e8
...
We found a memory leak at the array of the Object, which is referenced from java/util/Vector. But there's no information about who created the java/util/Vector. Unfortunately we can't figure out who created the local variable from the Java heap dump alone. Do you remember HeapExhaustionSimulator that we used to simulate Java heap exhaustion? We couldn't find out who created the vector in the Java heap dump either since the vector was a local variable (see Figure 22).
What do we do if we want to find out the method that created the local variable? In the following message, we can get some clues, method name, line numbers, and file names.
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at LocalVariableExample.init(LocalVariableExample.java:16)
at LocalVariableExample.<init>(LocalVariableExample.java:8)
at LocalVariableTest.main(LocalVariableTest.java:5)
If you have a javacore, we can take a look at the Java stack trace of the failing thread (see Figure 23).
We can find the method name (LocalVariableExample.init), line number, and file name (LocalVariableExample.java:16) from the stack trace (see Figure 24).
Simulation of Memory Leak in a Java Thread
Let's simulate one more scenario where excessive threads are created (see Listing 4). In this experiment, we are creating millions of threads.
IBM's Java virtual machine throws the following message:
JVMDG274: Dump Handler has Processed OutOfMemory.
Exception in thread "main" java.lang.OutOfMemoryError
at NativeMemoryExhaustionSimulator.main (NativeMemoryExhaustionSimulator.java:5)
There's not enough information in the message. Sun's JVM throws the following message:
java.lang.OutOfMemoryError: unable to create new native thread
Dumping heap to java_pid6790.hprof ...
Heap dump file created [45762946 bytes in 1.521 secs]
It indicates that it was unable to create a new native thread. Probably native memory exhaustion occurred due to the excessive number of thread creation. We can take a look at the line 5 in NativeMemoryExhaustionSimulator.java to see if we are creating too many threads.
Another scenario occurs when the application might make excessive use of Java finalizers, which is explained in one of my JDJ articles. [Anatomy of Java Finalizer, Volume 14 Issue 1, Java Developer's Journal 2009]
Analysis Java Heap Leak Patterns
After years of research, I have identified that there are three different basic Java heap leak patterns that are modeled inside of the IBM HeapAnalyzer so that it can detect each pattern. The IBM HeapAnalyzer can detect these different types of memory leaks and find leak suspects for you (see Figures 25 and 26).

1. Horizontal memory leak: Many objects are referenced from same objects
2. Vertical memory leak: Many objects are linked together
3. Diagonal memory leak: Combination of horizontal and vertical memory leak. It's often found in tree structures

Simulation of Horizontal Memory Leak
Believe it or not, we already have seen a horizontal memory leak in HeapExhaustionSimulator (see Listing 5). Different objects are added to an ArrayList. So many objects are referenced from the same object, ArrayList. We can clearly visualize the structure from the IBM HeapAnalyzer, which was able to detect a horizontal memory leak pattern (see Figure 27).
Simulation of a Vertical Memory Leak
We can utilize a linked list to simulate a vertical memory leak. Here's what I've come up with (see Listing 6). We are adding strings to a linked list until we get java.lang.OutOfMemoryError. The IBM HeapAnalyzer was able to detect a vertical memory leak pattern and provided a leak suspect (see Figure 28).
Simulation of a Diagonal Memory Leak
We can build up finalizable objects with the following application, DiagonalMemoryLeakSimulator (see Listing 7). From the IBM HeapAnalyzer, we can recognize a diagonal memory leak pattern. We are creating objects until java.lang.OutOfMemoryError is thrown. This is a typical tree structure thanks to java.lang.ref.Finalizer. Of course, the IBM HeapAnalzyer was able to recognize adiagonal memory leak pattern and detect a leak suspect (see Figure 29).
Conclusion
We've investigated different patterns of java.lang.OutOfMemoryError, artifacts and their possible solutions. When we see a java.lang.OutOfMemoryError and find Java heap dumps; however, it's not always necessary to analyze Java heap dumps. We often find many people blindly jumping on Java heap dumps without looking into other information. I don't think we should blame them. We just need to work smarter.
We can't diagnose all kinds of problems by analyzing the Java heap dumps alone. Java heap dumps are just snapshots of Java heap at specific times. Garbage collection trace is another source of information where we can figure out what's going on with the Java heap and garbage collector.
First, we need to determine whether java.lang.OutOfMemoryError is caused by Java heap, native memory or something else from error messages. If it's caused by native memory, we need to utilize operating system memory monitoring tools provided by an operating system such as svmon on AIX operating systems or your favorite third-party tools. If java.lang.OutOfMemoryError is related to Java heap, we need to make sure that the Java heap and parts of the Java heap are configured well enough to handle a given workload by analyzing garbage collection trace. If your JVM needs a little bit more Java heap and you have extra memory, you could endow more memory to the Java Virtual Machine in order to increase heap capacity, for example, by increasing the -Xmx command-line option. If you've already reached the address space limit, you could redistribute the workload with multiple horizontal clones of the Java Virtual Machine. If you really suspect there's Java heap leak, you are more than welcome to analyze the Java heap dumps with the IBM HeapAnalyzer or any of your favorite Java heap analysis tools.

© 2008 SYS-CON Media Inc.

Java Thread Dumps

Software maintenance is a tedious and challenging job. As long as the software functions as expected, it looks great. Imagine the situation, your pager keeps buzzing in the midnight (Not a happy feeling, right?).
Any software system, no matter how well it has been built and quality tested might get into run-time performance issues. The reasons can be within the functional boundary or can be from the external environment. Software systems are built on certain assumptions and pre-conceived notions. However, when they go live, the assumptions may turn out to be false causing the system to malfunction.
In enterprise J2EE systems which generally have very large user base and involve interactions between heterogeneous systems, one of the general runtime issues reported is the system slowdown or system "hang". In such situations, the general trouble shooting pattern would be to analyze the Java thread dumps to isolate the threads which are causing the slow-down or hang. This article discusses the Java stack traces; anatomy of Java Threads and how to read thread dumps in general.
Exceptions and Stack Traces
All of us encounter and have encountered exceptions during our learning/development phases. Exception is the way Java reports a runtime error. Exceptions have two parts. Message and the stack trace. Message tells you what has gone wrong. Stack Traces provide a complete call flow through all the classes involved top-down as part of the runtime error.
The following is an example stack trace for an ArrayIndexOutOfBoundsException
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 4
at Test.run(Test.java:13)
at Test.<init>(Test.java:5)
at Test.main(Test.java:20)
In the above exception, the first line "Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 4" tells you that the JVM encountered the reported exception when it tried to access the value at array index 4. The java thread which encountered the exception is "main" thread.
Next let's read through the stack trace. The rule of thumb for exceptions is to read from the top line (message line) to understand what the exception is and keep going down to understand the call flow. In the above example, the call flow started at Test.java line number 20 (main() method) and then it called the constructor of the Test class. Constructors are represented with <init> in the stack traces. Then it moved on to the run() method in the Test class and encountered the reported exception at line number 13.
From the above stack trace, we can conclude that in Test.java, an attempt is made to read past the size of the Array.
The Java Thread Dump
Java Thread dump can be defined as a snapshot of all threads running inside a Java Virtual Machine (JVM) at a given point of time. A thread dump might contain a single thread or multiple threads. In multi threaded environments like J2EE application servers, there will be many threads and Thread Groups. Each of these threads will have its own call stack and will be performing a separate function at a given time. Thread Dump will provide the stack traces of all the JVM threads and much more information about a particular thread.
The Java VM Process and Java Threads
The Java Virtual Machine or JVM is an operating system level process. Java Threads are child processes or Light-Weight Processes (in Solaris terms) of the JVM process.
Generating Java Thread Dumps
Thread Dump is generated by sending a SIGQUIT signal to the JVM process. There are different ways of sending this signal to the process.
In Unix, use "kill -3 <pid>" where pid is the Process ID of the JVM.
In Windows, press CTRL+BREAK on the window where the JVM is running
Java Thread State
Each Java Thread can be in one of the four states during its lifetime.
Runnable - The thread is either running or ready to run when it gets its CPU turn. JRockit thread dumps refer to this state as ACTIVE.
Waiting on Monitor - The thread is either sleeping or waiting on an object for said period of time or waiting to be notified by another thread. This will happen when any of the sleep() methods on Thread object or wait() method on an Object is called.
For example, in WebLogc server the idle execute threads are in this condition and they wait till a socket reader thread notify them of some new work to be done. The stack trace will look like this:
"ExecuteThread: '2' for queue: 'weblogic.admin.RMI'" daemon prio=5 tid=0x1752F040 nid=0x180c in Object.wait() [1887f000..1887fd8c]
at java.lang.Object.wait(Native Method) waiting on <04134D98> (a weblogic.kernel.ExecuteThread)
at java.lang.Object.wait(Object.java:426)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:126)
locked <04134D98> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:145)
Some other versions of the JVM call this state as CW, Object.wait() (as above). JRockit refer to this as state as WAITING.
Waiting for Monitor Entry - The thread is waiting to get the lock for an object (some other thread may be holding the lock). This happens if two or more threads try to execute synchronized code. Note that the lock is always for an object and not for individual methods.
Sample stack trace of a thread in this condition:
"ExecuteThread: '24' for queue: 'DisplayExecuteQueue'" daemon prio=5 tid=0x5541b0 nid=0x3b waiting for monitor entry [49b7f000..49b7fc24]
at weblogic.cluster.replication.ReplicationManager.createSecondary (ReplicationManager.java:908)
- waiting to lock <6c4b9130> (a java.lang.Object)
at weblogic.cluster.replication.ReplicationManager.updateSecondary (ReplicationManager.java:715)
at weblogic.servlet.internal.session.ReplicatedSessionData.syncSession (ReplicatedSessonData.java:459)
- locked <6c408700> (a weblogic.servlet.internal.session.ReplicatedSessionData)
at weblogic.servlet.internal.session.ReplicatedSessionContext.sync (ReplicatedSessionContext.java:134)
- locked <6c408700> (aweblogic.servlet.internal.session.ReplicatedSessionData)
at weblogic.servlet.internal.ServletRequestImpl.syncSession (ServletRequestImpl.java:2418)
at weblogic.servlet.internal.WebAppServletContext.invokeServlet (WebAppServletContext.java:3137)
at weblogic.servlet.internal.ServletRequestImpl.execute (ServletRequestImpl.java:2544)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:153)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:134)
In the above stacktrace, you can see that this thread holds the lock an object (6c408700) and waiting to lock another object (6c4b9130)
Some other JVMs may not give the object Ids with lock information in the stack trace. The same state may also be called as ‘MW'. JRockit refers to this state as LOCKED.
Anatomy of a Java Thread
To be able to read/analyze thread dump, it is important to understand various parts of a Thread dump. Let's take a simple Thread Stack example and read various parts of it.
"ExecuteThread: '1' " daemon prio=5 tid=0x628330 nid=0xf runnable [0xe4881000..0xe48819e0]
at com.vantive.vanjavi.VanJavi.VanCreateForm(Native Method)

CIO, CTO & Developer Resources

&lt;img src="http://ad.doubleclick.net/imp;v1;f;243338996;0-0;0;66803389;1|1;42967129|42984916|1;;cs=o;%3fhttp://ad.doubleclick.net/dot.gif?[timestamp] "&gt;

at com.vantive.vanjavi.VanMain.open(VanMain.java:53)
at jsp_servlet._so.__newServiceOrder.printSOSection( __newServiceOrder.java:3547)
at jsp_servlet._so.__newServiceOrder._jspService (__newServiceOrder.java:5652)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:27)
at weblogic.servlet.internal.ServletStubImpl.invokeServlet (ServletStubImpl.java:265)
at weblogic.servlet.internal.ServletStubImpl.invokeServlet (ServletStubImpl.java:200)
at weblogic.servlet.internal.WebAppServletContext.invokeServlet (WebAppServletContext.java:2495)
at weblogic.servlet.internal.ServletRequestImpl.execute (ServletRequestImpl.java:2204)
at weblogic.kernel.ExecuteThread.execute (ExecuteThread.java:139)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)
In the above Thread Dump, the interesting part to is the first line. The rest of the stuff is nothing more than a general stack trace. Lets analyze the first line here
Execute Thread : 1 This indicates the name of the thread.
daemon Indicates that this Thread is a daemon thread
prio=5 The priority of this thread (default is 5)
tid The Java Thread Id (Unique identifier for this thread in the running JVM instance).
nid Native Identifier of the thread. Indicates LWP id in Solaris. Identifier for this process
at the OS level.
runnable Indicate the Thread state (See above)
[x..y] Indicates the range of addresses in the heap where this thread is executing.
The remaining part of the thread dump show the call flow. In this example, the thread (Execute Thread 1) is a OS daemon which is currently executing a native method vanCreateForm().
Putting Thread Dumps to Use
In this section I describe few use cases in which Thread dumps are very useful.
High CPU consumption
Diagnosis
The application seems to consume almost 100% CPU and the throughput has come down drastically. Starts to perform very poorly with high load on the CPU.
Thread Dump
All threads in the dump generally show one or more threads struck doing the same operation through all the thread dumps.
Solution

• Take a series of thread dumps (around 5-7) for one particular call flow (a web form submit for example) before the call is completed.
• Looking for "runnable" threads in the Thread Dump. If each such thread seems to move (the method call in each thread dump is different), then the thread is progressing and probably not a culprit. If the thread is executing the same method (same line number) through all the thread dumps, then it is almost for sure the culprit. Go to the code and do the code level analysis. You are almost certain to catch an issue there.

Low CPU consumption and Poor Response time
Diagnosis
This happens in a high I/O bound systems under high load. The CPU consumption is low with only few threads consuming reasonable CPU chunk. However the application response time will be very slow.
Thread Dump
Some or all runnable threads seem to struck performing an I/O operation like a File read/write or a database operation.
Solution
Understand the I/O operation profile of your application. Use caches where applicable to reduce DB interactions.
Application/Server Hang
Diagnosis
An application or an server JVM hosting the application will hang (become unresponsive).
Thread Dump

• All runnable threads are struck doing the same operation through all the thread dumps taken. The server runs out of any more threads as all the runnable threads "never" complete their operation.
• There might be many threads waiting for monitor entry. This can happen if a "runnable" thread is holding a lock on an object and never returns it while others are waiting for the same lock.

Solution

• Check for dead lock. JVM generally detects a deadlock if it is simple enough (Thread A waiting for Thread B and vice versa). However, you need to understand the lock profile at the given moment to see if there is any complex deadlock situation involved.
• Revisit the synchronized methods/block in the code. Reduce the size of synchronization area to the extent possible.
• One issue could be too long a timeout while accessing a remote resource/component. Implement a reasonable timeout on the remote object clients so that the throw appropriate exception when a remote system does not respond in a reasonable amount of time.
• If all the threads are waiting for a resource (like an EJB/DB connection), consider increasing the object pool size for those resources.

Tools
There are both commercial and open source tools available for thread dump analysis. One such tool is Samurai. It is a light weight open source tool which runs as a Java Web Start application as well as from your command prompt. For more information and documentation on Samurai, visit
http://yusuke.homeip.net/samurai/en/index.html
Conclusion
Maintaining J2EE enterprise application in a production environment is a tough job. As the business dynamics change, J2EE application environments change which might cause runtime instability in the production application. One of the primary factors that effect an running J2EE application is high load. While most systems are designed to be scalable, environmental limitations might cause them to become non-responsive.
Java Thread Dumps is an excellent mechanism to identify, diagnose, detect and resolve typical production issues. While application profiling and other mechanisms do exist, analyzing Java Thread dumps will provide us an clear and early understanding of popular production level issues there by saving time and helping us provide better user experience with the production applications.

Sample Thread Dums:

Full thread dump Java HotSpot(TM) Client VM (1.5.0_10-b03 mixed mode):

"Thread-7" prio=4 tid=0x0b482220 nid=0x1570 in Object.wait() [0x0bbcf000..0x0bbcfae8]
 at java.lang.Object.wait(Native Method)
 - waiting on <0x03017960> (a concurrency.diners.Fork)
 at java.lang.Object.wait(Object.java:474)
 at concurrency.diners.Fork.get(Fork.java:22)
 - locked <0x03017960> (a concurrency.diners.Fork)
 at concurrency.diners.Philosopher.run(Philosopher.java:29)

"Thread-6" prio=4 tid=0x0b481808 nid=0xa84 in Object.wait() [0x0bb8f000..0x0bb8fb68]
 at java.lang.Object.wait(Native Method)
 - waiting on <0x030707e0> (a concurrency.diners.Fork)
 at java.lang.Object.wait(Object.java:474)
 at concurrency.diners.Fork.get(Fork.java:22)
 - locked <0x030707e0> (a concurrency.diners.Fork)
 at concurrency.diners.Philosopher.run(Philosopher.java:29)

"Thread-5" prio=4 tid=0x0b47e310 nid=0x167c in Object.wait() [0x0bb4f000..0x0bb4fbe8]
 at java.lang.Object.wait(Native Method)
 - waiting on <0x03070850> (a concurrency.diners.Fork)
 at java.lang.Object.wait(Object.java:474)
 at concurrency.diners.Fork.get(Fork.java:22)
 - locked <0x03070850> (a concurrency.diners.Fork)
 at concurrency.diners.Philosopher.run(Philosopher.java:29)

"Thread-4" prio=4 tid=0x0b47d808 nid=0x1730 in Object.wait() [0x0bb0f000..0x0bb0fc68]
 at java.lang.Object.wait(Native Method)
 - waiting on <0x030708c0> (a concurrency.diners.Fork)
 at java.lang.Object.wait(Object.java:474)
 at concurrency.diners.Fork.get(Fork.java:22)
 - locked <0x030708c0> (a concurrency.diners.Fork)
 at concurrency.diners.Philosopher.run(Philosopher.java:29)

"Thread-3" prio=4 tid=0x0b480cd8 nid=0x11c4 in Object.wait() [0x0bacf000..0x0bacfce8]
 at java.lang.Object.wait(Native Method)
 - waiting on <0x03017b38> (a concurrency.diners.Fork)
 at java.lang.Object.wait(Object.java:474)
 at concurrency.diners.Fork.get(Fork.java:22)
 - locked <0x03017b38> (a concurrency.diners.Fork)
 at concurrency.diners.Philosopher.run(Philosopher.java:29)

"AWT-EventQueue-1" prio=4 tid=0x0b46e1d0 nid=0x16c8 in Object.wait() [0x0ba4f000..0x0ba4fa68]
 at java.lang.Object.wait(Native Method)
 - waiting on <0x02ffa368> (a java.awt.EventQueue)
 at java.lang.Object.wait(Object.java:474)
 at java.awt.EventQueue.getNextEvent(EventQueue.java:345)
 - locked <0x02ffa368> (a java.awt.EventQueue)
 at java.awt.EventDispatchThread.pumpOneEventForHierarchy(EventDispatchThread.java:189)
 at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:163)
 at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:157)
 at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:149)
 at java.awt.EventDispatchThread.run(EventDispatchThread.java:110)

"DestroyJavaVM" prio=6 tid=0x00266dc0 nid=0x1118 waiting on condition [0x00000000..0x0006fae8]

"AWT-EventQueue-0" prio=6 tid=0x0b451f60 nid=0x124c in Object.wait() [0x0b82f000..0x0b82fbe8]
 at java.lang.Object.wait(Native Method)
 - waiting on <0x0300a858> (a java.awt.EventQueue)
 at java.lang.Object.wait(Object.java:474)
 at java.awt.EventQueue.getNextEvent(EventQueue.java:345)
 - locked <0x0300a858> (a java.awt.EventQueue)
 at java.awt.EventDispatchThread.pumpOneEventForHierarchy(EventDispatchThread.java:189)
 at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:163)
 at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:157)
 at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:149)
 at java.awt.EventDispatchThread.run(EventDispatchThread.java:110)

"thread applet-concurrency/diners/Diners.class" prio=4 tid=0x0b3cab40 nid=0x10a0 in Object.wait() [0x0b7ef000..0x0b7efb68]
 at java.lang.Object.wait(Native Method)
 - waiting on <0x0300a900> (a sun.applet.AppletViewerPanel)
 at java.lang.Object.wait(Object.java:474)
 at sun.applet.AppletPanel.getNextEvent(AppletPanel.java:282)
 - locked <0x0300a900> (a sun.applet.AppletViewerPanel)
 at sun.applet.AppletPanel.run(AppletPanel.java:332)
 at java.lang.Thread.run(Thread.java:595)

"AWT-Windows" daemon prio=6 tid=0x0ac90b38 nid=0x1124 runnable [0x0af0f000..0x0af0fce8]
 at sun.awt.windows.WToolkit.eventLoop(Native Method)
 at sun.awt.windows.WToolkit.run(WToolkit.java:269)
 at java.lang.Thread.run(Thread.java:595)

"AWT-Shutdown" prio=6 tid=0x0ac90780 nid=0x7dc in Object.wait() [0x0aecf000..0x0aecfd68]
 at java.lang.Object.wait(Native Method)
 - waiting on <0x02fb5830> (a java.lang.Object)
 at java.lang.Object.wait(Object.java:474)
 at sun.awt.AWTAutoShutdown.run(AWTAutoShutdown.java:259)
 - locked <0x02fb5830> (a java.lang.Object)
 at java.lang.Thread.run(Thread.java:595)

"Java2D Disposer" daemon prio=10 tid=0x0ac82aa8 nid=0x1014 in Object.wait() [0x0ae8f000..0x0ae8f9e8]
 at java.lang.Object.wait(Native Method)
 - waiting on <0x02fdd7a8> (a java.lang.ref.ReferenceQueue$Lock)
 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
 - locked <0x02fdd7a8> (a java.lang.ref.ReferenceQueue$Lock)
 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
 at sun.java2d.Disposer.run(Disposer.java:107)
 at java.lang.Thread.run(Thread.java:595)

"Low Memory Detector" daemon prio=6 tid=0x00a94e70 nid=0x1038 runnable [0x00000000..0x00000000]

"CompilerThread0" daemon prio=10 tid=0x00a93a70 nid=0x12c8 waiting on condition [0x00000000..0x0abcf8c8]

"Signal Dispatcher" daemon prio=10 tid=0x00a92e28 nid=0x16bc waiting on condition [0x00000000..0x00000000]

"Finalizer" daemon prio=8 tid=0x00a89cd0 nid=0x1044 in Object.wait() [0x0ab4f000..0x0ab4fc68]
 at java.lang.Object.wait(Native Method)
 - waiting on <0x02fdd950> (a java.lang.ref.ReferenceQueue$Lock)
 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
 - locked <0x02fdd950> (a java.lang.ref.ReferenceQueue$Lock)
 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
 at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x00a88860 nid=0x106c in Object.wait() [0x0ab0f000..0x0ab0fce8]
 at java.lang.Object.wait(Native Method)
 - waiting on <0x02fdd700> (a java.lang.ref.Reference$Lock)
 at java.lang.Object.wait(Object.java:474)
 at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
 - locked <0x02fdd700> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=10 tid=0x00a85d98 nid=0x1030 runnable 

"VM Periodic Task Thread" prio=10 tid=0x00a960c8 nid=0x14b4 waiting on condition 

Memory Management and OutofMemoryException Handling

n this article, I have tried to analyze the various causes which may lead to Memory Exception.
Once an ‘OutOfMemoryException’ is thrown, how best can it be handled has been discussed in this artilce.
OutOfMemoryException is thrown when there is not sufficient available memory to carry out a requested activity. In this article, I have tried to illustrate the different methods by which memory leaks can be handled in Web Applications.

Memory Management

When a program is loaded into memory, it is organized into three areas of memory, called segments: the text segment, stack segment, and heap segment. The text segment (sometimes also called the code segment) is where the compiled code of the program itself resides. This is the machine language representation of the program steps to be carried out, including all functions making up the program, both user defined and system.
The remaining two areas of system memory are where storage may be allocated by the compiler for data storage. The stack is where memory is allocated for automatic variables within functions. A stack is a Last In First Out (LIFO) storage device where new storage is allocated and deallocated at only one “end”, called the top of the stack.
When a program begins executing in the function main(), space is allocated on the stack for all variables declared within main(). If main() calls a function, func(), additional storage is allocated for the variables in func() at the top of the stack. It should be clear that memory allocated in this area will contain garbage values left over from previous usage.
The heap segment provides more stable storage of data for a program; memory allocated in the heap remains in existence for the duration of a program. Therefore, global variables (storage class external), and static variables are allocated on the heap. The memory allocated in the heap area, if initialized to zero at program start, remains zero until the program makes use of it. Thus, the heap area need not contain garbage.

OutofMemoryException Handling

Once an OutOfMemoryException has been thrown, the following checks should be carried out:
Handling Connection Objects: Check whether all objects pertaining to Connection, ResultSet, Statement and PreparedStatement have been properly closed in the finally block. Since the Connection objects are drawn from a Connection pool (the pool size depends on Server Configuration), if multiple Connection objects are created without closing them, it would result in reduction of the connection pool size and may throw an OutOfMemoryException if the pool is exhausted. Hence, even though there might be unused connection objects, the pool may be exhausted resulting in exception being thrown.
Handling OutputStream Objects: Check whether all OutputStream objects have been properly closed in the finally block. Streams are usually resource intensive objects and thus should be handled in a point-to-point manner. Hence it is imperative to close the streams individually to prevent memory leaks.
Checking for Static Variables: Static variables are stored in the heap and only one instance of variable is available throughout the lifetime of the application. Since static variables are not garbage collected till the class is unloaded or the variables are explicitly set to NULL, it needs to be checked whether all static variables are being used or some are unnecessarily occupying heap space.
Checking Third Party APIs: It is necessary to check the memory utilization of third party APIs. If the memory utilization is high (for example, if many static variables have been used), it may lead to OutOfMemoryException.
Usage of Singleton Classes: Adoption of the Singleton pattern results in creation of classes for which memory is allocated on the heap and not freed anywhere which may result in memory leaks. Hence, it is necessary to check the design pattern being followed, and in case of singleton pattern, check the number of classes which may be holding up space in the heap. Singleton pattern should not be used unless imperative.
Size of Session Objects: Since, it is possible to put large objects in a session, it is necessary to check the session size. Hence, it is important to check during memory leaks, the size of the session as well as the objects that might have been put in the session. To avoid memory leaks, only those objects should be put in the session which need to be put and should be removed when not required anymore.
If the above suggested checks do not throw a cause for memory leaks, the below mentioned methods may be carried out to determine the cause for memory leaks:
Using verbose:gc: The verbose:gc utility can be configured in startWebLogic.cmd to obtain information about the Java Object Heap in real time while running the Java application.
To activate the utility, Java must be run with -verbose:gc option.
For using verbose:gc; set the JVM size to 64 MB so that we may reach the maximum without load on the server. If the used memory does not drop to initial levels after full GC, then it would indicate a memory leak.

linked list in java and Operations on a singly-linked list

Singly-linked list

Linked list is a very important dynamic data structure. Basically, there are two types of linked list, singly-linked list and doubly-linked list. In a singly-linked list every element contains some data and a link to the next element, which allows keeping the structure. On the other hand, every node in a doubly-linked list also contains a link to the previous node. Linked list can be an underlying data structure to implement stack, queue or sorted list.

Example

Sketchy, singly-linked list can be shown like this:

Each cell is called a node of a singly-linked list. First node is called head and it's a dedicated node. By knowing it, we can access every other node in the list. Sometimes, last node, called tail, is also stored in order to speed up add operation.

Operations on a singly-linked list

Concrete implementation of operations on the singly-linked list depends on the purpose, it is used for. Following the links below, you can find descriptions of the common concepts, proper for every implementation.

LinkedList :

http://www.technicalypto.com/2010/01/java-program-to-reverse-singly-linked.html

class SinglyLinkedListNode {

      public int value;

      public SinglyLinkedListNode next;

      public SinglyLinkedListNode(int value) {

            this.value = value;

            next = null;

      }

}

public class SinglyLinkedList {

      private SinglyLinkedListNode head;

      private SinglyLinkedListNode tail;

      public SinglyLinkedList() {

            head = null;

            tail = null;

      }

      public void addLast(SinglyLinkedListNode newNode) {

            if (newNode == null)

                  return;

            else {

                  newNode.next = null;

                  if (head == null) {

                        head = newNode;

                        tail = newNode;

                  } else {

                        tail.next = newNode;

                        tail = newNode;

                  }

            }

      }

      public void addFirst(SinglyLinkedListNode newNode) {

            if (newNode == null)

                  return;

            else {

                  if (head == null) {

                        newNode.next = null;

                        head = newNode;

                        tail = newNode;

                  } else {

                        newNode.next = head;

                        head = newNode;

                  }

            }

      }

      public void insertAfter(SinglyLinkedListNode previous,

                  SinglyLinkedListNode newNode) {

            if (newNode == null)

                  return;

            else {

                  if (previous == null)

                        addFirst(newNode);

                  else if (previous == tail)

                        addLast(newNode);

                  else {

                        SinglyLinkedListNode next = previous.next;

                        previous.next = newNode;

                        newNode.next = next;

                  }

            }

      }

      public void removeFirst() {

            if (head == null)

                  return;

            else {

                  if (head == tail) {

                        head = null;

                        tail = null;

                  } else {

                        head = head.next;

                  }

            }

      }

      public void removeLast() {

            if (tail == null)

                  return;

            else {

                  if (head == tail) {

                        head = null;

                        tail = null;

                  } else {

                        SinglyLinkedListNode previousToTail = head;

                        while (previousToTail.next != tail)

                              previousToTail = previousToTail.next;

                        tail = previousToTail;

                        tail.next = null;

                  }

            }

      }

     

public void removeNext(SinglyLinkedListNode previous) {

            if (previous == null)

                  removeFirst();

            else if (previous.next == tail) {

                  tail = previous;

                  tail.next = null;

            } else if (previous == tail)

                  return;

            else {

previous.next = previous.next.next;

            }

      }

      public int traverse() {

        int sum = 0;

        SinglyLinkedListNode current = head;

        SinglyLinkedListNode previous = null;

        while (current != null) {

              sum += current.value;

              previous = current;

              current = current.next;

        }

        return sum;

  }

}

Java Program to reverse a Singly Linked List

I have written an efficient way to reverse a Singly Linked List using three pointers/nodes for traversal/temporary purposes. I have given a pictorial representation of what happens with near, mid and far pointer. The program is mostly self explanatory. Remember, we are using our implementation of Singly Linked List. You may visit this link if you need that class.

Note : You may click on the above image to get a clearer picture.

Now, for the source code.

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50

package dsa.linkedlist; /**  * Reverse a singly linked list  * @author Braga  */ public class ReverseAList {     public static void main(String args[]){   ReverseAList reverser = new ReverseAList();   SinglyLinkedList<Integer> originalList = reverser.getLabRatList(10);   System.out.println("Original List : "+originalList.toString());   reverser.reverseList(originalList);   System.out.println("Reversed List : "+originalList.toString());  }     public void reverseList(SinglyLinkedList<Integer> sourceList){   if(sourceList.size<=1){    return;   }   Node<Integer> nearNode = sourceList.start;   Node<Integer> midNode, farNode;       midNode = nearNode.next;   farNode = midNode.next;       nearNode.next = null;       while(farNode!=null){    midNode.next = nearNode;         nearNode = midNode;    midNode = farNode;    farNode = farNode.next;   }   midNode.next = nearNode;   sourceList.start = midNode;  }     private SinglyLinkedList<Integer> getLabRatList(int count){   SinglyLinkedList<Integer> sampleList = new SinglyLinkedList<Integer>();   for(int i=0;i<count;i++){    sampleList.add(i);   }   return sampleList;  } } //-----------------OUTPUT--------------------- //Original List : 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 //Reversed List : 9, 8, 7, 6, 5, 4, 3, 2, 1, 0

 

 

Loop in a Singly-Linked List

Question: How would you find a loop in a singly-linked list?
Answer: This is a very typical tech interview question that has a very elegant solution that runs in O(n) time and O(1) space. Also referred to as the “Tortoise and Hare Algorithm”,  the solution uses a fast pointer (traversing the list two items at a time) and a slow pointer (traversing the list one item at a time). If there is a loop, then the fast pointer will go around the loop twice as fast as the slow pointer. At one point, these two pointers will point to the same item. Here’s some code to make this clearer.

bool hasLoop( Node *startNode )

{

    Node *slowNode, *fastNode;

    slowNode = fastNode = startNode;

    while( slowNode && fastNode && fastNode->next )

    {

        if( slowNode == fastNode->next || slowNode == fastNode->next->next )

        {

            return true;

        }

        slowNode = slowNode->next;

        fastNode = fastNode->next->next;

    }

    return false;

}

Update: Thanks to RiderOfGiraffes for this solution.
The time complexity of this solution is still O(n), same as the solution provided above but this solution takes less number of steps at each point.

bool hasLoop( Node *startNode )

{

    Node *tortoise, *hare;

    int test_size = 2, steps_taken = 0;

    tortoise = hare = startNode;

    while( tortoise && hare->next )

    {

        hare = hare->next, steps_taken++;

        if( tortoise == hare )

        {

            return true;

        }

        if ( steps_taken>=test_size )

        {

            steps_taken = 0, test_size *= 2;

            tortoise = hare;

        }

    }

    return false;

}