This article will provide you with an equivalent approach, for JVM on Linux OS, of the powerful Solaris OS prstat command; allowing you to quickly pinpoint the high CPU Java VM Thread contributors.
One key concept to understand for a Java VM running on the Linux OS is that Java threads are implemented as native Threads, which results in each thread being a separate Linux process.
Ok thanks for the info but why is this related to prstat?
Well this key concept means that you don’t need a prstat command for Linux. Since each Java VM Thread is implemented as a native Thread, each Thread CPU % can simply be extracted out-of-the-box using the top command.
You still need to generate Thread Dump data of your JVM process in order to correlate with the Linux top command output.
Thanks for this explanation. Now please show me how to do this
Please simply follow the instructions below:
1) Execute the top command (press SHIFT-H to get the Threads toggle view) or use -H option (to show all Threads) and find the PID associated with your affected / high CPU WLS process(es) (remember, many may show up since each Java Thread is implemented as a separate Linux process)
2) Immediately after, generate a few Thread Dump snapshots using kill –3 <PID> of the parent WLS process. Thread Dump provides you with the complete list with associated Stack Trace of each Java Thread within your JVM process
3) Now, convert the PID(s) extracted from the top command output to HEX format
4) The next step is to search from the Thread Dump data for a match nid=<HEX PID>
5) The final step is to analyze the affected Thread(s) and analyze the Stack Trace so you can determine where in the code is the problem (application code, middleware itself, JDK etc.)
Example: top command captured of a Weblogic Server Java Thread running at 40% CPU utilization along with Thread Dump data via kill -3 <PID>
## top output sample
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
...........
22111 userWLS 9 0 86616 84M 26780 S 0.0 40.1 0:00 java
- Decimal to HEX conversion of Java Thread (native Thread) 22111 >> 0x565F
- Now using the HEX value, we can search within the Thread Dump for the following keyword: nid=0x565F
## Thread Dump output sample Thread as per the above search criteria nid=0x565F
"ExecuteThread: '0' for queue: 'default'" daemon prio=1 tid=0x83da550 nid=0x565F waiting on monitor [0x56138000..0x56138870]
at java.util.zip.ZipFile.getEntry(Native Method)
at java.util.zip.ZipFile.getEntry(ZipFile.java:172)
at java.util.jar.JarFile.getEntry(JarFile.java:269)
at java.util.jar.JarFile.getJarEntry(JarFile.java:252)
at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:989)
at sun.misc.URLClassPath$JarLoader.findResource(URLClassPath.java:967)
at sun.misc.URLClassPath.findResource(URLClassPath.java:262)
at java.net.URLClassLoader$4.run(URLClassLoader.java:763)
at java.security.AccessController.doPrivileged(AccessController.java:224)
at java.net.URLClassLoader.findResource(URLClassLoader.java:760)
at java.lang.ClassLoader.getResource(ClassLoader.java:444)
at java.lang.ClassLoader.getResourceAsStream(ClassLoader.java:504)
............................................
In the above example, the problem was related to an excessive class loading / IO problem.
As you can see, the approach did allow us to quickly pinpoint high CPU Thread contributor but you will need to spend additional time to analyze the root cause which is now your job.
Need any additional help?
I hope this short tutorial has helped you understand how you can pinpoint high CPU Thread contributors for your JVM running on the Linux OS.
For any question or additional help please simply post a comment or question below this article. You can also email me directly @[email protected].
