Compare jstat vs verbosegc vs visualvm vs HP J2EE Diagnostics !!

Below illustrates the comparison bet. verbose GC vs jstat (with gcutil option ) vs visualvm information vs jconsole information vs hp j2ee diagnostics

Before going further, the two GCs that are used in this example are ‘copy’ for young gen and ‘MarkSweepCompact’ for tenured/major gc

            VerboseGC:

            java -verbosegc

-XX:+PrintGCDetails

-XX:+PrintGCApplicationConcurrentTime

-XX:+PrintGCApplicationStoppedTime           

-XX:+PrintGCTimeStamps

Verbose GC log says tenured collections happened when ever the heap usage increased to 20Meg. Unfortunately the polling mechanism of visual vm and jconsole not able to capture the data points in that small interval. While jstat (polled at 1 sec interval) also says every sec there was/were FGC event/s happened and it also confirms that a total of 240 full gc events happened which coincides with the total number of Tenured collections indicated in the verbose gc log (jstat counts all tenured and full collections as FGC events)

The above fact is proven by uploading the raw data from jconsole to csv file. Which shows only 67 entries, that means it has not captured all 240 FGC events/major collections in verbose gc log. Infact it did not captured even 67 major collection points as those 67 data points are basically shows the heap usage points.. So in the jconsole graph below, all the major and Full GC events are not visible. Infact to be very strict, the graph is misrepresenting the major GC threshold (which is around 20 MB as per verbosegc logs & confirmed by jconsole VM Summary tab’s committed memory = 22MB, also in the heap usage graph legend). So, the graph most of the times representing the fullgc threshold as 14MB and showing in some instance(lucky) as around 20MB

Same applies to visualvm graph… we can not export visualvm graph’s raw data…

Click to see GC behaviour frm verbosegc log & exported JConsole Heap usage graph data

(in the above gc log, at timestamp 275.312: u see, its indicated as FULL GC (System) which means its a system trigged FGC.. Infact this was a manual forced GC done from visualvm. So FGC entry here indicates heap mem which is not occupied completely to the upper limit )

 JVisualVM:

 JConsole:

HP J2EE Diagnostics Probe:

Note: In the above comparison, the example program that was used has no memory leak. And the word FGC is actually refering to major GC or tenured GC. Infact, in the above test there was no Full GC happend though jstat convention is always FGC events even for major or real FGC.

And you can clearly identify when a real FGC happend by looking in to the verbose gc logs for the entry "Full GC".

Current JVM Settings..

To know JVM settings: open jconsole and attach to the pid running and then go to VM Summary page

Analysing Java Heap For Memory Leak Cont...

This blog is in continution to the previous one and illustrates the no mem leak cases..

Analysing Java Heap For Memory Leak

Analysing Heap dump - Two Examples: With memory leak, Without memory leak

· When to take a heap dump?

· Is it apt to take it when FullGC happens? (yes to observe more accurately.. pls refer to above stmnt on’confirm’ mem leak and below illustrated examples)

· How to say there is most probable leak in slow increasing heap applications ? (pls follow below examples..)

GC Type: 'Copy' for YGC and 'MarkSweepCompact' for tenured GC
In the Heap graph to a certain threshold, there were no FullGC events observed.. you can not notice them whether its FGC or just a tenured collection just by looking at the grapgh !.. Any way, by taking support from verbosegc, the non forced FGC happend after half-way through..

                                          (Please click on the images to view more closer)

                                                        And look at my next blog for no mem leak case...

A Java Memory Leak !

How come a memory leak can happen in JAVA as there a worker that do its job i.e. GC

GC, depends on the type selected (parallel gc,serialgc..refer to typesofGC.doc) does the job of cleaning the memory and moves objects in the spaces… However there will be some instances where unused reference objects that remains in memory till the main program exists as these programs are not called off in the main program..

Given the above conditions, not always an increasing heap usage suggests a memory leak ! Yes, GC won’t really perform a major/full GC unless it is required. And for some programs, even after exiting the load the GC won’t really perform the full GC as there will be still space for moving objects and new generations…

We may say memory leak is there when we see ‘OutofMemory’ exceptions in the logs OR some times the leak is very minimum so that it will not result in OOM exception so quickly. Also, be noticed OOM is not always a symptom of mem leak as it can happen when the basic enough memory is not there for the applications to start or what it has to be under rampup..

So, a best way to CONFIRM mem leak is by observing the Heap after two full GC activity as there will be a slight(depends) in increase in the base memory occupied in the Heap after the full GC. But this need to done under after reaching steady state.. Also, one has to note that this Major or Full GC (clearing old/tenured space), depending on the full gc algorithm, the full GC size can vary i.e. some times it waits to a specific threshold in Old space and performs a full gc and some other times it waits till more old gen space gets filled to perform a full gc.. However, if there is a (heap) memory leak, then what ever varying full gc happening over a long run you should see increasing base heap memory .. and may eventually turns up into OOM after 1 hour/1day/10days..!!

And there are some special category objects called roots which are special objects which GC won’t collect. The roots will into the below mentioned categories..

· Class - class loaded by system class loader. Such classes can never be unloaded. They can hold objects via static fields. Please note that classes loaded by custom class loaders are not roots, unless corresponding instances of java.lang.Class happen to be roots of other kind(s).

· Thread - live thread

· Stack Local - local variable or parameter of Java method

· JNI Local - local variable or parameter of JNI method

· JNI Global - global JNI reference

· Monitor Used - objects used as a monitor for synchronization

· Held by JVM - objects held from garbage collection by JVM for its purposes. Actually the list of such objects depends on JVM implementation. Possible known cases are: the system class loader, a few important exception classes which the JVM knows about, a few pre-allocated objects for exception handling, and custom class loaders when they are in the process of loading classes. Unfortunately, JVM provides absolutely no additional detail for such objects. Thus it is up to the analyst to decide to which case a certain "Held by JVM" belongs.

Understanding JVM Heap Spaces

JVM Heap memory is divided in to two main categories: young generation space and old generation space. As the names imply, young generation space is meant for recently created objects and old generation space stores surviving objects that have lived to some extent.

Young generation space is itself divided into three categories:

· Eden space and

· Two survivor spaces.

The JVM initially allocates objects in the Eden space, where most objects die and quickly are reclaimed. When Eden space fills up, it causes the JVM to perform a minor collection, when it moves some surviving objects to the old generation. (Note: Any new objects created inside the method go into Eden space and the objects space is reclaimed once the method exists. Class-level object variables hang around for the entire life of the objects.)

The two survivor spaces are for copying live objects, allowing young objects to remain in the young generation space longer. One survivor space is empty at any given time. It serves as the destination of the next copying collection of any living objects in the Eden space and the other survivor space

Old/Tenured generation space:

If objects get too old, or young generation space gets filled up, JVM promotes objects to the old generation space. When the old generation space gets filled up, the JVM performs a major collection (even FGC some times) to remove the unused objects and reclaim their space. A major GC collection takes a significant amount of time and can affect system performance. Therefore, developers must make sure that major collections do not happen too often in their applications.  Depending on the GC algorithm, if there is no enough tenured space and young gen space movement has to happen the GC alg. attempts a Full GC which scans across the complete heap and perm generation also..(In a fast mem leaking case, after certain threshold, u should see a change in steady and constant frequency FGC to a very frequent FGC happening and ultimately resulting in OOM.. )

Third generation – Permanent generation – A non heap memory

There is a third generation too - Permanent Generation. The permanent generation is special because it holds meta-data describing user classes (classes that are not part of the Java language). Examples of such meta-data are objects describing
classes and methods and they are stored in the Permanent Generation. Applications with large code-base can quickly fill up this segment which will cause java.lang.OutOfMemoryError: PermGen no matter how high your -Xmx and how much memory you have on the machine.

Sun JVMs allow you to resize the different generations of the heap, including the permanent generation. On a Sun JVM (1.3.1 and above) you can configure the initial permanent generation size and the maximum permanent generation size.

To set a new initial size on Sun JVM use the -XX:PermSize=64m option when starting the virtual machine. To set the maximum permanent generation size use -XX:MaxPermSize=128m option. If you set the initial size and maximum size to equal values you may be able to avoid some full garbage collections that may occur if/when the permanent generation needs to be resized. The default values differ from among different versions but for Sun JVMs upper limit is typically 64MB.

Native memory pool - A non heap memory

Ther is also one more component of memry of jvm that is called native memory pool which is used for compiling code and storing native code