Mateusz Sulima

JVM Garbage Collection — CQK Top 10

This article is a part of CQK Top 10 series.

Automatic memory management in Java greatly simplifies programs and helps to avoid dangerous errors like reading from unallocated memory. However if configured improperly, it can become a performance bottleneck. Here are some of our configuration advices and anti-patterns to avoid.

Did you set the maximum heap memory size? #

All class instances and arrays are allocated in the heap. The maximum amount of the memory which the JVM can use for this purpose, is specified by -Xmx*size* parameter, e.g. -Xmx3g. If an application requires more memory, the JVM throws an OutOfMemoryError and exits.

Remember that the JVM does not have to use all of this memory, this is just an upper limit. Memory usage may change with application load.

By default, the JVM calculates maximum heap size based on the amount of memory on the machine. In most cases it will assign 1/4 of the physical memory. This may lead to a very different performance, depending on what machine the application is run on.

You should always determine how much of the heap memory your application needs and pass this value in the -Xmx flag.

Did you set the initial heap memory size? #

Heap memory size may change during application’s lifetime. Allocated space may grow and shrink up to the upper limit specified by the -Xmx option.

You can also specify the initial heap size using the -Xms option. This may speed up your application on startup, because otherwise the JVM will need a few seconds or minutes to find the optimal memory size.

Do you use a Garbage Collection algorithm that matches your application’s behavior? #

There are four Garbage Collection (GC) algorithms available. To achieve the best performance, one should be picked based on application’s characteristics:

  • If response time is more important than overall throughput, then it is preferable to select the Concurrent Mark Sweep (CMS) Collector or Garbage-First (G1) Garbage Collector. Example application: a web service.
  • If throughput is most important, and pauses of 1 second and larger are acceptable, then it is preferable to select the Parallel Collector. Example application: a Spark job.
  • Serial Collector should be used if the application has a small data set (up to approximately 100 MB).

You should select the collector according to your specific needs. Remember that the parallel collector is default and it is not the best fit for the most of microservices. In this case you should use CMS (-XX:+UseConcMarkSweepGC flag) or G1 (-XX:+UseG1GC) collector.

Do your memory limits include the off-heap memory? #

The off-heap memory is composed of:

  • Metaspace — memory for class metadata.
  • Code cache — memory for the compiled code.
  • Thread stacks — every thread preallocates memory for a call stack, by default it is 1MB on 64-bit systems.
  • Memory allocated with sun.misc.Unsafe#allocateMemory, it may be used e.g. by caches to limit time spent on GC. It is also used by NIO objects to speed up operations on files or sockets.

The JVM reports only Metaspace and Code cache sizes, it does not tell how much space is occupied by thread stacks and other native memory consumers. The best estimate of a total memory used by a process is the resident set size (RSS). It can be read with standard operating system tools like ps.

You cannot set a limit for the off-heap memory in the JVM. If you run your application in a container, for example using Marathon, then the memory limit should be higher the maximum observed RSS. Otherwise the supervisor will kill the application.

Do you measure and monitor the memory usage and Garbage Collector behavior? #

It is good to know the behavior of garbage collector — how often it is called and how much time it takes. Memory usage is also crucial for capacity planning. These statistics can be easily reported with various tools, e.g. Dropwizard Metrics.

Does Garbage Collection have a significant impact on the performance of the application? #

A good metric of collector’s impact is CPU time spent on GC, which generally should not be more than a few percent. As a rule of thumb, time spent on young generation’s collection should be lower than 100ms per second.

Do you have some spare memory for unexpected situations? #

Sometimes the number of requests in the system may be higher than usual (e.g. due to greater interest in the application or slower database response times). This causes increased RAM usage, resulting in more GC. In the worst case, eventually the application will spend almost 100% of CPU time on Garbage Collection or die with OutOfMemoryError.

Setting approximately 25% of memory as slack should be enough to prevent these kinds of situations (assuming that queue sizes and request timeouts are properly configured).

Do you know what JVM flags you use? #

You should not add GC flags that have not proved to be helpful in a well-configured benchmark. Defaults generally work fine. There are a few dozen GC flags — manipulating with them can easily end with a loss of stability. It may also make it harder to migrate to newer JVM versions (even minor updates).

Do you keep a large cache in the heap memory? #

GC time grows proportionally to the number of objects in memory. It is possible that it is faster to fetch entities from a database or an external cache (like Redis) then to suffer from high GC pauses. As a rule of thumb, in-memory caches should be smaller than 0.5 GB.

Does your application have memory leaks? #

If your application does not free references to unused objects, it will eventually run out of memory. If it happens you should check your application’s heap dump with a specialized tool like Eclipse Memory Analyzer or JVisualVM.

Do you use proper tools to analyze GC problems? #

Analysis of GC logs is a tedious work, so we recommend using a specialized program. In our opinion Censum is a good tool to start.

You should consider having GC logs turned on also in production, because they might be helpful in a post-mortem analysis. These are flags that will log most important statistics and setup log rotation:

-Xloggc:gc.log -XX:GCLogFileSize=50M -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=2
-XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution
-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCCause

Where to go next: #

cqk-top-10 gc garbage-collection java

Discussion