JVM Tuning and Troubleshooting: Production Guide

JVM tuning is a data-driven discipline, not a guessing game. This article establishes the toolchain and methodology required to handle production crises—from OOM errors and frequent GCs to CPU spikes—without panic.

1. Common Production Anomalies

2. The Diagnostic Toolchain

2.1 Essential CLI Tools

• jps: Lists Java processes. (jps -lvm)
• jstat: Monitors GC statistics. (jstat -gcutil <pid> 1000 10)
• jinfo: Views or modifies JVM flags at runtime. (jinfo -flags <pid>)
• jmap: Generates Heap Dumps for memory analysis. (jmap -dump:format=b,file=heap.hprof <pid>)
• jstack: Dumps thread stacks to find deadlocks or CPU-hungry code. (jstack <pid>)
• jcmd: A multi-purpose Swiss Army knife for JVM diagnostics.

3. High-CPU Troubleshooting Workflow

If a Java process is consuming 100% CPU, follow these steps:

1. Find the process: top (Find the <pid>).
2. Find the thread: top -H -p <pid> (Identify the specific <tid> consuming CPU).
3. Convert ID: printf '%x\n' <tid> (Convert tid to hexadecimal).
4. Pinpoint Code: Search the hexadecimal ID in the jstack output: jstack <pid> | grep -A 30 '<hex_tid>'

This allows you to link a high-CPU thread directly to a specific line of Java code.

4. Memory Leak Analysis

1. Monitor Growth: jstat -gcutil <pid> 1000. If the O (Old Gen) column grows continuously despite GCs, a leak is likely.
2. Capture the Dump: jmap -dump:live,format=b,file=leak.hprof <pid>.
3. Analyze: Use MAT (Eclipse Memory Analyzer) or VisualVM. MAT’s "Leak Suspects" report can automatically identify the objects and GC Roots responsible for the leak.

5. Standard OOM Scenarios and Solutions

5.1 Java Heap Space

• Cause: Excess object creation or collection leaks (e.g., static Map caches).
• Solution: Analyze Heap Dump with MAT. Check for unclosed resources or static collections that grow without bounds.

5.2 Metaspace

• Cause: Too many dynamically generated classes (CGLIB, Groovy, Reflection).
• Solution: Set -XX:MaxMetaspaceSize to prevent system memory exhaustion. Inspect third-party libraries for proxy caching behavior.

5.3 Unable to create new native thread

• Cause: Thread count exceeds OS limits or thread pools are misconfigured.
• Solution: Check jstack for thread states. Review thread pool coreSize and maxSize. Check Linux ulimit -u limits.

6. Recommended Production Parameters

Memory Limits

-Xms4g -Xmx4g       # Keep initial and max heap identical to prevent resizing jitter
-Xss512k            # Thread stack size (Decrease to support more threads)

GC Selection

-XX:+UseG1GC        # Recommended for JDK 8/11+
-XX:+UseZGC         # Recommended for ultra-low latency on JDK 17+

Diagnostics

-XX:+HeapDumpOnOutOfMemoryError               # Auto-dump on OOM
-XX:HeapDumpPath=/var/logs/java_heap.hprof    # Path for dump
-Xlog:gc*:file=gc.log:time,level,tags         # Modern GC logging (JDK 9+)

7. Strategic Methodology

1. Code First: Most "JVM problems" are actually application code problems. Optimize your logic and eliminate leaks before touching VM flags.
2. Stability First: Keep -Xms = -Xmx to ensure a steady environment.
3. Measure, Don't Guess: Never apply a tuning parameter without observing its effect on a GC log analyzer like GCEasy.io.
4. Last Resort: JVM tuning is the last step of performance optimization, not the first.