Exited with status 137 (out of memory) happened sometimes

[gengivan]

Hi,

Our java application got crashed from time to time with following error:

app_name index: 1, reason: CRASHED, cell_id: 738d755-0a5a-441d-96de-
2bd283f4094, instance: Cb55b580-ebb0-425-4395-51ec, exit_description: APP/PROC/WEB: Exited with status 137 (out of memory)

And we have restricted the app at maximum memory at 1.8G and following are the detailed JVM configurations including heap and etc:

JDK Version (OpenJDK 64-Bit Server VM version 17.0.11+9-LTS)
-XX:+UnlockExperimentalVMOptions -XX:TrimNativeHeapInterval=8000 -XX:ReservedCodeCacheSize=240M -XX:CompressedClassSpaceSize=100m -Xms700M -Xmx700M -XX:MetaspaceSize=300M -XX:MaxMetaspaceSize=300M -Xss512K -XX:+HeapDumpOnOutOfMemoryError -XX:+UseG1GC -XX:MaxGCPauseMillis=400 -XX:OnOutOfMemoryError=/home/user/app/oom.sh -XX:MaxDirectMemorySize=466M

Originally, we found that the memory usage dropped if we manually trigger System.trim_native_heap, so added TrimNativeHeapInterval option. After then, the app crashes reduced.
Reference: https://docs.oracle.com/en/java/javase/17/docs/specs/man/jcmd.html

But issue was not gone, it still happens sometimes, e.g when load increases or the app kept running for some days. From the memory usage tracking charts, we didn't see the memory was reaching the limits. So just wonder, why it throws OOM error while there was still head room?

We didn't see any heap related out of memory error, nor heap dump generated, so assume this 137 OOM error should be related to native memory. But from our NMT tracking logs, didn't observe any category that took more memory than expected.

So any advice to address this crash issue?

[pivotal-david-osullivan]

we have restricted the app at maximum memory at 1.8G

How have you restricted the memory?

You seem to be explicitly setting JVM parameters that the Buildpack's memory calculator configures for you. Is there a reason you have configured these explicitly? The calculator is designed to work out a set of constant values based on your app and then set the heap based on this constant & the total memory available for the container.

[gengivan]

How have you restricted the #memory?

We have defined the app memory through app manifest: https://docs.cloudfoundry.org/devguide/deploy-apps/manifest-attributes.html#memory, which is a docker based application. And we can also see the value reflects from the app:

state since cpu memory disk logging details
#0 running 2024-10-30T15:01:06Z 1.5% 892.3M of 1.8G 1.4G of 4G 0B/s of unlimited

Actually, we don't use java build pack directly, but do tried https://github.com/cloudfoundry/java-buildpack-memory-calculator to get recommended JVM options and applied to our application, but without luck. Generally, I can see one difference is -XX:MaxDirectMemorySize uses 10M by default. And also tried to set to 10M for our app, still observed app crash once again.

Is there any recommended approach to collect more information for identifying the root cause? The issue was observed more times when we upgraded from JDK8 to JDK17. So do you have info from cf point of view for jdk itself mem usage.

With following native memory tracking log, do you know whether increasing reserved memory will cause OOM 137? Assume only the committed will take the physical memory and result in app crash.

Total: reserved=1576MB, committed=1363MB