Skip to main content

This content has been archived and is no longer being updated. Links may not function; however, this content may be relevant to outdated versions of the product.

Support Article

Out of memory error in logs

SA-2360

Summary



OutOfMemoryErrors occur in server logs despite the SMA reporting > 2GB of heapsize free.

Error Messages



[9/16/14 16:55:03:352 UTC] 0000001c SystemOut     O 2014-09-16 16:55:03,351 [j2ee14_ws,maxpri=10]] [  STANDARD] [                    ] (    mgmt.base.NodeRequestorMgt) ERROR   - An error occurred cleaning up requestor "A170A8749E23C9EAB16CFB93C0B6C57E0" (cleanup mode is 3); it will still be removed from this node's list

java.lang.OutOfMemoryError


Steps to Reproduce



NA


Root Cause



After reviewing the log files (PegaRULES and JVM) from the occurrences of the OOMErrors we can infer the following –
  1. In its steady state with max requestors online the application is not consuming very high heap memory.
  2. The OOMError that occurred were not due to heap exhaustion but rather due to insufficient native memory.
  3. There are two types of memory problems which will throw an OutOfMemoryError: If a Java object cannot be allocated or if there is no more memory available for native code allocations (c malloc) or threads. The first case is called Java heap exhaustion and the second, a native memory issue.
 
Summary – This issue has clear indication of Native memory exhaustion.
 

Resolution



Finding the culprits of native memory can be a lengthy process, as there is no way of knowing which code allocated the native memory. There are no comparable heapdumps for native memory.
 
However, following are the most common causes of native memory issues in WebSphere Application Server, so IBM’s recommendations is to try them first to resolve native memory issues. They are all trying to reduce the native memory used (the native memory footprint) so both the java heap memory and native memory can fit in the address space for the application server process.
 
1. WebContainer DirectByteBuffer use. The WebContainer may store a large number of buffers in native memory when sending large responses back to the requesters asynchronously. We can reduce the native memory footprint by setting a WebContainer property to send responses in synchronous mode. The downside of synchronous mode is that it will use a thread exclusively during the response..
See the following page for more detail and instructions:
Setting the WebSphere Application Server WebContainer to synchronous mode 


2. AIO Native Transport. (WebSphere Application Server 6.1 and higher).
In general, running the WebSphere Application Server TCP Channel in AIO mode gives better performance and scalability than using the TCP Channel in NIO mode. But this faster performance, different timing, and slightly different threading design can increase the native memory footprint. If the application server is short on native memory, sometimes disabling the AIO native library can increase the available address space for native memory.
See the following page for more detail and instructions: 

Disabling AIO (Asynchronous Input/Output) native transport in WebSphere Application Server


3. Thread pool issue. This native memory footprint issue can sometimes occur when data is attached to thread pool threads, via ThreadLocal java objects, and these threads are frequently created and destroyed. When a thread is released the memory it was holding onto is dereferenced for garbage collection. This includes heap and native memory. If there are any garbage collection problems or another object is referencing the ThreadLocal objects, then they cannot get garbage collected and therefore the native memory does not get freed either. To fix the problem, use a fixed size thread pool.
See the following page for more detail and instructions to set your thread pool size: 

Potential native memory use in WebSphere Application Server Thread Pools


4. Reflection delegating classloaders. When using Java reflection, if the JVM uses a Java bytecode accessor on the class being reflected, it will use native memory. If there is frequent use of Java reflection, there can be a significant amount of native memory use.
See the following page for more detail and instructions on how to limit the native memory use of the Java reflection accessors: 

Potential native memory use in reflection delegating classloaders
Suggest Edit

Published January 31, 2016 - Updated October 8, 2020

Did you find this content helpful? Yes No

100% found this useful

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.

We'd prefer it if you saw us at our best.

Pega Community has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice
Contact us