Support Article
Agents experiencing OutOfMemoryError: Failed to create a thread
SA-12061
Summary
Application complaining of not able to create threads and thus experiencing an “OutOfMemoryError: Failed to create a thread” and requiring restart of application.
Error Messages
[7/7/15 0:39:46:679 EDT] 0000009f SystemErr R java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 11
at java.lang.Error.<init>(Error.java:82)
at java.lang.VirtualMachineError.<init>(VirtualMachineError.java:64)
at java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:69)
at java.lang.Thread.startImpl(Native Method)
at java.lang.Thread.start(Thread.java:958)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:981)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1404)
at com.hazelcast.spi.impl.BasicOperationScheduler.execute(BasicOperationScheduler.java:148)
at com.hazelcast.spi.impl.BasicOperationService.receive(BasicOperationService.java:206)
at com.hazelcast.spi.impl.NodeEngineImpl.handlePacket(NodeEngineImpl.java:239)
at com.hazelcast.nio.NodeIOService.handleMemberPacket(NodeIOService.java:99)
at com.hazelcast.nio.SocketPacketReader.enqueueFullPacket(SocketPacketReader.java:54)
at com.hazelcast.nio.SocketPacketReader.access$200(SocketPacketReader.java:26)
at com.hazelcast.nio.SocketPacketReader$DefaultPacketReader.readPacket(SocketPacketReader.java:69)
at com.hazelcast.nio.SocketPacketReader.read(SocketPacketReader.java:49)
at com.hazelcast.nio.ReadHandler.handle(ReadHandler.java:72)
at com.hazelcast.nio.InSelectorImpl.handleSelectionKey(InSelectorImpl.java:33)
at com.hazelcast.nio.AbstractIOSelector.run(AbstractIOSelector.java:126)
Steps to Reproduce
Setup PRPC on Linux server running multiple applications.
Root Cause
This issue is due to less number of user processes configured at OS level. For most cases of older versions of Linux this value will be defaulted to around 2048. For out of the box Red Hat Enterprise Linux (RHEL) 6 the default value for nproc will be set to 1024. This low default setting for larger systems will not allow for enough threads in all processes.
Resolution
Increase NPROC to higher than default value in Operating system.
When using WebSphere Application Server, ulimits can be set to fix or tune around a number of problems. For more on how to set a ulimit refer to the "Guidelines for setting ulimits" Technote which goes into detail on setting different ulimits on various operating systems and the difference between the soft and hard limit. This article is concerned particularly with the "-u" ulimit or "nproc" limit which affects the number of processes allowed by a single user running WebSphere Application Server.
The nproc limit usually only counts processes on a server towards determining this number. Linux systems running WebSphere Application Server are a particular case. The nproc limit on Linux counts the number of threads within all processes that can exist for a given user. To determine the ulimit settings of a WebSphere Application Server process running on Linux refer to "How to determine the ulimit settings of a running WebSphere Application Server process on Linux".
Published July 13, 2015 - Updated October 8, 2020
Have a question? Get answers now.
Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.