SA-4659

Summary

2 nodes out of 13 became unresponsive and required rebooting. After rebooting, they function as expected.

Error Messages

OutOfMemoryError occurring within PassivationDaemon:

####<Dec 12, 2014 10:26:40 AM CST> <Notice> <Stdout> <holder_xxxx> <ManagedServer_5> <Dispatcher-Thread-22> <<WLS Kernel>> <> <> <1418401600248> <BEA-000000> <2014-12-12 10:26:40,237 [ES PassivationDaemon] [  STANDARD] [                    ] (ternal.async.PassivationDaemon) ERROR   - Failed to process passivation queue: 
java.lang.OutOfMemoryError: Java heap space
    at com.pega.pegarules.priv.factory.ByteArrayFactory.newProduct(ByteArrayFactory.java:76)
    at com.pega.pegarules.priv.factory.AbstractContainerFactory.acquireObject(AbstractContainerFactory.java:231)
    at com.pega.pegarules.priv.factory.ByteArrayFactory.acquire(ByteArrayFactory.java:102)
    at com.pega.pegarules.data.internal.clipboard.StandardEnvironmentAdapter.acquireByteArray(StandardEnvironmentAdapter.java:101)
    at com.pega.pegarules.data.internal.clipboard.directstream.ByteArrayUtil.ensureCapacity(ByteArrayUtil.java:665)
    at com.pega.pegarules.data.internal.clipboard.directstream.ByteArrayUtil.putStringUTF16(ByteArrayUtil.java:559)
    at com.pega.pegarules.data.internal.clipboard.DirectStreamEncoder$StreamSection.writeString(DirectStreamEncoder.java:2014)
    at com.pega.pegarules.data.internal.clipboard.DirectStreamEncoder$StringSection.writeLengthEncodedString(DirectStreamEncoder.java:1713)
    at com.pega.pegarules.data.internal.clipboard.DirectStreamEncoder$StringSection.forceAddString(DirectStreamEncoder.java:1636)
    at com.pega.pegarules.data.internal.clipboard.DirectStreamEncoder$StringSection.addString(DirectStreamEncoder.java:1615)
    at com.pega.pegarules.data.internal.clipboard.DirectStreamEncoder$PageSection.addProperty(DirectStreamEncoder.java:1166)
    at com.pega.pegarules.data.internal.clipboard.DirectStreamEncoder$PageSection.addPage(DirectStreamEncoder.java:727)
    at com.pega.pegarules.data.internal.clipboard.DirectStreamEncoder$PageListSection.addPageList(DirectStreamEncoder.java:1487)
    at com.pega.pegarules.data.internal.clipboard.DirectStreamEncoder$PageSection.addProperty(DirectStreamEncoder.java:1119)
    at com.pega.pegarules.data.internal.clipboard.DirectStreamEncoder$PageSection.addPage(DirectStreamEncoder.java:727)
    at com.pega.pegarules.data.internal.clipboard.DirectStreamEncoder.encode(DirectStreamEncoder.java:355)
    at com.pega.pegarules.data.internal.clipboard.StorageStreamCommonImpl.encode(StorageStreamCommonImpl.java:759)
    at com.pega.pegarules.data.internal.clipboard.ClipboardPageImpl.getStreamForm(ClipboardPageImpl.java:2794)
    at com.pega.pegarules.session.internal.mgmt.base.ThreadPassivation.getPageBytes(ThreadPassivation.java:616)
    at com.pega.pegarules.session.internal.mgmt.PRThreadImpl.getPageBytes(PRThreadImpl.java:1131)
    at com.pega.pegarules.session.internal.mgmt.base.FilesystemStorage.passivatePage(FilesystemStorage.java:366)
    at com.pega.pegarules.session.internal.mgmt.base.NodeRequestorMgt.savePage(NodeRequestorMgt.java:520)
    at com.pega.pegarules.session.internal.mgmt.base.NodeRequestorMgt.processPassivationQueue(NodeRequestorMgt.java:2085)
    at com.pega.pegarules.session.internal.async.PassivationDaemon.processPassivationQueue(PassivationDaemon.java:351)

Steps to Reproduce

N/A

Root Cause

Inspection of ALERT logs show a customer process which consumes over 1.5GB of heap space - this is an iterative process that attempts to perform a bulk operation and utilizes all available heap.

Resolution

The root cause of this problem is a defect in customer application code/rules.

Sent feedback and ALERT excerpts which outline the application process that is consuming large (1.5GB amounts of heap).  When the user responsible for this process goes idle (while the large data is still in their clipboard), the passivation daemon attempts to store their requestor data which then puts the JVM OutOfMemory.