Support Article

Campaign execution failed in data flow execution

SA-26621

Summary



Users are experiencing Campaign run failures.
The underlying data flow execution fails and the details indicate a failure with
Visual Business Director (VBD) and when looking at the logs, there's many out of memory errors on VBD.


Error Messages

) ERROR - Exception
com.pega.pegarules.pub.PRRuntimeException: Exception during data set execution
at com.pegarules.generated.dataset.ra_action_actuals_f6627700ad63a7a8a8339ef045fa7c95.perform(ra_action_actuals_f6627700ad63a7a8a8339ef045fa7c95.java:123)
at com.pega.pegarules.session.internal.mgmt.Executable.doAction(Executable.java:3310)
at com.pegarules.generated.activity.ra_action_initiatebatchoffer_dafe47fbe294811a1dcc25c988d56f3a.step8_3_circum0(ra_action_initiatebatchoffer_dafe47fbe294811a1dcc25c988d56f3a.java:1197)
at com.pegarules.generated.activity.ra_action_initiatebatchoffer_dafe47fbe294811a1dcc25c988d56f3a.step8_0_circum0(ra_action_initiatebatchoffer_dafe47fbe294811a1dcc25c988d56f3a.java:886)
at com.pegarules.generated.activity.ra_action_initiatebatchoffer_dafe47fbe294811a1dcc25c988d56f3a.perform(ra_action_initiatebatchoffer_dafe47fbe294811a1dcc25c988d56f3a.java:218)
at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Executable.java:3505)
at com.pegarules.generated.testgen.Rule_Decision_D_DF765bdfe0cc709247e6916c3485c95842f126a4e9_Action_20160718T074401_595_GMT$2$1.onNext(Rule_Decision__DF765bdfe0cc709247e6916c3485c95842f126a4e9_Action_20160718T074401_595_GMT.java:168)
at com.pega.dsm.dnode.api.dataflow.DataFlowStageBatchProcessor.commitBatchInternal(DataFlowStageBatchProcessor.java:102)
at com.pega.dsm.dnode.api.dataflow.DataFlowStageBatchProcessor.onNext(DataFlowStageBatchProcessor.java:57)
at com.pega.dsm.dnode.api.dataflow.DataFlowStageBatchProcessor.onNext(DataFlowStageBatchProcessor.java:17)
at com.pega.dsm.dnode.api.dataflow.DataFlowStage$StageInputSubscriber.onNext(DataFlowStage.java:259)
at com.pega.dsm.dnode.api.dataflow.DataFlowStage$StageInputSubscriber.onNext(DataFlowStage.java:184)
at com.pega.dsm.dnode.api.dataflow.DataFlowExecutor$QueueBasedDataFlowExecutor$2.process(DataFlowExecutor.java:189)
at com.pega.dsm.dnode.api.dataflow.DataFlowExecutor$QueueBasedDataFlowExecutor.runEventLoop(DataFlowExecutor.java:118)
at com.pega.dsm.dnode.api.dataflow.DataFlow$1.emit(DataFlow.java:162)
at com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:56)
at com.pega.dsm.dnode.impl.dataflow.manager.LocalRun.execute(LocalRun.java:145)
at com.pega.dsm.dnode.impl.dataflow.manager.LocalRun.lockAndRun(LocalRun.java:86)
at com.pega.dsm.dnode.impl.dataflow.manager.LocalRun.run(LocalRun.java:63)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at com.pega.dsm.dnode.impl.prpc.PrpcThreadFactory$PrpcThread.run(PrpcThreadFactory.java:81)
Caused by: com.pega.decision.vbd.VBDException: Server side exception is occured.
Exception:
----------
>>> Type: org.gridgain.grid.GridException
>>> Message: Remote job threw user exception (override or implement GridTask.result(..) method if you would like to have automatic failover for this exception).
>>> Documentation: http://wiki.gridgain.org
>>> Stack trace:
>>> at org.gridgain.grid.GridTaskAdapter.result(GridTaskAdapter.java:109)
>>> at org.gridgain.grid.kernal.processors.task.GridTaskWorker.result(GridTaskWorker.java:617)
>>> at org.gridgain.grid.kernal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:546)
>>> at org.gridgain.grid.kernal.processors.task.GridTaskProcessor$JobMessageListener.processJobExecuteResponse(GridTaskProcessor.java:886)
>>> at org.gridgain.grid.kernal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:851)
>>> at org.gridgain.grid.kernal.managers.communication.GridCommunicationManager.unwindMessageSet(GridCommunicationManager.java:767)
>>> at org.gridgain.grid.kernal.managers.communication.GridCommunicationManager.access$3300(GridCommunicationManager.java:45)
>>> at org.gridgain.grid.kernal.managers.communication.GridCommunicationManager$6.body(GridCommunicationManager.java:706)
>>> at org.gridgain.grid.util.runnable.GridRunnable$1.run(GridRunnable.java:142)
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>> at org.gridgain.grid.util.runnable.GridRunnable.run(GridRunnable.java:194)
>>> at org.gridgain.grid.util.runnable.GridRunnablePool$1.run(GridRunnablePool.java:80)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:745)

Caused By:
----------
>>> Type: org.gridgain.grid.GridUserUndeclaredException
>>> Message: Failed to execute job due to unexpected runtime exception [jobId=dc704d91-23da-42bd-bf04-d0e0b956c9cd, ses=GridTaskSessionImpl [taskName=com.pega.decision.vbd.gg.collection.tasks.InsertCollectionRecordTask, userVer=0, taskClsName=com.pega.decision.vbd.gg.collection.tasks.InsertCollectionRecordTask, sesId=e8c9a008-faff-441d-924b-78de97c18713, jobId=dc704d91-23da-42bd-bf04-d0e0b956c9cd, endTime=9223372036854775807, taskNodeId=7fa5fc56-f79e-4691-8675-ca2a0e19ed24, clsLdr=WebappClassLoader
context: /vbd
delegate: false
repositories:
/WEB-INF/classes/
----------> Parent Classloader:
org.apache.catalina.loader.StandardClassLoader@26c33140
, closed=false, topSpi=null, cpSpi=null, failSpi=null, loadSpi=null, seqNum=1]]
>>> Documentation: http://wiki.gridgain.org
>>> Stack trace:
>>> at org.gridgain.grid.kernal.processors.job.GridJobWorker.handleThrowable(GridJobWorker.java:469)
>>> at org.gridgain.grid.kernal.processors.job.GridJobWorker.body(GridJobWorker.java:426)
>>> at org.gridgain.grid.util.runnable.GridRunnable$1.run(GridRunnable.java:142)
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>> at org.gridgain.grid.util.runnable.GridRunnable.run(GridRunnable.java:194)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:745)

Caused By:
----------
>>> Type: java.lang.OutOfMemoryError
>>> Message: Java heap space

at com.pega.decision.vbd.service.impl.VBDServiceInterceptor.afterThrowing(VBDServiceInterceptor.java:30)
at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.framework.adapter.ThrowsAdviceInterceptor.invokeHandlerMethod(ThrowsAdviceInterceptor.java:145)
at org.springframework.aop.framework.adapter.ThrowsAdviceInterceptor.invoke(ThrowsAdviceInterceptor.java:130)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy38.insertFactRecords(Unknown Source)
at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.remoting.support.RemoteInvocationTraceInterceptor.invoke(RemoteInvocationTraceInterceptor.java:78)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy39.insertFactRecords(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.remoting.support.RemoteInvocation.invoke(RemoteInvocation.java:212)
at org.springframework.remoting.support.DefaultRemoteInvocationExecutor.invoke(DefaultRemoteInvocationExecutor.java:39)
at org.springframework.remoting.support.RemoteInvocationBasedExporter.invoke(RemoteInvocationBasedExporter.java:78)
at org.springframework.remoting.support.RemoteInvocationBasedExporter.invokeAndCreateResult(RemoteInvocationBasedExporter.java:114)
at org.springframework.remoting.httpinvoker.HttpInvokerServiceExporter.handleRequest(HttpInvokerServiceExporter.java:74)
at org.springframework.web.servlet.mvc.HttpRequestHandlerAdapter.handle(HttpRequestHandlerAdapter.java:51)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:943)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:877)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:868)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at org.springframework.remoting.support.RemoteInvocationUtils.fillInClientStackTraceIfPossible(RemoteInvocationUtils.java:45)
at org.springframework.remoting.support.RemoteInvocationResult.recreate(RemoteInvocationResult.java:149)
at org.springframework.remoting.support.RemoteInvocationBasedAccessor.recreateRemoteInvocationResult(RemoteInvocationBasedAccessor.java:85)
at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:150)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy36.insertFactRecords(Unknown Source)
at com.pegarules.generated.pxInsertRecordsFromPages_071017_cBXTCjrCfGalvt_3BxEN8w.pxInsertRecordsFromPages07_10_17(pxInsertRecordsFromPages_071017_cBXTCjrCfGalvt_3BxEN8w.java:200)
at com.pegarules.generated.pxInsertRecordsFromPages_071017_cBXTCjrCfGalvt_3BxEN8w.invoke(pxInsertRecordsFromPages_071017_cBXTCjrCfGalvt_3BxEN8w.java:91)
at com.pega.pegarules.generation.internal.library.LibraryRuntime.resolveAndinvokeFunctionViaReflection(LibraryRuntime.java:133)
at com.pega.pegarules.generation.internal.library.LibraryRuntime.invokeLibraryRuntime(LibraryRuntime.java:110)
at com.pega.pegarules.session.internal.mgmt.Executable.invokeLibraryRuntime(Executable.java:9000)
at com.pega.pegarules.priv.generator.LibrarySupport.resolveAndInvokeFunctionViaReflection(LibrarySupport.java:181)
at com.pegarules.generated.pega_decisionengine_vbd.pxInsertRecordsFromPages(pega_decisionengine_vbd.java:345)
at com.pegarules.generated.dataset.ra_action_actuals_f6627700ad63a7a8a8339ef045fa7c95$1$1.emit(ra_action_actuals_f6627700ad63a7a8a8339ef045fa7c95.java:152)
at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:320)
at com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:52)
at com.pega.dsm.dnode.impl.stream.DataObservableImpl.await(DataObservableImpl.java:98)
at com.pega.dsm.dnode.impl.stream.DataObservableImpl.toList(DataObservableImpl.java:77)
at com.pegarules.generated.dataset.ra_action_actuals_f6627700ad63a7a8a8339ef045fa7c95.perform(ra_action_actuals_f6627700ad63a7a8a8339ef045fa7c95.java:93)
... 21 more


Steps to Reproduce



Run a Pega Marketing Campaign.


Root Cause



The issue started to arise after lot of Pega Marketing Campaigns were run for testing a previous hotfix.

During those runs the Interaction History PR_DATA_IH_FACT has grown to 120 million records. VBD server loads the case ids into memory for all these responses.

On the VBD server, the heap dump of VBD JVM showed 'idsforactuals' collection taking 2 GB of space, other collections were taking another 1.5 GB. in total the 95% of the heap was already full. The allocated heap for VBD JVM was 4GB.


Resolution



Based on the expected volumes of Interaction History PR_DATA_IH_FACT to be seen on Production, it is informed to increase the VBD JVM heap size to 8GB to resolve memory exhaustion.

It is also recommended to perform regular Interaction History house keeping in order to prevent the Interaction History table getting too huge to cause performance problems.

Published August 10, 2016 - Updated August 23, 2017

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.