Support Article

System becomes unresponsive

SA-28958

Summary



After running  Qualsys vulnerability security scans in users  production servers that are hosting IBM WebSphere and pega on them.

This scan resulted in making two of the three pega nodes in production unresponsive requiring a restart of the production system before being able to access the system again. 

Error Messages



[9/19/16 20:57:10:117 EDT] 000003d6 DefaultChanne W org.jboss.netty.channel.DefaultChannelPipeline  An exception was thrown by a user handler while handling an exception event ([id: 0x00012edf, /10.88.155.100:55952 :> /10.88.212.21:7003] EXCEPTION: java.nio.channels.ClosedChannelException)
                                 java.lang.StackOverflowError
               at com.ibm.ws.util.ThreadPool.execute(ThreadPool.java:1334)
               at com.ibm.ws.util.ThreadPool.execute(ThreadPool.java:1260)
               at com.ibm.ws.management.event.NotificationDispatcher$DispatchANotificationToAllListeners.executeDispatchers(NotificationDispatcher.java:221)
               at com.ibm.ws.management.event.NotificationDispatcher$DispatchANotificationToAllListeners.dispatch(NotificationDispatcher.java:197)
               at com.ibm.ws.management.event.NotificationDispatcher.handleNotification(NotificationDispatcher.java:122)
               at com.ibm.ws.management.event.NotificationService.handleNotification(NotificationService.java:375)
               at com.ibm.ws.management.event.WsNotifDelegator.handleNotification(WsNotifDelegator.java:57)
               at com.ibm.ws.management.event.MBeanServerDelegateListener.handleNotification(MBeanServerDelegateListener.java:162)
               at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ListenerWrapper.handleNotification(DefaultMBeanServerInterceptor.java:1767)
               at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:286)
               at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:351)
               at javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:336)
               at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:259)
               at javax.management.modelmbean.RequiredModelMBean.sendNotification(RequiredModelMBean.java:2353)
               at com.ibm.websphere.management.RuntimeCollaborator.sendNotification(RuntimeCollaborator.java:101)
               at com.ibm.ws.runtime.mbean.RasLoggingServiceMBean.sendEvent(RasLoggingServiceMBean.java:177)
               at com.ibm.ws.runtime.mbean.RasLoggingServiceMBean.processEvent(RasLoggingServiceMBean.java:137)
               at com.ibm.ws.logging.WsHandlerWrapper.publish(WsHandlerWrapper.java:43)
               at java.util.logging.Logger.log(Logger.java:628)
               at com.ibm.ws.logging.WsLogger.deliverOrBuffer(WsLogger.java:407)
               at com.ibm.ws.logging.WsLogger.log(WsLogger.java:385)
               at com.ibm.ws.logging.WsLogger.logp(WsLogger.java:639)
               at org.jboss.netty.logging.JdkLogger.warn(JdkLogger.java:80)
               at org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:642)
               at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:566)
               at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
               at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
               at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:48)
               at org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658)
               at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:599)
               at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
               at org.jboss.netty.channel.Channels.write(Channels.java:704)
               at org.jboss.netty.channel.Channels.write(Channels.java:671)
               at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248)
               at com.pega.dsm.dnode.impl.server.BasicAuthenticationHandler.sendHttpResponse(BasicAuthenticationHandler.java:135)
               at com.pega.dsm.dnode.impl.server.BasicAuthenticationHandler.exceptionCaught(BasicAuthenticationHandler.java:62)
               at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
               at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
               at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
               at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
               at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
               at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
               at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
               at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
               at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:443)
               at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:128)
               at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:99)
               at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36)
               at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779)
               at org.jboss.netty.channel.Channels.write(Channels.java:725)
               at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
               at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
               at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
               at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
               at org.jboss.netty.channel.Channels.write(Channels.java:704)
               at org.jboss.netty.channel.Channels.write(Channels.java:671)
               at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248)
               at com.pega.dsm.dnode.impl.server.BasicAuthenticationHandler.sendHttpResponse(BasicAuthenticationHandler.java:135)
               at com.pega.dsm.dnode.impl.server.BasicAuthenticationHandler.exceptionCaught(BasicAuthenticationHandler.java:62)
 

Steps to Reproduce



Not Applicable 

Root Cause



The issue is caused by third party tool connecting to port 7003, which is used by Stream data set server. 
As the third party tools sends unknown requests, Stream data set causes stack overflow.

Resolution



HFix-29840 has been provided.

Published October 7, 2016 - Updated October 18, 2016

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.