Support Article

Huge numbers of I/O Worker threads due to third party tool

SA-29910

Summary



A datacenter required rebooting when 481 "New I/O worker #" appeared. 


Error Messages



"New I/O worker #479" prio=3 tid=0x000000011300d000 nid=0xddf24 runnable [0xfffffffdbecfe000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.PollArrayWrapper.poll0(Native Method)
at sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
at sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xfffffffe2df06550> (a sun.nio.ch.Util$2)
- locked <0xfffffffe2df06568> (a java.util.Collections$UnmodifiableSet)
- locked <0xfffffffe2e5f24e0> (a sun.nio.ch.PollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
...

 

<DATE> 0000006b DefaultChanne W org.jboss.netty.channel.DefaultChannelPipeline  An exception was thrown by a user handler while handling an exception event ([id: 0x6052d70a, /<IP:PORT> :> /<IP:PORT>] EXCEPTION: java.nio.channels.ClosedChannelException)
                                 java.lang.StackOverflowError
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at sun.misc.Launcher$ExtClassLoader.findClass(Launcher.java:229)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:295)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:438)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:422)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:410)
    at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:107)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at org.eclipse.hyades.logging.java.CommonBaseEventLogRecord.setThrown(CommonBaseEventLogRecord.java:1660)
    at com.ibm.ws.logging.object.CBEWsLogRecordAdapter.setThrown(CBEWsLogRecordAdapter.java:155)
    at com.ibm.ws.logging.object.WsLogRecordFactory.createWsLogRecord(WsLogRecordFactory.java:538)
    at com.ibm.ws.logging.WsLogger.createLogRecord(WsLogger.java:446)
    at com.ibm.ws.logging.WsLogger.logp(WsLogger.java:623)
    at org.jboss.netty.logging.JdkLogger.warn(JdkLogger.java:80)
    at org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:642)



Steps to Reproduce



Not applicable


Root Cause



A defect in Pegasystems’ code or rules.

The issue is caused by third party tool connecting to port 7003, which is used by Stream data set server. 
As the third party tools sends unknown requests, Stream data set causes stack overflow.


Resolution



Apply HFix-30186.

 

Published November 2, 2016 - Updated November 18, 2016

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.