Support Article
Huge numbers of I/O Worker threads due to third party tool
Summary
A datacenter required rebooting when 481 "New I/O worker #" appeared.
Error Messages
"New I/O worker #479" prio=3 tid=0x000000011300d000 nid=0xddf24 runnable [0xfffffffdbecfe000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.PollArrayWrapper.poll0(Native Method)
at sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
at sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xfffffffe2df06550> (a sun.nio.ch.Util$2)
- locked <0xfffffffe2df06568> (a java.util.Collections$UnmodifiableSet)
- locked <0xfffffffe2e5f24e0> (a sun.nio.ch.PollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
...
<DATE> 0000006b DefaultChanne W org.jboss.netty.channel.DefaultChannelPipeline An exception was thrown by a user handler while handling an exception event ([id: 0x6052d70a, /<IP:PORT> :> /<IP:PORT>] EXCEPTION: java.nio.channels.ClosedChannelException)
java.lang.StackOverflowError
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at sun.misc.Launcher$ExtClassLoader.findClass(Launcher.java:229)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:295)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:438)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:422)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:410)
at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:107)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at org.eclipse.hyades.logging.java.CommonBaseEventLogRecord.setThrown(CommonBaseEventLogRecord.java:1660)
at com.ibm.ws.logging.object.CBEWsLogRecordAdapter.setThrown(CBEWsLogRecordAdapter.java:155)
at com.ibm.ws.logging.object.WsLogRecordFactory.createWsLogRecord(WsLogRecordFactory.java:538)
at com.ibm.ws.logging.WsLogger.createLogRecord(WsLogger.java:446)
at com.ibm.ws.logging.WsLogger.logp(WsLogger.java:623)
at org.jboss.netty.logging.JdkLogger.warn(JdkLogger.java:80)
at org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:642)
Steps to Reproduce
Not applicable
Root Cause
A defect in Pegasystems’ code or rules.
The issue is caused by third party tool connecting to port 7003, which is used by Stream data set server.
As the third party tools sends unknown requests, Stream data set causes stack overflow.
Resolution
Apply HFix-30186.
Published November 18, 2016 - Updated October 8, 2020
Have a question? Get answers now.
Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.