Support Article
Application fails to start
SA-36807
Summary
During annual user clock change event user stopped the application and all components and four hours later tried to start it all up again.
During startup the application gives a transaction timeout in our startup bean (before it gets to the open for e-business message) and thus fails to start.
The issue in Production observed during the recovery:
- The application would not start within the JVM
- By stopping the Deployment Manager (Dmgr) the application was able to start
- Traces show that the application was attempting to connect to the Dmgr as part of its start up routine
User have observed from the prod and test traces that the application is intentionally connecting to the Dmgr on start up to perform some action so this is normal and it was not a network routing issue.
Also if the Dmgr is down the application code handles that and continues successfully.
So the main questions are :
- Why is the application, on startup, trying to talk to the deployment manager?
- What is the affect on the application if the deployment manager is not running?
- If user starts the deployment manager, will the application attempt to perform the same operation on start up and could it cause an impact on the stability of the cell
- Is there a way to debug what transactin is being run when user sees the timeout
The Introscope transaction trace captured during start up on Production, shows many EJB calls (EJBGatewayServiceBean and MasterStatelessServiceBean) and many small SQL calls for the same queries suggesting that it may be retrying.
Error Messages
The following is the error lines in the SysOut.log:
[26/03/17 20:25:22:588 BST] 0000000b TimeoutManage I WTRN0006W: Transaction TDApplication#TDACustomEJB.jar#TDAStartupCode 0000015B0C0FEF3800000001000000027407E1BADA6957DAC11DDEADCBE2E2543F2EDB470000015B0C0FEF3800000001000000027407E1BADA6957DAC11DDEADCBE2E2543F2EDB4700000001 has timed out after 300 seconds.
[26/03/17 20:25:22:590 BST] 0000000b TimeoutManage I WTRN0124I: When the timeout occurred the thread with which the transaction is, or was most recently, associated was Thread[server.startup : 0,5,main]. The stack trace of this thread when the timeout occurred was:
Steps to Reproduce
Try and start the application JVM with the WebSphere Deployment Manager running.
Resolution
Following are the answers to the queries:
1. Why is the application, on startup, trying to talk to the deployment manager ?
During the application start up few of the Chordiant services like (SystemTaskDispatcher and CacheService) are trying to register JNDI entries under cell persistence segment.
As it is trying to update the cell persistent segment, the deployment manager must be running during the server start up.
However, the cell persistent segment can be read even if the deployment manager is not running. Because server contains its own copy of the cell persistent partition, and look up locally for the objects bound in the cell persistent partition.
2. What is the effect on the application if the deployment manager is not running.
As mentioned under question 1, only during server start-up application will fail to setup Chordiant services and will throw exceptions.
But if application server is already running then there will be no impact even if the deployment manager is stopped.
3. If user starts the deployment manager, will the application attempt to perform the same operation on start up and could it cause an impact on the stability of the cell .
Setup method of Chordiant services like (SystemTaskDispatcher and CacheService) are called only once on server start-up. So application server restart is needed after starting deployment manager.
4. Is there a way to debug what transaction is being run when the timeout is observed.
[3/31/17 11:56:37:874 CDT] 0000009f SystemErr R javax.naming.NamingException: cell/persistent/CTIManagerContainer123 [Root exception is org.omg.CosNaming.NamingContextPackage.CannotProceed: IDL:omg.org/CosNaming/NamingContext/CannotProceed:1.0]
com.chordiant.service.NameServiceJNDIHelper.rebind(NameServiceJNDIHelper.java:170)
Observing the above error in OOTB if Deployment Manager is not running during server start–up.
cell/persistent/CTIManagerContainer123 is the JNDI name.
Tags:
Published May 10, 2017 - Updated October 8, 2020
Have a question? Get answers now.
Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.