RCA:IAC Links are loading very slow
SummaryThe Internet Application Composer (IAC) links are loading very slow and user is not able to access the application.
To recover, user had to restart the IAC app servers.
Error MessagesThere is no error as such in both IAC & prweb app nodes. Also on
LB (load balancer), there are no specific errors.
Steps to ReproduceHit the IAC URL in browser, it take pretty long time to load.
Root CauseJVM Capacity issues.
ResolutionBased on the data and investigation, the incident management team concluded that the issue was happening due to application servers not able to process incoming connections and that the max connections (maxThreads) on the prweb and prgateway server needs to be increased.
The team also found that the current heap memory set on IAC nodes was not sufficient and recommended to increase the heap memory.
The final recommendations were:
- Increase maxthreads on prweb nodes from 300 to 500
- Increase maxthreads on prgateway (IAC) nodes from 500 to 1000
- Upgrade instance type of prgateway (IAC) nodes and allocate 8GB of heap memory
Published January 16, 2017 - Updated May 24, 2017