Agents last runtime are not getting updated in SMA
SummaryAgents last run-time are not getting updated in SMA.
Error Messages1. 2017-04-13 22:31:25,261 [ PegaRULES-Batch-2] [ STANDARD] [ PegaRULES:06.03] ( async.agent.QueueManagerImpl) ERROR - Problem with browse of System-Work-Indexer
com.pega.pegarules.pub.database.DatabaseException: There was a problem getting a list: code: 1423 SQLState: 72000 Message: ORA-01423: error encountered while checking for extra rows in exact fetch
ORA-01089: immediate shutdown in progress - no operations are permitted
ORA-06512: at "RULES.SPPR_SYS_RESERVEQUEUEITEM_B", line 110
ORA-06512: at line 1
2. 2017-04-13 22:31:26,829 [ PegaRULES-Batch-2] [ STANDARD] [ PegaRULES:06.03] ( internal.access.Opener) ERROR - There was a problem opening a database instance YOUR_WORK_CLASS T-548527
com.pega.pegarules.pub.database.ConnectionException: Database-General There was a problem opening a database instance YOUR_WORK_CLASS T-548527 1089 08006 ORA-01089: immediate shutdown in progress - no operations are permitted
SQL: select pzPVStream from pc_YOUR_WORK_CLASS where pzInsKey = ?
SQL Inserts: <YOUR_WORK_CLASS T-548527>
Caused by SQL Problems.
Problem #1, SQLState 08006, Error code 1089: java.sql.SQLRecoverableException: ORA-01089: immediate shutdown in progress - no operations are permitted
3. 2017-04-13 13:43:10,008 [ PegaRULES-Batch-2] [ STANDARD] [ IPI:02.01.01] (.YOUR_WORK_CLASS.Action) ERROR - The case WA-220085 could not be updated because Fail: Database-Instance-NotFound pxObjClass = "Data-Admin-Operator-ID", pyUserIdentifier = "Digital"
2017-04-13 13:43:11,711 [ PegaRULES-Batch-2] [ STANDARD] [ IPI:02.01.01] (em_Queue_IPI_CreateCase.Action) ERROR - obj-save failed.
com.pega.pegarules.pub.database.LockGoneException: Save, Delete or Commit has failed because lock "YOUR_WORK_CLASS CS-3562" is not held
Steps to Reproduce
Launch the Agent page on SMA and observe that the Last Finish Date is not getting populated.
Root CauseA defect or configuration issue in the operating environment:
After looking into the PRPC logs we observed that the oracle shutdown got triggered abruptly, this made us to look into the Oracle logs and found that the below error got thrown on the Oracle side:Errors in file /smartpaas-db/oracle/app/diag/rdbms/prpc/PRPC/trace/PRPC_ora_18291.trc (incident=86692):
ORA-07445: exception encountered: core dump [kgghash()+367] [SIGSEGV] [ADDR:0x2B203B36D000] [PC:0x70B322F] [Address not mapped to object]  This can get thrown for various reasons, in this case we found it to be memory tuning issue. After comparing it with the the working environment, we observed that in working environment memory allocated to Oracle is 12gb and in this non-working environment it is 4GB.
Values which are of concern are:
ResolutionMake the following change to the operating environment:
1. Get the memory tuning done in the DB node.
2. For Pega (On premises instances) suggested values for Oracle are:
3. Restart Oracle instance and App server.
Published April 19, 2017 - Updated April 21, 2017