JMS integration items missing
After experiencing a outage there were a number of reports from users that work objects / items were missing in the system.
Where the items that were supposed to be present should have been generated by a JMS MDB Listener.
The reason for the outage was data pool exhaustion which ended with an application server restart. However, because the JMS MDB Listener was not using a managed container this led the administrators to the idea that the items that should have been created in the system were successfully being retrieved from the Queue and acknowledgded. But as no database connections were available these items simply remained in memory.
Hence when the server was restarted the items were effectively lost.
The origins of a problem was a JDBC driver problem which was causing bind variables to be resized.
14021941 JDBC program shows high cursor version count and high bind mismatch count
This was subsequently causing the database to re-parse the query which was generating a non-optimal execution plan, hence causing slow running queries.
Because there were many of these slow running queries running at the same time, this built a backlog of requests for database connections, which led to data pool exhaustions.
And this, combined with with the JMS MDB Listener not being resized was believed to be the source of items being removed from the Queue but not added to the system.
To avoid the problem in the future the JMS MDB Listener was modified to be Container Managed, and the JDBC driver has been changed to XA.