While he have running this production environments for a long time, we thought we have tuned this very well; on database as on application server. As I spoke earlier on Java Garbage collection on http://orasoa.blogspot.com/2011/09/soa-11g-improved-memory-and-garbage.html, and on tuning the SOA Suite 11g; http://orasoa.blogspot.com/2011/08/tuning-soa-11g-in-nutshell.html.
We first thought, it had to do the the applications we made; Composites, Custom Java Classes, Services. Is something changed in the meantime? We also thought it had to do with the amount of data we are processing, is there more data processed than normal? Something changed in the database? The network? The physical storage? All of them were not changed, looking in the program code; we did not find anything special.
We tried to analyse the a heap dump, digged into log files, Oracle Support Bug Database and many-many-other-things. I stared a long time to this
#
# A fatal error has been detected by the Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?
#
The I was triggered by a bug in the Oracle/Sun Java; http://bugs.sun.com/view_bug.do?bug_id=6973402 . For some reason I downloaded the latest Java 1.6 SDK and applied this on one managed server on one the SOA 11g Suite.
The result was amazing!
- Less CPU
- Less Heap usage
- Normal garbage collections
We see this difference also on de Oracle Service Bus (!). My advise, roll out Java 1.6.0 build 30 (or later) on your existing production environments, SOA Suite, WebCenter, ServiceBus, etc!
Here are the results of our instance running without Java 1.6.0_b30 and at the bottom with Java 1.6.0_b30. And see the incredible difference. I did not show the CPU graphs with the garbage collections, but those are also different.
Java Heap Before applying Java 1.6.0 Build 30
Java Heap After applying Java 1.6.0 Build 30
