Saturday, October 15, 2011

OSB: AdminServer spinning 100% CPU

Sometimes for no reason, or I did not find a reason..., the AdminServer is spinning into 100% CPU. Communication is impossible; deployment is failing or take a long time (hours...), Console will not start, etc.

The log files of the AdminServer will not give you a hint on this behavior. It will tell you that memory is reaching 100%:

AdminServer.log: reaching out of memory

Kill the AdminServer process and starting the AdminServer again, will not solve the issue. After a time the AdminServer is at 100% and communication is not possible.

A solution that worked for a short time was:
kill AdminServer process
delete all files and directories from the AdminServer:

cd $WLS_DOMAIN/servers/
cp -rv AdminServer/security .
rm -rf AdminServer/*
mv security AdminServer

Start the AdminServer
This solution resulted taht teh AdminServer was up and running, I could connect to the Service Bus console and the normal console. But when I started a deployment of new services, the AdminServer spinned into 100%.

After searching the log files and investigating why this suddenly happened, it did not occur on other environments, I did something radical. I removed all the OSB artifacts that were installed on the the AdminServer.

stop AdminServer
stop all the managed servers

cd $WLS_DOMAIN/osb
rm -rf *
start AdminServer
Now the AdminServer is up and running very quickly. Due to the fact there are now services deployed.
Start a deployment of services to the AdminSerer
After deployment, start all the managed services.
This workarround works! The AdminServer is not spinning anymore, reacts fast on the consoles.