Thanks for all the help. I am making progress. Also, I have to turn direct connect off to be able to do the refuse new sessions, so is there another way to figure out which app is hung so that it can be killed?
Look at the stack trace, like below. Also look for growing numbers of worker threads. If you have more than 16, you probably have a problem.
I have checked and this seems to be my last problem. Every once in a while I get an instance that spawns 256 worker threads. The statistics show that this does not even require a lot of transactions. I did a stack trace and there were a few funny looking things, but the one that stood out the most was the following appeared a lot:
Thread t@80131: (state = BLOCKED)
- java.lang.Class.forName0(java.lang.String, boolean, java.lang.ClassLoader) @bci=0 (Interpreted frame)
- java.lang.Class.forName(java.lang.String) @bci=5, line=164 (Interpreted frame)
- com.webobjects.foundation._NSUtilities._searchForClassInPackages(java.lang.String, com.webobjects.foundation.NSArray, boolean, boolean) @bci=92, line=417 (Interpreted frame)
- com.webobjects.foundation._NSUtilities._classWithPartialName(java.lang.String, boolean) @bci=154, line=375 (Interpreted frame)
- com.webobjects.foundation._NSUtilities.classWithName(java.lang.String) @bci=26, line=334 (Interpreted frame)
- com.webobjects.appserver._private.WOBundle.lookForClassInAllBundles(java.lang.String) @bci=5, line=92 (Interpreted frame)
- com.webobjects.appserver._private.WOActionRequestHandler._actionClassForName(java.lang.String) @bci=1, line=70 (Interpreted frame)
- com.webobjects.appserver._private.WOActionRequestHandler.getRequestActionClassAndNameForPath(com.webobjects.foundation.NSArray) @bci=114, line=389 (Interpreted frame)
- com.webobjects.appserver._private.WOActionRequestHandler._handleRequest(com.webobjects.appserver.WORequest) @bci=88, line=226 (Interpreted frame)
- com.webobjects.appserver._private.WOActionRequestHandler.handleRequest(com.webobjects.appserver.WORequest) @bci=79, line=161 (Interpreted frame)
- er.extensions.appserver.ERXDirectActionRequestHandler.handleRequest(com.webobjects.appserver.WORequest) @bci=222, line=123 (Interpreted frame)
- com.webobjects.appserver.WOApplication.dispatchRequest(com.webobjects.appserver.WORequest) @bci=32, line=1687 (Interpreted frame)
- er.extensions.appserver.ERXApplication.dispatchRequestImmediately(com.webobjects.appserver.WORequest) @bci=74, line=1791 (Interpreted frame)
- er.extensions.appserver.ERXApplication.dispatchRequest(com.webobjects.appserver.WORequest) @bci=13, line=1756 (Interpreted frame)
- com.webobjects.appserver._private.WOWorkerThread.runOnce() @bci=473, line=144 (Interpreted frame)
- com.webobjects.appserver._private.WOWorkerThread.run() @bci=129, line=226 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=613 (Interpreted frame)
So it appears that a lot of threads are getting locked up in Class.forName(). I want to print out some debug information to try to figure out which class is being instantiated, but I did not want to fill up the logs too much. I have already run out of disk space because of too much logging. So my question is, is there a call somewhere where I can find the number of worker threads or when the number of worker threads is increased to 32? That way I can turn on the logging when it really matters.
Thanks, Frank |