Re: WOSessionStore deadlocks - SOLVED
Re: WOSessionStore deadlocks - SOLVED
- Subject: Re: WOSessionStore deadlocks - SOLVED
- From: Maik Musall <email@hidden>
- Date: Thu, 08 Nov 2012 11:59:56 +0100
Hi Chuck,
a follow-up on this:
Am 19.10.2012 um 20:05 schrieb Chuck Hill <email@hidden>:
> Hi Maik,
>
> This can also indicate some other things too:
> - session did not get checked in (app threw OutOfMemory, sleep() threw an exception)
> - previous request for this session is still running (deadlock, waiting, infinite loop)
> - 2+ requests for the same session in rapid sequence where the first terminates the session
Looks like my answer that OutOfMemory would be "OutOfTheQuestion" was not true. I now discovered what lead to my application hanging every afternoon, after *once* it finally cared to log a proper message before hanging:
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: PermGen space
Doh, the PermGen. I totally forgot about that. I had the app at -Xmx24576m, but didn't adjust PermGen. Now with a PermGen limit of 512m (of which currently about 154m gets used max according to jvisualvm) everything is finally running smoothly. The app turns out to load about 12000 classes over a workday. I think I need to have a look at what those are sometime...
Maik
>
>
> Chuck
>
>
>
> On 2012-10-19, at 4:00 AM, Maik Musall wrote:
>
>> Hi,
>>
>> I recently discovered what may be responsible for frequent deadlocks of an application here. In the "jstack -l" output, I see almost all threads waiting on a single ReentrantLock, and this thread is what holds that lock:
>>
>>
>> "WorkerThread4" prio=5 tid=103bc9000 nid=0x132caf000 in Object.wait() [132cae000]
>> java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on <22711d098> (a com.webobjects.appserver.WOSessionStore$TimeoutEntry)
>> at java.lang.Object.wait(Object.java:485)
>> at com.webobjects.appserver.WOSessionStore.checkOutSessionWithID(WOSessionStore.java:191)
>> - locked <22711d098> (a com.webobjects.appserver.WOSessionStore$TimeoutEntry)
>> at com.webobjects.appserver.WOApplication.restoreSessionWithID(WOApplication.java:1913)
>> at er.extensions.appserver.ERXApplication.restoreSessionWithID(ERXApplication.java:2440)
>> at er.extensions.appserver.ERXComponentRequestHandler._dispatchWithPreparedApplication(ERXComponentRequestHandler.java:260)
>> at er.extensions.appserver.ERXComponentRequestHandler._handleRequest(ERXComponentRequestHandler.java:302)
>> at er.extensions.appserver.ERXComponentRequestHandler.handleRequest(ERXComponentRequestHandler.java:377)
>> at com.webobjects.appserver.WOApplication.dispatchRequest(WOApplication.java:1687)
>> at er.extensions.appserver.ERXApplication.dispatchRequestImmediately(ERXApplication.java:2139)
>> at er.extensions.appserver.ERXApplication.dispatchRequest(ERXApplication.java:2104)
>> at com.webobjects.appserver._private.WOWorkerThread.runOnce(WOWorkerThread.java:144)
>> at com.webobjects.appserver._private.WOWorkerThread.run(WOWorkerThread.java:226)
>> at java.lang.Thread.run(Thread.java:680)
>>
>> Locked ownable synchronizers:
>> - <20ce7bbc0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
>>
>>
>> Now, ERXApplication.restoreSessionWithID contains an interesting call to useSessionStoreDeadlockDetection(), but this detection only works in single threaded mode. I'm afraid I can't afford to switch off concurrent requests even for a testing period in production.
>>
>> I'm looking for someone with experience regarding this problem. The doc for that method mentions that it could help "to find cases when a session is checked out twice in a single RR-loop, which will lead to a session store lockup." Since I cannot switch on this detection, what in your experience could lead to that happening?
>>
>> Thanks
>> Maik
>> _______________________________________________
>> Do not post admin requests to the list. They will be ignored.
>> Webobjects-dev mailing list (email@hidden)
>> Help/Unsubscribe/Update your Subscription:
>>
>> This email sent to email@hidden
>
> --
> Chuck Hill Senior Consultant / VP Development
>
> Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems.
> http://www.global-village.net/gvc/practical_webobjects
>
> Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest Growing Companies in B.C!
> Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking of Canada’s Fastest-Growing Companies by PROFIT Magazine!
>
>
>
>
>
>
>
>
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden