Re: Immortal sessions in production
Re: Immortal sessions in production
- Subject: Re: Immortal sessions in production
- From: Matteo Centro <email@hidden>
- Date: Mon, 14 Jan 2013 12:35:01 +0100
Sure, here it is:
Hi Chuck,
I'm posting just to you, I can't have google to index this...
anyway, here is the stack trace:
java.lang.IllegalMonitorStateException
at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:127)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239)
at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431)
at com.webobjects.eocontrol.EOObjectStoreCoordinator.unlock(EOObjectStoreCoordinator.java:448)
at com.webobjects.eocontrol.EOEditingContext.unlockObjectStore(EOEditingContext.java:4684)
at er.extensions.eof.ERXEC.unlockObjectStore(ERXEC.java:805)
at com.webobjects.eocontrol.EOCustomObject.willReadRelationship(EOCustomObject.java:1281)
at er.extensions.eof.ERXGenericRecord.willReadRelationship(ERXGenericRecord.java:378)
at com.webobjects.eocontrol._EOMutableKnownKeyDictionary$Initializer$_LazyGenericRecordBinding.valueInObject(_EOMutableKnownKeyDictionary.java:614)
at com.webobjects.eocontrol.EOCustomObject.storedValueForKey(EOCustomObject.java:1634)
at com.tla.logic._RigaCarrello.clientiPerTicket(_RigaCarrello.java:275)
at com.tla.logic.RigaCarrello.setPosti(RigaCarrello.java:94)
at com.tla.logic.Carrello.cancellaTutto(Carrello.java:164)
at com.tla.calendar.Session.terminate(Session.java:81)
at com.webobjects.appserver.WOSession._terminateByTimeout(WOSession.java:610)
at com.webobjects.appserver.WOSessionStore$_SessionTimeoutManager.run(WOSessionStore.java:115)
at java.lang.Thread.run(Thread.java:662)
Sorry for the italian Class Names and Methods, as I said I inherited that...
The exception is caught, that printout is the printStackTrace of the
exception...
But anyway, I'm not sure that's the real problem, I found this morning
an instance with 295 sessions (still alive) and the only exception
logged was a weird
er.transaction.adaptor.Exceptions - Database Exception occured: No
operations allowed after connection closed.
Connection was closed due to the following exception:
** BEGIN NESTED EXCEPTION **
java.sql.SQLException
MESSAGE: Communication link failure: java.io.EOFException, underlying
cause: null
** BEGIN NESTED EXCEPTION **
java.io.EOFException
STACKTRACE:
java.io.EOFException
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1394)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:1538)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:1929)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1167)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1278)
at com.mysql.jdbc.Connection.execSQL(Connection.java:2247)
at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1371)
at com.webobjects.jdbcadaptor.JDBCChannel._bindInputVariablesWithBindingsAndExecute(JDBCChannel.java:265)
at com.webobjects.jdbcadaptor.JDBCChannel._evaluateExpression(JDBCChannel.java:337)
at com.webobjects.jdbcadaptor.JDBCChannel.evaluateExpression(JDBCChannel.java:296)
at com.webobjects.jdbcadaptor.JDBCChannel.selectAttributes(JDBCChannel.java:220)
at com.webobjects.eoaccess.EODatabaseChannel._selectWithFetchSpecificationEditingContext(EODatabaseChannel.java:897)
at com.webobjects.eoaccess.EODatabaseChannel.selectObjectsWithFetchSpecification(EODatabaseChannel.java:234)
at com.webobjects.eoaccess.EODatabaseContext._objectsWithFetchSpecificationEditingContext(EODatabaseContext.java:3055)
at com.webobjects.eoaccess.EODatabaseContext.objectsWithFetchSpecification(EODatabaseContext.java:3195)
at com.webobjects.eocontrol.EOObjectStoreCoordinator.objectsWithFetchSpecification(EOObjectStoreCoordinator.java:488)
at com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEditingContext.java:4069)
at er.extensions.eof.ERXEC.objectsWithFetchSpecification(ERXEC.java:1305)
at com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEditingContext.java:4444)
at com.tla.logic.Performance.fetchFreshPerformancesWithQualifier(Performance.java:414)
at com.tla.logic.Carrello.cancellaTutto(Carrello.java:154)
at com.tla.calendar.components.ALTComponent.invokeAction(ALTComponent.java:167)
at com.webobjects.appserver.WOSession.invokeAction(WOSession.java:1357)
at com.webobjects.appserver.WOApplication.invokeAction(WOApplication.java:1745)
at er.extensions.appserver.ajax.ERXAjaxApplication.invokeAction(ERXAjaxApplication.java:117)
at er.extensions.appserver.ERXApplication.invokeAction(ERXApplication.java:2018)
at er.extensions.appserver.ERXComponentRequestHandler._dispatchWithPreparedPage(ERXComponentRequestHandler.java:157)
at er.extensions.appserver.ERXComponentRequestHandler._dispatchWithPreparedSession(ERXComponentRequestHandler.java:235)
at er.extensions.appserver.ERXComponentRequestHandler._dispatchWithPreparedApplication(ERXComponentRequestHandler.java:268)
at er.extensions.appserver.ERXComponentRequestHandler._handleRequest(ERXComponentRequestHandler.java:302)
at er.extensions.appserver.ERXComponentRequestHandler.handleRequest(ERXComponentRequestHandler.java:377)
at com.webobjects.appserver.WOApplication.dispatchRequest(WOApplication.java:1687)
at er.extensions.appserver.ERXApplication.dispatchRequestImmediately(ERXApplication.java:2139)
at er.extensions.appserver.ERXApplication.dispatchRequest(ERXApplication.java:2104)
at com.webobjects.appserver._private.WOWorkerThread.runOnce(WOWorkerThread.java:144)
at com.webobjects.appserver._private.WOWorkerThread.run(WOWorkerThread.java:226)
at java.lang.Thread.run(Thread.java:662)
So I'm not that confident that I am on the right track...
Looks like if anything goes wrong in the life of the instance sessions
won't die... Which is kind of disturbing.
Matteo
On Sun, Jan 13, 2013 at 7:01 PM, Chuck Hill <email@hidden> wrote:
> Can you post the entire stack trace?
>
> At the very least, I would catch that exception and now allow it out of your finalize method.
On Sun, Jan 13, 2013 at 7:01 PM, Chuck Hill <email@hidden> wrote:
> Can you post the entire stack trace?
>
> At the very least, I would catch that exception and now allow it out of your finalize method.
>
>
> On 2013-01-12, at 2:01 PM, Matteo Centro wrote:
>
>> Thanks, I'll try that.
>> Anyway (I can't say it's that but this happens often in instances with
>> immortal sessions):
>> I get this in my terminate() method
>> java.lang.IllegalMonitorStateException
>> at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:127)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239)
>> at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431)
>> at com.webobjects.eocontrol.EOObjectStoreCoordinator.unlock(EOObjectStoreCoordinator.java:460)
>> at com.webobjects.eocontrol.EOEditingContext.unlockObjectStore(EOEditingContext.java:4684)
>> at er.extensions.eof.ERXEC.unlockObjectStore(ERXEC.java:805)
>> at com.webobjects.eocontrol.EOCustomObject.willReadRelationship(EOCustomObject.java:1281)
>> at er.extensions.eof.ERXGenericRecord.willReadRelationship(ERXGenericRecord.java:378)
>> at com.webobjects.eocontrol._EOMutableKnownKeyDictionary$Initializer$_LazyGenericRecordBinding.valueInObject(_EOMutableKnownKeyDictionary.java:614)
>> at com.webobjects.eocontrol.EOCustomObject.storedValueForKey(EOCustomObject.java:1634)
>>
>> I'm using ERXEC everywhere and I'm not locking explicitly, should I
>> lock the ec in the terminate method?
>>
>>
>> Matteo
>>
>> On Sat, Jan 12, 2013 at 8:39 PM, Simon <email@hidden> wrote:
>>> stick this in your session constructor, it will log out whenever a
>>> session is created. then you can start figuring out where you are
>>> creating a session outside the RR loop which is the normal culprit...
>>>
>>> StringWriter stringWriter = new StringWriter();
>>> PrintWriter printWriter = new PrintWriter(stringWriter);
>>> (new Throwable()).printStackTrace(printWriter);
>>> String trace = stringWriter.toString();
>>> log.debug("Session count = " + application().activeSessionsCount());
>>> ClickEventLogger2.getLogger().fatal(ClickEventCode.E00129, "Session
>>> Created:\n\n " + trace,
>>> this.getClass().getSimpleName());
>>>
>>> hmmm, you'll have to replace the "ClickEventLogger2" line because
>>> that's our logging mechanism. you could log.fatal it and set up an
>>> email appender in log4j.
>>>
>>> simon
>>>
>>>
>>> On 12 January 2013 17:43, Matteo Centro <email@hidden> wrote:
>>>> Sorry to resuscitate such an old discussion but I'm having the exact
>>>> same issue...
>>>> It's an old application that we "inherited", we wonderized it as much
>>>> as it's possible but something weird happens in production, sessions
>>>> on some instances simply won't die.
>>>> Some instances go out of memory and they hang there.
>>>> I'm in trouble and I needs some hints, both to fix the issue
>>>> temporarily and to fix it for good (of course in that case I assume
>>>> I'll have to rewrite the app, if only I could find the budget).
>>>> What are the most common causes of sessions not dying?
>>>>
>>>> Thanks,
>>>>
>>>>
>>>> Matteo
>>>>
>>>> On Thu, Aug 21, 2008 at 5:35 AM, Joe Little <email@hidden> wrote:
>>>>> I had something similar with sessions going bonkers on a public WO
>>>>> page that our internal google search engine completely trashed. In the
>>>>> end, robots.txt and explicit rules to deny certain patterns were added
>>>>> to prevent this.
>>>>>
>>>>>
>>>>> On Wed, Aug 20, 2008 at 8:17 PM, D Tim Cummings <email@hidden> wrote:
>>>>>> We have a couple of sessionless apps that have started showing this problem
>>>>>> with sessions that don't terminate. It turned out the sessions were being
>>>>>> created by malformed urls coming from malicious robot web crawlers. The
>>>>>> urls were of the form
>>>>>> http://www.courses.qut.edu.au/cgi-bin/WebObjects/Courses.woa/wa/cgi-bin/WebObjects/Courses.woa
>>>>>> Maybe see if you are getting incorrect links to your sessionless login page.
>>>>>> We solved the problem by catching unknown direct actions in
>>>>>> DirectAction.java
>>>>>> @Override
>>>>>> public WOActionResults performActionNamed(String actionName) {
>>>>>> try {
>>>>>> return super.performActionNamed(actionName);
>>>>>> } catch (NSForwardException nsfe) {
>>>>>> log.info("ns forward exception - prbalby no such method for " + actionName);
>>>>>> }
>>>>>> return defaultAction();
>>>>>> }
>>>>>> and in Application.java directing exceptions back to the Main page (for URLs
>>>>>> with more than one / after wa).
>>>>>> @Override
>>>>>> public WOComponent pageWithName(String namePage, WOContext context) {
>>>>>> if ( "WOExceptionPage".equals(namePage) ) {
>>>>>> namePage = "Main";
>>>>>> }
>>>>>> if ( "WOSessionRestorationError".equals(namePage) ) {
>>>>>> namePage = "Main";
>>>>>> }
>>>>>> return super.pageWithName(namePage, context);
>>>>>> }
>>>>>>
>>>>>> and in Main.java
>>>>>> public void setException ( Exception e ) {
>>>>>> log.error("an exception occurred " + e);
>>>>>> }
>>>>>>
>>>>>> We are running apps with embedded Wonder 4 and WebObjects 5.3.3 on Mac OS X
>>>>>> Server 10.5.4 with WebObjects 5.4.2 deployment. We didn't have the problem
>>>>>> before we went to this setup, but maybe we weren't getting hit with the same
>>>>>> url format then.
>>>>>> Tim
>>>>>> On 21/08/2008, at 4:02 AM, Chuck Hill wrote:
>>>>>>
>>>>>> On Aug 20, 2008, at 9:54 AM, Simon McLean wrote:
>>>>>>
>>>>>> Hi All -
>>>>>> Wondering if someone can throw me some ideas on solving what looks like an
>>>>>> immortal session problem.
>>>>>> - Start up 1 instance in java monitor.
>>>>>> - User lands on sessionless login page. No sessions.
>>>>>> - User logs in. 1 session.
>>>>>> - User logs out. 0 sessions.
>>>>>> - User logs in. 1 session.
>>>>>> - User does nothing. Session times out. 0 sessions.
>>>>>> All looks good. However, we get to the end of the day and the instance has
>>>>>> 376 active sessions. But I know for this particular app there are only 6
>>>>>> users, because they are all sitting next door :-) Also, If i now leave this
>>>>>> instance overnight none of those active sessions will go away.
>>>>>> So what could be going on ? Something appears to be creating immortal
>>>>>> sessions, but forced session termination (by the user logging out) and
>>>>>> session expiration seem to be behaving themselves.
>>>>>>
>>>>>> If the users don't notice any problems, my money would be on an exception
>>>>>> thrown from your Session.terminate(). Also check the sleep() method to
>>>>>> ensure it can never throw.
>>>>>> I have seen a related problem where two requests come in for the same
>>>>>> session (user double clicks, Ajax etc), and the first calls terminate() on
>>>>>> the session. The second thread is stuck and the app can't gracefully shut
>>>>>> down. IIRC, you would see zero sessions in this case. The "fix" for this
>>>>>> is to not call terminate() and instead set the session timeout to a small
>>>>>> number of seconds.
>>>>>> Chuck
>>>>>> --
>>>>>> Chuck Hill Senior Consultant / VP Development
>>>>>> Practical WebObjects - for developers who want to increase their overall
>>>>>> knowledge of WebObjects or who are trying to solve specific problems.
>>>>>> http://www.global-village.net/products/practical_webobjects
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>> Webobjects-dev mailing list (email@hidden)
>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>> This email sent to email@hidden
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>> Webobjects-dev mailing list (email@hidden)
>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>>
>>>>>> This email sent to email@hidden
>>>>>>
>>>>> _______________________________________________
>>>>> Do not post admin requests to the list. They will be ignored.
>>>>> Webobjects-dev mailing list (email@hidden)
>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>
>>>>> This email sent to email@hidden
>>>> _______________________________________________
>>>> Do not post admin requests to the list. They will be ignored.
>>>> Webobjects-dev mailing list (email@hidden)
>>>> Help/Unsubscribe/Update your Subscription:
>>>>
>>>> This email sent to email@hidden
>> _______________________________________________
>> Do not post admin requests to the list. They will be ignored.
>> Webobjects-dev mailing list (email@hidden)
>> Help/Unsubscribe/Update your Subscription:
>>
>> This email sent to email@hidden
>
> --
> Chuck Hill Senior Consultant / VP Development
>
> Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems.
> http://www.global-village.net/gvc/practical_webobjects
>
> Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest Growing Companies in B.C!
> Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking of Canada’s Fastest-Growing Companies by PROFIT Magazine!
>
>
>
>
>
>
>
>
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden