Re: Immortal sessions in production
Re: Immortal sessions in production
- Subject: Re: Immortal sessions in production
- From: Matteo Centro <email@hidden>
- Date: Tue, 15 Jan 2013 12:20:58 +0100
Unfortunately I don't have full control of the deployment
environment... It looks like at some times the DB stops responding
with no apparent reason.
Could a simple DB Connection glitch cause a EOObjectStoreCoordinator lock?
Maybe I can tweak the connection dictionary to enable auto reconnect.
I'll try
Matteo
On Mon, Jan 14, 2013 at 8:17 PM, Chuck Hill <email@hidden> wrote:
> Hi Matteo,
>
> Something locked the EOObjectStoreCoordinator and did not unlock it. Does the log for this instance show any exceptions?
>
> This exception could perhaps cause this problem:
>
>>> ava.io.EOFException
>>> at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1394)
>>> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:1538)
>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:1929)
>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1167)
>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1278)
>>> at com.mysql.jdbc.Connection.execSQL(Connection.java:2247)
>>> at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1371)
>>> at com.webobjects.jdbcadaptor.JDBCChannel._bindInputVariablesWithBindingsAndExecute(JDBCChannel.java:265)
>>> at com.webobjects.jdbcadaptor.JDBCChannel._evaluateExpression(JDBCChannel.java:337)
>>> at com.webobjects.jdbcadaptor.JDBCChannel.evaluateExpression(JDBCChannel.java:296)
>>> at com.webobjects.jdbcadaptor.JDBCChannel.selectAttributes(JDBCChannel.java:220)
>>> at com.webobjects.eoaccess.EODatabaseChannel._selectWithFetchSpecificationEditingContext(EODatabaseChannel.java:897)
>>> at com.webobjects.eoaccess.EODatabaseChannel.selectObjectsWithFetchSpecification(EODatabaseChannel.java:234)
>>> at com.webobjects.eoaccess.EODatabaseContext._objectsWithFetchSpecificationEditingContext(EODatabaseContext.java:3055)
>>> at com.webobjects.eoaccess.EODatabaseContext.objectsWithFetchSpecification(EODatabaseContext.java:3195)
>>> at com.webobjects.eocontrol.EOObjectStoreCoordinator.objectsWithFetchSpecification(EOObjectStoreCoordinator.java:488)
>>> at com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEditingContext.java:4069)
>>> at er.extensions.eof.ERXEC.objectsWithFetchSpecification(ERXEC.java:1305)
>>> at com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEditingContext.java:4444)
>
>
> Chuck
>
>
> On 2013-01-14, at 10:05 AM, Matteo Centro wrote:
>
>> Hi all, suspecting there was some sort of deadlock I ran jstack
>> against one of the immortal sessions instances, I'm attaching the
>> output, anything helps...
>>
>> Thanks,
>>
>>
>> Matteo
>>
>> On Mon, Jan 14, 2013 at 12:35 PM, Matteo Centro <email@hidden> wrote:
>>> Sure, here it is:
>>> Hi Chuck,
>>>
>>> I'm posting just to you, I can't have google to index this...
>>>
>>> anyway, here is the stack trace:
>>> java.lang.IllegalMonitorStateException
>>> at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:127)
>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239)
>>> at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431)
>>> at com.webobjects.eocontrol.EOObjectStoreCoordinator.unlock(EOObjectStoreCoordinator.java:448)
>>> at com.webobjects.eocontrol.EOEditingContext.unlockObjectStore(EOEditingContext.java:4684)
>>> at er.extensions.eof.ERXEC.unlockObjectStore(ERXEC.java:805)
>>> at com.webobjects.eocontrol.EOCustomObject.willReadRelationship(EOCustomObject.java:1281)
>>> at er.extensions.eof.ERXGenericRecord.willReadRelationship(ERXGenericRecord.java:378)
>>> at com.webobjects.eocontrol._EOMutableKnownKeyDictionary$Initializer$_LazyGenericRecordBinding.valueInObject(_EOMutableKnownKeyDictionary.java:614)
>>> at com.webobjects.eocontrol.EOCustomObject.storedValueForKey(EOCustomObject.java:1634)
>>> at com.tla.logic._RigaCarrello.clientiPerTicket(_RigaCarrello.java:275)
>>> at com.tla.logic.RigaCarrello.setPosti(RigaCarrello.java:94)
>>> at com.tla.logic.Carrello.cancellaTutto(Carrello.java:164)
>>> at com.tla.calendar.Session.terminate(Session.java:81)
>>> at com.webobjects.appserver.WOSession._terminateByTimeout(WOSession.java:610)
>>> at com.webobjects.appserver.WOSessionStore$_SessionTimeoutManager.run(WOSessionStore.java:115)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> Sorry for the italian Class Names and Methods, as I said I inherited that...
>>>
>>> The exception is caught, that printout is the printStackTrace of the
>>> exception...
>>> But anyway, I'm not sure that's the real problem, I found this morning
>>> an instance with 295 sessions (still alive) and the only exception
>>> logged was a weird
>>>
>>> er.transaction.adaptor.Exceptions - Database Exception occured: No
>>> operations allowed after connection closed.
>>>
>>> Connection was closed due to the following exception:
>>>
>>> ** BEGIN NESTED EXCEPTION **
>>>
>>> java.sql.SQLException
>>> MESSAGE: Communication link failure: java.io.EOFException, underlying
>>> cause: null
>>>
>>> ** BEGIN NESTED EXCEPTION **
>>>
>>> java.io.EOFException
>>>
>>> STACKTRACE:
>>>
>>> java.io.EOFException
>>> at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1394)
>>> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:1538)
>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:1929)
>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1167)
>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1278)
>>> at com.mysql.jdbc.Connection.execSQL(Connection.java:2247)
>>> at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1371)
>>> at com.webobjects.jdbcadaptor.JDBCChannel._bindInputVariablesWithBindingsAndExecute(JDBCChannel.java:265)
>>> at com.webobjects.jdbcadaptor.JDBCChannel._evaluateExpression(JDBCChannel.java:337)
>>> at com.webobjects.jdbcadaptor.JDBCChannel.evaluateExpression(JDBCChannel.java:296)
>>> at com.webobjects.jdbcadaptor.JDBCChannel.selectAttributes(JDBCChannel.java:220)
>>> at com.webobjects.eoaccess.EODatabaseChannel._selectWithFetchSpecificationEditingContext(EODatabaseChannel.java:897)
>>> at com.webobjects.eoaccess.EODatabaseChannel.selectObjectsWithFetchSpecification(EODatabaseChannel.java:234)
>>> at com.webobjects.eoaccess.EODatabaseContext._objectsWithFetchSpecificationEditingContext(EODatabaseContext.java:3055)
>>> at com.webobjects.eoaccess.EODatabaseContext.objectsWithFetchSpecification(EODatabaseContext.java:3195)
>>> at com.webobjects.eocontrol.EOObjectStoreCoordinator.objectsWithFetchSpecification(EOObjectStoreCoordinator.java:488)
>>> at com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEditingContext.java:4069)
>>> at er.extensions.eof.ERXEC.objectsWithFetchSpecification(ERXEC.java:1305)
>>> at com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEditingContext.java:4444)
>>> at com.tla.logic.Performance.fetchFreshPerformancesWithQualifier(Performance.java:414)
>>> at com.tla.logic.Carrello.cancellaTutto(Carrello.java:154)
>>> at com.tla.calendar.components.ALTComponent.invokeAction(ALTComponent.java:167)
>>> at com.webobjects.appserver.WOSession.invokeAction(WOSession.java:1357)
>>> at com.webobjects.appserver.WOApplication.invokeAction(WOApplication.java:1745)
>>> at er.extensions.appserver.ajax.ERXAjaxApplication.invokeAction(ERXAjaxApplication.java:117)
>>> at er.extensions.appserver.ERXApplication.invokeAction(ERXApplication.java:2018)
>>> at er.extensions.appserver.ERXComponentRequestHandler._dispatchWithPreparedPage(ERXComponentRequestHandler.java:157)
>>> at er.extensions.appserver.ERXComponentRequestHandler._dispatchWithPreparedSession(ERXComponentRequestHandler.java:235)
>>> at er.extensions.appserver.ERXComponentRequestHandler._dispatchWithPreparedApplication(ERXComponentRequestHandler.java:268)
>>> at er.extensions.appserver.ERXComponentRequestHandler._handleRequest(ERXComponentRequestHandler.java:302)
>>> at er.extensions.appserver.ERXComponentRequestHandler.handleRequest(ERXComponentRequestHandler.java:377)
>>> at com.webobjects.appserver.WOApplication.dispatchRequest(WOApplication.java:1687)
>>> at er.extensions.appserver.ERXApplication.dispatchRequestImmediately(ERXApplication.java:2139)
>>> at er.extensions.appserver.ERXApplication.dispatchRequest(ERXApplication.java:2104)
>>> at com.webobjects.appserver._private.WOWorkerThread.runOnce(WOWorkerThread.java:144)
>>> at com.webobjects.appserver._private.WOWorkerThread.run(WOWorkerThread.java:226)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> So I'm not that confident that I am on the right track...
>>> Looks like if anything goes wrong in the life of the instance sessions
>>> won't die... Which is kind of disturbing.
>>>
>>>
>>> Matteo
>>>
>>> On Sun, Jan 13, 2013 at 7:01 PM, Chuck Hill <email@hidden> wrote:
>>>> Can you post the entire stack trace?
>>>>
>>>> At the very least, I would catch that exception and now allow it out of your finalize method.
>>>
>>>
>>> On Sun, Jan 13, 2013 at 7:01 PM, Chuck Hill <email@hidden> wrote:
>>>> Can you post the entire stack trace?
>>>>
>>>> At the very least, I would catch that exception and now allow it out of your finalize method.
>>>>
>>>>
>>>> On 2013-01-12, at 2:01 PM, Matteo Centro wrote:
>>>>
>>>>> Thanks, I'll try that.
>>>>> Anyway (I can't say it's that but this happens often in instances with
>>>>> immortal sessions):
>>>>> I get this in my terminate() method
>>>>> java.lang.IllegalMonitorStateException
>>>>> at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:127)
>>>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239)
>>>>> at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431)
>>>>> at com.webobjects.eocontrol.EOObjectStoreCoordinator.unlock(EOObjectStoreCoordinator.java:460)
>>>>> at com.webobjects.eocontrol.EOEditingContext.unlockObjectStore(EOEditingContext.java:4684)
>>>>> at er.extensions.eof.ERXEC.unlockObjectStore(ERXEC.java:805)
>>>>> at com.webobjects.eocontrol.EOCustomObject.willReadRelationship(EOCustomObject.java:1281)
>>>>> at er.extensions.eof.ERXGenericRecord.willReadRelationship(ERXGenericRecord.java:378)
>>>>> at com.webobjects.eocontrol._EOMutableKnownKeyDictionary$Initializer$_LazyGenericRecordBinding.valueInObject(_EOMutableKnownKeyDictionary.java:614)
>>>>> at com.webobjects.eocontrol.EOCustomObject.storedValueForKey(EOCustomObject.java:1634)
>>>>>
>>>>> I'm using ERXEC everywhere and I'm not locking explicitly, should I
>>>>> lock the ec in the terminate method?
>>>>>
>>>>>
>>>>> Matteo
>>>>>
>>>>> On Sat, Jan 12, 2013 at 8:39 PM, Simon <email@hidden> wrote:
>>>>>> stick this in your session constructor, it will log out whenever a
>>>>>> session is created. then you can start figuring out where you are
>>>>>> creating a session outside the RR loop which is the normal culprit...
>>>>>>
>>>>>> StringWriter stringWriter = new StringWriter();
>>>>>> PrintWriter printWriter = new PrintWriter(stringWriter);
>>>>>> (new Throwable()).printStackTrace(printWriter);
>>>>>> String trace = stringWriter.toString();
>>>>>> log.debug("Session count = " + application().activeSessionsCount());
>>>>>> ClickEventLogger2.getLogger().fatal(ClickEventCode.E00129, "Session
>>>>>> Created:\n\n " + trace,
>>>>>> this.getClass().getSimpleName());
>>>>>>
>>>>>> hmmm, you'll have to replace the "ClickEventLogger2" line because
>>>>>> that's our logging mechanism. you could log.fatal it and set up an
>>>>>> email appender in log4j.
>>>>>>
>>>>>> simon
>>>>>>
>>>>>>
>>>>>> On 12 January 2013 17:43, Matteo Centro <email@hidden> wrote:
>>>>>>> Sorry to resuscitate such an old discussion but I'm having the exact
>>>>>>> same issue...
>>>>>>> It's an old application that we "inherited", we wonderized it as much
>>>>>>> as it's possible but something weird happens in production, sessions
>>>>>>> on some instances simply won't die.
>>>>>>> Some instances go out of memory and they hang there.
>>>>>>> I'm in trouble and I needs some hints, both to fix the issue
>>>>>>> temporarily and to fix it for good (of course in that case I assume
>>>>>>> I'll have to rewrite the app, if only I could find the budget).
>>>>>>> What are the most common causes of sessions not dying?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>>
>>>>>>> Matteo
>>>>>>>
>>>>>>> On Thu, Aug 21, 2008 at 5:35 AM, Joe Little <email@hidden> wrote:
>>>>>>>> I had something similar with sessions going bonkers on a public WO
>>>>>>>> page that our internal google search engine completely trashed. In the
>>>>>>>> end, robots.txt and explicit rules to deny certain patterns were added
>>>>>>>> to prevent this.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Aug 20, 2008 at 8:17 PM, D Tim Cummings <email@hidden> wrote:
>>>>>>>>> We have a couple of sessionless apps that have started showing this problem
>>>>>>>>> with sessions that don't terminate. It turned out the sessions were being
>>>>>>>>> created by malformed urls coming from malicious robot web crawlers. The
>>>>>>>>> urls were of the form
>>>>>>>>> http://www.courses.qut.edu.au/cgi-bin/WebObjects/Courses.woa/wa/cgi-bin/WebObjects/Courses.woa
>>>>>>>>> Maybe see if you are getting incorrect links to your sessionless login page.
>>>>>>>>> We solved the problem by catching unknown direct actions in
>>>>>>>>> DirectAction.java
>>>>>>>>> @Override
>>>>>>>>> public WOActionResults performActionNamed(String actionName) {
>>>>>>>>> try {
>>>>>>>>> return super.performActionNamed(actionName);
>>>>>>>>> } catch (NSForwardException nsfe) {
>>>>>>>>> log.info("ns forward exception - prbalby no such method for " + actionName);
>>>>>>>>> }
>>>>>>>>> return defaultAction();
>>>>>>>>> }
>>>>>>>>> and in Application.java directing exceptions back to the Main page (for URLs
>>>>>>>>> with more than one / after wa).
>>>>>>>>> @Override
>>>>>>>>> public WOComponent pageWithName(String namePage, WOContext context) {
>>>>>>>>> if ( "WOExceptionPage".equals(namePage) ) {
>>>>>>>>> namePage = "Main";
>>>>>>>>> }
>>>>>>>>> if ( "WOSessionRestorationError".equals(namePage) ) {
>>>>>>>>> namePage = "Main";
>>>>>>>>> }
>>>>>>>>> return super.pageWithName(namePage, context);
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> and in Main.java
>>>>>>>>> public void setException ( Exception e ) {
>>>>>>>>> log.error("an exception occurred " + e);
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> We are running apps with embedded Wonder 4 and WebObjects 5.3.3 on Mac OS X
>>>>>>>>> Server 10.5.4 with WebObjects 5.4.2 deployment. We didn't have the problem
>>>>>>>>> before we went to this setup, but maybe we weren't getting hit with the same
>>>>>>>>> url format then.
>>>>>>>>> Tim
>>>>>>>>> On 21/08/2008, at 4:02 AM, Chuck Hill wrote:
>>>>>>>>>
>>>>>>>>> On Aug 20, 2008, at 9:54 AM, Simon McLean wrote:
>>>>>>>>>
>>>>>>>>> Hi All -
>>>>>>>>> Wondering if someone can throw me some ideas on solving what looks like an
>>>>>>>>> immortal session problem.
>>>>>>>>> - Start up 1 instance in java monitor.
>>>>>>>>> - User lands on sessionless login page. No sessions.
>>>>>>>>> - User logs in. 1 session.
>>>>>>>>> - User logs out. 0 sessions.
>>>>>>>>> - User logs in. 1 session.
>>>>>>>>> - User does nothing. Session times out. 0 sessions.
>>>>>>>>> All looks good. However, we get to the end of the day and the instance has
>>>>>>>>> 376 active sessions. But I know for this particular app there are only 6
>>>>>>>>> users, because they are all sitting next door :-) Also, If i now leave this
>>>>>>>>> instance overnight none of those active sessions will go away.
>>>>>>>>> So what could be going on ? Something appears to be creating immortal
>>>>>>>>> sessions, but forced session termination (by the user logging out) and
>>>>>>>>> session expiration seem to be behaving themselves.
>>>>>>>>>
>>>>>>>>> If the users don't notice any problems, my money would be on an exception
>>>>>>>>> thrown from your Session.terminate(). Also check the sleep() method to
>>>>>>>>> ensure it can never throw.
>>>>>>>>> I have seen a related problem where two requests come in for the same
>>>>>>>>> session (user double clicks, Ajax etc), and the first calls terminate() on
>>>>>>>>> the session. The second thread is stuck and the app can't gracefully shut
>>>>>>>>> down. IIRC, you would see zero sessions in this case. The "fix" for this
>>>>>>>>> is to not call terminate() and instead set the session timeout to a small
>>>>>>>>> number of seconds.
>>>>>>>>> Chuck
>>>>>>>>> --
>>>>>>>>> Chuck Hill Senior Consultant / VP Development
>>>>>>>>> Practical WebObjects - for developers who want to increase their overall
>>>>>>>>> knowledge of WebObjects or who are trying to solve specific problems.
>>>>>>>>> http://www.global-village.net/products/practical_webobjects
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>>>>> Webobjects-dev mailing list (email@hidden)
>>>>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>>>>> This email sent to email@hidden
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>>>>> Webobjects-dev mailing list (email@hidden)
>>>>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>>>>>
>>>>>>>>> This email sent to email@hidden
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>>>> Webobjects-dev mailing list (email@hidden)
>>>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>>>>
>>>>>>>> This email sent to email@hidden
>>>>>>> _______________________________________________
>>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>>> Webobjects-dev mailing list (email@hidden)
>>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>>>
>>>>>>> This email sent to email@hidden
>>>>> _______________________________________________
>>>>> Do not post admin requests to the list. They will be ignored.
>>>>> Webobjects-dev mailing list (email@hidden)
>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>
>>>>> This email sent to email@hidden
>>>>
>>>> --
>>>> Chuck Hill Senior Consultant / VP Development
>>>>
>>>> Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems.
>>>> http://www.global-village.net/gvc/practical_webobjects
>>>>
>>>> Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest Growing Companies in B.C!
>>>> Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking of Canada’s Fastest-Growing Companies by PROFIT Magazine!
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>> <jstackoutput.txt>
>
> --
> Chuck Hill Senior Consultant / VP Development
>
> Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems.
> http://www.global-village.net/gvc/practical_webobjects
>
> Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest Growing Companies in B.C!
> Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking of Canada’s Fastest-Growing Companies by PROFIT Magazine!
>
>
>
>
>
>
>
>
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden