Re: Immortal sessions in production
Re: Immortal sessions in production
- Subject: Re: Immortal sessions in production
- From: Matteo Centro <email@hidden>
- Date: Wed, 30 Jan 2013 17:56:11 +0100
From what I see some instances don't log the DB Connection error and
still have immortal sessions.
The main problems with immortal sessions is that eventually they end
up with a memory problem, I recall there is a way to make the instance
restart if memory errors happen but I can't remember how.
I'm still in dire straits, no idea how to fix this... no idea what's
happening either. Any suggestions?
Matteo
On Tue, Jan 15, 2013 at 6:14 PM, Chuck Hill <email@hidden> wrote:
>
> On 2013-01-15, at 3:20 AM, Matteo Centro wrote:
>
>> Unfortunately I don't have full control of the deployment
>> environment... It looks like at some times the DB stops responding
>> with no apparent reason.
>> Could a simple DB Connection glitch cause a EOObjectStoreCoordinator lock?
>
> I recall a bug at some level related to this. I could be wrong.
>
>
>> Maybe I can tweak the connection dictionary to enable auto reconnect.
>> I'll try
>
> Chuck
>
>
>> On Mon, Jan 14, 2013 at 8:17 PM, Chuck Hill <email@hidden> wrote:
>>> Hi Matteo,
>>>
>>> Something locked the EOObjectStoreCoordinator and did not unlock it. Does the log for this instance show any exceptions?
>>>
>>> This exception could perhaps cause this problem:
>>>
>>>>> ava.io.EOFException
>>>>> at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1394)
>>>>> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:1538)
>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:1929)
>>>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1167)
>>>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1278)
>>>>> at com.mysql.jdbc.Connection.execSQL(Connection.java:2247)
>>>>> at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1371)
>>>>> at com.webobjects.jdbcadaptor.JDBCChannel._bindInputVariablesWithBindingsAndExecute(JDBCChannel.java:265)
>>>>> at com.webobjects.jdbcadaptor.JDBCChannel._evaluateExpression(JDBCChannel.java:337)
>>>>> at com.webobjects.jdbcadaptor.JDBCChannel.evaluateExpression(JDBCChannel.java:296)
>>>>> at com.webobjects.jdbcadaptor.JDBCChannel.selectAttributes(JDBCChannel.java:220)
>>>>> at com.webobjects.eoaccess.EODatabaseChannel._selectWithFetchSpecificationEditingContext(EODatabaseChannel.java:897)
>>>>> at com.webobjects.eoaccess.EODatabaseChannel.selectObjectsWithFetchSpecification(EODatabaseChannel.java:234)
>>>>> at com.webobjects.eoaccess.EODatabaseContext._objectsWithFetchSpecificationEditingContext(EODatabaseContext.java:3055)
>>>>> at com.webobjects.eoaccess.EODatabaseContext.objectsWithFetchSpecification(EODatabaseContext.java:3195)
>>>>> at com.webobjects.eocontrol.EOObjectStoreCoordinator.objectsWithFetchSpecification(EOObjectStoreCoordinator.java:488)
>>>>> at com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEditingContext.java:4069)
>>>>> at er.extensions.eof.ERXEC.objectsWithFetchSpecification(ERXEC.java:1305)
>>>>> at com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEditingContext.java:4444)
>>>
>>>
>>> Chuck
>>>
>>>
>>> On 2013-01-14, at 10:05 AM, Matteo Centro wrote:
>>>
>>>> Hi all, suspecting there was some sort of deadlock I ran jstack
>>>> against one of the immortal sessions instances, I'm attaching the
>>>> output, anything helps...
>>>>
>>>> Thanks,
>>>>
>>>>
>>>> Matteo
>>>>
>>>> On Mon, Jan 14, 2013 at 12:35 PM, Matteo Centro <email@hidden> wrote:
>>>>> Sure, here it is:
>>>>> Hi Chuck,
>>>>>
>>>>> I'm posting just to you, I can't have google to index this...
>>>>>
>>>>> anyway, here is the stack trace:
>>>>> java.lang.IllegalMonitorStateException
>>>>> at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:127)
>>>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239)
>>>>> at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431)
>>>>> at com.webobjects.eocontrol.EOObjectStoreCoordinator.unlock(EOObjectStoreCoordinator.java:448)
>>>>> at com.webobjects.eocontrol.EOEditingContext.unlockObjectStore(EOEditingContext.java:4684)
>>>>> at er.extensions.eof.ERXEC.unlockObjectStore(ERXEC.java:805)
>>>>> at com.webobjects.eocontrol.EOCustomObject.willReadRelationship(EOCustomObject.java:1281)
>>>>> at er.extensions.eof.ERXGenericRecord.willReadRelationship(ERXGenericRecord.java:378)
>>>>> at com.webobjects.eocontrol._EOMutableKnownKeyDictionary$Initializer$_LazyGenericRecordBinding.valueInObject(_EOMutableKnownKeyDictionary.java:614)
>>>>> at com.webobjects.eocontrol.EOCustomObject.storedValueForKey(EOCustomObject.java:1634)
>>>>> at com.tla.logic._RigaCarrello.clientiPerTicket(_RigaCarrello.java:275)
>>>>> at com.tla.logic.RigaCarrello.setPosti(RigaCarrello.java:94)
>>>>> at com.tla.logic.Carrello.cancellaTutto(Carrello.java:164)
>>>>> at com.tla.calendar.Session.terminate(Session.java:81)
>>>>> at com.webobjects.appserver.WOSession._terminateByTimeout(WOSession.java:610)
>>>>> at com.webobjects.appserver.WOSessionStore$_SessionTimeoutManager.run(WOSessionStore.java:115)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> Sorry for the italian Class Names and Methods, as I said I inherited that...
>>>>>
>>>>> The exception is caught, that printout is the printStackTrace of the
>>>>> exception...
>>>>> But anyway, I'm not sure that's the real problem, I found this morning
>>>>> an instance with 295 sessions (still alive) and the only exception
>>>>> logged was a weird
>>>>>
>>>>> er.transaction.adaptor.Exceptions - Database Exception occured: No
>>>>> operations allowed after connection closed.
>>>>>
>>>>> Connection was closed due to the following exception:
>>>>>
>>>>> ** BEGIN NESTED EXCEPTION **
>>>>>
>>>>> java.sql.SQLException
>>>>> MESSAGE: Communication link failure: java.io.EOFException, underlying
>>>>> cause: null
>>>>>
>>>>> ** BEGIN NESTED EXCEPTION **
>>>>>
>>>>> java.io.EOFException
>>>>>
>>>>> STACKTRACE:
>>>>>
>>>>> java.io.EOFException
>>>>> at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1394)
>>>>> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:1538)
>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:1929)
>>>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1167)
>>>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1278)
>>>>> at com.mysql.jdbc.Connection.execSQL(Connection.java:2247)
>>>>> at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1371)
>>>>> at com.webobjects.jdbcadaptor.JDBCChannel._bindInputVariablesWithBindingsAndExecute(JDBCChannel.java:265)
>>>>> at com.webobjects.jdbcadaptor.JDBCChannel._evaluateExpression(JDBCChannel.java:337)
>>>>> at com.webobjects.jdbcadaptor.JDBCChannel.evaluateExpression(JDBCChannel.java:296)
>>>>> at com.webobjects.jdbcadaptor.JDBCChannel.selectAttributes(JDBCChannel.java:220)
>>>>> at com.webobjects.eoaccess.EODatabaseChannel._selectWithFetchSpecificationEditingContext(EODatabaseChannel.java:897)
>>>>> at com.webobjects.eoaccess.EODatabaseChannel.selectObjectsWithFetchSpecification(EODatabaseChannel.java:234)
>>>>> at com.webobjects.eoaccess.EODatabaseContext._objectsWithFetchSpecificationEditingContext(EODatabaseContext.java:3055)
>>>>> at com.webobjects.eoaccess.EODatabaseContext.objectsWithFetchSpecification(EODatabaseContext.java:3195)
>>>>> at com.webobjects.eocontrol.EOObjectStoreCoordinator.objectsWithFetchSpecification(EOObjectStoreCoordinator.java:488)
>>>>> at com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEditingContext.java:4069)
>>>>> at er.extensions.eof.ERXEC.objectsWithFetchSpecification(ERXEC.java:1305)
>>>>> at com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEditingContext.java:4444)
>>>>> at com.tla.logic.Performance.fetchFreshPerformancesWithQualifier(Performance.java:414)
>>>>> at com.tla.logic.Carrello.cancellaTutto(Carrello.java:154)
>>>>> at com.tla.calendar.components.ALTComponent.invokeAction(ALTComponent.java:167)
>>>>> at com.webobjects.appserver.WOSession.invokeAction(WOSession.java:1357)
>>>>> at com.webobjects.appserver.WOApplication.invokeAction(WOApplication.java:1745)
>>>>> at er.extensions.appserver.ajax.ERXAjaxApplication.invokeAction(ERXAjaxApplication.java:117)
>>>>> at er.extensions.appserver.ERXApplication.invokeAction(ERXApplication.java:2018)
>>>>> at er.extensions.appserver.ERXComponentRequestHandler._dispatchWithPreparedPage(ERXComponentRequestHandler.java:157)
>>>>> at er.extensions.appserver.ERXComponentRequestHandler._dispatchWithPreparedSession(ERXComponentRequestHandler.java:235)
>>>>> at er.extensions.appserver.ERXComponentRequestHandler._dispatchWithPreparedApplication(ERXComponentRequestHandler.java:268)
>>>>> at er.extensions.appserver.ERXComponentRequestHandler._handleRequest(ERXComponentRequestHandler.java:302)
>>>>> at er.extensions.appserver.ERXComponentRequestHandler.handleRequest(ERXComponentRequestHandler.java:377)
>>>>> at com.webobjects.appserver.WOApplication.dispatchRequest(WOApplication.java:1687)
>>>>> at er.extensions.appserver.ERXApplication.dispatchRequestImmediately(ERXApplication.java:2139)
>>>>> at er.extensions.appserver.ERXApplication.dispatchRequest(ERXApplication.java:2104)
>>>>> at com.webobjects.appserver._private.WOWorkerThread.runOnce(WOWorkerThread.java:144)
>>>>> at com.webobjects.appserver._private.WOWorkerThread.run(WOWorkerThread.java:226)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> So I'm not that confident that I am on the right track...
>>>>> Looks like if anything goes wrong in the life of the instance sessions
>>>>> won't die... Which is kind of disturbing.
>>>>>
>>>>>
>>>>> Matteo
>>>>>
>>>>> On Sun, Jan 13, 2013 at 7:01 PM, Chuck Hill <email@hidden> wrote:
>>>>>> Can you post the entire stack trace?
>>>>>>
>>>>>> At the very least, I would catch that exception and now allow it out of your finalize method.
>>>>>
>>>>>
>>>>> On Sun, Jan 13, 2013 at 7:01 PM, Chuck Hill <email@hidden> wrote:
>>>>>> Can you post the entire stack trace?
>>>>>>
>>>>>> At the very least, I would catch that exception and now allow it out of your finalize method.
>>>>>>
>>>>>>
>>>>>> On 2013-01-12, at 2:01 PM, Matteo Centro wrote:
>>>>>>
>>>>>>> Thanks, I'll try that.
>>>>>>> Anyway (I can't say it's that but this happens often in instances with
>>>>>>> immortal sessions):
>>>>>>> I get this in my terminate() method
>>>>>>> java.lang.IllegalMonitorStateException
>>>>>>> at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:127)
>>>>>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239)
>>>>>>> at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431)
>>>>>>> at com.webobjects.eocontrol.EOObjectStoreCoordinator.unlock(EOObjectStoreCoordinator.java:460)
>>>>>>> at com.webobjects.eocontrol.EOEditingContext.unlockObjectStore(EOEditingContext.java:4684)
>>>>>>> at er.extensions.eof.ERXEC.unlockObjectStore(ERXEC.java:805)
>>>>>>> at com.webobjects.eocontrol.EOCustomObject.willReadRelationship(EOCustomObject.java:1281)
>>>>>>> at er.extensions.eof.ERXGenericRecord.willReadRelationship(ERXGenericRecord.java:378)
>>>>>>> at com.webobjects.eocontrol._EOMutableKnownKeyDictionary$Initializer$_LazyGenericRecordBinding.valueInObject(_EOMutableKnownKeyDictionary.java:614)
>>>>>>> at com.webobjects.eocontrol.EOCustomObject.storedValueForKey(EOCustomObject.java:1634)
>>>>>>>
>>>>>>> I'm using ERXEC everywhere and I'm not locking explicitly, should I
>>>>>>> lock the ec in the terminate method?
>>>>>>>
>>>>>>>
>>>>>>> Matteo
>>>>>>>
>>>>>>> On Sat, Jan 12, 2013 at 8:39 PM, Simon <email@hidden> wrote:
>>>>>>>> stick this in your session constructor, it will log out whenever a
>>>>>>>> session is created. then you can start figuring out where you are
>>>>>>>> creating a session outside the RR loop which is the normal culprit...
>>>>>>>>
>>>>>>>> StringWriter stringWriter = new StringWriter();
>>>>>>>> PrintWriter printWriter = new PrintWriter(stringWriter);
>>>>>>>> (new Throwable()).printStackTrace(printWriter);
>>>>>>>> String trace = stringWriter.toString();
>>>>>>>> log.debug("Session count = " + application().activeSessionsCount());
>>>>>>>> ClickEventLogger2.getLogger().fatal(ClickEventCode.E00129, "Session
>>>>>>>> Created:\n\n " + trace,
>>>>>>>> this.getClass().getSimpleName());
>>>>>>>>
>>>>>>>> hmmm, you'll have to replace the "ClickEventLogger2" line because
>>>>>>>> that's our logging mechanism. you could log.fatal it and set up an
>>>>>>>> email appender in log4j.
>>>>>>>>
>>>>>>>> simon
>>>>>>>>
>>>>>>>>
>>>>>>>> On 12 January 2013 17:43, Matteo Centro <email@hidden> wrote:
>>>>>>>>> Sorry to resuscitate such an old discussion but I'm having the exact
>>>>>>>>> same issue...
>>>>>>>>> It's an old application that we "inherited", we wonderized it as much
>>>>>>>>> as it's possible but something weird happens in production, sessions
>>>>>>>>> on some instances simply won't die.
>>>>>>>>> Some instances go out of memory and they hang there.
>>>>>>>>> I'm in trouble and I needs some hints, both to fix the issue
>>>>>>>>> temporarily and to fix it for good (of course in that case I assume
>>>>>>>>> I'll have to rewrite the app, if only I could find the budget).
>>>>>>>>> What are the most common causes of sessions not dying?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Matteo
>>>>>>>>>
>>>>>>>>> On Thu, Aug 21, 2008 at 5:35 AM, Joe Little <email@hidden> wrote:
>>>>>>>>>> I had something similar with sessions going bonkers on a public WO
>>>>>>>>>> page that our internal google search engine completely trashed. In the
>>>>>>>>>> end, robots.txt and explicit rules to deny certain patterns were added
>>>>>>>>>> to prevent this.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Aug 20, 2008 at 8:17 PM, D Tim Cummings <email@hidden> wrote:
>>>>>>>>>>> We have a couple of sessionless apps that have started showing this problem
>>>>>>>>>>> with sessions that don't terminate. It turned out the sessions were being
>>>>>>>>>>> created by malformed urls coming from malicious robot web crawlers. The
>>>>>>>>>>> urls were of the form
>>>>>>>>>>> http://www.courses.qut.edu.au/cgi-bin/WebObjects/Courses.woa/wa/cgi-bin/WebObjects/Courses.woa
>>>>>>>>>>> Maybe see if you are getting incorrect links to your sessionless login page.
>>>>>>>>>>> We solved the problem by catching unknown direct actions in
>>>>>>>>>>> DirectAction.java
>>>>>>>>>>> @Override
>>>>>>>>>>> public WOActionResults performActionNamed(String actionName) {
>>>>>>>>>>> try {
>>>>>>>>>>> return super.performActionNamed(actionName);
>>>>>>>>>>> } catch (NSForwardException nsfe) {
>>>>>>>>>>> log.info("ns forward exception - prbalby no such method for " + actionName);
>>>>>>>>>>> }
>>>>>>>>>>> return defaultAction();
>>>>>>>>>>> }
>>>>>>>>>>> and in Application.java directing exceptions back to the Main page (for URLs
>>>>>>>>>>> with more than one / after wa).
>>>>>>>>>>> @Override
>>>>>>>>>>> public WOComponent pageWithName(String namePage, WOContext context) {
>>>>>>>>>>> if ( "WOExceptionPage".equals(namePage) ) {
>>>>>>>>>>> namePage = "Main";
>>>>>>>>>>> }
>>>>>>>>>>> if ( "WOSessionRestorationError".equals(namePage) ) {
>>>>>>>>>>> namePage = "Main";
>>>>>>>>>>> }
>>>>>>>>>>> return super.pageWithName(namePage, context);
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> and in Main.java
>>>>>>>>>>> public void setException ( Exception e ) {
>>>>>>>>>>> log.error("an exception occurred " + e);
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> We are running apps with embedded Wonder 4 and WebObjects 5.3.3 on Mac OS X
>>>>>>>>>>> Server 10.5.4 with WebObjects 5.4.2 deployment. We didn't have the problem
>>>>>>>>>>> before we went to this setup, but maybe we weren't getting hit with the same
>>>>>>>>>>> url format then.
>>>>>>>>>>> Tim
>>>>>>>>>>> On 21/08/2008, at 4:02 AM, Chuck Hill wrote:
>>>>>>>>>>>
>>>>>>>>>>> On Aug 20, 2008, at 9:54 AM, Simon McLean wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi All -
>>>>>>>>>>> Wondering if someone can throw me some ideas on solving what looks like an
>>>>>>>>>>> immortal session problem.
>>>>>>>>>>> - Start up 1 instance in java monitor.
>>>>>>>>>>> - User lands on sessionless login page. No sessions.
>>>>>>>>>>> - User logs in. 1 session.
>>>>>>>>>>> - User logs out. 0 sessions.
>>>>>>>>>>> - User logs in. 1 session.
>>>>>>>>>>> - User does nothing. Session times out. 0 sessions.
>>>>>>>>>>> All looks good. However, we get to the end of the day and the instance has
>>>>>>>>>>> 376 active sessions. But I know for this particular app there are only 6
>>>>>>>>>>> users, because they are all sitting next door :-) Also, If i now leave this
>>>>>>>>>>> instance overnight none of those active sessions will go away.
>>>>>>>>>>> So what could be going on ? Something appears to be creating immortal
>>>>>>>>>>> sessions, but forced session termination (by the user logging out) and
>>>>>>>>>>> session expiration seem to be behaving themselves.
>>>>>>>>>>>
>>>>>>>>>>> If the users don't notice any problems, my money would be on an exception
>>>>>>>>>>> thrown from your Session.terminate(). Also check the sleep() method to
>>>>>>>>>>> ensure it can never throw.
>>>>>>>>>>> I have seen a related problem where two requests come in for the same
>>>>>>>>>>> session (user double clicks, Ajax etc), and the first calls terminate() on
>>>>>>>>>>> the session. The second thread is stuck and the app can't gracefully shut
>>>>>>>>>>> down. IIRC, you would see zero sessions in this case. The "fix" for this
>>>>>>>>>>> is to not call terminate() and instead set the session timeout to a small
>>>>>>>>>>> number of seconds.
>>>>>>>>>>> Chuck
>>>>>>>>>>> --
>>>>>>>>>>> Chuck Hill Senior Consultant / VP Development
>>>>>>>>>>> Practical WebObjects - for developers who want to increase their overall
>>>>>>>>>>> knowledge of WebObjects or who are trying to solve specific problems.
>>>>>>>>>>> http://www.global-village.net/products/practical_webobjects
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>>>>>>> Webobjects-dev mailing list (email@hidden)
>>>>>>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>>>>>>> This email sent to email@hidden
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>>>>>>> Webobjects-dev mailing list (email@hidden)
>>>>>>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>>>>>>>
>>>>>>>>>>> This email sent to email@hidden
>>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>>>>>> Webobjects-dev mailing list (email@hidden)
>>>>>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>>>>>>
>>>>>>>>>> This email sent to email@hidden
>>>>>>>>> _______________________________________________
>>>>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>>>>> Webobjects-dev mailing list (email@hidden)
>>>>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>>>>>
>>>>>>>>> This email sent to email@hidden
>>>>>>> _______________________________________________
>>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>>> Webobjects-dev mailing list (email@hidden)
>>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>>>
>>>>>>> This email sent to email@hidden
>>>>>>
>>>>>> --
>>>>>> Chuck Hill Senior Consultant / VP Development
>>>>>>
>>>>>> Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems.
>>>>>> http://www.global-village.net/gvc/practical_webobjects
>>>>>>
>>>>>> Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest Growing Companies in B.C!
>>>>>> Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking of Canada’s Fastest-Growing Companies by PROFIT Magazine!
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> <jstackoutput.txt>
>>>
>>> --
>>> Chuck Hill Senior Consultant / VP Development
>>>
>>> Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems.
>>> http://www.global-village.net/gvc/practical_webobjects
>>>
>>> Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest Growing Companies in B.C!
>>> Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking of Canada’s Fastest-Growing Companies by PROFIT Magazine!
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>
> --
> Chuck Hill Senior Consultant / VP Development
>
> Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems.
> http://www.global-village.net/gvc/practical_webobjects
>
> Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest Growing Companies in B.C!
> Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking of Canada’s Fastest-Growing Companies by PROFIT Magazine!
>
>
>
>
>
>
>
>
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden