Re: Immortal sessions in production
Re: Immortal sessions in production
- Subject: Re: Immortal sessions in production
- From: Matteo Centro <email@hidden>
- Date: Wed, 30 Jan 2013 17:59:57 +0100
Regarding the deployment environment (which is not under my control,
as I said before) There are 6 boxes:
2 webservers (one each for 2 different apps)
2 WO nodes with wonder's latest wotaskd
1 Mysql DB
1 JavaMonitor box
I noticed that the box that runs JavaMonitor doesn't have wotaskd
running, is this normal?
Matteo
On Wed, Jan 30, 2013 at 5:56 PM, Matteo Centro <email@hidden> wrote:
> From what I see some instances don't log the DB Connection error and
> still have immortal sessions.
> The main problems with immortal sessions is that eventually they end
> up with a memory problem, I recall there is a way to make the instance
> restart if memory errors happen but I can't remember how.
>
> I'm still in dire straits, no idea how to fix this... no idea what's
> happening either. Any suggestions?
>
> Matteo
>
> On Tue, Jan 15, 2013 at 6:14 PM, Chuck Hill <email@hidden> wrote:
>>
>> On 2013-01-15, at 3:20 AM, Matteo Centro wrote:
>>
>>> Unfortunately I don't have full control of the deployment
>>> environment... It looks like at some times the DB stops responding
>>> with no apparent reason.
>>> Could a simple DB Connection glitch cause a EOObjectStoreCoordinator lock?
>>
>> I recall a bug at some level related to this. I could be wrong.
>>
>>
>>> Maybe I can tweak the connection dictionary to enable auto reconnect.
>>> I'll try
>>
>> Chuck
>>
>>
>>> On Mon, Jan 14, 2013 at 8:17 PM, Chuck Hill <email@hidden> wrote:
>>>> Hi Matteo,
>>>>
>>>> Something locked the EOObjectStoreCoordinator and did not unlock it. Does the log for this instance show any exceptions?
>>>>
>>>> This exception could perhaps cause this problem:
>>>>
>>>>>> ava.io.EOFException
>>>>>> at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1394)
>>>>>> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:1538)
>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:1929)
>>>>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1167)
>>>>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1278)
>>>>>> at com.mysql.jdbc.Connection.execSQL(Connection.java:2247)
>>>>>> at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1371)
>>>>>> at com.webobjects.jdbcadaptor.JDBCChannel._bindInputVariablesWithBindingsAndExecute(JDBCChannel.java:265)
>>>>>> at com.webobjects.jdbcadaptor.JDBCChannel._evaluateExpression(JDBCChannel.java:337)
>>>>>> at com.webobjects.jdbcadaptor.JDBCChannel.evaluateExpression(JDBCChannel.java:296)
>>>>>> at com.webobjects.jdbcadaptor.JDBCChannel.selectAttributes(JDBCChannel.java:220)
>>>>>> at com.webobjects.eoaccess.EODatabaseChannel._selectWithFetchSpecificationEditingContext(EODatabaseChannel.java:897)
>>>>>> at com.webobjects.eoaccess.EODatabaseChannel.selectObjectsWithFetchSpecification(EODatabaseChannel.java:234)
>>>>>> at com.webobjects.eoaccess.EODatabaseContext._objectsWithFetchSpecificationEditingContext(EODatabaseContext.java:3055)
>>>>>> at com.webobjects.eoaccess.EODatabaseContext.objectsWithFetchSpecification(EODatabaseContext.java:3195)
>>>>>> at com.webobjects.eocontrol.EOObjectStoreCoordinator.objectsWithFetchSpecification(EOObjectStoreCoordinator.java:488)
>>>>>> at com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEditingContext.java:4069)
>>>>>> at er.extensions.eof.ERXEC.objectsWithFetchSpecification(ERXEC.java:1305)
>>>>>> at com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEditingContext.java:4444)
>>>>
>>>>
>>>> Chuck
>>>>
>>>>
>>>> On 2013-01-14, at 10:05 AM, Matteo Centro wrote:
>>>>
>>>>> Hi all, suspecting there was some sort of deadlock I ran jstack
>>>>> against one of the immortal sessions instances, I'm attaching the
>>>>> output, anything helps...
>>>>>
>>>>> Thanks,
>>>>>
>>>>>
>>>>> Matteo
>>>>>
>>>>> On Mon, Jan 14, 2013 at 12:35 PM, Matteo Centro <email@hidden> wrote:
>>>>>> Sure, here it is:
>>>>>> Hi Chuck,
>>>>>>
>>>>>> I'm posting just to you, I can't have google to index this...
>>>>>>
>>>>>> anyway, here is the stack trace:
>>>>>> java.lang.IllegalMonitorStateException
>>>>>> at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:127)
>>>>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239)
>>>>>> at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431)
>>>>>> at com.webobjects.eocontrol.EOObjectStoreCoordinator.unlock(EOObjectStoreCoordinator.java:448)
>>>>>> at com.webobjects.eocontrol.EOEditingContext.unlockObjectStore(EOEditingContext.java:4684)
>>>>>> at er.extensions.eof.ERXEC.unlockObjectStore(ERXEC.java:805)
>>>>>> at com.webobjects.eocontrol.EOCustomObject.willReadRelationship(EOCustomObject.java:1281)
>>>>>> at er.extensions.eof.ERXGenericRecord.willReadRelationship(ERXGenericRecord.java:378)
>>>>>> at com.webobjects.eocontrol._EOMutableKnownKeyDictionary$Initializer$_LazyGenericRecordBinding.valueInObject(_EOMutableKnownKeyDictionary.java:614)
>>>>>> at com.webobjects.eocontrol.EOCustomObject.storedValueForKey(EOCustomObject.java:1634)
>>>>>> at com.tla.logic._RigaCarrello.clientiPerTicket(_RigaCarrello.java:275)
>>>>>> at com.tla.logic.RigaCarrello.setPosti(RigaCarrello.java:94)
>>>>>> at com.tla.logic.Carrello.cancellaTutto(Carrello.java:164)
>>>>>> at com.tla.calendar.Session.terminate(Session.java:81)
>>>>>> at com.webobjects.appserver.WOSession._terminateByTimeout(WOSession.java:610)
>>>>>> at com.webobjects.appserver.WOSessionStore$_SessionTimeoutManager.run(WOSessionStore.java:115)
>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> Sorry for the italian Class Names and Methods, as I said I inherited that...
>>>>>>
>>>>>> The exception is caught, that printout is the printStackTrace of the
>>>>>> exception...
>>>>>> But anyway, I'm not sure that's the real problem, I found this morning
>>>>>> an instance with 295 sessions (still alive) and the only exception
>>>>>> logged was a weird
>>>>>>
>>>>>> er.transaction.adaptor.Exceptions - Database Exception occured: No
>>>>>> operations allowed after connection closed.
>>>>>>
>>>>>> Connection was closed due to the following exception:
>>>>>>
>>>>>> ** BEGIN NESTED EXCEPTION **
>>>>>>
>>>>>> java.sql.SQLException
>>>>>> MESSAGE: Communication link failure: java.io.EOFException, underlying
>>>>>> cause: null
>>>>>>
>>>>>> ** BEGIN NESTED EXCEPTION **
>>>>>>
>>>>>> java.io.EOFException
>>>>>>
>>>>>> STACKTRACE:
>>>>>>
>>>>>> java.io.EOFException
>>>>>> at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1394)
>>>>>> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:1538)
>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:1929)
>>>>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1167)
>>>>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1278)
>>>>>> at com.mysql.jdbc.Connection.execSQL(Connection.java:2247)
>>>>>> at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1371)
>>>>>> at com.webobjects.jdbcadaptor.JDBCChannel._bindInputVariablesWithBindingsAndExecute(JDBCChannel.java:265)
>>>>>> at com.webobjects.jdbcadaptor.JDBCChannel._evaluateExpression(JDBCChannel.java:337)
>>>>>> at com.webobjects.jdbcadaptor.JDBCChannel.evaluateExpression(JDBCChannel.java:296)
>>>>>> at com.webobjects.jdbcadaptor.JDBCChannel.selectAttributes(JDBCChannel.java:220)
>>>>>> at com.webobjects.eoaccess.EODatabaseChannel._selectWithFetchSpecificationEditingContext(EODatabaseChannel.java:897)
>>>>>> at com.webobjects.eoaccess.EODatabaseChannel.selectObjectsWithFetchSpecification(EODatabaseChannel.java:234)
>>>>>> at com.webobjects.eoaccess.EODatabaseContext._objectsWithFetchSpecificationEditingContext(EODatabaseContext.java:3055)
>>>>>> at com.webobjects.eoaccess.EODatabaseContext.objectsWithFetchSpecification(EODatabaseContext.java:3195)
>>>>>> at com.webobjects.eocontrol.EOObjectStoreCoordinator.objectsWithFetchSpecification(EOObjectStoreCoordinator.java:488)
>>>>>> at com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEditingContext.java:4069)
>>>>>> at er.extensions.eof.ERXEC.objectsWithFetchSpecification(ERXEC.java:1305)
>>>>>> at com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEditingContext.java:4444)
>>>>>> at com.tla.logic.Performance.fetchFreshPerformancesWithQualifier(Performance.java:414)
>>>>>> at com.tla.logic.Carrello.cancellaTutto(Carrello.java:154)
>>>>>> at com.tla.calendar.components.ALTComponent.invokeAction(ALTComponent.java:167)
>>>>>> at com.webobjects.appserver.WOSession.invokeAction(WOSession.java:1357)
>>>>>> at com.webobjects.appserver.WOApplication.invokeAction(WOApplication.java:1745)
>>>>>> at er.extensions.appserver.ajax.ERXAjaxApplication.invokeAction(ERXAjaxApplication.java:117)
>>>>>> at er.extensions.appserver.ERXApplication.invokeAction(ERXApplication.java:2018)
>>>>>> at er.extensions.appserver.ERXComponentRequestHandler._dispatchWithPreparedPage(ERXComponentRequestHandler.java:157)
>>>>>> at er.extensions.appserver.ERXComponentRequestHandler._dispatchWithPreparedSession(ERXComponentRequestHandler.java:235)
>>>>>> at er.extensions.appserver.ERXComponentRequestHandler._dispatchWithPreparedApplication(ERXComponentRequestHandler.java:268)
>>>>>> at er.extensions.appserver.ERXComponentRequestHandler._handleRequest(ERXComponentRequestHandler.java:302)
>>>>>> at er.extensions.appserver.ERXComponentRequestHandler.handleRequest(ERXComponentRequestHandler.java:377)
>>>>>> at com.webobjects.appserver.WOApplication.dispatchRequest(WOApplication.java:1687)
>>>>>> at er.extensions.appserver.ERXApplication.dispatchRequestImmediately(ERXApplication.java:2139)
>>>>>> at er.extensions.appserver.ERXApplication.dispatchRequest(ERXApplication.java:2104)
>>>>>> at com.webobjects.appserver._private.WOWorkerThread.runOnce(WOWorkerThread.java:144)
>>>>>> at com.webobjects.appserver._private.WOWorkerThread.run(WOWorkerThread.java:226)
>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> So I'm not that confident that I am on the right track...
>>>>>> Looks like if anything goes wrong in the life of the instance sessions
>>>>>> won't die... Which is kind of disturbing.
>>>>>>
>>>>>>
>>>>>> Matteo
>>>>>>
>>>>>> On Sun, Jan 13, 2013 at 7:01 PM, Chuck Hill <email@hidden> wrote:
>>>>>>> Can you post the entire stack trace?
>>>>>>>
>>>>>>> At the very least, I would catch that exception and now allow it out of your finalize method.
>>>>>>
>>>>>>
>>>>>> On Sun, Jan 13, 2013 at 7:01 PM, Chuck Hill <email@hidden> wrote:
>>>>>>> Can you post the entire stack trace?
>>>>>>>
>>>>>>> At the very least, I would catch that exception and now allow it out of your finalize method.
>>>>>>>
>>>>>>>
>>>>>>> On 2013-01-12, at 2:01 PM, Matteo Centro wrote:
>>>>>>>
>>>>>>>> Thanks, I'll try that.
>>>>>>>> Anyway (I can't say it's that but this happens often in instances with
>>>>>>>> immortal sessions):
>>>>>>>> I get this in my terminate() method
>>>>>>>> java.lang.IllegalMonitorStateException
>>>>>>>> at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:127)
>>>>>>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239)
>>>>>>>> at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431)
>>>>>>>> at com.webobjects.eocontrol.EOObjectStoreCoordinator.unlock(EOObjectStoreCoordinator.java:460)
>>>>>>>> at com.webobjects.eocontrol.EOEditingContext.unlockObjectStore(EOEditingContext.java:4684)
>>>>>>>> at er.extensions.eof.ERXEC.unlockObjectStore(ERXEC.java:805)
>>>>>>>> at com.webobjects.eocontrol.EOCustomObject.willReadRelationship(EOCustomObject.java:1281)
>>>>>>>> at er.extensions.eof.ERXGenericRecord.willReadRelationship(ERXGenericRecord.java:378)
>>>>>>>> at com.webobjects.eocontrol._EOMutableKnownKeyDictionary$Initializer$_LazyGenericRecordBinding.valueInObject(_EOMutableKnownKeyDictionary.java:614)
>>>>>>>> at com.webobjects.eocontrol.EOCustomObject.storedValueForKey(EOCustomObject.java:1634)
>>>>>>>>
>>>>>>>> I'm using ERXEC everywhere and I'm not locking explicitly, should I
>>>>>>>> lock the ec in the terminate method?
>>>>>>>>
>>>>>>>>
>>>>>>>> Matteo
>>>>>>>>
>>>>>>>> On Sat, Jan 12, 2013 at 8:39 PM, Simon <email@hidden> wrote:
>>>>>>>>> stick this in your session constructor, it will log out whenever a
>>>>>>>>> session is created. then you can start figuring out where you are
>>>>>>>>> creating a session outside the RR loop which is the normal culprit...
>>>>>>>>>
>>>>>>>>> StringWriter stringWriter = new StringWriter();
>>>>>>>>> PrintWriter printWriter = new PrintWriter(stringWriter);
>>>>>>>>> (new Throwable()).printStackTrace(printWriter);
>>>>>>>>> String trace = stringWriter.toString();
>>>>>>>>> log.debug("Session count = " + application().activeSessionsCount());
>>>>>>>>> ClickEventLogger2.getLogger().fatal(ClickEventCode.E00129, "Session
>>>>>>>>> Created:\n\n " + trace,
>>>>>>>>> this.getClass().getSimpleName());
>>>>>>>>>
>>>>>>>>> hmmm, you'll have to replace the "ClickEventLogger2" line because
>>>>>>>>> that's our logging mechanism. you could log.fatal it and set up an
>>>>>>>>> email appender in log4j.
>>>>>>>>>
>>>>>>>>> simon
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 12 January 2013 17:43, Matteo Centro <email@hidden> wrote:
>>>>>>>>>> Sorry to resuscitate such an old discussion but I'm having the exact
>>>>>>>>>> same issue...
>>>>>>>>>> It's an old application that we "inherited", we wonderized it as much
>>>>>>>>>> as it's possible but something weird happens in production, sessions
>>>>>>>>>> on some instances simply won't die.
>>>>>>>>>> Some instances go out of memory and they hang there.
>>>>>>>>>> I'm in trouble and I needs some hints, both to fix the issue
>>>>>>>>>> temporarily and to fix it for good (of course in that case I assume
>>>>>>>>>> I'll have to rewrite the app, if only I could find the budget).
>>>>>>>>>> What are the most common causes of sessions not dying?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Matteo
>>>>>>>>>>
>>>>>>>>>> On Thu, Aug 21, 2008 at 5:35 AM, Joe Little <email@hidden> wrote:
>>>>>>>>>>> I had something similar with sessions going bonkers on a public WO
>>>>>>>>>>> page that our internal google search engine completely trashed. In the
>>>>>>>>>>> end, robots.txt and explicit rules to deny certain patterns were added
>>>>>>>>>>> to prevent this.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Aug 20, 2008 at 8:17 PM, D Tim Cummings <email@hidden> wrote:
>>>>>>>>>>>> We have a couple of sessionless apps that have started showing this problem
>>>>>>>>>>>> with sessions that don't terminate. It turned out the sessions were being
>>>>>>>>>>>> created by malformed urls coming from malicious robot web crawlers. The
>>>>>>>>>>>> urls were of the form
>>>>>>>>>>>> http://www.courses.qut.edu.au/cgi-bin/WebObjects/Courses.woa/wa/cgi-bin/WebObjects/Courses.woa
>>>>>>>>>>>> Maybe see if you are getting incorrect links to your sessionless login page.
>>>>>>>>>>>> We solved the problem by catching unknown direct actions in
>>>>>>>>>>>> DirectAction.java
>>>>>>>>>>>> @Override
>>>>>>>>>>>> public WOActionResults performActionNamed(String actionName) {
>>>>>>>>>>>> try {
>>>>>>>>>>>> return super.performActionNamed(actionName);
>>>>>>>>>>>> } catch (NSForwardException nsfe) {
>>>>>>>>>>>> log.info("ns forward exception - prbalby no such method for " + actionName);
>>>>>>>>>>>> }
>>>>>>>>>>>> return defaultAction();
>>>>>>>>>>>> }
>>>>>>>>>>>> and in Application.java directing exceptions back to the Main page (for URLs
>>>>>>>>>>>> with more than one / after wa).
>>>>>>>>>>>> @Override
>>>>>>>>>>>> public WOComponent pageWithName(String namePage, WOContext context) {
>>>>>>>>>>>> if ( "WOExceptionPage".equals(namePage) ) {
>>>>>>>>>>>> namePage = "Main";
>>>>>>>>>>>> }
>>>>>>>>>>>> if ( "WOSessionRestorationError".equals(namePage) ) {
>>>>>>>>>>>> namePage = "Main";
>>>>>>>>>>>> }
>>>>>>>>>>>> return super.pageWithName(namePage, context);
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> and in Main.java
>>>>>>>>>>>> public void setException ( Exception e ) {
>>>>>>>>>>>> log.error("an exception occurred " + e);
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> We are running apps with embedded Wonder 4 and WebObjects 5.3.3 on Mac OS X
>>>>>>>>>>>> Server 10.5.4 with WebObjects 5.4.2 deployment. We didn't have the problem
>>>>>>>>>>>> before we went to this setup, but maybe we weren't getting hit with the same
>>>>>>>>>>>> url format then.
>>>>>>>>>>>> Tim
>>>>>>>>>>>> On 21/08/2008, at 4:02 AM, Chuck Hill wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> On Aug 20, 2008, at 9:54 AM, Simon McLean wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi All -
>>>>>>>>>>>> Wondering if someone can throw me some ideas on solving what looks like an
>>>>>>>>>>>> immortal session problem.
>>>>>>>>>>>> - Start up 1 instance in java monitor.
>>>>>>>>>>>> - User lands on sessionless login page. No sessions.
>>>>>>>>>>>> - User logs in. 1 session.
>>>>>>>>>>>> - User logs out. 0 sessions.
>>>>>>>>>>>> - User logs in. 1 session.
>>>>>>>>>>>> - User does nothing. Session times out. 0 sessions.
>>>>>>>>>>>> All looks good. However, we get to the end of the day and the instance has
>>>>>>>>>>>> 376 active sessions. But I know for this particular app there are only 6
>>>>>>>>>>>> users, because they are all sitting next door :-) Also, If i now leave this
>>>>>>>>>>>> instance overnight none of those active sessions will go away.
>>>>>>>>>>>> So what could be going on ? Something appears to be creating immortal
>>>>>>>>>>>> sessions, but forced session termination (by the user logging out) and
>>>>>>>>>>>> session expiration seem to be behaving themselves.
>>>>>>>>>>>>
>>>>>>>>>>>> If the users don't notice any problems, my money would be on an exception
>>>>>>>>>>>> thrown from your Session.terminate(). Also check the sleep() method to
>>>>>>>>>>>> ensure it can never throw.
>>>>>>>>>>>> I have seen a related problem where two requests come in for the same
>>>>>>>>>>>> session (user double clicks, Ajax etc), and the first calls terminate() on
>>>>>>>>>>>> the session. The second thread is stuck and the app can't gracefully shut
>>>>>>>>>>>> down. IIRC, you would see zero sessions in this case. The "fix" for this
>>>>>>>>>>>> is to not call terminate() and instead set the session timeout to a small
>>>>>>>>>>>> number of seconds.
>>>>>>>>>>>> Chuck
>>>>>>>>>>>> --
>>>>>>>>>>>> Chuck Hill Senior Consultant / VP Development
>>>>>>>>>>>> Practical WebObjects - for developers who want to increase their overall
>>>>>>>>>>>> knowledge of WebObjects or who are trying to solve specific problems.
>>>>>>>>>>>> http://www.global-village.net/products/practical_webobjects
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>>>>>>>> Webobjects-dev mailing list (email@hidden)
>>>>>>>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>>>>>>>> This email sent to email@hidden
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>>>>>>>> Webobjects-dev mailing list (email@hidden)
>>>>>>>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>>>>>>>>
>>>>>>>>>>>> This email sent to email@hidden
>>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>>>>>>> Webobjects-dev mailing list (email@hidden)
>>>>>>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>>>>>>>
>>>>>>>>>>> This email sent to email@hidden
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>>>>>> Webobjects-dev mailing list (email@hidden)
>>>>>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>>>>>>
>>>>>>>>>> This email sent to email@hidden
>>>>>>>> _______________________________________________
>>>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>>>> Webobjects-dev mailing list (email@hidden)
>>>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>>>>
>>>>>>>> This email sent to email@hidden
>>>>>>>
>>>>>>> --
>>>>>>> Chuck Hill Senior Consultant / VP Development
>>>>>>>
>>>>>>> Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems.
>>>>>>> http://www.global-village.net/gvc/practical_webobjects
>>>>>>>
>>>>>>> Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest Growing Companies in B.C!
>>>>>>> Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking of Canada’s Fastest-Growing Companies by PROFIT Magazine!
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>> <jstackoutput.txt>
>>>>
>>>> --
>>>> Chuck Hill Senior Consultant / VP Development
>>>>
>>>> Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems.
>>>> http://www.global-village.net/gvc/practical_webobjects
>>>>
>>>> Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest Growing Companies in B.C!
>>>> Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking of Canada’s Fastest-Growing Companies by PROFIT Magazine!
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>
>> --
>> Chuck Hill Senior Consultant / VP Development
>>
>> Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems.
>> http://www.global-village.net/gvc/practical_webobjects
>>
>> Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest Growing Companies in B.C!
>> Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking of Canada’s Fastest-Growing Companies by PROFIT Magazine!
>>
>>
>>
>>
>>
>>
>>
>>
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden