Re: Immortal sessions in production
Re: Immortal sessions in production
- Subject: Re: Immortal sessions in production
- From: Amedeo Mantica <email@hidden>
- Date: Wed, 30 Jan 2013 18:10:02 +0100
Sent from my iPhone
On 30/gen/2013, at 17:59, Matteo Centro <email@hidden> wrote:
> Regarding the deployment environment (which is not under my control,
> as I said before) There are 6 boxes:
> 2 webservers (one each for 2 different apps)
> 2 WO nodes with wonder's latest wotaskd
> 1 Mysql DB
> 1 JavaMonitor box
>
> I noticed that the box that runs JavaMonitor doesn't have wotaskd
> running, is this normal?
Yes. JavaMonitor don't have to be in the same machine that runs wotaskd
>
> Matteo
>
> On Wed, Jan 30, 2013 at 5:56 PM, Matteo Centro <email@hidden> wrote:
>> From what I see some instances don't log the DB Connection error and
>> still have immortal sessions.
>> The main problems with immortal sessions is that eventually they end
>> up with a memory problem, I recall there is a way to make the instance
>> restart if memory errors happen but I can't remember how.
>>
>> I'm still in dire straits, no idea how to fix this... no idea what's
>> happening either. Any suggestions?
>>
>> Matteo
>>
>> On Tue, Jan 15, 2013 at 6:14 PM, Chuck Hill <email@hidden> wrote:
>>>
>>> On 2013-01-15, at 3:20 AM, Matteo Centro wrote:
>>>
>>>> Unfortunately I don't have full control of the deployment
>>>> environment... It looks like at some times the DB stops responding
>>>> with no apparent reason.
>>>> Could a simple DB Connection glitch cause a EOObjectStoreCoordinator lock?
>>>
>>> I recall a bug at some level related to this. I could be wrong.
>>>
>>>
>>>> Maybe I can tweak the connection dictionary to enable auto reconnect.
>>>> I'll try
>>>
>>> Chuck
>>>
>>>
>>>> On Mon, Jan 14, 2013 at 8:17 PM, Chuck Hill <email@hidden> wrote:
>>>>> Hi Matteo,
>>>>>
>>>>> Something locked the EOObjectStoreCoordinator and did not unlock it. Does the log for this instance show any exceptions?
>>>>>
>>>>> This exception could perhaps cause this problem:
>>>>>
>>>>>>> ava.io.EOFException
>>>>>>> at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1394)
>>>>>>> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:1538)
>>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:1929)
>>>>>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1167)
>>>>>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1278)
>>>>>>> at com.mysql.jdbc.Connection.execSQL(Connection.java:2247)
>>>>>>> at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1371)
>>>>>>> at com.webobjects.jdbcadaptor.JDBCChannel._bindInputVariablesWithBindingsAndExecute(JDBCChannel.java:265)
>>>>>>> at com.webobjects.jdbcadaptor.JDBCChannel._evaluateExpression(JDBCChannel.java:337)
>>>>>>> at com.webobjects.jdbcadaptor.JDBCChannel.evaluateExpression(JDBCChannel.java:296)
>>>>>>> at com.webobjects.jdbcadaptor.JDBCChannel.selectAttributes(JDBCChannel.java:220)
>>>>>>> at com.webobjects.eoaccess.EODatabaseChannel._selectWithFetchSpecificationEditingContext(EODatabaseChannel.java:897)
>>>>>>> at com.webobjects.eoaccess.EODatabaseChannel.selectObjectsWithFetchSpecification(EODatabaseChannel.java:234)
>>>>>>> at com.webobjects.eoaccess.EODatabaseContext._objectsWithFetchSpecificationEditingContext(EODatabaseContext.java:3055)
>>>>>>> at com.webobjects.eoaccess.EODatabaseContext.objectsWithFetchSpecification(EODatabaseContext.java:3195)
>>>>>>> at com.webobjects.eocontrol.EOObjectStoreCoordinator.objectsWithFetchSpecification(EOObjectStoreCoordinator.java:488)
>>>>>>> at com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEditingContext.java:4069)
>>>>>>> at er.extensions.eof.ERXEC.objectsWithFetchSpecification(ERXEC.java:1305)
>>>>>>> at com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEditingContext.java:4444)
>>>>>
>>>>>
>>>>> Chuck
>>>>>
>>>>>
>>>>> On 2013-01-14, at 10:05 AM, Matteo Centro wrote:
>>>>>
>>>>>> Hi all, suspecting there was some sort of deadlock I ran jstack
>>>>>> against one of the immortal sessions instances, I'm attaching the
>>>>>> output, anything helps...
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>>
>>>>>> Matteo
>>>>>>
>>>>>> On Mon, Jan 14, 2013 at 12:35 PM, Matteo Centro <email@hidden> wrote:
>>>>>>> Sure, here it is:
>>>>>>> Hi Chuck,
>>>>>>>
>>>>>>> I'm posting just to you, I can't have google to index this...
>>>>>>>
>>>>>>> anyway, here is the stack trace:
>>>>>>> java.lang.IllegalMonitorStateException
>>>>>>> at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:127)
>>>>>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239)
>>>>>>> at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431)
>>>>>>> at com.webobjects.eocontrol.EOObjectStoreCoordinator.unlock(EOObjectStoreCoordinator.java:448)
>>>>>>> at com.webobjects.eocontrol.EOEditingContext.unlockObjectStore(EOEditingContext.java:4684)
>>>>>>> at er.extensions.eof.ERXEC.unlockObjectStore(ERXEC.java:805)
>>>>>>> at com.webobjects.eocontrol.EOCustomObject.willReadRelationship(EOCustomObject.java:1281)
>>>>>>> at er.extensions.eof.ERXGenericRecord.willReadRelationship(ERXGenericRecord.java:378)
>>>>>>> at com.webobjects.eocontrol._EOMutableKnownKeyDictionary$Initializer$_LazyGenericRecordBinding.valueInObject(_EOMutableKnownKeyDictionary.java:614)
>>>>>>> at com.webobjects.eocontrol.EOCustomObject.storedValueForKey(EOCustomObject.java:1634)
>>>>>>> at com.tla.logic._RigaCarrello.clientiPerTicket(_RigaCarrello.java:275)
>>>>>>> at com.tla.logic.RigaCarrello.setPosti(RigaCarrello.java:94)
>>>>>>> at com.tla.logic.Carrello.cancellaTutto(Carrello.java:164)
>>>>>>> at com.tla.calendar.Session.terminate(Session.java:81)
>>>>>>> at com.webobjects.appserver.WOSession._terminateByTimeout(WOSession.java:610)
>>>>>>> at com.webobjects.appserver.WOSessionStore$_SessionTimeoutManager.run(WOSessionStore.java:115)
>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> Sorry for the italian Class Names and Methods, as I said I inherited that...
>>>>>>>
>>>>>>> The exception is caught, that printout is the printStackTrace of the
>>>>>>> exception...
>>>>>>> But anyway, I'm not sure that's the real problem, I found this morning
>>>>>>> an instance with 295 sessions (still alive) and the only exception
>>>>>>> logged was a weird
>>>>>>>
>>>>>>> er.transaction.adaptor.Exceptions - Database Exception occured: No
>>>>>>> operations allowed after connection closed.
>>>>>>>
>>>>>>> Connection was closed due to the following exception:
>>>>>>>
>>>>>>> ** BEGIN NESTED EXCEPTION **
>>>>>>>
>>>>>>> java.sql.SQLException
>>>>>>> MESSAGE: Communication link failure: java.io.EOFException, underlying
>>>>>>> cause: null
>>>>>>>
>>>>>>> ** BEGIN NESTED EXCEPTION **
>>>>>>>
>>>>>>> java.io.EOFException
>>>>>>>
>>>>>>> STACKTRACE:
>>>>>>>
>>>>>>> java.io.EOFException
>>>>>>> at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1394)
>>>>>>> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:1538)
>>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:1929)
>>>>>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1167)
>>>>>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1278)
>>>>>>> at com.mysql.jdbc.Connection.execSQL(Connection.java:2247)
>>>>>>> at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1371)
>>>>>>> at com.webobjects.jdbcadaptor.JDBCChannel._bindInputVariablesWithBindingsAndExecute(JDBCChannel.java:265)
>>>>>>> at com.webobjects.jdbcadaptor.JDBCChannel._evaluateExpression(JDBCChannel.java:337)
>>>>>>> at com.webobjects.jdbcadaptor.JDBCChannel.evaluateExpression(JDBCChannel.java:296)
>>>>>>> at com.webobjects.jdbcadaptor.JDBCChannel.selectAttributes(JDBCChannel.java:220)
>>>>>>> at com.webobjects.eoaccess.EODatabaseChannel._selectWithFetchSpecificationEditingContext(EODatabaseChannel.java:897)
>>>>>>> at com.webobjects.eoaccess.EODatabaseChannel.selectObjectsWithFetchSpecification(EODatabaseChannel.java:234)
>>>>>>> at com.webobjects.eoaccess.EODatabaseContext._objectsWithFetchSpecificationEditingContext(EODatabaseContext.java:3055)
>>>>>>> at com.webobjects.eoaccess.EODatabaseContext.objectsWithFetchSpecification(EODatabaseContext.java:3195)
>>>>>>> at com.webobjects.eocontrol.EOObjectStoreCoordinator.objectsWithFetchSpecification(EOObjectStoreCoordinator.java:488)
>>>>>>> at com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEditingContext.java:4069)
>>>>>>> at er.extensions.eof.ERXEC.objectsWithFetchSpecification(ERXEC.java:1305)
>>>>>>> at com.webobjects.eocontrol.EOEditingContext.objectsWithFetchSpecification(EOEditingContext.java:4444)
>>>>>>> at com.tla.logic.Performance.fetchFreshPerformancesWithQualifier(Performance.java:414)
>>>>>>> at com.tla.logic.Carrello.cancellaTutto(Carrello.java:154)
>>>>>>> at com.tla.calendar.components.ALTComponent.invokeAction(ALTComponent.java:167)
>>>>>>> at com.webobjects.appserver.WOSession.invokeAction(WOSession.java:1357)
>>>>>>> at com.webobjects.appserver.WOApplication.invokeAction(WOApplication.java:1745)
>>>>>>> at er.extensions.appserver.ajax.ERXAjaxApplication.invokeAction(ERXAjaxApplication.java:117)
>>>>>>> at er.extensions.appserver.ERXApplication.invokeAction(ERXApplication.java:2018)
>>>>>>> at er.extensions.appserver.ERXComponentRequestHandler._dispatchWithPreparedPage(ERXComponentRequestHandler.java:157)
>>>>>>> at er.extensions.appserver.ERXComponentRequestHandler._dispatchWithPreparedSession(ERXComponentRequestHandler.java:235)
>>>>>>> at er.extensions.appserver.ERXComponentRequestHandler._dispatchWithPreparedApplication(ERXComponentRequestHandler.java:268)
>>>>>>> at er.extensions.appserver.ERXComponentRequestHandler._handleRequest(ERXComponentRequestHandler.java:302)
>>>>>>> at er.extensions.appserver.ERXComponentRequestHandler.handleRequest(ERXComponentRequestHandler.java:377)
>>>>>>> at com.webobjects.appserver.WOApplication.dispatchRequest(WOApplication.java:1687)
>>>>>>> at er.extensions.appserver.ERXApplication.dispatchRequestImmediately(ERXApplication.java:2139)
>>>>>>> at er.extensions.appserver.ERXApplication.dispatchRequest(ERXApplication.java:2104)
>>>>>>> at com.webobjects.appserver._private.WOWorkerThread.runOnce(WOWorkerThread.java:144)
>>>>>>> at com.webobjects.appserver._private.WOWorkerThread.run(WOWorkerThread.java:226)
>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> So I'm not that confident that I am on the right track...
>>>>>>> Looks like if anything goes wrong in the life of the instance sessions
>>>>>>> won't die... Which is kind of disturbing.
>>>>>>>
>>>>>>>
>>>>>>> Matteo
>>>>>>>
>>>>>>> On Sun, Jan 13, 2013 at 7:01 PM, Chuck Hill <email@hidden> wrote:
>>>>>>>> Can you post the entire stack trace?
>>>>>>>>
>>>>>>>> At the very least, I would catch that exception and now allow it out of your finalize method.
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Jan 13, 2013 at 7:01 PM, Chuck Hill <email@hidden> wrote:
>>>>>>>> Can you post the entire stack trace?
>>>>>>>>
>>>>>>>> At the very least, I would catch that exception and now allow it out of your finalize method.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2013-01-12, at 2:01 PM, Matteo Centro wrote:
>>>>>>>>
>>>>>>>>> Thanks, I'll try that.
>>>>>>>>> Anyway (I can't say it's that but this happens often in instances with
>>>>>>>>> immortal sessions):
>>>>>>>>> I get this in my terminate() method
>>>>>>>>> java.lang.IllegalMonitorStateException
>>>>>>>>> at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:127)
>>>>>>>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239)
>>>>>>>>> at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431)
>>>>>>>>> at com.webobjects.eocontrol.EOObjectStoreCoordinator.unlock(EOObjectStoreCoordinator.java:460)
>>>>>>>>> at com.webobjects.eocontrol.EOEditingContext.unlockObjectStore(EOEditingContext.java:4684)
>>>>>>>>> at er.extensions.eof.ERXEC.unlockObjectStore(ERXEC.java:805)
>>>>>>>>> at com.webobjects.eocontrol.EOCustomObject.willReadRelationship(EOCustomObject.java:1281)
>>>>>>>>> at er.extensions.eof.ERXGenericRecord.willReadRelationship(ERXGenericRecord.java:378)
>>>>>>>>> at com.webobjects.eocontrol._EOMutableKnownKeyDictionary$Initializer$_LazyGenericRecordBinding.valueInObject(_EOMutableKnownKeyDictionary.java:614)
>>>>>>>>> at com.webobjects.eocontrol.EOCustomObject.storedValueForKey(EOCustomObject.java:1634)
>>>>>>>>>
>>>>>>>>> I'm using ERXEC everywhere and I'm not locking explicitly, should I
>>>>>>>>> lock the ec in the terminate method?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Matteo
>>>>>>>>>
>>>>>>>>> On Sat, Jan 12, 2013 at 8:39 PM, Simon <email@hidden> wrote:
>>>>>>>>>> stick this in your session constructor, it will log out whenever a
>>>>>>>>>> session is created. then you can start figuring out where you are
>>>>>>>>>> creating a session outside the RR loop which is the normal culprit...
>>>>>>>>>>
>>>>>>>>>> StringWriter stringWriter = new StringWriter();
>>>>>>>>>> PrintWriter printWriter = new PrintWriter(stringWriter);
>>>>>>>>>> (new Throwable()).printStackTrace(printWriter);
>>>>>>>>>> String trace = stringWriter.toString();
>>>>>>>>>> log.debug("Session count = " + application().activeSessionsCount());
>>>>>>>>>> ClickEventLogger2.getLogger().fatal(ClickEventCode.E00129, "Session
>>>>>>>>>> Created:\n\n " + trace,
>>>>>>>>>> this.getClass().getSimpleName());
>>>>>>>>>>
>>>>>>>>>> hmmm, you'll have to replace the "ClickEventLogger2" line because
>>>>>>>>>> that's our logging mechanism. you could log.fatal it and set up an
>>>>>>>>>> email appender in log4j.
>>>>>>>>>>
>>>>>>>>>> simon
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 12 January 2013 17:43, Matteo Centro <email@hidden> wrote:
>>>>>>>>>>> Sorry to resuscitate such an old discussion but I'm having the exact
>>>>>>>>>>> same issue...
>>>>>>>>>>> It's an old application that we "inherited", we wonderized it as much
>>>>>>>>>>> as it's possible but something weird happens in production, sessions
>>>>>>>>>>> on some instances simply won't die.
>>>>>>>>>>> Some instances go out of memory and they hang there.
>>>>>>>>>>> I'm in trouble and I needs some hints, both to fix the issue
>>>>>>>>>>> temporarily and to fix it for good (of course in that case I assume
>>>>>>>>>>> I'll have to rewrite the app, if only I could find the budget).
>>>>>>>>>>> What are the most common causes of sessions not dying?
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Matteo
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Aug 21, 2008 at 5:35 AM, Joe Little <email@hidden> wrote:
>>>>>>>>>>>> I had something similar with sessions going bonkers on a public WO
>>>>>>>>>>>> page that our internal google search engine completely trashed. In the
>>>>>>>>>>>> end, robots.txt and explicit rules to deny certain patterns were added
>>>>>>>>>>>> to prevent this.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Aug 20, 2008 at 8:17 PM, D Tim Cummings <email@hidden> wrote:
>>>>>>>>>>>>> We have a couple of sessionless apps that have started showing this problem
>>>>>>>>>>>>> with sessions that don't terminate. It turned out the sessions were being
>>>>>>>>>>>>> created by malformed urls coming from malicious robot web crawlers. The
>>>>>>>>>>>>> urls were of the form
>>>>>>>>>>>>> http://www.courses.qut.edu.au/cgi-bin/WebObjects/Courses.woa/wa/cgi-bin/WebObjects/Courses.woa
>>>>>>>>>>>>> Maybe see if you are getting incorrect links to your sessionless login page.
>>>>>>>>>>>>> We solved the problem by catching unknown direct actions in
>>>>>>>>>>>>> DirectAction.java
>>>>>>>>>>>>> @Override
>>>>>>>>>>>>> public WOActionResults performActionNamed(String actionName) {
>>>>>>>>>>>>> try {
>>>>>>>>>>>>> return super.performActionNamed(actionName);
>>>>>>>>>>>>> } catch (NSForwardException nsfe) {
>>>>>>>>>>>>> log.info("ns forward exception - prbalby no such method for " + actionName);
>>>>>>>>>>>>> }
>>>>>>>>>>>>> return defaultAction();
>>>>>>>>>>>>> }
>>>>>>>>>>>>> and in Application.java directing exceptions back to the Main page (for URLs
>>>>>>>>>>>>> with more than one / after wa).
>>>>>>>>>>>>> @Override
>>>>>>>>>>>>> public WOComponent pageWithName(String namePage, WOContext context) {
>>>>>>>>>>>>> if ( "WOExceptionPage".equals(namePage) ) {
>>>>>>>>>>>>> namePage = "Main";
>>>>>>>>>>>>> }
>>>>>>>>>>>>> if ( "WOSessionRestorationError".equals(namePage) ) {
>>>>>>>>>>>>> namePage = "Main";
>>>>>>>>>>>>> }
>>>>>>>>>>>>> return super.pageWithName(namePage, context);
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> and in Main.java
>>>>>>>>>>>>> public void setException ( Exception e ) {
>>>>>>>>>>>>> log.error("an exception occurred " + e);
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> We are running apps with embedded Wonder 4 and WebObjects 5.3.3 on Mac OS X
>>>>>>>>>>>>> Server 10.5.4 with WebObjects 5.4.2 deployment. We didn't have the problem
>>>>>>>>>>>>> before we went to this setup, but maybe we weren't getting hit with the same
>>>>>>>>>>>>> url format then.
>>>>>>>>>>>>> Tim
>>>>>>>>>>>>> On 21/08/2008, at 4:02 AM, Chuck Hill wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Aug 20, 2008, at 9:54 AM, Simon McLean wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi All -
>>>>>>>>>>>>> Wondering if someone can throw me some ideas on solving what looks like an
>>>>>>>>>>>>> immortal session problem.
>>>>>>>>>>>>> - Start up 1 instance in java monitor.
>>>>>>>>>>>>> - User lands on sessionless login page. No sessions.
>>>>>>>>>>>>> - User logs in. 1 session.
>>>>>>>>>>>>> - User logs out. 0 sessions.
>>>>>>>>>>>>> - User logs in. 1 session.
>>>>>>>>>>>>> - User does nothing. Session times out. 0 sessions.
>>>>>>>>>>>>> All looks good. However, we get to the end of the day and the instance has
>>>>>>>>>>>>> 376 active sessions. But I know for this particular app there are only 6
>>>>>>>>>>>>> users, because they are all sitting next door :-) Also, If i now leave this
>>>>>>>>>>>>> instance overnight none of those active sessions will go away.
>>>>>>>>>>>>> So what could be going on ? Something appears to be creating immortal
>>>>>>>>>>>>> sessions, but forced session termination (by the user logging out) and
>>>>>>>>>>>>> session expiration seem to be behaving themselves.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If the users don't notice any problems, my money would be on an exception
>>>>>>>>>>>>> thrown from your Session.terminate(). Also check the sleep() method to
>>>>>>>>>>>>> ensure it can never throw.
>>>>>>>>>>>>> I have seen a related problem where two requests come in for the same
>>>>>>>>>>>>> session (user double clicks, Ajax etc), and the first calls terminate() on
>>>>>>>>>>>>> the session. The second thread is stuck and the app can't gracefully shut
>>>>>>>>>>>>> down. IIRC, you would see zero sessions in this case. The "fix" for this
>>>>>>>>>>>>> is to not call terminate() and instead set the session timeout to a small
>>>>>>>>>>>>> number of seconds.
>>>>>>>>>>>>> Chuck
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Chuck Hill Senior Consultant / VP Development
>>>>>>>>>>>>> Practical WebObjects - for developers who want to increase their overall
>>>>>>>>>>>>> knowledge of WebObjects or who are trying to solve specific problems.
>>>>>>>>>>>>> http://www.global-village.net/products/practical_webobjects
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>>>>>>>>> Webobjects-dev mailing list (email@hidden)
>>>>>>>>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>>>>>>>>> This email sent to email@hidden
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>>>>>>>>> Webobjects-dev mailing list (email@hidden)
>>>>>>>>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>>>>>>>>>
>>>>>>>>>>>>> This email sent to email@hidden
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>>>>>>>> Webobjects-dev mailing list (email@hidden)
>>>>>>>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>>>>>>>>
>>>>>>>>>>>> This email sent to email@hidden
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>>>>>>> Webobjects-dev mailing list (email@hidden)
>>>>>>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>>>>>>>
>>>>>>>>>>> This email sent to email@hidden
>>>>>>>>> _______________________________________________
>>>>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>>>>> Webobjects-dev mailing list (email@hidden)
>>>>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>>>>>
>>>>>>>>> This email sent to email@hidden
>>>>>>>>
>>>>>>>> --
>>>>>>>> Chuck Hill Senior Consultant / VP Development
>>>>>>>>
>>>>>>>> Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems.
>>>>>>>> http://www.global-village.net/gvc/practical_webobjects
>>>>>>>>
>>>>>>>> Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest Growing Companies in B.C!
>>>>>>>> Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking of Canada’s Fastest-Growing Companies by PROFIT Magazine!
>>>>>> <jstackoutput.txt>
>>>>>
>>>>> --
>>>>> Chuck Hill Senior Consultant / VP Development
>>>>>
>>>>> Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems.
>>>>> http://www.global-village.net/gvc/practical_webobjects
>>>>>
>>>>> Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest Growing Companies in B.C!
>>>>> Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking of Canada’s Fastest-Growing Companies by PROFIT Magazine!
>>>
>>> --
>>> Chuck Hill Senior Consultant / VP Development
>>>
>>> Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems.
>>> http://www.global-village.net/gvc/practical_webobjects
>>>
>>> Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest Growing Companies in B.C!
>>> Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking of Canada’s Fastest-Growing Companies by PROFIT Magazine!
>
> _______________________________________________
> Do not post admin requests to the list. They will be ignored.
> Webobjects-dev mailing list (email@hidden)
> Help/Unsubscribe/Update your Subscription:
>
> This email sent to email@hidden
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden