On Feb 19, 2008, at 8:33 PM, Chuck Hill wrote:
We are thinking about a different thing : could it be a memory problem ?
We are allocating "only" 256 Mb and are dealing concurrently with thousands of objects in background threads, components and wolongresponses tasks....
Could it lead to a deadlock ?
Possibly. If the JVM runs out of memory, almost anything can happen. But it sounds like this is the only problem you are seeing and that it happens somewhat consistently. That does not sound like an out of memory exception.
Might explain our situation then ....
Is there anything special to do with EOObjectStoreCoordinator?
I saw properties about pools of EOObjectStoreCoordinator, are they useful in my case?
Probably not.
Just a quick note : I now undertand what all those
[1] sun.misc.Unsafe.park (native method)
are for : threads waiting for the database lock to be able to do something on database.
At least this will have clarified some essential WO behavior to me.
That made me give a deeper look at it.
Under heavy load, my threads keep waiting for those locks.
In a typical situation, I have 3 threads waiting for a lock and one is actually accessing database.
Is it in that case that EOObjectStoreCoordinator pools are useful to get a better performance ?
Is that dangerous to use?
Here are some of the stack traces (I reduced the size of traces a bit and removed the threads sleeping, if you really want all threads, tell me) :
ApplicationHealthTask:
[1] sun.misc.Unsafe.park (native method)
[2] java.util.concurrent.locks.LockSupport.park (LockSupport.java:118)
[3] java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt (AbstractQueuedSynchronizer.java:716)
[4] java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued (AbstractQueuedSynchronizer.java:746)
[5] java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire (AbstractQueuedSynchronizer.java:1,076)
[6] java.util.concurrent.locks.ReentrantLock$NonfairSync.lock (ReentrantLock.java:184)
[7] java.util.concurrent.locks.ReentrantLock.lock (ReentrantLock.java:256)
[8] com.webobjects.eocontrol.EOObjectStoreCoordinator.lock (EOObjectStoreCoordinator.java:420)
[9] com.webobjects.eocontrol.EOEditingContext.lockObjectStore (EOEditingContext.java:4,650)
[10] com.webobjects.eocontrol.EOCustomObject.willRead (EOCustomObject.java:1,165)
[11] com.webobjects.eocontrol._EOMutableKnownKeyDictionary$Initializer$_GenericRecordBinding.valueInObject (_EOMutableKnownKeyDictionary.java:570)
[12] com.webobjects.eocontrol.EOCustomObject.storedValueForKey (EOCustomObject.java:1,634)
[13] com.easyshadow.server.base._Area.type (_Area.java:80)
...
Was the app totally deadlocked when you make these dumps? If not, these dumps might not mean much.
Maybe not 100% deadlock, I am not sure.
But totally slow (a human user does not really make the difference between deadlock and sloooooow)
And afterwards, the app crashed (I could not reproduce the bug since that moment)
If it was totally deadlockes, here, I think, is your bug:
...
AWT-AppKit:
Monitoring Thread:
[1] java.net.SocketOutputStream.socketWrite0 (native method)
[2] java.net.SocketOutputStream.socketWrite (SocketOutputStream.java:92)
[3] java.net.SocketOutputStream.write (SocketOutputStream.java:136)
[4] java.io.BufferedOutputStream.flushBuffer (BufferedOutputStream.java:65)
[5] java.io.BufferedOutputStream.flush (BufferedOutputStream.java:123)
[6] com.mysql.jdbc.MysqlIO.send (MysqlIO.java:3,119)
[7] com.mysql.jdbc.MysqlIO.sendCommand (MysqlIO.java:1,818)
[8] com.mysql.jdbc.MysqlIO.sqlQueryDirect (MysqlIO.java:1,961)
[9] com.mysql.jdbc.ConnectionImpl.execSQL (ConnectionImpl.java:2,543)
[10] com.mysql.jdbc.PreparedStatement.executeInternal (PreparedStatement.java:1,737)
[11] com.mysql.jdbc.PreparedStatement.execute (PreparedStatement.java:998)
[12] com.webobjects.jdbcadaptor.JDBCChannel._bindInputVariablesWithBindingsAndExecute (JDBCChannel.java:265)
[13] com.webobjects.jdbcadaptor.JDBCChannel._evaluateExpression (JDBCChannel.java:337)
[14] com.webobjects.jdbcadaptor.JDBCChannel.evaluateExpression (JDBCChannel.java:296)
[15] com.webobjects.jdbcadaptor.JDBCChannel.selectAttributes (JDBCChannel.java:220)
[16] com.webobjects.eoaccess.EODatabaseChannel._selectWithFetchSpecificationEditingContext (EODatabaseChannel.java:898)
[17] com.webobjects.eoaccess.EODatabaseChannel.selectObjectsWithFetchSpecification (EODatabaseChannel.java:234)
[18] com.webobjects.eoaccess.EODatabaseContext._objectsWithFetchSpecificationEditingContext (EODatabaseContext.java:3,055)
[19] com.webobjects.eoaccess.EODatabaseContext.objectsWithFetchSpecification (EODatabaseContext.java:3,195)
[20] com.webobjects.eoaccess.EODatabaseContext._fireFault (EODatabaseContext.java:4,187)
[21] com.webobjects.eoaccess.EOAccessFaultHandler.completeInitializationOfObject (EOAccessFaultHandler.java:89)
[22] com.webobjects.eocontrol.EOCustomObject.willRead (EOCustomObject.java:1,172)
[23] com.webobjects.eocontrol._EOMutableKnownKeyDictionary$Initializer$_GenericRecordBinding.valueInObject (_EOMutableKnownKeyDictionary.java:570)
[24] com.webobjects.eocontrol.EOCustomObject.storedValueForKey (EOCustomObject.java:1,634)
...
Whatever is happening here has the EODatabaseContext locked which will prevent any other thread from locking the EOObjectStoreCoordinator.
I now uderstand why both you and Nike point that thread.
But what it does there does not seem strange and I see no reason for a deadlock here.
Excepted that it accesses some of the database objects that make the app slow (too many of them).
And after testing more, reducing the number of eo involved and augmenting the memory allocated to the wo instance, this does not seem to happen anymore.
Or at least the app crashes frankly with a OutOfMemory or with an error that is impossible without a OutOfMemory like :
Object o = new Object();
o.doSmthg(); ------> NullPointerException .....
Thanks a lot for your help guys !
Fabrice Pipart