Re: Progressively all threads go into deadlock
Re: Progressively all threads go into deadlock
- Subject: Re: Progressively all threads go into deadlock
- From: "Mr. Pierre Frisch" <email@hidden>
- Date: Wed, 20 Feb 2008 19:16:09 -0800
Hi Fabrice,
How many stacks have you created for your multithreaded application?
Are you using only one? Then you may have a contention as EOF is
single threaded. Can you elaborate a little bit more on how your
application works?
Cheers
Pierre
--
Pierre Frisch
email@hidden
On Feb 20, 2008, at 13:35, Chuck Hill wrote:
On Feb 20, 2008, at 5:33 AM, Fabrice Pipart wrote:
On Feb 19, 2008, at 8:33 PM, Chuck Hill wrote:
We are thinking about a different thing : could it be a memory
problem ?
We are allocating "only" 256 Mb and are dealing concurrently with
thousands of objects in background threads, components and
wolongresponses tasks....
Could it lead to a deadlock ?
Possibly. If the JVM runs out of memory, almost anything can
happen. But it sounds like this is the only problem you are
seeing and that it happens somewhat consistently. That does not
sound like an out of memory exception.
Might explain our situation then ....
Is there anything special to do with EOObjectStoreCoordinator?
I saw properties about pools of EOObjectStoreCoordinator, are
they useful in my case?
Probably not.
Just a quick note : I now undertand what all those
[1] sun.misc.Unsafe.park (native method)
are for : threads waiting for the database lock to be able to do
something on database.
At least this will have clarified some essential WO behavior to me.
Yes, EOF is single threaded below the EOObjectStoreCoordinator.
That made me give a deeper look at it.
Under heavy load, my threads keep waiting for those locks.
In a typical situation, I have 3 threads waiting for a lock and one
is actually accessing database.
Is it in that case that EOObjectStoreCoordinator pools are useful
to get a better performance ?
Is that dangerous to use?
The one in Wonder? I have never used it, but if it is in Wonder it
is probably safe to use. It sounds like you have some database
intensive operations and having multiple EOObjectStoreCoordinators
(i.e. EOF stacks) will improve concurrency. Sometimes I offload
these to another application. That also effectively gives them
their own stack and also moves the load of resulting garbage
collection etc to a different JVM.
Here are some of the stack traces (I reduced the size of traces a
bit and removed the threads sleeping, if you really want all
threads, tell me) :
ApplicationHealthTask:
[1] sun.misc.Unsafe.park (native method)
[2] java.util.concurrent.locks.LockSupport.park (LockSupport.java:
118)
[3]
java
.util
.concurrent
.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt
(AbstractQueuedSynchronizer.java:716)
[4]
java
.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued
(AbstractQueuedSynchronizer.java:746)
[5] java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire
(AbstractQueuedSynchronizer.java:1,076)
[6] java.util.concurrent.locks.ReentrantLock$NonfairSync.lock
(ReentrantLock.java:184)
[7] java.util.concurrent.locks.ReentrantLock.lock
(ReentrantLock.java:256)
[8] com.webobjects.eocontrol.EOObjectStoreCoordinator.lock
(EOObjectStoreCoordinator.java:420)
[9] com.webobjects.eocontrol.EOEditingContext.lockObjectStore
(EOEditingContext.java:4,650)
[10] com.webobjects.eocontrol.EOCustomObject.willRead
(EOCustomObject.java:1,165)
[11] com.webobjects.eocontrol._EOMutableKnownKeyDictionary
$Initializer$_GenericRecordBinding.valueInObject
(_EOMutableKnownKeyDictionary.java:570)
[12] com.webobjects.eocontrol.EOCustomObject.storedValueForKey
(EOCustomObject.java:1,634)
[13] com.easyshadow.server.base._Area.type (_Area.java:80)
...
Was the app totally deadlocked when you make these dumps? If not,
these dumps might not mean much.
Maybe not 100% deadlock, I am not sure.
But totally slow (a human user does not really make the difference
between deadlock and sloooooow)
And afterwards, the app crashed (I could not reproduce the bug
since that moment)
If it was totally deadlockes, here, I think, is your bug:
...
AWT-AppKit:
Monitoring Thread:
[1] java.net.SocketOutputStream.socketWrite0 (native method)
[2] java.net.SocketOutputStream.socketWrite
(SocketOutputStream.java:92)
[3] java.net.SocketOutputStream.write (SocketOutputStream.java:136)
[4] java.io.BufferedOutputStream.flushBuffer
(BufferedOutputStream.java:65)
[5] java.io.BufferedOutputStream.flush (BufferedOutputStream.java:
123)
[6] com.mysql.jdbc.MysqlIO.send (MysqlIO.java:3,119)
[7] com.mysql.jdbc.MysqlIO.sendCommand (MysqlIO.java:1,818)
[8] com.mysql.jdbc.MysqlIO.sqlQueryDirect (MysqlIO.java:1,961)
[9] com.mysql.jdbc.ConnectionImpl.execSQL (ConnectionImpl.java:
2,543)
[10] com.mysql.jdbc.PreparedStatement.executeInternal
(PreparedStatement.java:1,737)
[11] com.mysql.jdbc.PreparedStatement.execute
(PreparedStatement.java:998)
[12]
com
.webobjects
.jdbcadaptor
.JDBCChannel._bindInputVariablesWithBindingsAndExecute
(JDBCChannel.java:265)
[13] com.webobjects.jdbcadaptor.JDBCChannel._evaluateExpression
(JDBCChannel.java:337)
[14] com.webobjects.jdbcadaptor.JDBCChannel.evaluateExpression
(JDBCChannel.java:296)
[15] com.webobjects.jdbcadaptor.JDBCChannel.selectAttributes
(JDBCChannel.java:220)
[16]
com
.webobjects
.eoaccess
.EODatabaseChannel._selectWithFetchSpecificationEditingContext
(EODatabaseChannel.java:898)
[17]
com
.webobjects
.eoaccess.EODatabaseChannel.selectObjectsWithFetchSpecification
(EODatabaseChannel.java:234)
[18]
com
.webobjects
.eoaccess
.EODatabaseContext._objectsWithFetchSpecificationEditingContext
(EODatabaseContext.java:3,055)
[19]
com
.webobjects
.eoaccess.EODatabaseContext.objectsWithFetchSpecification
(EODatabaseContext.java:3,195)
[20] com.webobjects.eoaccess.EODatabaseContext._fireFault
(EODatabaseContext.java:4,187)
[21]
com
.webobjects
.eoaccess.EOAccessFaultHandler.completeInitializationOfObject
(EOAccessFaultHandler.java:89)
[22] com.webobjects.eocontrol.EOCustomObject.willRead
(EOCustomObject.java:1,172)
[23] com.webobjects.eocontrol._EOMutableKnownKeyDictionary
$Initializer$_GenericRecordBinding.valueInObject
(_EOMutableKnownKeyDictionary.java:570)
[24] com.webobjects.eocontrol.EOCustomObject.storedValueForKey
(EOCustomObject.java:1,634)
...
Whatever is happening here has the EODatabaseContext locked which
will prevent any other thread from locking the
EOObjectStoreCoordinator.
I now uderstand why both you and Nike point that thread.
But what it does there does not seem strange and I see no reason
for a deadlock here.
Excepted that it accesses some of the database objects that make
the app slow (too many of them).
That sounds like what is happening. You get too many database (EOF)
intensive processes at the same time and the app slows way down.
And after testing more, reducing the number of eo involved and
augmenting the memory allocated to the wo instance, this does not
seem to happen anymore.
Or at least the app crashes frankly with a OutOfMemory or with an
error that is impossible without a OutOfMemory like :
Object o = new Object();
o.doSmthg(); ------> NullPointerException .....
Thanks a lot for your help guys !
Fabrice Pipart
You are welcome!
Chuck
www.easyshadow.com
EasyMediaOnline
Digital Signage Software
Easyshadow
Palais de la Scala
1 avenue Henri Dunant
Suite 1155
MC - 98000 Monaco
Skype: fabrice.pipart
Tel. +377 97 98 21 04 (direct)
Fax. +377 97 70 88 07
--
Practical WebObjects - for developers who want to increase their
overall knowledge of WebObjects or who are trying to solve specific
problems.
http://www.global-village.net/products/practical_webobjects
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden