• Open Menu Close Menu
  • Apple
  • Shopping Bag
  • Apple
  • Mac
  • iPad
  • iPhone
  • Watch
  • TV
  • Music
  • Support
  • Search apple.com
  • Shopping Bag

Lists

Open Menu Close Menu
  • Terms and Conditions
  • Lists hosted on this site
  • Email the Postmaster
  • Tips for posting to public mailing lists
Re: Progressively all threads go into deadlock
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Progressively all threads go into deadlock


  • Subject: Re: Progressively all threads go into deadlock
  • From: Fabrice Pipart <email@hidden>
  • Date: Wed, 20 Feb 2008 14:33:11 +0100


On Feb 19, 2008, at 8:33 PM, Chuck Hill wrote:

We are thinking about a different thing : could it be a memory problem ?
We are allocating "only" 256 Mb and are dealing concurrently with thousands of objects in background threads, components and wolongresponses tasks....
Could it lead to a deadlock ?

Possibly.  If the JVM runs out of memory, almost anything can happen.  But it sounds like this is the only problem you are seeing and that it happens somewhat consistently.  That does not sound like an out of memory exception.

Might explain our situation then ....

Is there anything special to do with EOObjectStoreCoordinator?
I saw properties about pools of EOObjectStoreCoordinator, are they useful in my case?

Probably not.

Just a quick note : I now undertand what all those
[1] sun.misc.Unsafe.park (native method)
are for : threads waiting for the database lock to be able to do something on database.
At least this will have clarified some essential WO behavior to me.

That made me give a deeper look at it.
Under heavy load, my threads keep waiting for those locks.
In a typical situation, I have 3 threads waiting for a lock and one is actually accessing database.
Is it in that case that EOObjectStoreCoordinator pools are useful to get a better performance ?
Is that dangerous to use?


Here are some of the stack traces (I reduced the size of traces a bit and removed the threads sleeping, if you really want all threads, tell me) :



ApplicationHealthTask:
 [1] sun.misc.Unsafe.park (native method)
 [2] java.util.concurrent.locks.LockSupport.park (LockSupport.java:118)
 [3] java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt (AbstractQueuedSynchronizer.java:716)
 [4] java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued (AbstractQueuedSynchronizer.java:746)
 [5] java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire (AbstractQueuedSynchronizer.java:1,076)
 [6] java.util.concurrent.locks.ReentrantLock$NonfairSync.lock (ReentrantLock.java:184)
 [7] java.util.concurrent.locks.ReentrantLock.lock (ReentrantLock.java:256)
 [8] com.webobjects.eocontrol.EOObjectStoreCoordinator.lock (EOObjectStoreCoordinator.java:420)
 [9] com.webobjects.eocontrol.EOEditingContext.lockObjectStore (EOEditingContext.java:4,650)
 [10] com.webobjects.eocontrol.EOCustomObject.willRead (EOCustomObject.java:1,165)
 [11] com.webobjects.eocontrol._EOMutableKnownKeyDictionary$Initializer$_GenericRecordBinding.valueInObject (_EOMutableKnownKeyDictionary.java:570)
 [12] com.webobjects.eocontrol.EOCustomObject.storedValueForKey (EOCustomObject.java:1,634)
 [13] com.easyshadow.server.base._Area.type (_Area.java:80)
...

Was the app totally deadlocked when you make these dumps?  If not, these dumps might not mean much.

Maybe not 100% deadlock, I am not sure.
But totally slow (a human user does not really make the difference between deadlock and sloooooow)
And afterwards, the app crashed (I could not reproduce the bug since that moment)

If it was totally deadlockes, here, I think, is your bug:

...

AWT-AppKit:
Monitoring Thread:
 [1] java.net.SocketOutputStream.socketWrite0 (native method)
 [2] java.net.SocketOutputStream.socketWrite (SocketOutputStream.java:92)
 [3] java.net.SocketOutputStream.write (SocketOutputStream.java:136)
 [4] java.io.BufferedOutputStream.flushBuffer (BufferedOutputStream.java:65)
 [5] java.io.BufferedOutputStream.flush (BufferedOutputStream.java:123)
 [6] com.mysql.jdbc.MysqlIO.send (MysqlIO.java:3,119)
 [7] com.mysql.jdbc.MysqlIO.sendCommand (MysqlIO.java:1,818)
 [8] com.mysql.jdbc.MysqlIO.sqlQueryDirect (MysqlIO.java:1,961)
 [9] com.mysql.jdbc.ConnectionImpl.execSQL (ConnectionImpl.java:2,543)
 [10] com.mysql.jdbc.PreparedStatement.executeInternal (PreparedStatement.java:1,737)
 [11] com.mysql.jdbc.PreparedStatement.execute (PreparedStatement.java:998)
 [12] com.webobjects.jdbcadaptor.JDBCChannel._bindInputVariablesWithBindingsAndExecute (JDBCChannel.java:265)
 [13] com.webobjects.jdbcadaptor.JDBCChannel._evaluateExpression (JDBCChannel.java:337)
 [14] com.webobjects.jdbcadaptor.JDBCChannel.evaluateExpression (JDBCChannel.java:296)
 [15] com.webobjects.jdbcadaptor.JDBCChannel.selectAttributes (JDBCChannel.java:220)
 [16] com.webobjects.eoaccess.EODatabaseChannel._selectWithFetchSpecificationEditingContext (EODatabaseChannel.java:898)
 [17] com.webobjects.eoaccess.EODatabaseChannel.selectObjectsWithFetchSpecification (EODatabaseChannel.java:234)
 [18] com.webobjects.eoaccess.EODatabaseContext._objectsWithFetchSpecificationEditingContext (EODatabaseContext.java:3,055)
 [19] com.webobjects.eoaccess.EODatabaseContext.objectsWithFetchSpecification (EODatabaseContext.java:3,195)
 [20] com.webobjects.eoaccess.EODatabaseContext._fireFault (EODatabaseContext.java:4,187)
 [21] com.webobjects.eoaccess.EOAccessFaultHandler.completeInitializationOfObject (EOAccessFaultHandler.java:89)
 [22] com.webobjects.eocontrol.EOCustomObject.willRead (EOCustomObject.java:1,172)
 [23] com.webobjects.eocontrol._EOMutableKnownKeyDictionary$Initializer$_GenericRecordBinding.valueInObject (_EOMutableKnownKeyDictionary.java:570)
 [24] com.webobjects.eocontrol.EOCustomObject.storedValueForKey (EOCustomObject.java:1,634)
...

Whatever is happening here has the EODatabaseContext locked which will prevent any other thread from locking the EOObjectStoreCoordinator.

I now uderstand why both you and Nike point that thread.
But what it does there does not seem strange and I see no reason for a deadlock here.
Excepted that it accesses some of the database objects that make the app slow (too many of them).

And after testing more, reducing the number of eo involved and augmenting the memory allocated to the wo instance, this does not seem to happen anymore.
Or at least the app crashes frankly with a OutOfMemory or with an error that is impossible without a OutOfMemory like :
Object o = new Object();
o.doSmthg();          ------> NullPointerException .....

Thanks a lot for your help guys !

Fabrice Pipart


www.easyshadow.com
EasyMediaOnline
Digital Signage Software

Easyshadow
Palais de la Scala
1 avenue Henri Dunant
Suite 1155
MC - 98000 Monaco

Skype: fabrice.pipart
Tel.  +377 97 98 21 04 (direct)
Fax. +377 97 70 88 07


 _______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list      (email@hidden)
Help/Unsubscribe/Update your Subscription:

This email sent to email@hidden

  • Follow-Ups:
    • Re: Progressively all threads go into deadlock
      • From: Chuck Hill <email@hidden>
References: 
 >Progressively all threads go into deadlock (From: Fabrice Pipart <email@hidden>)
 >Re: Progressively all threads go into deadlock (From: Chuck Hill <email@hidden>)
 >Re: Progressively all threads go into deadlock (From: Fabrice Pipart <email@hidden>)
 >Re: Progressively all threads go into deadlock (From: Chuck Hill <email@hidden>)

  • Prev by Date: Re: java heap trouble after 5.4.1 upgrade, opensnoop to the rescue
  • Next by Date: Cannot rollbackTransaction() while a fetch is in progress (Was: MySQL 5.0 - note)
  • Previous by thread: Re: Progressively all threads go into deadlock
  • Next by thread: Re: Progressively all threads go into deadlock
  • Index(es):
    • Date
    • Thread