Re: Deadlock using EOObjectStore.invalidateObjectsWithGlobalIDs(NSArray)
Re: Deadlock using EOObjectStore.invalidateObjectsWithGlobalIDs(NSArray)
- Subject: Re: Deadlock using EOObjectStore.invalidateObjectsWithGlobalIDs(NSArray)
- From: Jean-François Veillette <email@hidden>
- Date: Mon, 25 Sep 2006 17:00:31 -0400
Hi Timo,
The system I'm working on is working fine with the change
notification framework (modified to invalidate based on global ids).
But my co-worker's system (different project from mine) is having
heavy deadlock problem. I'm thinking one the the lock is coming from
the shared editing context, as your stack trace is showing. My
system doesn't use the SharedEditingContext at all (and the change
notification framework by gid work perfectly). The other system does
use the shared editing context heavily (and has major problem with
the same change notification framework). The other lock must be
somewhere in the eof stack. Here is a high level explanation of it :
ThreadA does the sequence :
- ask shared editing context to lock
- ask the eof stack to lock
ThreadB does the sequence :
- ask the eof stack to lock
- ask shared editing context to lock
As we can see, ThreadA and ThreadB can deadlock. This is what I
think is happening ... somehow.
I have 2 ideas but tried none of them. I think you already tried #1
when you said you disabled the sharedEC, if so, try #2 :
1- null the shared editing context from the editing context used for
invalidation :
...
EOEditingContext ec = new EOEditingContext();
=> ec.setSharedEditingContext(null);
ec.lock();
...
2- by looking at the de-compiled classes, it seemed that
EOSharedEditingContext should be synchronized. We could use
Javassist (never tried, only read the doc) to fix the implementation
(see http://www.javassist.org ).
Here is roughly what it would look like to fix the lock method using
javassist.
We would need to put this code early in the application
initialization (probably in Application.main()), before the class is
really loaded by the jvm.
ClassPool pool = ClassPool.getDefault();
CtClass cc = pool.get("com.webobjects.eocontrol.EOEditingContext");
CtMethod cm = cc.getDeclaredMethod("lock");
cm.instrument(
new ExprEditor() {
public void edit(MethodCall m) throws CannotCompileException {
if (m.getClassName().equals("EOEditingContext") && m.getMethodName
().equals("lock")) {
m.replace("{ synchronize (this) { $proceed($$); } }");
}
}
}
);
- jfv
Le 06-09-20 à 13:35, Timo Hoepfner a écrit :
On application load, I get deadlocks on
"_invalidateObjectsWithGlobalIDs(editingContext, globalIDs)" line.
Unfortunately, I couldn't get the stack trace of the thread: A
"kill -3 <pid>" is not enought, I must perform a "kill -9 <pid>"
to stop the application.
So, here is my question: is there a safer way to invalidate
globalIDs rather than using "editingContext.parentObjectStore
().invalidateObjectsWithGlobalIDs(globalIDs);" with properly
locked context and object store? Any hints or ideas to avoid these
deadlocks?
I did try ec.rootObjectStore.invalidate... with correct locking and
also got deadlocks. I was able to get some Thread dumps. In almost
all cases, the SharedEC was involved. Sample at the bottom of the
message.
Even when the sharedEC was disabled, I got IllegalStateExceptions.
I posted a message concerning that a few days ago regarding that:
Somehow it appears impossible to me to go the "invalidate global
IDs" route because I'm getting IllegalStateExceptions during
saveChanges (see bottom of post). I made some reproduction code
which reproduces the problem quite reliably (also see bottom of
post).
I tried tons of alternatives for the locking (ec.lockObjectectStore
() instead ec.rootObjectStore().lock(), getting down to the
database context and locking it as well, ...) but the problem
remains. Also tried EODatabase.forgetSnapshotForGlobalID with the
same result.
Before I finally give up on the subject, could someone with a
deeper understanding of the locking stuff (Chuck?) take a look at
it. I think it should work this way. If I'm not missing something,
I currently see no way in getting any kind of change notification
framework work reliably with the "invalidate global IDs" approach.
And the "snapshot" approach is already known for causing deadlocks.
Locally I'm testing on Intel/Tiger/WO 5.3.1, but have seen the
same Exceptions on PPC/Panther Server/WO 5.2.3.
Any help would be greatly appreciated.
Timo
PS: As Apple is going out of the tools business for WO and wants
to concentrate more on the framework, I *really* hope they take a
deep look into EOF concurrency and cross-VM cache synchronization.
I guess that will remain a wish but would make life soooooo much
easier...
PPS:
The meat of the code is:
a) in some thread read and write data concurrently (with correct
locking)
b) simulate incoming change notifications in another tread
EOEditingContext ec = new EOEditingContext();
try {
ec.lock();
ec.rootObjectStore().lock();
ec.rootObjectStore().invalidateAllObjects(); // just for testing,
real code would only invalidate certain GIDs
} finally {
ec.rootObjectStore().unlock();
ec.unlock();
ec.dispose();
}
Result is e.g.:
java.lang.IllegalStateException:
rowDiffsForAttributes:
snapshot in com.webobjects.eoaccess.EODatabaseOperation {
_dbSnapshot = {};
_entity = "TestEntity";
_newRow = {id = 45; testValue = "Thread-2/2200"; lastUpdate =
2006-09-13 15:12:40 Etc/GMT; };
_object = "{values = {lastUpdate = 2006-09-13 15:12:40 Etc/GMT;
testValue = "Thread-2/2200"; };
this = "<timo.testet.TestEntity 5b8e6c _EOIntegralKeyGlobalID
[TestEntity (java.lang.Integer)45]>"; }";
_globalID = _EOIntegralKeyGlobalID[TestEntity (java.lang.Integer)45];
_databaseOperator = "EODatabaseUpdateOperator";
}
does not contain value for attribute named id with snapshot key: id
Sample of deadlock with sharedEC involved while invalidating
objects on rootObjectStore:
"ActiveMQ Session Task" daemon prio=7 tid=0x005d7bb0
nid=0x30ad8e00 in Object.wait() [f181e000..f1820b20]
at java.lang.Object.wait(Native Method)
- waiting on <0x999cb18> (a
com.webobjects.foundation.NSMultiReaderLock$ConditionLock)
at java.lang.Object.wait(Object.java:429)
at com.webobjects.foundation.NSMultiReaderLock$ConditionLock.await
(NSMultiReaderLock.java:505)
- locked <0x999cb18> (a
com.webobjects.foundation.NSMultiReaderLock$ConditionLock)
at com.webobjects.foundation.NSMultiReaderLock._lockForWriting
(NSMultiReaderLock.java:204)
at com.webobjects.foundation.NSMultiReaderLock.lockForWriting
(NSMultiReaderLock.java:165)
at com.webobjects.eocontrol.EOSharedEditingContext.lock
(EOSharedEditingContext.java:700)
at com.webobjects.eocontrol.EOEditingContext.lockObjectStore
(EOEditingContext.java:4673)
at com.webobjects.eocontrol.EOEditingContext.refaultObject
(EOEditingContext.java:3980)
at er.extensions.ERXEC.refaultObject(ERXEC.java:753)
at
com.webobjects.eocontrol.EOEditingContext._refaultObjectWithGlobalID(
EOEditingContext.java:3232)
at
com.webobjects.eocontrol.EOEditingContext._newChangesFromInvalidating
ObjectsWithGlobalIDs(EOEditingContext.java:3461)
at
com.webobjects.eocontrol.EOEditingContext._processObjectStoreChanges(
EOEditingContext.java:3497)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at com.webobjects.foundation.NSSelector.invoke(NSSelector.java:354)
at com.webobjects.foundation.NSSelector._safeInvokeSelector
(NSSelector.java:108)
at
com.webobjects.eocontrol.EOEditingContext._processNotificationQueue
(EOEditingContext.java:4750)
at com.webobjects.eocontrol.EOEditingContext.lock
(EOEditingContext.java:4627)
at er.extensions.ERXEC.lock(ERXEC.java:259)
at com.webobjects.eocontrol.EOEditingContext.tryLock
(EOEditingContext.java:4639)
at
com.webobjects.eocontrol.EOEditingContext._sendOrEnqueueNotification(
EOEditingContext.java:4714)
at
com.webobjects.eocontrol.EOEditingContext._objectsChangedInStore
(EOEditingContext.java:3537)
at er.extensions.ERXEC._objectsChangedInStore(ERXEC.java:919)
at sun.reflect.GeneratedMethodAccessor88.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at com.webobjects.foundation.NSSelector._safeInvokeMethod
(NSSelector.java:120)
at com.webobjects.foundation.NSNotificationCenter
$_Entry.invokeMethod(NSNotificationCenter.java:598)
at com.webobjects.foundation.NSNotificationCenter.postNotification
(NSNotificationCenter.java:542)
at com.webobjects.foundation.NSNotificationCenter.postNotification
(NSNotificationCenter.java:572)
at
com.webobjects.eocontrol.EOObjectStoreCoordinator._objectsChangedInSu
bStore(EOObjectStoreCoordinator.java:744)
at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at com.webobjects.foundation.NSSelector._safeInvokeMethod
(NSSelector.java:120)
at com.webobjects.foundation.NSNotificationCenter
$_Entry.invokeMethod(NSNotificationCenter.java:598)
at com.webobjects.foundation.NSNotificationCenter.postNotification
(NSNotificationCenter.java:542)
at com.webobjects.foundation.NSNotificationCenter.postNotification
(NSNotificationCenter.java:572)
at
com.webobjects.eoaccess.EODatabaseContext._snapshotsChangedInDatabase
(EODatabaseContext.java:3812)
at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at com.webobjects.foundation.NSSelector._safeInvokeMethod
(NSSelector.java:120)
at com.webobjects.foundation.NSNotificationCenter
$_Entry.invokeMethod(NSNotificationCenter.java:598)
at com.webobjects.foundation.NSNotificationCenter.postNotification
(NSNotificationCenter.java:542)
at com.webobjects.foundation.NSNotificationCenter.postNotification
(NSNotificationCenter.java:572)
at com.webobjects.eoaccess.EODatabase.forgetSnapshotsForGlobalIDs
(EODatabase.java:951)
at
com.webobjects.eoaccess.EODatabaseContext.forgetSnapshotsForGlobalIDs
(EODatabaseContext.java:2397)
at
com.webobjects.eoaccess.EODatabaseContext.invalidateObjectsWithGlobal
IDs(EODatabaseContext.java:3781)
at
com.webobjects.eocontrol.EOObjectStoreCoordinator.invalidateObjectsWi
thGlobalIDs(EOObjectStoreCoordinator.java:718)
at er.changenotification.ERCNSubscriber._processUpdates
(ERCNSubscriber.java:166)
at er.changenotification.ERCNSubscriber.onMessage
(ERCNSubscriber.java:103)
at org.apache.activemq.ActiveMQMessageConsumer.dispatch
(ActiveMQMessageConsumer.java:795)
at org.apache.activemq.ActiveMQSessionExecutor.dispatch
(ActiveMQSessionExecutor.java:96)
at org.apache.activemq.ActiveMQSessionExecutor.iterate
(ActiveMQSessionExecutor.java:149)
at org.apache.activemq.thread.PooledTaskRunner.runTask
(PooledTaskRunner.java:110)
at org.apache.activemq.thread.PooledTaskRunner.access$100
(PooledTaskRunner.java:25)
at org.apache.activemq.thread.PooledTaskRunner$1.run
(PooledTaskRunner.java:43)
at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:650)
at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:552)
Timo
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden