• Open Menu Close Menu
  • Apple
  • Shopping Bag
  • Apple
  • Mac
  • iPad
  • iPhone
  • Watch
  • TV
  • Music
  • Support
  • Search apple.com
  • Shopping Bag

Lists

Open Menu Close Menu
  • Terms and Conditions
  • Lists hosted on this site
  • Email the Postmaster
  • Tips for posting to public mailing lists
Re: many-to-many, worker-threads, save problems
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: many-to-many, worker-threads, save problems


  • Subject: Re: many-to-many, worker-threads, save problems
  • From: Chuck Hill <email@hidden>
  • Date: Tue, 13 Jun 2006 13:10:03 -0700

Been there! Here is an old message that I have saved. I have not tested this, instead I have been using another workaround. I won't publish that unless this one does not work as this fix is far better than my ugly hack.

Chuck

	From: 	  email@hidden
	Subject: 	Re: EOF Updating object I'm trying to delete
	Date: 	November 21, 2004 9:34:42 AM PST (CA)
	To: 	  email@hidden
	Cc: 	  email@hidden

Chuck,

Thanks so much for shedding some light on this. It was driving me crazy!!

On Friday, Nov 19, 2004, at 18:08 America/New_York, Chuck Hill wrote:


Lenny,

Welcome to Hell. :-( What we have here is a persistent and long term EOF bug. I've never found a work around. The reason that you only see it on the deployment machine is that it is concurrency related. Sigh, well, you're not going to like this...

The problematic situation occurs when ObjectStore changes for registered objects are processed while the method <code>processRecentChanges()</code> is running. If this happens, the editing context state for inserted, deleted, and updated objects is lost. This has unpredictable results including attempting to save as updates those records being deleted which then fails validateForSave. The request to process ObjectStore changes can come as a result of faults firing during processRecentChanged() or thetriggering of a refetch of data as the snapshot is older than what the editing context permits and the fetched row being different that what was in the ObjectStore.



That was it. I did some experimenting and was finally able to reliably reproduce the bug.


Ex.
Employee <<-> Department

I created a simple app that lists employees with there departments. There are links to delete each employee. In the deleteEmployee() action method I refault the department and update the ec.fetchTimestamp to force the condition.

 public WOComponent deleteEmployee() {
        EOEditingContext ec = employeeItem.editingContext();

 	ec.setFetchTimestamp(System.currentTimeMillis());

//force a refault. The relationship may happen under normal
//circumstances in response to an ObjectChangedInStoreNotification.
ec.refaultObject((EOEnterpriseObject)employeeItem.valueForKey ("department"));


        ec.deleteObject(employeeItem);
        ec.saveChanges();

        return null;
    }

1. load page
2. update department directly in database
3. click to delete employee
4. Success

If I perform the above steps with more than one session under the application, it fails every time. I can see that for whatever reason, if there is only one ec with the department registered in in it, then there's only one ObjectsChangedInStoreNotification posted from the OSC, which includes both the deleted employee, and the updated department. If there is more than one ec, with the department registered then I see a notification immediately containing only the updated department. Then the errror.

com.webobjects.eoaccess.EOGeneralAdaptorException: EvaluateExpression failed: <com.webobjects.jdbcadaptor.OpenBasePlugIn $OpenBaseExpression: "UPDATE EMPLOYEE SET DEPARTMENT_ID = NULL WHERE (ID = ? AND FIRST_NAME = ? AND LAST_NAME = ? AND TITLE is NULL AND DEPARTMENT_ID = ?)" withBindings: 1:2(id), 2:"Joe"(firstName), 3:"Smith"(lastName), 4:1(departmentId)>:
Next exception:SQL State:42000 -- error code: 0 -- msg: ERROR - Value can not be NULL for column 'DEPARTMENT_ID'


I can see from logging messages I've inserted, and various stack dumps, that the problem occurs when _processObjectStoreChanges is called while _preparePushForChanges is executing. The problem disappears if I remove the back relationship from department- >employees. ***Also, this happens all within the same thread. The application does not have to be handling concurrent requests. I guess, if you do make sure to pre-fault all involved relationships before the delete as you mentioned, and the app is single threaded, the problem would be avoided because there would be no other threads changing the object store.


What I have done to work around this, as much as possible, is to fire all the fault that might get touched during saveChanges (mostly a problem when evaluating delete rules for an EO about to be deleted. I also override public void _processObjectStoreChanges(NSDictionary changes) to detect this condition and throw which at least gives me a fighting chance of cleaning up (dispose the EC etc.). Very nasty.



Very nasty is right! Definitely some race conditions going on. I came up with the following workaround that seems to work, without forcing changes to the model or having to make sure all relationships faults have been fired. (not well tested yet though) I've spent way too much time on this one. I'm planning on packaging up my sample app and submitting it with a bug report to Apple, but it will have to wait till I catch up a bit.


public class WorkAroundEc extends EOEditingContext {
    private boolean savingChanges;
    private NSMutableArray queuedNotifications = new NSMutableArray();

    public void saveChanges() {

        _EOAssertSafeMultiThreadedAccess("saveChanges()");
        savingChanges = true;
        try {
            super.saveChanges();
        } finally {
            savingChanges = false;
        }

        processQueuedNotifications();
    }

    public void _objectsChangedInStore(NSNotification nsnotification) {
        if(savingChanges) {
            queuedNotifications.addObject(nsnotification);
        } else {
            super._objectsChangedInStore(nsnotification);
        }
    }

    private void processQueuedNotifications() {
        System.out.println("processQueuedNotifications()");

        synchronized(queuedNotifications) {
            for(Enumeration e = queuedNotifications.objectEnumerator();
                e.hasMoreElements();) {

                NSNotification n = (NSNotification)e.nextElement();

                _objectsChangedInStore(n);

            }
            queuedNotifications.removeAllObjects();
        }
    }

}

-lenny



Chuck

P.S. There was (or is, I don't recall exactly) another related problem. When an editing context finishes saving it sends a notification that the other editing contexts listen to. The other editing contexts listen for this notification and update their part of the object graph if any relevant objects have changed. The bug takes place when one EO is registered in two (or more!) editing contexts and they both get saved at around the same time. The first save finishes and sends the notifications. The receipt of these notification causes the second EC to lose its internal state and get confused about what is an insert and what is a delete and it starts to treat the half finished operations as updated. I _think_ proper locking prevents this bug, but I'm not 100% certain at the moment



On Nov 19, 2004, at 2:24 PM, Lenny Marks wrote:


This one's got me stumped!

We have a WO5.2.3 app that runs inside Tomcat. There is an entity called RefereTag which has a mandatory to-one 'tagger' relationship to an Individual. In the database(which happens to be Oracle), there is a non null constraint on TAG.TAGGER_IND_ID. Via the WO app, there is a page where you can view, add, and delete the tags. On our live deployment machine, AND ONLY on our live deployment, there are sporadic errors when attempting to delete a tag. Some users claim up to one and every 3 times fails. The stacktrace in the error log is below. It is an EOGeneralAdaptorException. The database rejecting EOF's attempt to UPDATE the tag setting its tagger to null. The thing is that no where in the application are tags updated. Only added and deleted. I've been absolutely unable to reproduce this in our testing environment(which is the same as live) or on my development server. I've even gone so far as to use HttpUnit to repetitively add and remove tags through the app. I added some code to dump the stack inside Tag.setTagger whenever the new value is null. Surprisingly this showed that relationships are always nulled out by EOClassDescription.propagateDeleteForObject() up the stack from ec.saveChanges() . It seems like EOF is making it past the nulling out of the relationships, but somehow ends up thinking that the object is updated instead of deleted. I found a thread from a while back that seemed related.

http://www.omnigroup.com/mailman/archive/webobjects-dev/2004-March/ 038915.html

In it, Chuck, mentions that there is a bug in EOEditingContext where if validateForDelete fails, things go awry. In this case, I don't think validateForDelete is failing since EOF is actually trying to update the object in the DB. Also, I see no validation exceptions. I think its scary that if not for the DB constraint, It looks like EOF would have happily updated the object with incorrect data.

Vanilla code used to delete the object:

EOEditingContext ec = refereeTag.editingContext();
ec.deleteObject(refereeTag);
ec.saveChanges();

Other clues/theories:

- Maybe something related to weak references being reclaimed by the garbage collector which may explain why this only happens on the live server.
- In this case, there is no undo manager for the editing context.
- Notifications?
- There is only one editing context involved in the action, and it is definitely properly locked.
- The tags are always fetched via EOUtilities.objectsMatchingKeyAndValue because there is a
legacy app that can also add and remove tags.
- To cope with tags changing out from under, I actually include the globalId of the tag in the url which is checked in the action to ensure that say the 2nd tag in the worepetion is the same object that was there
when the response was generated.


I have a couple of ideas about things might help:
-manually nulling out tags relationships.
-fetching into brand new ec just for delete
-using an undo manager.
-just manually delete object with EOAdaptorChannel.deleteRowDescribedByQualifier(drastic)


Its just a pain since it can only be reproduced on our live box. I hate to just throw stuff in and have to wait to here if anyone still complains to know if it's fixed. I'd love to know the the cause of the problem and which workaround was the winner.

ANYWAY(I know this was long winded), any input would be greatly appreciated.

-lenny

com.webobjects.eoaccess.EOGeneralAdaptor
Exception: EvaluateExpression failed: <com.webobjects.jdbcadaptor.OraclePlugIn$OracleExpression: "UP
DATE MANUSCRIPTS.REFEREE_TAG SET TAGGER_IND_ID = NULL WHERE (REFTAG_ID = ? AND DATE_TAGGED = ?)" wit
hBindings: 1:55387(reftagId), 2:2004-11-19 15:36:32(dateTagged)>:
Next exception:SQL State:72000 -- error code: 1407 -- msg: ORA-01407: cannot update ("MANUSCRIPT
S"."REFEREE_TAG"."TAGGER_IND_ID") to NULL


at com.webobjects.eoaccess.EODatabaseContext._exceptionWithDatabaseConte xtInformationAdded(E
ODatabaseContext.java:4657)
at com.webobjects.eoaccess.EODatabaseContext.performChanges (EODatabaseContext.java:6365)
at com.webobjects.eocontrol.EOObjectStoreCoordinator.saveChangesInEditin gContext(EOObjectSto
reCoordinator.java:411)
at com.webobjects.eocontrol.EOEditingContext.saveChanges (EOEditingContext.java:3148)
at prism.refview.RefTagsComponent.deleteTag(Unknown Source)



Lenny Marks
American Physical Society


On Jun 13, 2006, at 12:32 PM, Christian Pekeler wrote:

It seems that not just ec.saveChanges is resetting the EOObserverCenter (so that it notifies the ec about changes to my objects again), but also ec.processRecentChanges. At the end of WorkerThread3 (the one that creates my object), ec.processRecentChanges is not being invoked. Calling it explicitly (from my component's sleep()) appears to fix my problem (i.e. once WorkerThread3 becomes active again, changes to my object are properly recognized and saved).

I've found this four year old thread
http://www.wodeveloper.com/omniLists/eof/2002/January/msg00015.html
which talks about ec.processRecentChanges not always being called at the end of events. Has anyone come up with a good rule of thumb when processRecentChanges should be called explicitly? Is it OK so just always call it from my component superclass's sleep()?


Thanks,
Christian


On Jun 13, 2006, at 12:02, Christian Pekeler wrote:
I'm having a strange problem with EOF that I think I've never seen before. Every now and then, the editing context will not save the changes I've done to a many-to-many relationship. I see that the addTo and removeFrom methods are always called on my object, which are always calling willChange, but then it looks like EOObserverCenter.notifyObserversObjectWillChange (which is called by willChange) does not always call EOEditingContext.objectWillChange(). So, right before calling ec.saveChanges I see that my object's many-to-many has changes which are different from the snapshot at the EODatabase level, however, my ec doesn't contain my object in its updatedObjects array.

Digging deeper I've noticed that this problem only happens when the worker thread that first created my object is being reused. So I'm seeing the following pattern:

WorkerThread3 (creates my object, but doesn't save yet)
WorkerThread4 (saves successfully)
WorkerThread5 (doing anything, for example saving changes to my object's many-to-many)
WorkerThread6 (dito)
...until...
WorkerThread3 (saving changes to my object's many-to-many - this one fails - always)



I'm always using the same ec. I can only reproduce this with changes to many-to-many relationships. I'm not creating my object in an unusual way. I'm not catching any saveChanges exceptions.


Inspecting EOObserverCenter, it appears that it uses an internal hashmap with threads as keys and uses it to prevent calling ec.objectWillChange for the same object more than once in the same transaction. I believe the ec is resetting the EOObserverCenter after saveChanges. So for some reason, the ec is not resetting the EOObserverCenter for the worker thread that first created my object. Has anyone seen this before? Anyone have an idea what's going on here?

_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
40global-village.net


This email sent to email@hidden

--
Coming sometime... - an introduction to web applications using WebObjects and Xcode http://www.global-village.net/wointro


Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems. http://www.global-village.net/products/practical_webobjects




_______________________________________________ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (email@hidden) Help/Unsubscribe/Update your Subscription: This email sent to email@hidden
  • Follow-Ups:
    • Re: many-to-many, worker-threads, save problems
      • From: Christian Pekeler <email@hidden>
References: 
 >many-to-many, worker-threads, save problems (From: Christian Pekeler <email@hidden>)
 >Re: many-to-many, worker-threads, save problems (From: Christian Pekeler <email@hidden>)

  • Prev by Date: Re: many-to-many, worker-threads, save problems
  • Next by Date: Re: many-to-many, worker-threads, save problems
  • Previous by thread: Re: many-to-many, worker-threads, save problems
  • Next by thread: Re: many-to-many, worker-threads, save problems
  • Index(es):
    • Date
    • Thread