Re: CoreData performance issues
Re: CoreData performance issues
- Subject: Re: CoreData performance issues
- From: Tomek Piatek <email@hidden>
- Date: Wed, 22 Jun 2005 17:36:53 +1200
On 22/06/2005, at 5:29 PM, Chris Hanson wrote: On Jun 21, 2005, at 5:10 PM, Tomek Piatek wrote:
We have written a simple test application which reads in a line at a time from the text file and for each record creates a managed object instance. The managed object context is saved only once after all records have been processed. We want to avoid duplicates and in fact we want to reuse existing managed objects. For example, when ingesting a new scene we want to create a new project object only if one with the given name doesn't already exist.
It sounds like what you're doing is something like this:
1. Read a line of input. 2. Decide which entity to instantiate for it. 3. Execute a fetch request to see if an instance with that name already exists. a. If it does, do nothing. b. If it doesn't, create it. 4. Connect its relationships. 5. Repeat.
That is exactly what we're doing. I'm not sure if you're familiar with Perl's Class::DBI modules. In a way they are similar to CoreData. Class::DBI contains a convenience method called "find_or_create". We use it all the time for this sort of thing with no adverse effects. Typically we develop against MySQL. This isn't the most efficient way to do this kind of batch import. Most importantly, don't issue fetch requests for step 3. Instead, just keep track of the objects you create during the import by name in a dictionary. If you can run subsequent imports, then do a fetch once at the very beginning to "prime" your dictionary, and use that to determine uniqueness for the rest of the import.
This makes sense and someone on the list has already suggested this approach. The problem I see is with the size of this lookup structure. We create hundreds of thousands of objects. They will all end up in the managed context which stores them in memory. I'm not sure at what point the context flushes managed objects back into persistent store. I know it does it when you send it a "save" message. Our application has to run on a laptop computer with limited memory and it's very important that this import does not take hours. Minutes is ok, but not hours.
-tomek
|
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Cocoa-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden