Core Data Fetch request very poor performance ??!
Core Data Fetch request very poor performance ??!
- Subject: Core Data Fetch request very poor performance ??!
- From: Aurélien Hugelé <email@hidden>
- Date: Tue, 29 Nov 2005 08:22:39 +0100
Hi list !
i've modified my source code to use Core Data instead of NSArchiving/
NSUnarchiving to/from disk.
Basically, my code parse all mbox files (the .emlx files) in the ~/
Library/Mail folder using Pantomime for decoding, and store metadatas
informations like subject, date, sender, recipients, number of files
etc... in an object of class GCMKEmail that should be saved to disk.
At the end of the parsing i get about 50 000 objects of class GCMKEmail
Once everything is parsed, i used to save the emails "metatadas" in
*several* archive files using NSArchiver. The process was not fast
enough for my needs, and of course, the more emails i had the worsw
was the performance.
Now using core data in SQL format, it is even worse, i mean REALLY
worse because it is extremly slow, not to save, but to use to
retrieve my objects!
Fetching for all emails is *extremly* slow (i mean more than 1 min
for about 50 000 objects on a G5 2x2Ghz with 3GB of RAM!)
Only 2 fetches requests are executed, and they basically ask for all
stored emails.
The code of -[GCMKEmailManager storeAllObjectsOfMailBoxAtPath:] is
like this :
NSFetchRequest* fr = [[NSFetchRequest alloc] init];
[fr setEntity:[NSEntityDescription entityForName:@"Email"
inManagedObjectContext:[self managedObjectContext]]]; // the fetch
request does not have any predicate, because we want ALL objects of
the database
NSError* error;
NSArray* allEmails = [[self managedObjectContext]
executeFetchRequest:fr error:&error]; // this takes more than a
minute to execute !!
if(!allEmails)
{
NSLog(@"NSError:%@",error);
}
NSDictionary* uniqueIdsToEmails = [NSDictionary
dictionaryWithObjects:allEmails forKeys:[allEmails
valueForKey:@"uniqueId"]]; // cache the fetched objects
....
the email entity is very simple : it only has attributes (no
relationship at all!) (about 28 attributes, of "native" types :
string, number, bool)
i've joined an excerpt of the Shark report :
| | | | | | | | | | | | | | | | | | + 10.4 s -[GCMKEmailManager
upgradeCommunicationsToV3Format:] (GumiComKit)
| | | | | | | | | | | | | | | | | | | + 10.4 s -[GCMKEmailManager
storeAllObjectsOfMailBoxAtPath:] (GumiComKit)
| | | | | | | | | | | | | | | | | | | | + 9.0 s -
[NSManagedObjectContext executeFetchRequest:error:] (CoreData)
| | | | | | | | | | | | | | | | | | | | | + 8.4 s -
[NSPersistentStoreCoordinator(_NSInternalMethods)
executeRequest:withContext:] (CoreData)
| | | | | | | | | | | | | | | | | | | | | | + 8.4 s -[NSSQLCore
executeRequest:withContext:] (CoreData)
| | | | | | | | | | | | | | | | | | | | | | | + 8.4 s -[NSSQLCore
objectsForFetchRequest:inContext:] (CoreData)
| | | | | | | | | | | | | | | | | | | | | | | | + 7.9 s -[NSSQLCore
_newRowsForSelector:withArgument:andLimit:] (CoreData)
| | | | | | | | | | | | | | | | | | | | | | | | | + 7.4 s -
[NSSQLChannel newFetchedRowObjectIDsOnly:] (CoreData)
| | | | | | | | | | | | | | | | | | | | | | | | | | + 5.5 s -
[NSSQLiteConnection newFetchedRowObjectIDOnly:] (CoreData)
| | | | | | | | | | | | | | | | | | | | | | | | | | | - 1.6 s -
[NSSQLiteConnection _newValueForColumnOfSQLType:atIndex:inStatement:]
(CoreData)
| | | | | | | | | | | | | | | | | | | | | | | | | | | + 1.5 s -
[NSSQLiteConnection _execute] (CoreData)
| | | | | | | | | | | | | | | | | | | | | | | | | | | | + 1.5 s
_sqlite3_step (libsqlite3.0.dylib)
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | + 1.5 s
_sqlite3VdbeExec (libsqlite3.0.dylib)
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | + 264.1
ms _sqlite3RunVacuum (libsqlite3.0.dylib)
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | + 66.5
ms _sqlite3VdbeMemRelease (libsqlite3.0.dylib)
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
49.4 ms _szone_size (libSystem.B.dylib)
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | +
12.1 ms _free (libSystem.B.dylib)
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
1.0 ms _szone_size (libSystem.B.dylib)
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 64.5
ms _szone_free (libSystem.B.dylib)
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 14.1
ms ___spin_lock (commpage [libSystem.B.dylib])
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 6.0
ms _free (libSystem.B.dylib)
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 4.0
ms _dyld_stub_free (libsqlite3.0.dylib)
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1.0
ms ___spin_unlock (commpage [libSystem.B.dylib])
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | + 245.0
ms _sqlite3BtreeNext (libsqlite3.0.dylib)
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | + 218.8
ms _sqlite3BtreeDataFetch (libsqlite3.0.dylib)
....
can anyone point me on any direction to improve performance ? or is
SQLite a very poor choice from apple ?
it seems 99% of the time is spent in low level sqlite obscure
calls.... did i designed any thing wrong ? Core Data can't support
more that one thousands of objects ??
i would be happy to give anything to people that would like to help
(my data model, my database which is quite big : about 15MB, my
source code, and even possibly a benchmarking code)
thanks
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Cocoa-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden