Re: CoreData pagination
Re: CoreData pagination
- Subject: Re: CoreData pagination
- From: Ben Trumbull <email@hidden>
- Date: Mon, 23 Jun 2008 19:07:29 -0700
Anyway, is there any way to query CoreData in chunks? I have a very
large CoreData SQl store (>1 million) of managed objects and it's
taking for ever (well, tens of minutes) to load it all in, never mind
to search through it all.
(1) Why do you want to load it all in ?
(2) Why won't you let Core Data search it with a fetch request+predicate ?
(3) What are you trying to accomplish ? Why do you want to show the
user millions of records simultaneously ?
(4)
2008-06-23 18:34:41.098 CoreDataFetchTool[459:10b] CoreData:
annotation: sql connection fetch time: 0.7982s
2008-06-23 18:34:41.493 CoreDataFetchTool[459:10b] CoreData:
annotation: total fetch execution time: 1.2037s for 1000000 rows.
Fetched 1000000 items at 789505 rows per second.
Total test time: 1.2666s
So, you're running 3+ orders of magnitude off the lower bound. You
should try looking at things with Shark and Instruments.
(5) As Jim suggests, you can get back just the object IDs with:
[request setResultType:NSManagedObjectIDResultType];
[request setIncludesPropertyValues:NO];
then I get:
2008-06-23 18:36:58.495 CoreDataFetchTool[521:10b] CoreData:
annotation: sql connection fetch time: 0.2418s
2008-06-23 18:36:58.505 CoreDataFetchTool[521:10b] CoreData:
annotation: total fetch execution time: 0.2575s for 1000000 rows.
Fetched 1000000 items at 3608102 rows per second.
Total test time: 0.2772s
Then you can use an IN predicate with a sliding window of objectIDs.
Perhaps in batches of a thousand.
Any more than a few tens of thousands of results and it's slow down city.
I can get 10,000 items back 3 times per user event (100ms).
You're probably thrashing the VM system. That's the easiest way to
burn a few orders of magnitude in performance.
In addition to the above, you should consider your model (schema) and
break up your primary entity so ancillary attributes are not a part
of this large fetch.
I'm about to embark on a testing run with using multiple stores
You should use Shark and Instruments to fix the current problems
first. It sounds like you're memory bound. Use less of it.
While multiple stores can be useful, they also prevent us from
leveraging the maximal number of optimizations. Results may need to
be sorted in memory (since none of the databases have all of them),
and various other issues.
--
-Ben
_______________________________________________
Cocoa-dev mailing list (email@hidden)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden