Re: Garbage collection, core data, and tight loops
Re: Garbage collection, core data, and tight loops
- Subject: Re: Garbage collection, core data, and tight loops
- From: "John R. Timmer" <email@hidden>
- Date: Wed, 31 Oct 2007 08:37:08 -0400
On Oct 30, 2007, at 5:56 PM, Bill Bumgarner wrote:
This sounds like you might have a situation where allocations are
outrunning the collector (a known issue that is being worked on).
Specifically, if you are allocating tons and tons of memory in a
tight loop, the collector can't scan and purge the garbage quickly
enough and, thus, you can cause this kind of behavior (there is more
information on this in the documentation).
However, that could easily not be the case -- I would need more
information to know.
Specifically -- are you periodically saving changes in your loop?
I have been periodically saving changes in the loop.
Are you asking Core Data to purge objects?
I neglected to manually purge the objects - i assume that would
involve using the ManagedObjectContext's -reset method?
Does your object graph allow partial subgraphs to exist in memory
without eventually faulting everything into memory?
At this point, I'm only importing a single type of managed object to
see how well things work with millions of records, and to make sure
the project's feasible as designed before going further, so this is a
non-issue.
... do you have any kind of caches or hashes that are effectively
keeping the objects around?
No, nothing should be retaining the objects - i'm setting values and
moving on.
I'll give it another shot with resetting the ManagedObjectContext and
see if that does the trick.
Thanks,
John
On Oct 30, 2007, at 2:03 PM, John R.Timmer wrote:
I'm trying to process an enormous XML file that should ultimately
fill a Core Data sqlite store with millions of records. This
should only happen once, so i'm willing to let the app go non-
responsive; i'm doing all the parsing in an extremely tight loop.
My first time through, i did things with manual memory management,
but the memory use kept going up. So, i switched to mandatory
Garbage Collection, leaving my autorelease pool statements in
place, since they should be converted to no-ops. I confirmed that
a collector thread existed when i launched the app.
It still ran out of memory:
malloc: *** auto malloc[333]: Zone::Can not allocate 0x8200000 bytes
I'm assuming the problem is an interaction between the tight loop
and either Core Data or the Garbage Collector:
The GC doesn't recognize which objects to collect while the loop is
still chugging away.
The sqlite store can't send objects out to disk and release their
memory while the loop is still chugging away.
I'm leaning towards the latter explanation.
I'm going to redesign things so that the main runloop has a chance
to act sporadically, but i was just curious as to which of these is
a problem so that i can design specifically to avoid this in the
future.
This sounds like you might have a situation where allocations are
outrunning the collector (a known issue that is being worked on).
Specifically, if you are allocating tons and tons of memory in a
tight loop, the collector can't scan and purge the garbage quickly
enough and, thus, you can cause this kind of behavior (there is more
information on this in the documentation).
However, that could easily not be the case -- I would need more
information to know.
Specifically -- are you periodically saving changes in your loop?
Are you asking Core Data to purge objects?
Does your object graph allow partial subgraphs to exist in memory
without eventually faulting everything into memory?
... do you have any kind of caches or hashes that are effectively
keeping the objects around?
b.bum
_______________________________________________
Cocoa-dev mailing list (email@hidden)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden