Core Data caching very large property values in 10.6
Core Data caching very large property values in 10.6
- Subject: Core Data caching very large property values in 10.6
- From: Quincey Morris <email@hidden>
- Date: Mon, 19 Jul 2010 16:47:43 -0700
I'm not sure if I can describe this problem clearly enough, but here goes ...
I have a Core Data GC application that has been working fine under Leopard, but now crashes in a certain operation under Snow Leopard. It appears to be running out of memory (the 32-bit version produces a console error from 'malloc'; the 64-bit version just crashes in 'malloc'). The GC is being invoked explicitly ('collectExhaustively') from time to time, and it seems that the collector is *not* being outrun.
The operation that fails is a kind of image import. (I'm simplifying here, but that's the essence.) The images are compressed PNGs, typically about 200-400 KB each, and the total size of the import can be anything from a couple of hundred MB of image data to a couple of GB. As it's imported, each compressed image is stored as a NSData object property in a Core Data persistent store.
Obviously that amount of data can't be in memory at once, so there is a caching scheme. The images are represented by an Image entity, which has metadata (like the width and height) plus a 1-1 relationship to an ImageCache entity whose single property is the above-mentioned NSData with the compressed image data.
This arrangement allows image data to be dumped out of memory by faulting out the ImageCache object and allowing the NSData to be collected, while the metadata stays available all the time. (That's why ImageCache has to be a separate entity -- if the image data were a property of Image, there'd be no way to fault out just the image data.)
All of this worked well enough under Leopard. However, under Snow Leopard the faulted-out ImageCache image data property is still alive and *not* being collected.
From the debugger, I followed the lifetime of one of the NSData objects. Initially, it's kept alive by its owning ImageCache object, which is turn is kept alive by being in the managed context's "inserted objects" data structures. After each group of images, the import code saves the Core Data context, then attempts to fault out all of the recently created Core Data objects (there are other objects whose existence is supposed to be transitory, too), then invokes the garbage collector.
At this point, the NSData is unreferenced by application code, but unfortunately still alive, being rooted like this:
> Root:
> 0 Kind: stack rc: 0 Address: 0xb020f804 Frame level: 0 Symbol: <unknown>
> 1 Kind: object rc: 0 Address: 0x0137e3c0 Class: NSManagedObjectContext ivar: _objectStore
> 2 Kind: object rc: 1 Address: 0x013708e0 Class: NSPersistentStoreCoordinator ivar: _persistentStores
> 3 Kind: object rc: 1 Address: 0x013708c0 Offset: 0x00000014 Class: NSCFArray
> 4 Kind: bytes rc: 0 Address: 0x0137d9a0 Offset: 0x00000010
> 5 Kind: object rc: 0 Address: 0x01377450 Offset: 0x00000044 Class: NSSQLCore
> 6 Kind: object rc: 0 Address: 0x0137d290 Offset: 0x00000004 Class: NSSQLRowCache
> 7 Kind: object rc: 0 Address: 0x0137cf70 Offset: 0x0000001c Class: NSCFDictionary
> 8 Kind: bytes rc: 0 Address: 0x011f0c00 Offset: 0x00000554
> 9 Kind: bytes rc: 0 Address: 0x012f2e30 Offset: 0x0000002c
> 10 Kind: object rc: 0 Address: 0x01317280 Offset: 0x00000000 Class: NSConcreteData
AFAICT, this is telling me that Core Data has itself cached the NSData object in a "row cache". That's very bad news, because it seems to persist in there indefinitely, until the managed object context is finally disposed of (which I can't do in the middle of the import).
Does anyone know of a way to empty or disable the row cache, or can anyone provide a better insight into what's going wrong?
_______________________________________________
Cocoa-dev mailing list (email@hidden)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden