Re: move large data structure to Core Data?
Re: move large data structure to Core Data?
- Subject: Re: move large data structure to Core Data?
- From: James Maxwell <email@hidden>
- Date: Sun, 03 Mar 2013 14:57:16 -0800
Hi Uli,
Thanks for the thoughtful reply. I do think I could probably reduce the data load of my graph by wider use of primitives, but it would require a fair bit of re-engineering, and there's not a lot of time to do that… (Have to finish my PhD someday!) Also, since the system should, at least in theory, be able to learn from absurdly huge corpora, I think I'd still likely hit the memory wall at some point. Which is simply to say that freeing myself from memory limitations is probably a good idea anyway. Having the data structures on disk will, of course, slow things down considerably, but I think will be worth the trade-off. I'll keep poking around with Core Data until I figure out a reasonable solution.
That said, I'll also put further thought into how I might reduce the memory footprint of my model, as is. I'm sure it's possible.
Thanks,
J.
On 2013-03-03, at 1:57 AM, Uli Kusterer <email@hidden> wrote:
> Just a question: Do you really *need* NSDictionaries? I.e. is this something where arbitrary string keys need to be mapped to objects? Because otherwise this might be one of the cases where NSDictionary is too generic, and you may just want to create your own class.
>
> E.g. when I started out, I did a game engine with all NSDictionaries, NSNumbers etc. Rewriting the dictionaries to be simple objects with NSPoint iVars etc. yielded a huge speed-up (because I was obviously kinda abusing NSDictionary).
>
> Objects are preferable, but they do have a trade-off: For each of them, you allocate a new block on the heap, which on most platform has an overhead of at least 8 bytes (32-bit) or 16 bytes (64-bit) to keep a pointer to the class (isa) and the block's length. Often it uses more to allow the system to keep track of empty spots on the heap.
>
> If your payload is a 4-byte int, that means each NSNumber "wastes" about 2x to 4x the memory it actually needs (and that's a conservative estimate). So if it makes sense to group several ints or floats in ivars of a single object, that should quickly reduce your memory use. Similarly, if you have a list of primitive types, not objects, you can use CFArray with custom callbacks to store ints directly instead of boxing each one in an NSNumber.
>
> Or just grab one of the numerous IntArray classes out there that mallocs its own buffer, if all you need is really a straight, linear, indexed array w/o all the optimizations that NSArray uses to speed up search for a particular value etc., but you want the convenience of an ObjC API that CFArray doesn't quite offer.
>
> Cheers,
> -- Uli Kusterer
> "The Witnesses of TeachText are everywhere..."
>
>
> On 03.03.2013, at 05:23, James Maxwell <email@hidden> wrote:
>> Hello All,
>>
>> I have an app that creates a large and complex graph structure. It's working nicely now, after a fairly lengthy development process, but I'm having scaling issues -- the data structure gets too big to keep in memory. So, I'm thinking about Core Data. The thing is, the process of building the object graph is very specific (it's a complex undirected graph structure, built via machine learning) and since I've got it working properly I really don't want to port the whole thing over to Core Data, since that would just be begging for a huge headache, I'm sure.
>>
>> All of the data is held in NSMutableDictionaries (basically working as adjacency lists), owned by one particular class. What I'm wondering is whether there's some way to use Core Data to perform the work of the dictionaries, without necessarily rebuilding all of the object relationships etc. using Core Data entities? Looking around a bit it looks like "Transformable" attributes might offer a solution. What I'd like to do is to basically swap out the NSMutableDictionaries for Core Data entities that store my custom objects as transformable attributes. Is that at all feasible? Worth trying?
>>
>> Thanks in advance.
>>
>> J.
>> _______________________________________________
>>
>> Cocoa-dev mailing list (email@hidden)
>>
>> Please do not post admin requests or moderator comments to the list.
>> Contact the moderators at cocoa-dev-admins(at)lists.apple.com
>>
>> Help/Unsubscribe/Update your Subscription:
>>
>> This email sent to email@hidden
>
_______________________________________________
Cocoa-dev mailing list (email@hidden)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden