Re: Recursion taking up far too much memory... any ideas?
Re: Recursion taking up far too much memory... any ideas?
- Subject: Re: Recursion taking up far too much memory... any ideas?
- From: John Stiles <email@hidden>
- Date: Tue, 5 Jun 2007 10:29:24 -0700
A couple of thoughts:
- 50MB is really not very much RAM. Using 50MB of RAM is highly
preferable to creating 200 threads.
- You don't describe your algorithm enough for me to say for sure,
but most kinds of recursion (particularly tail recursion) can be
converted into non-recursive code. For instance, where you would
normally recurse to do something, you could instead add an entry to
the end of an queue of "things to do," and then continually drain
that task queue.
On Jun 3, 2007, at 10:37 AM, Sam Stigler wrote:
Hi,
I'm writing a program to automatically, recursively, fetch every
single Amazon BrowseNode from BrowseNodes.com (I subscribed to
their web service) into a Core Data persistent store that I can use
in another program. What it does is go through every single
category, then every single category within each of those
categories, etc., and builds a Core Data-based tree out of them.
The problem I'm having is that the first time I ran the program,
(1) it took forever (but Shark says this is due mainly to Cocoa API
calls that I can't get rid of unfortunately), but more importantly,
(2) it took up a whole lot of memory -- up to around 50 MB by the
time it was far less than halfway through. The heavy lifting is
being done in a separate thread with its own autorelease pool; it
looks to me like the problem is that the autorelease pools aren't
being released until the recursions (if you can call a void method
a recursion) are finished.
Can anyone confirm if I'm right about this; and give any ideas on
how to correct the problem (whatever the problem is)? Can it be
corrected? I'm currently spinning off all of the recursive calls
(that is, all but the first one) into the second thread, but I
tried spinning those all off into their own thread and while it
took up less memory, I got hundreds of threads and ended up having
to deal with racetrack conditions -- not ideal. I've posted my
relevant code to http://homepage.mac.com/sstigler1985/
recursionProblem.html .
Some additional notes: Firstly, I've tried searching the archives
for this issue and found something on it, but the message thread
was very hard to follow, and it's unclear if the issue was actually
solved in the end. I'm hoping someone could (please?) clear it up
for me. Second, when I mentioned "racetrack conditions" for me
using all threads instead of just going halfway, I meant that I'm
getting seemingly-random mutateErrors appending a certain string in
the middle of the addInfoAboutNode method, somewhere in the middle
of the run of the program. It's my hypothesis that this is due to
racetrack conditions with at least 170-odd threads competing for
the variables, but I've put one NSRecursiveLock over each of the
methods (see http://homepage.mac.com/sstigler1985/
recursionProblem_allThreads.html) and that didn't make a
difference. Has anyone seen this problem before? The weird thing
is that in the racetrack conditions, I'm getting the exception at
the same line each time:
[currentStatus appendString:nodeTitle]; .
This is my first major use of recursion (I'm finally getting my
head around it), so it's very possible that I've missed something;
but having said that, I think the general algorithm is sound.
Please let me know, though, if there's something glaringly obvious
that I've left out.
Thanks for any help you can give me,
Sam
_______________________________________________
Cocoa-dev mailing list (email@hidden)
Do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
40blizzard.com
This email sent to email@hidden
_______________________________________________
Cocoa-dev mailing list (email@hidden)
Do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden