Re: Search Kit memory management
Re: Search Kit memory management
- Subject: Re: Search Kit memory management
- From: Michael Thon <email@hidden>
- Date: Sun, 1 Aug 2010 18:10:02 +0200
On Aug 1, 2010, at 3:12 PM, Ken Thomases wrote:
> On Aug 1, 2010, at 1:43 AM, Michael Thon wrote:
>
>> I am indexing possibly tens of thousands of documents. No matter what I tried, I could not get search kit to release its memory, even after closing the index. The only way I could prevent memory usage from growing too much is to do the indexing in a separate thread and shut down the thread. I have 'solved' the problem by indexing documents batch-wise, every 1000 or so in a separate thread using an NSOperation.
>
> That doesn't make much sense. In general, "shutting down" a thread doesn't release objects. There is the relatively rarely used feature known as thread-local storage, implemented in Cocoa as -[NSThread threadDictionary], which is released when the thread exits. I'd be fairly surprised if Search Kit was using that for much.
>
> So, if exiting out of a thread releases a significant number of objects, that's probably more due to simply exiting out of each stack frame of the thread in an orderly fashion, cleaning up as you go. For example, releasing autorelease pools.
>
> I would also point out that using an NSOperationQueue, if that's what you're currently using to run your NSOperation, does not give you any guarantees about thread lifetimes. So, you may not actually know that threads are being terminated.
>
Yes I am using NSOperationQueue and I add an operation using addOperationWithBlock: Then I wait for that operation to finish using waitUntilAllOperationsAreFinished before adding a new one. In fact I don't know if if threads are being reused, etc. I should probably just stick to my observations and state that "Real Mem" in Activity Monitor is reduced by several hundred MB when the NSOperation finishes indexing a few thousand documents.
> I reiterate my suggestion to use the Object Allocations instrument, configured to record reference counts, to investigate. Even if you investigate your current solution, it will illuminate why and when the objects are released, which can help you understand why it seems to fix the issue and lead to a better fix.
>
I did check the retain counts in Instruments and I see that there are sometimes more than 1000 SKDocument objects with a retain count of 1, even though my code creates these objects and releases them one at a time. In Instruments I cannot see who has actually retained these objects, but I will consult the documentation to see where this info is hidden. At the time, I concluded that they must be retained by Search Kit, which was out of my control anyway, so I abandoned this line of investigation.
The comments by Aaron make me think that perhaps I should not be so concerned about the memory usage reported by Activity Monitor (and that I'd better understand how it reports memory in the first place). When I watch object allocations in Instruments, I see that the total number of living objects (# living for the All Allocations row) oscillates between a maximum number and minimum number and this is reflected in the graph. After a minute or so the oscillations appear to even out and the number # living tend to stabilize and even slowly decreases over the next few minutes. The memory listed in the Live Bytes column is much lower than that reported by Activity Monitor and reflects exactly what I see in # living objects. Memory usage in the Live Bytes column stabilizes at less than 100 MB. My application seems to be indexing documents at a more or less constant rate. All the while Activity Monitor reports that my application is using more and more "Real Mem" .
My interpretation is that the memory usage of my application is fine, according to Instruments. I don't know if it is safe to ignore what Activity Monitory reports though. I will find some more documentation and try to figure out what it is telling me.
Best,
Mike
> Regards,
> Ken
>
_______________________________________________
Cocoa-dev mailing list (email@hidden)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden