Re: System resources and number of operations on an NSOperationQueue
Re: System resources and number of operations on an NSOperationQueue
- Subject: Re: System resources and number of operations on an NSOperationQueue
- From: Antonio Nunes <email@hidden>
- Date: Sun, 14 Nov 2010 18:39:14 +0000
On 14 Nov 2010, at 16:25, Ken Thomases wrote:
> The system can't predict the future. When an operation is pending in a queue, there's no way for the system to know if it's going to be CPU intensive, memory intensive, and/or I/O intensive.
>
> Let's suppose the system primarily governs its thread pool based on available CPU time. Let's also suppose that your operations start with some blocking I/O but then switch to computation. When there are operations queued, but most of the already-running operations are blocked doing I/O, then the CPU may have idle capacity. So, the system starts another operation (or several). It will have effectively "overbooked" the CPU -- in short order, as I/O requests are satisfied, there will be more operations doing CPU-based computation then there is CPU capacity to handle them.
>
> It's also the case that disk I/O degrades significantly with contention. So, even just having a bunch of I/O-intensive operations queued can burden the system, even while there's plenty of spare CPU and memory capacity.
Ok, thanks for the clear explanation. You've probably hit the nail on the head. The tasks do some initial computation, but that is really, really fast. The next bit is writing the result to a file on disk, and that is really slow, especially with big documents.
> However, one approach to working around the system's limitations is to separate your operations into pure I/O operations and pure CPU operations. (If one logical operation involves both, you can split it and use dependencies to have the two halves run in the appropriate order.)
I'm not sure that is an option, but I don't think that matters anyway, since the computational part of the operation is almost negligible. It's writing the data to disk that is most likely causing contention.
>> Also, is there a way to find out the number of cores on a machine so that I can set that as a hard limit of number of operations on an NSOperationQueue?
>
> -[NSProcessInfo activeProcessorCount]
Short and sweet. Thanks!
I see there is also processorCount. The docs totally don't explain what the difference is. Is an active processor one that is available for processing, but not necessarily being used, or is it one that is definitely in the process of churning bits?
> It appears that your program is launching copies of itself to process individual files. I suspect you're "bringing the whole system to a crawl" by a fork bomb. Have you checked that your program is not infinitely-recursively launching subprocesses?
Yes, it is not recursively launching subprocesses.
> Also, why are you re-running your program in a subprocess, instead of just having your operation do the work directly? (I suspect you do it that way in case processing of a particular file causes a crash. You want to limit the crash to just the process handling that file.)
Well, that is one benefit of this approach, but it wasn't the main reason. It looks though, like I will need to re-asses my decision and see if i can get better results by having the operation doing the work directly.
> Have you considered letting launchd monitor the directories, launch your daemon, and deal with any crashes?
Nope. Did not even know that was possible. Another interesting option to look into.
> I'll also say that having NSOperations launch an NSTask and block waiting for it to exit is nearly a worst case scenario for NSOperationQueue. It has a thread pool with idle threads, so it figures it can pull a new operation off the queue to start it with minimal new resources. That operation not only monopolizes the thread, it does effectively nothing with it. Instead, it launches a whole separate process, which is a significant use of new resources.
Yes, that would explain the behaviour I'm seeing quite well, I believe. Back to the drawing board. Thanks so much for so clearly expounding the issues involved.
António
-----------------------------------------------------------
Don't believe everything you think
-----------------------------------------------------------
_______________________________________________
Cocoa-dev mailing list (email@hidden)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden