September 2007 - Darwin-kernel

Re: panic routine?
by Alexander von Below 10 Sep '07

10 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Is panic () in debug.h not what you are looking for? Alex Am 10.09.2007 um 17:03 schrieb Stefan Jeglinski: I read Amit Singh's article here: http://www.osxbook.com/book/bonus/chapter5/panic/ Stefan Jeglinski _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/below%40mac.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… I don't have immediate access to his larger treatise, perhaps he talks about it more there, but in the above article, he includes the panic routine for UNIX circa 1973. I was wondering if there is a single panic routine in Darwin, akin to panic.c in Linux, and if so, where is it? I note, for example, that the Apple source for Libc has a panic.c and a panic routine within. But the panic routine in debug.c in the xnu source *looks* like it might be the "main" one. It seems that Apple makes it very hard to download the entire source tree in one shot - I haven't gotten around to searching the whole source tree for examples. I'm just looking for answers to very simple questions. I've been tasked with a small presentation to a mostly non-technical audience, but one that can make sense of a small amount of source code and wants to get a slightly deeper flavor than the casual user. I of course want to know more about it as well. This email sent to below(a)mac.com This email sent to site_archiver(a)lists.apple.com

1 0

panic routine?
by Stefan Jeglinski 10 Sep '07

10 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com I read Amit Singh's article here: http://www.osxbook.com/book/bonus/chapter5/panic/ Stefan Jeglinski _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… I don't have immediate access to his larger treatise, perhaps he talks about it more there, but in the above article, he includes the panic routine for UNIX circa 1973. I was wondering if there is a single panic routine in Darwin, akin to panic.c in Linux, and if so, where is it? I note, for example, that the Apple source for Libc has a panic.c and a panic routine within. But the panic routine in debug.c in the xnu source *looks* like it might be the "main" one. It seems that Apple makes it very hard to download the entire source tree in one shot - I haven't gotten around to searching the whole source tree for examples. I'm just looking for answers to very simple questions. I've been tasked with a small presentation to a mostly non-technical audience, but one that can make sense of a small amount of source code and wants to get a slightly deeper flavor than the casual user. I of course want to know more about it as well. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Yielding the processor in a kext?
by Terry Lambert 08 Sep '07

08 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com So by just using thread_block(), we might not surrender the timeslice. I'll do some more real-life testing. Anything in P=1? No. Look in next lower band. Anything in P=2? No. Look in next lower band. Anything in P=3? Yes -> Run thing in P=3 band again. -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Sep 7, 2007, at 4:41 PM, Régis Duchesne wrote: Indeed, sched_yield() calls the swtch_pri() system call with zero as only argument (the argument is not used by swtch_pri() so it does not matter anyway). That in turn is connected to thread_block_reason() which you cannot call form a kext but that is called from thread_block() which I suggested to you in my previous email so my answer of using thread_block(THREAD_CONTINUE_NULL) may well turn out to be a winner here! I did the same investigation, but came to a different conclusion: swtch_pri actually uses thread_block_reason(THREAD_CONTINUE_NULL, NULL, AST_YIELD) and the only difference is that: 1) thread_block_reason is not exported 2) AST_YIELD forces the timeslice of the current thread to 0 I have say 20 priority bands, and the thread I want to yield is off all by it's lonesome at say P=3, and the things you want to yield to are all in P=6, then going to the back of the line in the P=3 priority band and rescheduling will get you exactly: ...in other words, you aren't yielding to what you think you are yield to, unless you jack around the priority. Your only input, other than using an FSA (which I see you say you can't use because you are using a "black box thread") is to control your thread priority. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Yielding the processor in a kext?
by Terry Lambert 08 Sep '07

08 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com 1) When a long-running task is being performed at an inappropriate priority, and it might be "nice" to let other threads have the CPU. In this case, I argue that the task should simply be moved to another thread that runs at an appropriate priority. So let's say I realize that one of my threads is too far ahead of the others in its computation. I want to slow it down, so I'll reduce its priority. o User A starts the app with 2 threads TA1 and TA2, and this scenario occurs, TA1 is way ahead of TA2, so TA1 lowers is priority. o User B starts the app with only 1 thread TB. -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Sep 7, 2007, at 10:10 AM, Régis Duchesne wrote: Michael, Now imagine that 2 different users start 2 different instances of the app: At this point, user A loses: TA1 will almost never be scheduled anymore, because TB will always have a higher priority. This is only true if your thread priority is PTHREAD_SCOPE_SYSTEM, rather than PTHREAD_SCOPE_PROCESS, and therefore each of the threads in your process compete on an equal footing with every process in the system. In your example above, you want your processes to compete equally as units, irrespective of number of threads, so your use is inapprpriate; I suggest you look at adding a pthread_attr_t with process scoping for scheduling. I really want to lower the priority of my thread _within_ its task (because the scheduling of that thread relative to the other in the same task is a decision that can be best made by my app). But the APIs that I find only allow me to do it system-wide. This is not exatly what you are saying here, if I am interpreting "will almost never be scheduled" as a complaint about the scope value for your competing for quantum. So as a workaround, I want a way to yield the processor, i.e. go at the end of the scheduling queue but with the exact same priority. This is a hack; there are many different ways you could avoid this hack, almost all of which are a better fit for the problem of geting process-scope "fair share" while keeping SMP scalability. Realize also that User A is always going to get more quanta on an SMP system because their implementation is not SMP scalable; it's impossible to guarantee fairness in this situation without hitting user B in the head with a shovel and telling them to use the sysctl's to find out how many threads they should be starting, instead of the incorred number they are. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Yielding the processor in a kext?
by Terry Lambert 08 Sep '07

08 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com It's not your business to make scheduling decisions; don't even try. In other words, I'm trying to write my own little scheduler (just for all the threads within my task), on top (not instead of) the xnu system-wide scheduler. I'm the only one who can do that because xnu obviously doesn't know the specifics of my application. -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Sep 7, 2007, at 9:59 AM, Régis Duchesne wrote: Michael Smith wrote: That is just not true. In the example I gave in my previous email, I show that sometimes, the application knows better than the OS how things must be scheduled. It does not matter how much progress all my threads make compared to the rest of the system, but it does matter how much progress they make relative to each other. Your scheduler should be a "work-to-do" cooperative multitasking scheduler, with your large thread pool scheduled onto a smaller set of real backing threads to do the work. This would allow you to queue up work units and manage "number of work units per thread" as a figure of merit on your own work queues. Terry Lambert wrote: You block on the mutex held by the other thread Well that does not work in my case either: as I explained, I don't want to stop the thread that is too far ahead in the computation. I just want to slow it down. So blocking is not an option. Do not move it from your implementation of the "too many work units completed" to your implementation of "needs to complete more work units now" queues so that when an idle backing thread becomes available and takes the next item of your "needs to complete more work units now", it won't see the object. I want to hint the scheduler that "hey, you made me run, but at the app level I know that it is better if I do nothing at this moment, so I would like to voluntarily relinquish my time slice." Actually, you want to control the scheduler queue order, which you are not permitted to do. What you _are_ permitted to do is abstract the system scheduler from your workload using your own queues and a finite state automata, effectively implementing "scope=my_task " prioritization of threads within your task. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Yielding the processor in a kext?
by Anton Altaparmakov 08 Sep '07

08 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Hi Régis, So by just using thread_block(), we might not surrender the timeslice. I'll do some more real-life testing. But yes, real-life testing will show what it really does. (-: Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer, http://www.linux-ntfs.org/ _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On 8 Sep 2007, at 00:41, Régis Duchesne wrote: Indeed, sched_yield() calls the swtch_pri() system call with zero as only argument (the argument is not used by swtch_pri() so it does not matter anyway). That in turn is connected to thread_block_reason() which you cannot call form a kext but that is called from thread_block() which I suggested to you in my previous email so my answer of using thread_block(THREAD_CONTINUE_NULL) may well turn out to be a winner here! I did the same investigation, but came to a different conclusion: swtch_pri actually uses thread_block_reason(THREAD_CONTINUE_NULL, NULL, AST_YIELD) and the only difference is that: 1) thread_block_reason is not exported 2) AST_YIELD forces the timeslice of the current thread to 0 Correct but due to 1) it is irrelevant as I cannot see any way to specify AST_YIELD from a kext (I am assuming you will not manage to get Apple to export thread_block_reason()) and as to 2) I am not convinced it matters. Looking at the code, thread_select() only cares about the timeslice value (which is the only thing set to zero by AST_YIELD) when the current process priority is at REALTIME so that is not an issue (unless your threads run realtime I suppose but I am assuming here that they are not). Further choose_thread() does not care about the timeslice at all. It unconditionally switches to the next thread in the run queue even going to the trouble of stealing a task from a different CPU run queue if needed. So perhaps what actually happens is that calling with AST_YIELD kills the current time slice and starts a new one whereas AST_NONE (used by thread_block()) hands the remainder of the current time slice to a new thread. Again, please take what I say with a pinch of salt as I am not exactly an OSX scheduler expert having never looked at the code before this evening! This email sent to site_archiver(a)lists.apple.com

1 0

Re: Yielding the processor in a kext?
by Régis Duchesne 07 Sep '07

07 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com User-agent: Icedove 1.5.0.9 (X11/20061220) Anton, > Indeed, sched_yield() calls the swtch_pri() system call with zero as > only argument (the argument is not used by swtch_pri() so it does not > matter anyway). That in turn is connected to thread_block_reason() > which you cannot call form a kext but that is called from thread_block() > which I suggested to you in my previous email so my answer of using > thread_block(THREAD_CONTINUE_NULL) may well turn out to be a winner here! I did the same investigation, but came to a different conclusion: swtch_pri actually uses thread_block_reason(THREAD_CONTINUE_NULL, NULL, AST_YIELD) and the only difference is that: 1) thread_block_reason is not exported 2) AST_YIELD forces the timeslice of the current thread to 0 So by just using thread_block(), we might not surrender the timeslice. I'll do some more real-life testing. -- hpreg _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Yielding the processor in a kext?
by Anton Altaparmakov 07 Sep '07

07 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Hi Régis, You are welcome! Great, looking forward to them. (-: Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer, http://www.linux-ntfs.org/ _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On 7 Sep 2007, at 23:36, Régis Duchesne wrote: (void)thread_block(THREAD_CONTINUE_NULL); And best of all thread_block() is exported in Mach.exports so it is a stable kernel API. Definitely worth a try I would think! If it hangs (I don't think it will but I may have misunderstood!) or if it always schedules the same process back (I don't think it will do that either!) then you haven't lost anything but if my reading of the scheduling code is correct this will indeed perform a context switch to the next in line task on the run queue and will keep doing this until your task gets to be the next in line and the context switch then brings your task back and thread_block() returns. Note as I read the code when your task is the only task on the run queue thread_block() will simply return immediately. PS. If you try this I would be very grateful if you tell me how it worked out as I have already spent too long looking at the code when I have lots of other work to do! Looks promising. Thanks for your investigation. I'll definitely give this a try and report my findings. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Yielding the processor in a kext?
by Anton Altaparmakov 07 Sep '07

07 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer, http://www.linux-ntfs.org/ _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On 7 Sep 2007, at 23:29, Régis Duchesne wrote: Jonas wrote: You could have a look at to which kernel api sched_yield() is connected. Good idea. Indeed, sched_yield() calls the swtch_pri() system call with zero as only argument (the argument is not used by swtch_pri() so it does not matter anyway). That in turn is connected to thread_block_reason() which you cannot call form a kext but that is called from thread_block () which I suggested to you in my previous email so my answer of using thread_block(THREAD_CONTINUE_NULL) may well turn out to be a winner here! This email sent to site_archiver(a)lists.apple.com

1 0

Re: Yielding the processor in a kext?
by Régis Duchesne 07 Sep '07

07 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com User-agent: Icedove 1.5.0.9 (X11/20061220) Anton, > (void)thread_block(THREAD_CONTINUE_NULL); > And best of all thread_block() is exported in Mach.exports so it is a > stable kernel API. Definitely worth a try I would think! If it hangs > (I don't think it will but I may have misunderstood!) or if it always > schedules the same process back (I don't think it will do that either!) > then you haven't lost anything but if my reading of the scheduling code > is correct this will indeed perform a context switch to the next in line > task on the run queue and will keep doing this until your task gets to > be the next in line and the context switch then brings your task back > and thread_block() returns. > > Note as I read the code when your task is the only task on the run queue > thread_block() will simply return immediately. > > PS. If you try this I would be very grateful if you tell me how it > worked out as I have already spent too long looking at the code when I > have lots of other work to do! Looks promising. Thanks for your investigation. I'll definitely give this a try and report my findings. -- hpreg _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0