Re: Long latencies in realtime threads when doing BSD syscalls -- Bug or expected behaviour?
Re: Long latencies in realtime threads when doing BSD syscalls -- Bug or expected behaviour?
- Subject: Re: Long latencies in realtime threads when doing BSD syscalls -- Bug or expected behaviour?
- From: Mario Kleiner <email@hidden>
- Date: Tue, 03 Feb 2004 20:46:16 +0100
- Organization: Max Planck Institute for Biological Cybernetics
Hello Mr. Quinn,
Thanks a lot for your quick & helpful answer! I came to the same
conclusions after my first extended safari through the xnu-kernel
source, but as i am mostly from a Linux background - used to the way how
Linux handles locking/smp/preemption et al. - and quite new to OS-X, i
needed some confirmation regarding xnu's locking et al.
So, am i right that the kernel_funnel has this effect on
all BSD syscalls, even on uniprocessor machines?
Yes.
Ah ja, needs to be on uniprocessor as well, because of the preemptible
kernel functions, which pose the same requirements as really parallel
(==smp) threads...
Does this mean, that the kernel funnel is automatically
released if a thread sleeps in the kernel,
Yes. See the routine thread_block_reason in the following file.
<http://www.opensource.apple.com/darwinsource/Current/xnu-517.3.7/osfmk/kern/sched_prim.c>
[This link requires an APSL <http://www.opensource.apple.com/apsl/>
account.]
but that it is
not released, if a thread gets preempted by a realtime
thread -- which would cause the realtime thread to get
delayed/blocked on the lock, as soon as it does a BSD
syscall -- basically some kind of priority inversion?
Yes.
Let me just say on behalf of Apple that we recognise that funnels are
bad. We're actively working to eliminate them and introduce finer
grained locking to the BSD portions of the kernel. However, with
current versions of Mac OS X, they are unavoidable.
I understand. But as introducing fine grained locking in an existing
kernel is quite a time consuming and incremental endeavour, we probably
shouldn't expect this to be completed in the near future? Can i assume
that it will take rather a couple of months or years instead of a couple
of weeks or months?
That would make perfect sense to me, but why doesn't the
same thing happen to the gettimeofday() syscall that i use
for timing and the usleep() when it gets back from sleep?
My guess (didn't have time to check) is that these routines are
implemented on top of Mach primitives, not BSD. Non-BSD syscalls don't
need to take any funnels.
I've checked now: usleep is implemented on top of the Mach-API and
gettimeofday is one of a few BSD-calls that doesn't need the funnels.
Is there a way to avoid this, e.g. by only using Apples
Frameworks or Mach API's instead of Posix stuff?
Virtually all frameworks use BSD at some point (you can't access the file
system or the network without it), so you can't draw the general
conclusion that Apple frameworks won't end up blocking on a funnel.
However, there are some frameworks that are safe. For example, all of
Mach and all of I/O Kit don't need the funnel.
What about things like OpenGL, collecting keyboard responses or mouse
button presses? Do the frameworks use Unix device files for
communication with the device drivers (-> ioctls/read/write/mmap -> BSD
calls -> block on funnel) or do they use the Mach-API? The main concern
is not file i/o but querying the mouse and keyboard for user responses
and OpenGL graphics output with well controlled timing, see below for
reason.
Our current -- ugly -- solution is to kill the update()
process during realtime parts of the application. Is there
something better, e.g. the equivalent of SCHED_FIFO
scheduling on other Unices?
The best solution is to avoid doing BSD stuff in your realtime thread.
For example, let's say you're playing an audio file, reading from disk
and playing via Core Audio (which calls you back in a real-time
thread). The best way to do this is to separate the disk read and audio
play into two threads, insert a buffer between the two, and make the
disk thread non-realtime.
If you can explain exactly what your realtime thread does, I can perhaps
offer some alternative suggestions.
Avoiding BSD stuff in this case is difficult: The application is called
Psychtoolbox for MacOS-X (see
http://www.psychtoolbox.org/osx.html ). It
is an extension to/on top of Mathworks Matlab. It is used for
programming psychophysical experiments on human visual perception in
Matlab -- basically presentation of visual stimuli via OpenGL and
collection of keyboard and mouse responses with controlled timing. Its
very popular under vision scientists on the old OS-9 Macs and Windows
and they are porting it to OS-X at the moment.
On OS-9 they achieved "good enough" realtime behaviour by blocking
processor interrupts and deferred actions and switching off virtual memory.
In the OS-X implementation, "realtime" behaviour is realized by
promoting the priority of the Matlab thread to THREAD_CONSTRAINT_POLICY
during the experiments, but the content of the realtime part of an
experiment can be any kind of mixture of OpenGL rendering, sound output,
keyboard/mouse/hid checks, computation and blocking for a defined amount
of time. The only thing it probably doesn't need to do in the realtime
part is file i/o and large scale memory allocation. As a thread needs to
block for some time on each iteration of the experiment loop, in order
to not get demoted by the scheduler to normal timesharing mode due to
overusage of cpu, it can always happen that some other thread does
syscalls inbetween, which take the funnel lock.
Another problem: Even if they could avoid any BSD syscalls in their
code, the Matlab interpreter loop contains periodic calls to
sigsetjmp(), which in turn calls sigprocmask() - which is a BSD syscall
and causes the delays on the kernel funnel. As Mathworks seems to be not
too enthusiastic about rewriting Matlab to avoid this function, the
psychtoolbox guys have a problem. The whole history of the bug can be
found under:
http://www.psychtoolbox.org/osxTiming.html
But anyway, knowing the root cause of the problem may help them to find
a solution or workaround -- even if it boils down to killing the update
process and calming down other sources of noise in the system.
And a last question: Is there an equivalent to mlockall()
to prevent paging for a realtime app?
My stock answer to this question is that, if the system is short enough
on memory that it's paging out parts of your app that you're touching at
realtime, things are sufficiently messed up that the wiring wouldn't
have helped.
Coming from Linux - maybe the VM implementation is completely different
in behaviour on OS-X - i would assume that you get delays even if you
have enough memory, due to things like copy-on-write and zerofill on
demand allocation of memory pages on first access and "faulting in"
executable code, when a code path is executed the first time after
application startup. On Linux, a mlockall(MCL_CURRENT | MCL_FUTURE)
command would guarantee that all parts of the adress space are in core
when the call returns.
However, this appropriateness of this answer depends on
what the negative effect of missing your realtime goal is. For example,
glitching audio is not too bad on the overall scale of things, but
failing to lower the rods on your nuclear reactor is bad (sufficiently
bad that you shouldn't be using Mac OS X to control it).
Messing up a few trials in an experiment is not that catastrophic,
although too much unwanted delays can pose problems for fMRI experiments
AFAIK, but how did you know that i have a nuclear reactor ;)
Any more helpful comments are appreciated and thanks again.
ciao,
-mario
--
*********************************************************************
Mario Kleiner
Max Planck Institute for Biological Cybernetics
Spemannstr. 38
72076 Tuebingen
Germany
e-mail: email@hidden
office: +49 (0)7071/601-734
fax: +49 (0)7071/601-616
www:
http://www.kyb.tuebingen.mpg.de/bu/people/kleinerm/index.html
*********************************************************************
_______________________________________________
darwin-kernel mailing list | email@hidden
Help/Unsubscribe/Archives:
http://www.lists.apple.com/mailman/listinfo/darwin-kernel
Do not post admin requests to the list. They will be ignored.