February 2004 - Darwin-kernel

RE: Assembly macros with Gcc
by Brad Post (MACBU) 04 Feb '04

04 Feb '04

Sorry, let me clarify, I am writing assembly code, using a .s file with GCC, I don't need to know how to do macros with c or cpp files, but with assembly files. Thanks. Brad -----Original Message----- From: darwin-kernel-admin(a)lists.apple.com [mailto:darwin-kernel-admin@lists.apple.com] On Behalf Of Brad Post (MACBU) Sent: Wednesday, February 04, 2004 10:48 AM To: darwin-kernel(a)lists.apple.com Subject: Assembly macros with Gcc Anyone have a good reference on how to do macros in assembly? I'm trying to get some macros to work with gcc and Project Builder (not Xcode) and having some troubles, so does anyone know a good reference with decent examples? I'm trying to do something like this: #define SAVE_STATISTICS(stat, reg) lwz reg, stat(r3) addi reg, reg, 1 stw reg, stat(r3) I've tried a bunch of stuff, but gcc doesn't seem to like it and I'm hoping someone here might have a good reference book or website.... Thanks. Brad _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored. _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Long latencies in realtime threads when doing BSD syscalls -- Bug or expected behaviour?
by Jim Magee 04 Feb '04

04 Feb '04

On Feb 3, 2004, at 2:46 PM, Mario Kleiner wrote: Our current -- ugly -- solution is to kill the update() process during realtime parts of the application. Is there something better, e.g. the equivalent of SCHED_FIFO scheduling on other Unices? The best solution is to avoid doing BSD stuff in your realtime thread. For example, let's say you're playing an audio file, reading from disk and playing via Core Audio (which calls you back in a real-time thread). The best way to do this is to separate the disk read and audio play into two threads, insert a buffer between the two, and make the disk thread non-realtime. If you can explain exactly what your realtime thread does, I can perhaps offer some alternative suggestions. Avoiding BSD stuff in this case is difficult: The application is called Psychtoolbox for MacOS-X (see http://www.psychtoolbox.org/osx.html ). It is an extension to/on top of Mathworks Matlab. It is used for programming psychophysical experiments on human visual perception in Matlab -- basically presentation of visual stimuli via OpenGL and collection of keyboard and mouse responses with controlled timing. Its very popular under vision scientists on the old OS-9 Macs and Windows and they are porting it to OS-X at the moment. Note that avoiding BSD calls is not really sufficient anyway. A BSD funnel is just a special form of Mach mutex (that gets dropped when you voluntarily sleep), but a mutex just the same. And those are the same things that Mach uses to guard many of its (and IOKit's) resources. It may be more likely to conflict with someone holding the BSD funnel (mutex) than say the Mach port namespace or address space lock for your process/task, but eventually you will hit that one already locked too. The point is not to eliminate conflicts, but to "bound" them in duration. This takes two parts: The first part falls on all Apple/Darwin kernel developers - and that is to limit singular lock holds to an acceptable latency value. This is an ongoing process. And yes, as we are eliminating reliance on the funnels we are also trying to bound the holds on the new locks that replace it. But this will be an ongoing effort for some time to come. And doing things like file I/O operations will likely always fall into the category of having too high a latency for time critical threads. The second part is to assure these lock hold latencies don't stay unbounded because of priority inversions (the lock hold is low priority - starved of the CPU by a medium priority thread - which means your high priority thread will never get the lock it needs). We already have priority inheritance mechanisms in place - with more and better ones to come in the future (for instance we can't take mutexes out of their FIFO ordering yet because of of potential starvation issues - but we are working on remedying that). --Jim _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: How to write Network Kernel Extensions
by Koryn Grant 04 Feb '04

04 Feb '04

On Wed, 4 Feb 2004 mamuneeb(a)hss.hns.com wrote: > Hi, > > I am not sure if this is the right place to ask this question !! > > I am looking for a "netfilter" kind of utility in MAC OS X or above. > How can I write a protocol handler NKE, which diverts all the packets of a > protocol, say TCP, packets to a handler program. You don't need to write an NKE for this - try using an ipfw rule that sends the desired packets to a divert socket, and have a userland application open the divert socket and process the packets. 'man divert' and 'man ipfw' will give more info. Cheers, Koryn -- Dr Koryn Grant phone +64 7 8346729 Senior Software Engineer mobile +64 21 488386 Endace Technology Ltd email koryn(a)endace.com Hamilton, New Zealand web http://www.endace.com/ _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Assembly macros with Gcc
by Jonas Maebe 04 Feb '04

04 Feb '04

On 4 feb 2004, at 19:48, Brad Post (MACBU) wrote: Anyone have a good reference on how to do macros in assembly? And less arcane: <http://developer.apple.com/documentation/DeveloperTools/gcc-3.3/gcc/ Asm-Blocks-and-Functions.html#Asm%20Blocks%20and%20Functions> :) Jonas _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Assembly macros with Gcc
by Jonas Maebe 04 Feb '04

04 Feb '04

On 4 feb 2004, at 19:48, Brad Post (MACBU) wrote: Anyone have a good reference on how to do macros in assembly? <http://developer.apple.com/documentation/DeveloperTools/gcc-3.3/gcc/ Extended-Asm.html#Extended%20Asm> Jonas _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Assembly macros with Gcc
by Brad Post (MACBU) 04 Feb '04

04 Feb '04

Anyone have a good reference on how to do macros in assembly? I'm trying to get some macros to work with gcc and Project Builder (not Xcode) and having some troubles, so does anyone know a good reference with decent examples? I'm trying to do something like this: #define SAVE_STATISTICS(stat, reg) lwz reg, stat(r3) addi reg, reg, 1 stw reg, stat(r3) I've tried a bunch of stuff, but gcc doesn't seem to like it and I'm hoping someone here might have a good reference book or website.... Thanks. Brad _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

How to write Network Kernel Extensions
by mamuneeb＠hss.hns.com 04 Feb '04

04 Feb '04

Hi, I am not sure if this is the right place to ask this question !! I am looking for a "netfilter" kind of utility in MAC OS X or above. How can I write a protocol handler NKE, which diverts all the packets of a protocol, say TCP, packets to a handler program.In linux the netfilter provides the facility to write hooks to diverts the packets for packet filtering or mangling. Data link NKEs,Protocol handler NKEs are provided in MAC. But I am not sure how they help me. Or how to use them to write a handler function. regards, -muneeb- _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Long latencies in realtime threads when doing BSD syscalls -- Bug or expected behaviour?
by Quinn 04 Feb '04

04 Feb '04

At 20:46 +0100 3/2/04, Mario Kleiner wrote: Ah ja, needs to be on uniprocessor as well, because of the preemptible kernel functions, which pose the same requirements as really parallel (==smp) threads... Right. We also need it because I/O Kit executes most 'interrupt' code on a workloop thread. These threads call up into BSD for things like disk I/O completion and network packet delivery. They must acquire the appropriate funnel on the way up to keep BSD data structures consistent. It's worth noting that the splxxx routines (eg splnet) are NOPs on Mac OS X. [Actually, they do some helpful debugging checks but no actual work.] This is fine because all BSD code runs under a funnel and all interrupt delivery is done by I/O Kit threads that acquire the funnel before calling up into BSD. There's a very short (and information dense) section of Technote 2028 that explains this. <http://developer.apple.com/technotes/tn/tn2028.html#MacOSXKernelThreading> But as introducing fine grained locking in an existing kernel is quite a time consuming and incremental endeavour, we probably shouldn't expect this to be completed in the near future? Can i assume that it will take rather a couple of months or years instead of a couple of weeks or months? We stated at WWDC 2003 that finer grain locking will be shipped as part of a post-Panther major OS release. At this time we've made no public schedule commitment, so I can't really comment. What about things like OpenGL, collecting keyboard responses or mouse button presses? Do the frameworks use Unix device files for communication with the device drivers (-> ioctls/read/write/mmap -> BSD calls -> block on funnel) or do they use the Mach-API? The main concern is not file i/o but querying the mouse and keyboard for user responses and OpenGL graphics output with well controlled timing, see below for reason. Most user space driver access on Mac OS X is done through obvious BSD API (networking and file system) or via the I/O Kit "user client" mechanism. See "IMOSX: Accessing Hardware From Applications" for information about this. <http://developer.apple.com/documentation/DeviceDrivers/Conceptual/Accessing…> However, my concern is that the frameworks might end up hitting BSD for silly reasons. For example, even something as trivial as getpid takes the kernel funnel (the first time you call it; the response is cached in user space). Frameworks, in general, are complex things, and it's very hard to predict how they behave in this respect. Worse yet, even if they do happen to avoid funnels in the current implementation, that's not part of their documented API and there's no guarantee that they'll maintain this behaviour in the future. * * * I didn't have time to read and respond to the rest of your post tonight. I'll try to get to it tomorrow. S+E -- Quinn "The Eskimo!" <http://www.apple.com/developer/> Apple Developer Technical Support * Networking, Communications, Hardware _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Long latencies in realtime threads when doing BSD syscalls -- Bug or expected behaviour?
by Mario Kleiner 03 Feb '04

03 Feb '04

Hello Mr. Quinn, Thanks a lot for your quick & helpful answer! I came to the same conclusions after my first extended safari through the xnu-kernel source, but as i am mostly from a Linux background - used to the way how Linux handles locking/smp/preemption et al. - and quite new to OS-X, i needed some confirmation regarding xnu's locking et al. So, am i right that the kernel_funnel has this effect on all BSD syscalls, even on uniprocessor machines? Yes. Ah ja, needs to be on uniprocessor as well, because of the preemptible kernel functions, which pose the same requirements as really parallel (==smp) threads... Does this mean, that the kernel funnel is automatically released if a thread sleeps in the kernel, Yes. See the routine thread_block_reason in the following file. <http://www.opensource.apple.com/darwinsource/Current/xnu-517.3.7/osfmk/kern…> [This link requires an APSL <http://www.opensource.apple.com/apsl/> account.] but that it is not released, if a thread gets preempted by a realtime thread -- which would cause the realtime thread to get delayed/blocked on the lock, as soon as it does a BSD syscall -- basically some kind of priority inversion? Yes. Let me just say on behalf of Apple that we recognise that funnels are bad. We're actively working to eliminate them and introduce finer grained locking to the BSD portions of the kernel. However, with current versions of Mac OS X, they are unavoidable. I understand. But as introducing fine grained locking in an existing kernel is quite a time consuming and incremental endeavour, we probably shouldn't expect this to be completed in the near future? Can i assume that it will take rather a couple of months or years instead of a couple of weeks or months? That would make perfect sense to me, but why doesn't the same thing happen to the gettimeofday() syscall that i use for timing and the usleep() when it gets back from sleep? My guess (didn't have time to check) is that these routines are implemented on top of Mach primitives, not BSD. Non-BSD syscalls don't need to take any funnels. I've checked now: usleep is implemented on top of the Mach-API and gettimeofday is one of a few BSD-calls that doesn't need the funnels. Is there a way to avoid this, e.g. by only using Apples Frameworks or Mach API's instead of Posix stuff? Virtually all frameworks use BSD at some point (you can't access the file system or the network without it), so you can't draw the general conclusion that Apple frameworks won't end up blocking on a funnel. However, there are some frameworks that are safe. For example, all of Mach and all of I/O Kit don't need the funnel. What about things like OpenGL, collecting keyboard responses or mouse button presses? Do the frameworks use Unix device files for communication with the device drivers (-> ioctls/read/write/mmap -> BSD calls -> block on funnel) or do they use the Mach-API? The main concern is not file i/o but querying the mouse and keyboard for user responses and OpenGL graphics output with well controlled timing, see below for reason. Our current -- ugly -- solution is to kill the update() process during realtime parts of the application. Is there something better, e.g. the equivalent of SCHED_FIFO scheduling on other Unices? The best solution is to avoid doing BSD stuff in your realtime thread. For example, let's say you're playing an audio file, reading from disk and playing via Core Audio (which calls you back in a real-time thread). The best way to do this is to separate the disk read and audio play into two threads, insert a buffer between the two, and make the disk thread non-realtime. If you can explain exactly what your realtime thread does, I can perhaps offer some alternative suggestions. Avoiding BSD stuff in this case is difficult: The application is called Psychtoolbox for MacOS-X (see http://www.psychtoolbox.org/osx.html ). It is an extension to/on top of Mathworks Matlab. It is used for programming psychophysical experiments on human visual perception in Matlab -- basically presentation of visual stimuli via OpenGL and collection of keyboard and mouse responses with controlled timing. Its very popular under vision scientists on the old OS-9 Macs and Windows and they are porting it to OS-X at the moment. On OS-9 they achieved "good enough" realtime behaviour by blocking processor interrupts and deferred actions and switching off virtual memory. In the OS-X implementation, "realtime" behaviour is realized by promoting the priority of the Matlab thread to THREAD_CONSTRAINT_POLICY during the experiments, but the content of the realtime part of an experiment can be any kind of mixture of OpenGL rendering, sound output, keyboard/mouse/hid checks, computation and blocking for a defined amount of time. The only thing it probably doesn't need to do in the realtime part is file i/o and large scale memory allocation. As a thread needs to block for some time on each iteration of the experiment loop, in order to not get demoted by the scheduler to normal timesharing mode due to overusage of cpu, it can always happen that some other thread does syscalls inbetween, which take the funnel lock. Another problem: Even if they could avoid any BSD syscalls in their code, the Matlab interpreter loop contains periodic calls to sigsetjmp(), which in turn calls sigprocmask() - which is a BSD syscall and causes the delays on the kernel funnel. As Mathworks seems to be not too enthusiastic about rewriting Matlab to avoid this function, the psychtoolbox guys have a problem. The whole history of the bug can be found under: http://www.psychtoolbox.org/osxTiming.html But anyway, knowing the root cause of the problem may help them to find a solution or workaround -- even if it boils down to killing the update process and calming down other sources of noise in the system. And a last question: Is there an equivalent to mlockall() to prevent paging for a realtime app? My stock answer to this question is that, if the system is short enough on memory that it's paging out parts of your app that you're touching at realtime, things are sufficiently messed up that the wiring wouldn't have helped. Coming from Linux - maybe the VM implementation is completely different in behaviour on OS-X - i would assume that you get delays even if you have enough memory, due to things like copy-on-write and zerofill on demand allocation of memory pages on first access and "faulting in" executable code, when a code path is executed the first time after application startup. On Linux, a mlockall(MCL_CURRENT | MCL_FUTURE) command would guarantee that all parts of the adress space are in core when the call returns. However, this appropriateness of this answer depends on what the negative effect of missing your realtime goal is. For example, glitching audio is not too bad on the overall scale of things, but failing to lower the rods on your nuclear reactor is bad (sufficiently bad that you shouldn't be using Mac OS X to control it). Messing up a few trials in an experiment is not that catastrophic, although too much unwanted delays can pose problems for fMRI experiments AFAIK, but how did you know that i have a nuclear reactor ;) Any more helpful comments are appreciated and thanks again. ciao, -mario -- ********************************************************************* Mario Kleiner Max Planck Institute for Biological Cybernetics Spemannstr. 38 72076 Tuebingen Germany e-mail: mario.kleiner(a)tuebingen.mpg.de office: +49 (0)7071/601-734 fax: +49 (0)7071/601-616 www: http://www.kyb.tuebingen.mpg.de/bu/people/kleinerm/index.html ********************************************************************* _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

IOMedia driver
by Herry Wiputra 02 Feb '04

02 Feb '04

Hi guys, I'm sorry for a rather long post. I need to develop an IOMedia driver which has its own bdevsw table, something similar to IOMediaBSDClient. If it works properly, this driver should sit above all IOMedia devices that have a specific signature in their superblock label. That way the driver would replace IOMediaBSDClient for these devices. I have approached this problem from 3 angles, but none of them gave me any success: 1. I copied IOMediaBSDClient, IOMedia and IOBlockStorageDriver code and renamed it to IOMediaBSDClientTest, IOMediaTest and IOBlockStorageDriverTest, respectively. IOMediaBSDClientTest is a subclass of IOMediaBSDClient and its getProvider() method returns IOMediaTest. IOMediaTest is a subclass of IOMedia and its getProvider() method returns IOStorage. IOBlockStorageDriverTest is a subclass of IOBlockStorageDriver and its getProvider() method returns IOBlockStorageDevice. This is the plist for it: <dict> ... <key>IOKitPersonalities</key> <dict> <key>IOBlockStorageDriverTest</key> <dict> <key>CFBundleIdentifier</key> <string>com.test.iokit.IOMediaBSDClientTest</string> <key>IOClass</key> <string>IOBlockStorageDriverTest</string> <key>IOPropertyMatch</key> <dict> <key>device-type</key> <string>Generic</string> </dict> <key>IOProviderClass</key> <string>IOBlockStorageDevice</string> </dict> <key>IOMediaBSDClientTest</key> <dict> <key>CFBundleIdentifier</key> <string>com.test.iokit.IOMediaBSDClientTest</string> <key>IOClass</key> <string>IOMediaBSDClientTest</string> <key>IOMatchCategory</key> <string>IOMediaBSDClient</string> <key>IOProbeScore</key> <integer>60001</integer> <key>IOProviderClass</key> <string>IOMediaTest</string> <key>IOResourceMatch</key> <string>IOBSD</string> </dict> </dict> ... <key>OSBundleLibraries</key> <dict> <key>com.apple.iokit.IOStorageFamily</key> <string>1.2.3</string> <key>com.apple.kernel.bsd</key> <string>1.1</string> <key>com.apple.kernel.iokit</key> <string>1.1</string> <key>com.apple.kernel.libkern</key> <string>1.1</string> </dict> <key>OSBundleRequired</key> <string>Local-Root</string> </dict> After I put this kext in /System/Library/Extensions, and rebooted with boot-args="-v", I got a panic almost immediately in my IOKit kext. This indicates to me that my kext gets loaded and matched and actually gets used but something is seriously wrong, although I haven't changed anything from the original code except for the names. Anyway, I can't debug it because it is too early in the startup sequence. 2. I copied the IOMediaBSDClient class only and renamed it to IOMediaBSDClientTest. I also made this kext depend on IOStorageFamily kext, so it doesn't have undefined symbols. IOMediaBSDClientTest is a subclass of IOMediaBSDClient and its getProvider() method returns IOMedia. This is the plist for it: <dict> ... <key>IOKitPersonalities</key> <dict> <key>IOMediaBSDClientTest</key> <dict> <key>CFBundleIdentifier</key> <string>com.test.iokit.IOMediaBSDClientTest</string> <key>IOClass</key> <string>IOMediaBSDClientTest</string> <key>IOMatchCategory</key> <string>IOMediaBSDClient</string> <key>IOProbeScore</key> <integer>65536</integer> <key>IOProviderClass</key> <string>IOMedia</string> <key>IOResourceMatch</key> <string>IOBSD</string> <key>Leaf</key> <true/> </dict> </dict> ... <key>OSBundleLibraries</key> <dict> <key>com.apple.iokit.IOStorageFamily</key> <string>1.2.3</string> <key>com.apple.kernel.bsd</key> <string>1.1</string> <key>com.apple.kernel.iokit</key> <string>1.1</string> <key>com.apple.kernel.libkern</key> <string>1.1</string> </dict> <key>OSBundleRequired</key> <string>Local-Root</string> </dict> This kext loaded with no problems but it never got used. kextstat shows the kext has already been loaded. I suspect it is because the IOStorageFamily kext has loaded first and thus IOMediaBSDClient has a higher probability of getting used, even though I have given this kext a much higher IOProbeScore. 3. I copied IOCDStorageFamily project and renamed all occurences of IOCD to IOTest. This kext loaded with no problems when done manually, but if you reboot the box, the box won't even be able to boot. It gives me the apple logo with a black box on top of it. This is the plist for it: <dict> ... <key>IOKitPersonalities</key> <dict> <key>IOTestPartitionScheme</key> <dict> <key>CFBundleIdentifier</key> <string>com.test.iokit.IOTestStorageFamily</string> <key>Content Mask</key> <string>CD_partition_scheme</string> <key>Content Table</key> <dict> ... </dict> <key>IOClass</key> <string>IOTestPartitionScheme</string> <key>IOMatchCategory</key> <string>IOStorage</string> <key>IOProviderClass</key> <string>IOTestMedia</string> </dict> <key>IOTestBlockStorageDriver</key> <dict> <key>CFBundleIdentifier</key> <string>com.test.iokit.IOTestStorageFamily</string> <key>IOClass</key> <string>IOTestBlockStorageDriver</string> <key>IOPropertyMatch</key> <dict> ... </dict> <key>IOProviderClass</key> <string>IOTestBlockStorageDevice</string> </dict> <key>IOTestMediaBSDClient</key> <dict> <key>CFBundleIdentifier</key> <string>com.test.iokit.IOTestStorageFamily</string> <key>IOClass</key> <string>IOTestMediaBSDClient</string> <key>IOMatchCategory</key> <string>IOMediaBSDClient</string> <key>IOProbeScore</key> <integer>61000</integer> <key>IOProviderClass</key> <string>IOTestMedia</string> </dict> </dict> <key>OSBundleLibraries</key> <dict> <key>com.apple.iokit.IOStorageFamily</key> <string>1.2.3</string> <key>com.apple.kernel.bsd</key> <string>1.1</string> <key>com.apple.kernel.libkern</key> <string>1.1</string> <key>com.apple.kernel.mach</key> <string>1.1</string> </dict> <key>OSBundleRequired</key> <string>Local-Root</string> </dict> If everything goes according to plan, this driver should sit above all CD media, replacing IOCDMediaBSDClient, correct? Has someone done this before? or maybe can help me with this? Thanks, Herry Wiputra _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0