October 2002 - Darwin-kernel

More on KEXTs & Altivec stack corruption
by Mike Vannorsdel 12 Oct '02

12 Oct '02

After doing more tests, I found some more information on the problem. I tested with building my KEXT with the -faltivec flag. None of the KEXT sources contains any vector code at this point. The final binary had some extra functions added to it: saveVEC, restVEC, saveVEC_vr11, and rest_VEC_vr10. This is not a big deal. However, in one of my functions in the middle of the binary, there are calls to save_VEC_vr11 and rest_VEC_vr10. I have no idea why these were inserted in this function and why only this function. Seeing this, I commented out all calls to this function in my KEXT and loaded it. The stack corruption panics did not occur. It seems these functions are the source of the problem. Here's the dumps of each function: saveVEC_vr11: 00022fd0 li r11,0xff40 00022fd4 stvx v20,r11,r0 00022fd8 li r11,0xff50 00022fdc stvx v21,r11,r0 00022fe0 li r11,0xff60 00022fe4 stvx v22,r11,r0 00022fe8 li r11,0xff70 00022fec stvx v23,r11,r0 00022ff0 li r11,0xff80 00022ff4 stvx v24,r11,r0 00022ff8 li r11,0xff90 00022ffc stvx v25,r11,r0 00023000 li r11,0xffa0 00023004 stvx v26,r11,r0 00023008 li r11,0xffb0 0002300c stvx v27,r11,r0 00023010 li r11,0xffc0 00023014 stvx v28,r11,r0 00023018 li r11,0xffd0 0002301c stvx v29,r11,r0 00023020 li r11,0xffe0 00023024 stvx v30,r11,r0 00023028 li r11,0xfff0 0002302c stvx v31,r11,r0 00023030 mfspr r11,VRsave 00023034 blr restVEC_vr10: 00023038 li r11,0xff40 0002303c lvx v20,r11,r0 00023040 li r11,0xff50 00023044 lvx v21,r11,r0 00023048 li r11,0xff60 0002304c lvx v22,r11,r0 00023050 li r11,0xff70 00023054 lvx v23,r11,r0 00023058 li r11,0xff80 0002305c lvx v24,r11,r0 00023060 li r11,0xff90 00023064 lvx v25,r11,r0 00023068 li r11,0xffa0 0002306c lvx v26,r11,r0 00023070 li r11,0xffb0 00023074 lvx v27,r11,r0 00023078 li r11,0xffc0 0002307c lvx v28,r11,r0 00023080 li r11,0xffd0 00023084 lvx v29,r11,r0 00023088 li r11,0xffe0 0002308c lvx v30,r11,r0 00023090 li r11,0xfff0 00023094 lvx v31,r11,r0 00023098 mtspr VRsave,r10 0002309c blr The only oddity I see here is with rest_VEC_vr10. All the load immediates are being done to r11, but it uses r10 for the VRSAVE. Does this look like a compiler bug and could this be the source of the stack corruption? _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

KEXT & Altivec stack corruption?
by Mike Vannorsdel 11 Oct '02

11 Oct '02

I have been implementing some Altivec routines in a KEXT and have been suffering some stack corruption panics (in stack_detach), despite scouring the code for bugs. So I then went back to the plain scalar sources which has already been thoroughly tested and performs fine. Yet, I still had the same panic problems. It turns out that I left the -faltivec compiler flag on. Removing the flag and rebuilding the scalar version ceased the panics. I tried this with other KEXTs I had; adding the -faltivec flag even though the KEXT had no Altivec code. They then began panicing as well. It this a known bug? Anyone else seen this? _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: NKE Manager
by Joshua Graessley 11 Oct '02

11 Oct '02

On Friday, October 11, 2002, at 01:00 PM, Justin C. Walker wrote: On Friday, October 11, 2002, at 11:32 AM, Joshua Graessley wrote: Yes; PF_NKE (pre 10.2, in any case) is what you want. There was some discussion of merging the (somewhat immature) PF_NKE implementation into PF_SYSTEM, and improving it, but I don't know how far that got (and I haven't spent any time with 10.2, to see whether it's been done). In Mac OS X 10.2, the header sys/kern_control.h defines the AF_SYS_CONTROL protocol of PF_SYSTEM. This can be used to communicate with and control kernel extensions. It has some limitations. sys/kern_control.h has some documentation that should help you get started. Unlike the NKE manager (PF_NKE), there is no sample code. -josh _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: NKE Manager
by Justin C. Walker 11 Oct '02

11 Oct '02

On Friday, October 11, 2002, at 11:32 AM, Joshua Graessley wrote: On Friday, October 11, 2002, at 10:40 AM, Michael Cashwell wrote: Greetings, I'm working on a datalink NKE which is a port of a body of code that works on Solaris and Linux. At this point I'm trying to get the darwin-specific parts of the kernel/userland IPC mechanism figured > out. The Linux code seems to blindly hijack the PF_KEY domain and excuses doing so with a rather lame comment that anything else using it would conflict with the module anyway so it's OK. I'd rather not do this on darwin / MacOS X and am eyeing the NKE Manager. Conceptually I think I get it. It marshals connections to a domain specifically create to allow userland code to find an NKE based on its handle. Yes; PF_NKE (pre 10.2, in any case) is what you want. There was some discussion of merging the (somewhat immature) PF_NKE implementation into PF_SYSTEM, and improving it, but I don't know how far that got (and I haven't spent any time with 10.2, to see whether it's been done). The NKE Manager is the support for PF_NKE, so that should get you started. You're right, hijacking PF_KEY would be bad since PF_KEY is used for Mac OS Xs IPSec implementation. The NKE Manager is one way to go. With Jaguar, there is a better way using a PF_SYSTEM socket. I can not recall the details unfortunately. Perhaps someone else in the know will respond? As for the NKE Manager, there should be a sample NKE that makes use of it. Again, I can't recall the details. I'll ask around and see if anyone remembers. 'TCPLogger' is the NKE; the user program that uses it is 'tcplog'. Regards, Justin -- Justin C. Walker, Curmudgeon-At-Large * Institute for General Semantics | If you're not confused, | You're not paying attention *--------------------------------------*-------------------------------* _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: NKE Manager
by Joshua Graessley 11 Oct '02

11 Oct '02

On Friday, October 11, 2002, at 10:40 AM, Michael Cashwell wrote: Greetings, I'm working on a datalink NKE which is a port of a body of code that works on Solaris and Linux. At this point I'm trying to get the darwin-specific parts of the kernel/userland IPC mechanism figured > out. The Linux code seems to blindly hijack the PF_KEY domain and excuses doing so with a rather lame comment that anything else using it would conflict with the module anyway so it's OK. I'd rather not do this on darwin / MacOS X and am eyeing the NKE Manager. Conceptually I think I get it. It marshals connections to a domain specifically create to allow userland code to find an NKE based on its handle. You're right, hijacking PF_KEY would be bad since PF_KEY is used for Mac OS Xs IPSec implementation. The NKE Manager is one way to go. With Jaguar, there is a better way using a PF_SYSTEM socket. I can not recall the details unfortunately. Perhaps someone else in the know will respond? As for the NKE Manager, there should be a sample NKE that makes use of it. Again, I can't recall the details. I'll ask around and see if anyone remembers. -josh _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

NKE Manager
by Michael Cashwell 11 Oct '02

11 Oct '02

Greetings, I'm working on a datalink NKE which is a port of a body of code that works on Solaris and Linux. At this point I'm trying to get the darwin-specific parts of the kernel/userland IPC mechanism figured out. The Linux code seems to blindly hijack the PF_KEY domain and excuses doing so with a rather lame comment that anything else using it would conflict with the module anyway so it's OK. I'd rather not do this on darwin / MacOS X and am eyeing the NKE Manager. Conceptually I think I get it. It marshals connections to a domain specifically create to allow userland code to find an NKE based on its handle. My problem is in the details. I assume my NKE it must register for connections against the same PF_NKE domain. I also assume that "protocol" in this case is the distinction between a socket NKE and a datalink NKE. What is this distinction for as far as the NKE Manager stuff is concerned? Why would I use one over the other? (I understand what protocol NKE v. datalink NKE means for actually plumbing into the networking stack. That's not what's confusing me. I don't understand why a control connection to an NKE would need to consider this.) My other confusion is from userland, when I create a raw socket to the PF_NKE domain and specify the socket or dlink protocol (again, ?) when do I specify the NKE handle I'm looking for? And what is it that such a socket connects to? Does the NKE manager magically bridge the socket over to the NKE with the matching handle? Docs or sample code would answer all of these questions, of course, but there's precious little out there on the NKE Manager (other than its source code which does not describe how to _use_ it from either my NKE or from userland). Thanks for any assistance! Mike _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Weird source MAC address on a virtual Ethernet interface
by Joshua Zhao 08 Oct '02

08 Oct '02

I ran into a puzzling problem with an NKE that presents itself as a virtual Ethernet device. I wish that some Mac OS kernel guru out there would be able to shed some light on it. My NKE code is based on the Mac OS VLAN NKE code from the Darwin web site and the tunnel KEXT code (based on FreeBSD's tun driver). What the driver does is to function as an Ethernet device, except that frames sent through it are actually looped back to an application program where they are encapsulated before being transmitted through a real Ethernet interface, much like the ethertap driver does under Linux. I have set the source MAC address of the virtual Ethernet interface to be the same as its underlying real Ethernet device, and "ifconfig" seems to indicate that the MAC address is set properly. However, the problem is that the frames, such as ARP requests sent through this virtual Ethernet interface, have the source MAC address being something different from what it was set to. To be more specific, the source MAC address in the Ethernet header of the virtual Ethernet interface resembles the MAC address of the real Ethernet interface, i.e. 2 higher in the last digit. Changing the unit number of the virtual Ethernet interface doesn't make any difference at least on my ibook. I'm quite puzzled by this problem, and I'm hoping someone more familiar with Mac OS kernel internals could offer some advices on things for me to check. Any suggestion would be greatly appreciated! Joshua _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: fork/exec performance?
by Andrew Gallatin 07 Oct '02

07 Oct '02

Jim Magee writes: > > On Thursday, October 3, 2002, at 05:39 PM, Andrew Gallatin wrote: > > % pwd > > /private/var/tmp > > % cp /home/gallatin/hacks/LMbench/bin/powerpc-apple-darwin1.4/lat_proc > > . > > % ./lat_proc fork > > Process fork+exit: 7577.0000 microseconds > > % /home/gallatin/hacks/LMbench/bin/powerpc-apple-darwin1.4/lat_proc > > fork > > Process fork+exit: 7529.0000 microseconds > > Have you tried it from an account that has a local home directory? NFS > does such weird things to performance numbers that I like to keep it > completely out of the mix (unless I'm trying to benchmark NFS). > Interestingly, there were significant NFS changes in those timeframes. Earlier I replied privately to Jim that an account with a local home forked faster than an account with an NFS home... Well, its all due to a difference in the local and NFS .cshrc. The NFS .cshrc contains the line 'unlimit stacksize' I guess Darwin must REALLY unlimit the stacksize to something obscene. From the appended few datapoints, it looks like the fork perf is directly proportional to the stacksize limit. At any rate, removing that line (conditional on darwin-powerpc*), gets fork performance back to what I remember it being a long time ago. (probably before I needed to add that line for nameless OS which artificially limits stacksize too low). The install script that prompted the question by taking over 3 minutes is now down to 20 seconds (within an acceptable factor of 2 of linux). I'm willing to plead to pilot error on this one. But it does seem rather strange that upping a *limit* will wreak havok with fork performance. Are process' stacks pre-allocated up to the limit or something? Drew [g4:~] drew% limit cputime unlimited filesize unlimited datasize unlimited stacksize 512 kbytes coredumpsize 0 kbytes memoryuse unlimited descriptors 256 memorylocked unlimited maxproc 100 [g4:~] drew% ~gallatin/hacks/LMbench/bin/powerpc-apple-darwin6.0/lat_proc fork Process fork+exit: 1648.2500 microseconds [g4:~] drew% limit stacksize 16384k [g4:~] drew% ~gallatin/hacks/LMbench/bin/powerpc-apple-darwin6.0/lat_proc fork Process fork+exit: 4530.0000 microseconds [g4:~] drew% exit logout Connection to g4.sw closed. <3:15pm>whisper/gallatin:~>ssh g4.sw -l drew Last login: Mon Oct 7 15:14:36 2002 from 172.31.193.10 Welcome to Darwin! [g4:~] drew% unlimit stacksize [g4:~] drew% ~gallatin/hacks/LMbench/bin/powerpc-apple-darwin6.0/lat_proc fork [g4:~] drew% limit stacksize 8k Process fork+exit: 7648.0000 microseconds [g4:~] drew% ~gallatin/hacks/LMbench/bin/powerpc-apple-darwin6.0/lat_proc fork Process fork+exit: 1574.7500 microseconds _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: fork/exec performance?
by Jim Magee 03 Oct '02

03 Oct '02

On Thursday, October 3, 2002, at 10:52 AM, Andrew Gallatin wrote: FWIW, fork _used_ to be faster. I have some old lmbench numbers laying around: ---------------------------------------------------------------- Host OS Mhz null null open selct sig sig fork exec sh call I/O stat clos TCP inst hndl proc proc proc --------- ------------- ---- ---- ---- ---- ---- ----- ---- ---- ---- ---- ---- g4 Darwin 1.4 799 2.21 3.19 12.6 16.4 30.4 3.78 9.88 1576 4095 13.K g4 Darwin 5.5 797 2.26 3.15 14.5 17.8 30.4 3.82 10.2 5685 12.K 40.K g4.local. Darwin 6.0 797 1.47 2.73 17.2 20.7 27.2 3.00 10.5 7517 17.K 41.K g4 Linux 2.4.18- 799 0.42 0.69 2.52 3.79 33.6 1.23 3.08 659. 2642 12.K Interesting. My numbers don't show nearly the dramatic drop-off over the releases that yours do: Host OS Mhz null null open selct sig sig fork exec sh call I/O stat clos TCP inst hndl proc proc proc --------- ------------- ---- ---- ---- ---- ---- ----- ---- ---- ---- ---- ---- jim Darwin 1.4 448 2.85 4.23 9.53 17 4.83 14 2402 6817 19K jim.local Darwin 6.1 448 1.89 3.71 21 29 43 3.85 15 2832 8955 19K And more interesting is that I am very familiar with what changed, and nothing would explain going from 1576 to 7517. There seems to be something dramatically different about our configurations. Can you send me more info on your configuration (privately to avoid clogging the list) and in particular, anything you can think of that might have changed in your configuration between the time of the 1.4 numbers and the 5.5 numbers? When I track down what is going on there, I'll summarize to the list. --Jim _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: fork/exec performance?
by Andrew Gallatin 03 Oct '02

03 Oct '02

Ragnar Sundblad writes: > On Mon, 23 Sep 2002 18:06:53 -0400 Jim Magee <jmagee(a)apple.com> wrote: > > > vfork() yields 100X (yes, 100 times, not percent more) the performance > > of fork() in most situations on Mac OS X. So, if you can live with > > vfork() requirements, please do so. Getting common tools to use it > > where appropriate appears to be the trick in your case. I don't think > > Linux has vfork(), so people tend to stick with just fork() in portable > > code. > > Is there any chance of improving fork+exec performance in the OS > since, as you say, many programs currently use fork? > FWIW, fork _used_ to be faster. I have some old lmbench numbers laying around: ---------------------------------------------------------------- Host OS Mhz null null open selct sig sig fork exec sh call I/O stat clos TCP inst hndl proc proc proc --------- ------------- ---- ---- ---- ---- ---- ----- ---- ---- ---- ---- ---- g4 Darwin 1.4 799 2.21 3.19 12.6 16.4 30.4 3.78 9.88 1576 4095 13.K g4 Darwin 5.5 797 2.26 3.15 14.5 17.8 30.4 3.82 10.2 5685 12.K 40.K g4.local. Darwin 6.0 797 1.47 2.73 17.2 20.7 27.2 3.00 10.5 7517 17.K 41.K g4 Linux 2.4.18- 799 0.42 0.69 2.52 3.79 33.6 1.23 3.08 659. 2642 12.K So it looks like somebody "fixed" something with a sledgehammer sometime between 10.1 and 10.1.5. Eg, 1.4 was only a factor of 2 slower than linux for fork and fork+exec, and just a small percent slower for running a shell. Does the update process save the old kernels off someplace? If so, I'll try to narrow this down further. I the meantime, I'm thinking about pulling the kernel off the original installation media and running 1.4 for developement. I can't afford to wait an hour for a compile which takes < 10 minutes on linux (on the same hardware). Drew _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0