October 2002 - Darwin-kernel

Accessing the PPC performance counters
by Paul Ripke 31 Oct '02

31 Oct '02

As part of an Honours thesis, I extended the Mach processor_info APIs to allow pretty much full access to the PPC performance counter registers, including on SMP systems. I also made use of the performance monitor interrupt to maintain a set of per-cpu software updated 64-bit counters, wrote a set of Mach/BSD command line tools for access and control, and a Cocoa app which graphs deltas of the 64-bit counters. I'm now releasing the code to the wider public. The guts of the code is a kernel patch, which I've successfully applied (with minor modifications) to xnu from Mac OS X 10.0.* all the way through to Mac OS X 10.2.1 (with the proviso that I haven't tested an SMP system since around 10.1.3). It's all available at my home web site <http://stix.homeunix.net:8080/>. There's no compiled kernels there, if anyone is really interested, I'd be able to put up a Mac OS X 10.2.1 (xnu 344.2) kernel. Enjoy. -- Paul Ripke Unix/OpenVMS/DBA 101 reasons why you can't find your Sysadmin: 67: It's 9AM and they're not working that late. -- Koos van den Hout _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Address Space Limit - 2.25GB Per Process?
by Jeremy T. Fox 31 Oct '02

31 Oct '02

I tried to find the most memory I could allocate via Fortran (g77) and I got 2.25GB (on a G4 with 512MB of physical memory). Does Mac OS X have a limit of 2.25GB per process, or is this something strange with Fortran and/or my machine? For comparison, Windows has a limit of 2GB per process (tunable to 3GB if you shell out for Advanced Server), while I believe the default on Linux is 3GB, which can be upgraded to 3.5GB with the right patch. I don't know about FreeBSD. Clearly 64-bit machines do not have this type of limitation. Jeremy, who hopes that one day Apple will support 1GB DIMMs :) -------------------- Jeremy T. Fox jerfox(a)stanford.edu _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Interface Filter NKE Packet Handling
by Michael Cashwell 30 Oct '02

30 Oct '02

Greetings, I'm working on a port of a body of networking code as an Interface filter NKE. As it runs on several platforms it already has the concept of abstracting things like network packets and the actions one can do with them. But I'm having problems figuring out how to best map some of these abstractions to Darwin. The first is handling network packets themselves. The operations are things like allocating, copying, freeing, prefixing and postfixing data, etc. One specific issue is which mbuf allocator to use. If found functions like these * MGET(struct mbuf *m, int how, int type) * allocates an mbuf and initializes it to contain internal data. * * MGETHDR(struct mbuf *m, int how, int type) * allocates an mbuf and initializes it to contain a packet header * and internal data. If I'm reading this right the header in question is a struct pkthdr. That struct has many members but it's not clear to me if this it represents the ethernet media's header or is something else? And either way what's the right way to set its contents? And more general that just that, does anyone know of Darwin-accurate documentation for these mbuf functions? The names are intriguing but what does m_prepend, m_prepend_2, or m_pulldown, actually do, for example? The NKE.pdf file lists their names but says no more. The second area involves actions I can perform with packets. Here I'm looking at functions like dlil_inject_if_input() to send packets up the network stack and dlil_inject_if_output() to send packets out the wire. The input function takes a frame header as well as an mbuf. Does that mean its mbufs should be allocated by MGET to avoid putting another header in it? Does the output function assume the reverse (ie that the mbuf has come from MGETHDR)? If so, again, how is it to be filled in (things like MAC addresses and ether-types)? And lastly, when I hook in as an an interface filter and my filter_if_input() gets called with a received frame, does it include the media header? Any advise on where to look for this kind of information or what sample code or projects might help would be appreciate. Thanks! Mike _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Using ifnet pointers safely
by Peter Lovell 25 Oct '02

25 Oct '02

Hi Michael, in working on socket NKEs, I've found "TCP/IP Illustrated, Vol 2" by Wright & Stevens to be a great help, in addition to Stevens's other books. There's a section in there on interrupt levels and concurrency. This deals with BSD in general, rather than OS X in particular. The same general rules apply though because otherwise the whole stack would be failing. The splnet(), splx() etc routines in their original BSD implementation managed concurrency by controlling interrupt levels. As has been noted, this is insufficient for MP systems. On OS X, they are calls into the funnel mechanism and no longer work by adjusting interrupt levels. Note that there are no levels - the network lock is an all-or-nothing lock. The kernel lock is similar. And you can't hold both of them, only one or the other. You'll also see code which switches between, starting out with kernel lock and switching to network lock when dealing with network structures. The code for processing ioctls is a good example of this behaviour. On Thursday, October 24, 2002, at 08:17 PM, Michael Cashwell wrote: During this section of code I have the network funnel. If I understand correctly, that protects me from any other running thread (even on MP systems). Again, can someone confirm? Yes, you're protected. My real worry is about my own thread's behavior. I understand that kernel threads can block and if they do then I loose and regain the funnel. But since I can avoid making calls that block I can handle that. But what about preemption? If my thread runs out of its time quantum might it get preempted (and again loose and regain the funnel)? My understanding is that the time quantum runout is deferred until the kernel processing is finished. I'm not certain of that, though, so hopefully someone else on the list can confirm. But sanity dictates that this must be so. Otherwise you'd always be running the risk of leaving kernel structures in inconsistent states. Think about what would happen if time ran out while tcp_output was part way through processing an mbuf to end. Ouch. And lastly, whatever the solution to that is ("kernel threads can't be preempted" or "use splnet() / splx()", or something else), what about paths into my NKE that are really user threads that have called into the kernel (through the network stack)? (Can that happen or is everything in the kernel run on a kernel thread with the "call" handled via a kernel thread blocking on a queue?) If it can happen, do I need to consider such threads as different from native kernel threads as far as preemption in my NKE goes? Output and input processing are handled differently. It helps to think of output as a synchronous process, done in response to a syscall from user-land. It operates in kernel mode, but in a way it's something like an extended function call. Input is handled on an interrupt-driven basis, and not in the context of the target user or process. Part of the processing is done directly in response to the hardware interrupt and some is queued, as I understand, but it's not in the user thread or context etc. Regards.....Peter _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Using ifnet pointers safely
by Michael Cashwell 25 Oct '02

25 Oct '02

On Friday, October 25, 2002, at 05:03 AM, Stiphane Sudre wrote: On Friday, October 25, 2002, at 02:17 AM, Michael Cashwell wrote: And lastly, whatever the solution to that is ("kernel threads can't be preempted" or "use splnet() / splx()", or something else)... If I remember correctly what I saw in xnu, splnet and splx are just doing nothing. I haven't looked up their source but I also recall statements that they are not sufficient for synchronization on MP systems. I think they represent some sort of interrupt disable which only applies to the one CPU executing them. Or something like that... But that's using them by themselves. I'm wondering if they might help _if_ one also holds the network funnel. In that case as long as I don't actively block perhaps splnet()/splx() would prevent preemption long enough to finish iterating the interface list. (I could also modify my thread's quantum to achieve the same thing I guess.) But that sounds bad. It's using a sledge hammer to swat a fly. I only care about other threads modifying the interface list. Blocking them all just to protect that one data structure seems heavy handed and is not being a good citizen. What I would like is a mutex (or even better a read lock) I could take that would protect the "ifnet.tqh_first; ...if_link.tqe_next interface linked list (like the funnel) but that would not be taken from me if I blocked or was preempted. It seems to me that many of the kernel's data structures would have this peril and yet it works. I must be missing something fundamental. Any ideas would be appreciated. Mike _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Using ifnet pointers safely
by Stéphane Sudre 25 Oct '02

25 Oct '02

On Friday, October 25, 2002, at 02:17 AM, Michael Cashwell wrote: And lastly, whatever the solution to that is ("kernel threads can't be preempted" or "use splnet() / splx()", or something else), what about paths into my NKE that are really user threads that have called into the kernel (through the network stack)? (Can that happen or is everything in the kernel run on a kernel thread with the "call" handled via a kernel thread blocking on a queue?) If it can happen, do I need to consider such threads as different from native kernel threads as far as preemption in my NKE goes? If I remember correctly what I saw in xnu, splnet and splx are just doing nothing. _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Using ifnet pointers safely
by Michael Cashwell 25 Oct '02

25 Oct '02

On Thursday, October 24, 2002, at 11:50 PM, Christophe Allie wrote: > No. The refcount doesn't count the filters, but count the protocols. > If an interface is detached, the detach function of your filter is > called, > and then interface detachment continues. You have no way to prevent > that > from a filter. OK. Since my only copy of the ifnet* is in a member of an object I can reach by casting the cookie I can handle that. >> for (ifp = ifnet.tqh_first; ifp; ifp = ifp->if_link.tqe_next) { >> ... new InterfaceObject( ... ifp ...); >> } > > I think walking the list of ifnet is unfortunately the only way to > discover > the all ifnet. However, if you do an allocation like new > InterfaceObject, > you can loose the funnel, and potentially the list can be updated by an > other code. So "new" allocations are done as M_WAITOK? That's unfortunate. (One work around might be to preallocate them. Or I could override the new operator.) But the problem remains if preemption is possible since the thread could exhaust its time quantum. >> And if that happens, couldn't the interface list change while I'm >> iterating over it? > > That's possible. This sounds like a kernel panic just waiting to happen. Yet I see these sorts of loops in many places in xnu. (Admittedly they likely don't try to allocate a C++ instance while iterating.) Surely there must be some way to safely iterate over the interface list. Anyone know of a solution to this? Thanks, Mike _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Using ifnet pointers safely
by Christophe Allie 25 Oct '02

25 Oct '02

> In a nutshell, I have a C++ class whose instances each encapsulate a > network interface of interest. Each has a member that holds its > associated ifnet*. My plan is to put the dlil_attach_interface_filter() > and dlil_detach_filter() calls in the constructor and destructor of > these interface objects. > > In looking at the code in xnu it seems that there is a refcount that > will prevent the system from freeing an interface with such an attached > filter. And my hope it that this will make it OK for me to hold onto > the ifnet* for as long as that filter is in place. Can anyone confirm > this? No. The refcount doesn't count the filters, but count the protocols. If an interface is detached, the detach function of your filter is called, and then interface detachment continues. You have no way to prevent that from a filter. > My second question is how to create these objects safely in the first > place. Again, cribbing from xnu I've come up with a loop like this: > > struct ifnet *ifp; > > // read through list of devices and build a list of adapters that > we care about > for (ifp = ifnet.tqh_first; ifp; ifp = ifp->if_link.tqe_next) { > ... new InterfaceObject( ... ifp ...); > } > > During this section of code I have the network funnel. If I understand > correctly, that protects me from any other running thread (even on MP > systems). Again, can someone confirm? I think walking the list of ifnet is unfortunately the only way to discover the all ifnet. However, if you do an allocation like new InterfaceObject, you can loose the funnel, and potentially the list can be updated by an other code. > My real worry is about my own thread's behavior. I understand that > kernel threads can block and if they do then I loose and regain the > funnel. But since I can avoid making calls that block I can handle > that. But what about preemption? If my thread runs out of its time > quantum might it get preempted (and again loose and regain the funnel)? > > And if that happens, couldn't the interface list change while I'm > iterating over it? That's possible. > And lastly, whatever the solution to that is ("kernel threads can't be > preempted" or "use splnet() / splx()", or something else), what about > paths into my NKE that are really user threads that have called into > the kernel (through the network stack)? (Can that happen or is > everything in the kernel run on a kernel thread with the "call" handled > via a kernel thread blocking on a queue?) If it can happen, do I need > to consider such threads as different from native kernel threads as far > as preemption in my NKE goes? > > I hate to ask so many questions but I do want to get this right. Any > hints would be appreciated. > > Thanks, > Mike _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Using ifnet pointers safely
by Michael Cashwell 25 Oct '02

25 Oct '02

Greetings all, I've got a few design issues with a data-link NKE I'm working on. They all center around thread safety and refcounting. In a nutshell, I have a C++ class whose instances each encapsulate a network interface of interest. Each has a member that holds its associated ifnet*. My plan is to put the dlil_attach_interface_filter() and dlil_detach_filter() calls in the constructor and destructor of these interface objects. In looking at the code in xnu it seems that there is a refcount that will prevent the system from freeing an interface with such an attached filter. And my hope it that this will make it OK for me to hold onto the ifnet* for as long as that filter is in place. Can anyone confirm this? My second question is how to create these objects safely in the first place. Again, cribbing from xnu I've come up with a loop like this: struct ifnet *ifp; // read through list of devices and build a list of adapters that we care about for (ifp = ifnet.tqh_first; ifp; ifp = ifp->if_link.tqe_next) { ... new InterfaceObject( ... ifp ...); } During this section of code I have the network funnel. If I understand correctly, that protects me from any other running thread (even on MP systems). Again, can someone confirm? My real worry is about my own thread's behavior. I understand that kernel threads can block and if they do then I loose and regain the funnel. But since I can avoid making calls that block I can handle that. But what about preemption? If my thread runs out of its time quantum might it get preempted (and again loose and regain the funnel)? And if that happens, couldn't the interface list change while I'm iterating over it? And lastly, whatever the solution to that is ("kernel threads can't be preempted" or "use splnet() / splx()", or something else), what about paths into my NKE that are really user threads that have called into the kernel (through the network stack)? (Can that happen or is everything in the kernel run on a kernel thread with the "call" handled via a kernel thread blocking on a queue?) If it can happen, do I need to consider such threads as different from native kernel threads as far as preemption in my NKE goes? I hate to ask so many questions but I do want to get this right. Any hints would be appreciated. Thanks, Mike _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Socket problems in NKE
by Christophe Allie 19 Oct '02

19 Oct '02

Michael, What you do is correct, but apparently the ctl_enqueuedata() function is buggy. It creates an mbuf with you data but doesn't set the len of the mbuf. Using ctl_enqueuembuf() for the time being. I should just work fine. Christophe. On 10/18/02 4:16 PM, "Michael Cashwell" <mboards(a)prograde.net> wrote: > Greetings all, > > I can almost talk with my NKE from userland but I'm having socket > problems. I can't get any data from the NKE. > > I'm using the AF_SYSTEM domain from <sys/kern_control.h>. It > essentially is registering a protocol handler as a SOCK_DGRAM type. > > Then from userland I do this (I've deleted the error checking code > below for brevity): > > struct sockaddr_ctl sf; > memset(&sf, 0, sizeof(sf)); > > m_rpcSock = socket(AF_SYSTEM, SOCK_DGRAM, SYSPROTO_CONTROL); > > sf.sc_len = sizeof(sf); > sf.sc_family = AF_SYSTEM; > sf.ss_sysaddr = AF_SYS_CONTROL; > sf.sc_id = A1_DARWIN_NKE_SIG; > sf.sc_unit = A1_NKE_UNIT_RPC; > > connect(m_rpcSock, (struct sockaddr *)&sf, sizeof(sf); > > This all works. I also successfully send one byte of data to the NKE > via: > > char dummy = 0; > error = send(m_rpcSock, &dummy, 1, 0); > > But then when the userland code call recv() with zero flags it blocks > forever. I then added this diagnostic loop just to verify and recv() > always returns -1 and EAGAIN: > > do { > sleep(2); > error = recv(m_rpcSock, ver, sizeof(ver), MSG_DONTWAIT | > MSG_PEEK); > printf("version MSG_PEEK returned %08X, error %d\n", error, > errno); > } while (error != sizeof(ver)); > > But the log shows that the NKE does receive the one zero byte and in > response does this: > > unsigned short ver[2]; > > ver[0] = IPC_VERSION; > ver[1] = IPC_MIN_SUPPORTED_VERSION; > > error = ctl_enqueuedata(ctlref, ver, sizeof(ver), 0); > > printf("RPC version send result = %d\n", error); > > The log also shows that the ctl_enqueuedata() call returns zero. (I've > also verified that ctlref is the same one the NKE registered.) So it > should have written data to the socket. I can't figure out why the > userland code cannot receive anything. > > I've done this sort of thing in the past using SOCK_RAW but the > infrastructure is forcing me to use SOCK_DGRAM. I expect this to raise > some issues in terms of how I read data from the socket but I can > handle that with MSG_PEEK. But only if I can get data to arrive. If > anyone's done this sort of socket code and could assist I'd appreciate > it. _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0