February 2006 - Darwin-kernel

Re: kmem_alloc replacement in Tiger
by Terry Lambert 02 Feb '06

02 Feb '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Feb 1, 2006, at 3:58 PM, Andrew Gallatin wrote: You've confused me. What 32-bit value? I wasn't asking about how to handle a 64-bit user virtual address. We already do this. It sounded like you were trying to use a 32 bit physical address... My concern was with avoiding bouncing a 64-bit physical address which backs a 32-bit kernel virtual address (or a 32- or 64-bit user virtual address). -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Terry Lambert writes: [...] This should not be a problem, and should not require bouncing, unless there's a hardware limitation. The kernel itself only really deals with virtual addresses. Physical addresses are handled by the pmap layer. In practice, it's impossible to have a machine where you are using memory that has a pmap entry referring to it that can't refer to it because it didn't have enough bits to establish the mapping in the first place. The attempt to map it would have failed. This email sent to site_archiver(a)lists.apple.com

1 0

Re: kernel dump server
by Sam Vaughan 02 Feb '06

02 Feb '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On 02/02/2006, at 1:30 AM, Derek Kumar <drk(a)apple.com> wrote: Something else that I wanted to note was that a feature that was asked for last WWDC, the ability to examine the live, running kernel on the same machine, has been in use internally for quite some time, and should be made available externally in a forthcoming tools release. Sam _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Tremendous news, Derek! I remember asking for KPI support for walking kernel threads live at WWDC '04. I'd love to see an Apple- supported tool made available to do it. I haven't had the time to port my own 10.3 implementation to 10.4 and I really miss it when I need to look for deadlocks or stuck threads, especially on remote test machines in far off lands. Remote debugging isn't always feasible and I've never found Shark quite up to the task. This email sent to site_archiver(a)lists.apple.com

1 0

Re: kmem_alloc replacement in Tiger
by Andrew Gallatin 01 Feb '06

01 Feb '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Terry Lambert writes: > > Please tell me that there is some way to coax a full, 64-bit DMA > > address out of IOKit on a DARTless system. OS-bypass doesn't make any > > sense when the DMA involves bounce buffers. It looks like > > getPhysicalSegment64() will only really return a 64-bit address on a > > DARTful system (where it is useless for DMA), but perhaps I'm reading > > the code wrong. > > The specific answer to this question depends on the answer to > questions I can't speak on. This is part of where I potentially > needed correction, so take the following with a grain of salt, until > other people have weighed in on the matter, or you have contacted DTS > and gotten an authoritative answer. > > > The generic answer is as follows: If the platform you are on supports > the full address range in the I/O bus, and the I/O bus memory > controller is capable (the controller in the "T1" development systems > that were leased is *not*, but neither is that system capable of > supporting large amounts of physical memory), then there will be no > problem. > > Therefore, practically, at the IOKit layer, in the absolutely worst > case, you could use the 32 bit value as an address token, and handle > it in your driver. But as of Tiger, this is in theory taken care of > for you in the iovec/uio structures, where we use user_addr_t's, which > are 64 bit values. How you handle their processing is up to you, and > you can override their remainder processing, and effectively take > control of everything yourself, and skip the intervening code path > between the request and your way back up the device stack. Doing this > would be tricky, so you should be careful. You've confused me. What 32-bit value? I wasn't asking about how to handle a 64-bit user virtual address. We already do this. My concern was with avoiding bouncing a 64-bit physical address which backs a 32-bit kernel virtual address (or a 32- or 64-bit user virtual address). Right now, I'm thinking about a "32-bit" intel system with PAE and >4GB of RAM. > On DART-ful system, it is my understanding that the issue will be one > of performance, as the DART remaps system physical addresses into > different location in the I/O bus's idea of the physical address > space, and dispersing the memory used for this over the physical > memory address space will require additional remapping that may > otherwise not be necessary (mapping is done in chunks, and contiguous > chunks all in the window size would guarantee that mapping changes > would not be necessary). Solaris on SPARC will suffer this same > issue, since it has DART-like hardware for I/O memory window management. Yes, the DART is a pain. Tiger is much better, and can do our allocations about 25% faster than Panther. > > - Now for some opinion on design for this specific type of application: > > My first comment is that it's not a good idea to grab a chunk of > kernel virtual address space for OS-bypass type uses. G5's, for a <....> This is for a small (a few MB/procss) area to use to copy small messages to/from. <....> > So for both these cases, you would be generally better off with > nominally portable code like this by wiring down sections of the > process virtual memory while it's being used as a DMA target, and > communicating those addresses to your driver. This, rather than > allocating a huge chunk of the available kernel virtual memory, and > making it unavailable to the system. I believe this approach was used > in a number of the "zero copy" TCP implementations in FreeBSD, when > that code was first being attempted. This is exactly what we do, on all platforms. Large messages are pinned on demand. Small messages are copied to the copyblock which is mapped into the user process from the kernel. This is the buffer that I was talking about earlier. It is amusing that the ideal cutoff point between small/large messages is directly dependant on the operating system's system call latency, and memory pinning latency. MacOSX has by far the worst system call overhead of any platform we've seen. Sometimes by as much as a factor of 8 (when compared to ppc64 linux). At least Tiger redeems itself by being fairly adept with the DART, and winds up making up all the time it lost doing the system call by actually doing the DART setup quicker. If you take the DART out of the picture, linux blows macosx out of the water. Drew _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: kmem_alloc replacement in Tiger
by Terry Lambert 01 Feb '06

01 Feb '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Feb 1, 2006, at 2:38 PM, Nikita Danilov wrote: Terry Lambert writes: [...] -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Likewise, at least with FreeBSD's unified VM and buffer cache, and in NetBSD's case, their UVM implementation, and Windows, it's also likely better to avoid the possibility of fragmentation of the kernel virtual address map, even if it's a full 64 bits. The main issue is the number of pmap entries needed to support that large an address space grows as the address space becomes fragmented, even if the amount of available physical memory is relatively small. I believe this will also be a problem on Linux, unless something happened very recently to change things there; it may or may not be a problem on Solaris, In overwhelming majority of cases Linux uses 1:1 mapping of physical to virtual memory in the kernel mode, and kernel `pmap' is modified quite rarely. All major memory consumers (file system, anonymous memory, slab allocator) work in 1:1 mode. So it's still an issue because of the mapping into and out of the user process address spaces taking pmap entries. I was aware that it would be a problem for Linux the last time I read the sources, but it's been a while since I've followed the LKML closely enough to speak authoritatively on the answer. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Re(5): kernel dump server
by Peter Lovell 01 Feb '06

01 Feb '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Feb 1, 2006, at 9:29 AM, Derek Kumar wrote: If it helps, I'll revise the patch and add it to the radar. Cheers.....Peter _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Hi Derek, the ARP request routine will allow the control transfer, in the same way that takes place during the dump itself. The "flag_panic_dump_in_progress" flag is already set and the routine kdp_arp_request (line 1277) calls kdp_poll which will cause the transfer (test around line 678) if a packet is received. Hello, kdp_poll() doesn't actually perform a control transfer if a debugger packet arrives in the course of a crashdump - it merely clears some globals via abort_panic_transfer() and returns to the caller, which would be your ARP request routine in this case. It is the caller's responsibility to then cause an exit from kdp_panic_dump(), which will then switch to the debugger, i.e. kdp_connection_wait(), if certain conditions are met. In your implementation, you'd address this by checking for a return of -2 from your arp request routine, and then branching to the panic_dump_exit label. But it's a minor point. An issue with the way this is done is that the original debugger packet is discarded, so it will only be the next debugger connection retry that makes it through. My arp request routine was deliberately cloned from kdp_send_panic_pkt and has the same breakout logic. But you're right - it should check specifically for -2 and go to panic_dump_exit. Of course, it doesn't try for very long so debugger will get there, just be delayed a bit. A port number for the crashdump server could certainly be a separate boot-arg (or the parser could be changed to allow IP:port notation, if that's what you meant); I'd have to check if the server allows you to specify the port it would bind to, but if not, that can be changed. The IP:port notation was my original intent, to keep things simple. The parse routine has a hard limit at 16 though, ignoring a limit greater than that if so specified. I can appreciate that this is a reasonable default, but it shouldn't override a specific limit provided by the caller. The port number value for the server can be changed readily - it's specified in the xinetd config file. ... (since the lack of ARP resolution will be addressed in a future update This is actually the "biggie" for us, and I suspect other developers too. If you have a full-blown network with routers then you can put the dump server on another network and things "just work". But for those of us with smaller networks (even large-ish companies with fairly simple networks) this is a killer. It makes kernel-dump entirely useless, so please can we have it in a "sooner" update. This email sent to site_archiver(a)lists.apple.com

1 0

Re: kmem_alloc replacement in Tiger
by Nikita Danilov 01 Feb '06

01 Feb '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Terry Lambert writes: [...] > Likewise, at least with FreeBSD's unified VM and buffer cache, and in > NetBSD's case, their UVM implementation, and Windows, it's also likely > better to avoid the possibility of fragmentation of the kernel virtual > address map, even if it's a full 64 bits. The main issue is the > number of pmap entries needed to support that large an address space > grows as the address space becomes fragmented, even if the amount of > available physical memory is relatively small. I believe this will > also be a problem on Linux, unless something happened very recently to > change things there; it may or may not be a problem on Solaris, In overwhelming majority of cases Linux uses 1:1 mapping of physical to virtual memory in the kernel mode, and kernel `pmap' is modified quite rarely. All major memory consumers (file system, anonymous memory, slab allocator) work in 1:1 mode. Nikita. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: kmem_alloc replacement in Tiger
by Terry Lambert 01 Feb '06

01 Feb '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Feb 1, 2006, at 6:40 AM, Andrew Gallatin wrote: Here is my rational for using kmem_alloc(). My driver is cross platform (linux, windows, solaris, freebsd, macosx) and all allocations are done from platform independent code, which calls a platform dependent allocator. I'm using kmem_alloc() (via IOMallocAligned()) inside my macosx allocator. The allocator has no idea how the memory will be used. Practically, all large allocations are subsequently wired for DMA and mapped into user space via platform dependent code (IOKit's memory descriptor routines on macosx). There are some allocations which are much smaller, which do not need to be wired. Your comment about bouncing concerns me, especially because we wire userspace memory for DMA so changes to our driver's memory allocator would not guarantee all memory we need to deal with comes from the lower 4GB. The 32-bit nature of IOKit is frustrating, because our device is DAC capable and our older products have been for as long as I can remember. Please tell me that there is some way to coax a full, 64-bit DMA address out of IOKit on a DARTless system. OS-bypass doesn't make any sense when the DMA involves bounce buffers. It looks like getPhysicalSegment64() will only really return a 64-bit address on a DARTful system (where it is useless for DMA), but perhaps I'm reading the code wrong. - Now for some opinion on design for this specific type of application: -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Terry Lambert writes: Even if you *are* talking to a device driver, though, you should use IOKit routines to do it. In the case of a G5 or other large physical memory system, physical memory above 4G will either need to be remapped via moving the DART window in the memory controller, or by bouncing, if you didn't use an IOKit routine to allocate the memory in the DART window or below 4G to avoid needing the bounce. The specific answer to this question depends on the answer to questions I can't speak on. This is part of where I potentially needed correction, so take the following with a grain of salt, until other people have weighed in on the matter, or you have contacted DTS and gotten an authoritative answer. The generic answer is as follows: If the platform you are on supports the full address range in the I/O bus, and the I/O bus memory controller is capable (the controller in the "T1" development systems that were leased is *not*, but neither is that system capable of supporting large amounts of physical memory), then there will be no problem. Therefore, practically, at the IOKit layer, in the absolutely worst case, you could use the 32 bit value as an address token, and handle it in your driver. But as of Tiger, this is in theory taken care of for you in the iovec/uio structures, where we use user_addr_t's, which are 64 bit values. How you handle their processing is up to you, and you can override their remainder processing, and effectively take control of everything yourself, and skip the intervening code path between the request and your way back up the device stack. Doing this would be tricky, so you should be careful. On DART-ful system, it is my understanding that the issue will be one of performance, as the DART remaps system physical addresses into different location in the I/O bus's idea of the physical address space, and dispersing the memory used for this over the physical memory address space will require additional remapping that may otherwise not be necessary (mapping is done in chunks, and contiguous chunks all in the window size would guarantee that mapping changes would not be necessary). Solaris on SPARC will suffer this same issue, since it has DART-like hardware for I/O memory window management. My first comment is that it's not a good idea to grab a chunk of kernel virtual address space for OS-bypass type uses. G5's, for a specific example, have a 32 bit kernel virtual address space even though they support a 64 bit (in 51 bit chunks at a time) process virtual address space. What this means is that a huge amount of physical memory can be addressed by a 64 bit process, and that mapping all of it or a large enough chunk of it to be meaningful would mean swamping the kernel virtual address space map (the pmap layer is 64 bit clean; the kernel map is still 32 bits on the G5). Likewise, at least with FreeBSD's unified VM and buffer cache, and in NetBSD's case, their UVM implementation, and Windows, it's also likely better to avoid the possibility of fragmentation of the kernel virtual address map, even if it's a full 64 bits. The main issue is the number of pmap entries needed to support that large an address space grows as the address space becomes fragmented, even if the amount of available physical memory is relatively small. I believe this will also be a problem on Linux, unless something happened very recently to change things there; it may or may not be a problem on Solaris, depending on which allocation facilities you use, since it's my understanding that Solaris 9+ has at least some ability to defragment the address map on the fly for the two level page reclaiming allocator (Dynix was capable of the same thing). So for both these cases, you would be generally better off with nominally portable code like this by wiring down sections of the process virtual memory while it's being used as a DMA target, and communicating those addresses to your driver. This, rather than allocating a huge chunk of the available kernel virtual memory, and making it unavailable to the system. I believe this approach was used in a number of the "zero copy" TCP implementations in FreeBSD, when that code was first being attempted. If you go for the kernel virtual address space solution, the amount of memory you have available for mapping will be limited to a total of ~2G, given shared segments, com pages, video drivers, etc., and it will be physically discontiguous. The only way to get usefully large chunks of memory would be to do what VT did in support of their HyperTransport cards in their cluster, which was to grab all the memory up front, and manage the physical allocations themselves. So you are still better off mapping in user space, unless, as Jim pointed out, there are specific performance issues that can't be solved going that route. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Darwin-kernel Digest, Vol 3, Issue 24
by Derek Kumar 01 Feb '06

01 Feb '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Feb 1, 2006, at 11:13 AM, Jason Seng wrote: Hi Derek, Peter, Also, it's possible to do both crashdumps and kernel debugging over a firewire transport, which can be pretty useful in some cases. I'll check to see if those tools can/have been made available externally. Cheers, -- J-Sun Seng Derek _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Yes they are. Look in /Developer/Extras/Kernel Debugging for the bits needed to enable gdb over FireWire. Another interesting tool is FWMemView in the FireWire SDK. This tool allows you to view (and manipulate!) the physical memory of a running system connected by FireWire. That's great to hear, thanks. I don't, however, see "FireWireCoreDumper" (a crashdump server that listens on firewire) or the gdb plugin which bypasses the KDP protocol (so it doesn't need an active processor, as is the case when you can't force the machine into the debugger - useful in certain pathological situations where processors are spinning with interrupts off, or the processor has entered a state such as checkstop) and performs a hashtable walk manually, so that you can look at virtual addresses (since Firewire DMA, of course, operates in physical mode). It would be good if we can get those out, although the latter is probably not particularly useful externally. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Darwin-kernel Digest, Vol 3, Issue 24
by Jason Seng 01 Feb '06

01 Feb '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Hi Derek, Peter, Also, it's possible to do both crashdumps and kernel debugging over a firewire transport, which can be pretty useful in some cases. I'll check to see if those tools can/have been made available externally. Cheers, -- J-Sun Seng Networking & Communications / Hardware Developer Technical Support Apple Computer, Inc. #1 Infinite Loop M/S: 303-2T Cupertino, CA 95014 _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Yes they are. Look in /Developer/Extras/Kernel Debugging for the bits needed to enable gdb over FireWire. Another interesting tool is FWMemView in the FireWire SDK. This tool allows you to view (and manipulate!) the physical memory of a running system connected by FireWire. This email sent to site_archiver(a)lists.apple.com

1 0

Re: kmem_alloc replacement in Tiger
by Nikita Danilov 01 Feb '06

01 Feb '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Quinn writes: > At 20:06 -0800 30/1/06, Herbert wrote: > >P.S. Bad Apple, for moving kmem_alloc to a private place, and > >removing it's symbol!!! > > But we give you so many other memory allocators to choose from (-: > > I've did a bunch of research into kernel memory allocation over the > last couple of years. I took copious notes, which I haven't had time > to productise yet. I've pasted my notes into the end of this email. > I hope folks find them useful. > > btw I highly recommend using Graphviz to render the diagram. The > resulting graph really illuminates the kernel memory allocation Big > Picture (tm). Cool diagram, but accidental line-wraps have to be fixed before using. > > S+E > -- > Quinn "The Eskimo!" <http://www.apple.com/developer/> > Apple Developer Technical Support * Networking, Communications, Hardware Nikita. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0