January 2006 - Darwin-kernel

Re: kmem_alloc replacement in Tiger
by Terry Lambert 31 Jan '06

31 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Jan 31, 2006, at 6:00 AM, Andrew Gallatin wrote: I am trying to replace kmem_alloc with another equivalent call in Tiger. It was suggested that I used OSMalloc instead, but this won't work since it does not get it's memory from the same place. Basically, I have to allocate a huge chunk (4+ MB) for an array, so I need basic access. We are currently looking at IOBufferMemory etc., but we're not sure if it will work. At least for non pageable memory, OSMalloc will allocate memory using kalloc. The kalloc() routine uses the kalloc_map submap, which is only 16MB in size. Other people can correct me, if I make a mistake here, but... -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Terry Lambert writes: On Jan 30, 2006, at 8:06 PM, Herbert wrote: Has anyone had to deal with this? Unless you are creating your own submap, the memory all comes from the same place: kernel_map and the underlying implementation is all the same code. This is on purpose, to discourage large non-pageable allocations - allocations requiring non-movable commitments of physical RAM. Predominantly, people should allocate pageable pages; pageable pages have kernel virtual address mappings, but no backing physical pages initially associated with them until they are needed. You can explicitly associate physical pages with these by explicitly calling appropriate vm_ routines (documented on developer.apple.com; you just have to look for them), and there are IOKit calls that will wire them down for the duration of I/O operations. The only case where this is less useful is if a large physically contiguous region is required because we are talking about a device driver incapable of scatter/gather operations, or for which such operations result in a large performance penalty on the flip side (e.g. video capture at close to hardware clock limits, etc.). As you point out in another posting, you can also use IOMallocAligned (); but if he has a driver like this, he probably actually wants IOMallocContiguous(); the latter would be the best bet for the scatter/ gather restricted hardware, and the pageable nature of the aligned memory in the OSMalloc() case being the only issue, probably means he doesn't need IOMallocAligned(). In general, if the memory is not going to be a DMA target, then it should not matter what the backing physical memory is, since you aren't communicating the physical memory address to a device driver. In other words, pageable memory should be OK for most non-driver usage. Even if you *are* talking to a device driver, though, you should use IOKit routines to do it. In the case of a G5 or other large physical memory system, physical memory above 4G will either need to be remapped via moving the DART window in the memory controller, or by bouncing, if you didn't use an IOKit routine to allocate the memory in the DART window or below 4G to avoid needing the bounce. Since Herb was talking about using the memory for a large array, I recommended the API that I'd normally tell people to use for that usage. If this array was actually a driver DMA target, well, then one of the other options is probably a better bet. This email sent to site_archiver(a)lists.apple.com

1 0

RE: Problem using OSAtomic functions
by Carlos Silva 31 Jan '06

31 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Thread-index: AcYmnjRAvSa5VbFdSwmSgsk7n5dg5AAEqfBA _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Problem using OSAtomic functions
by Derek Kumar 31 Jan '06

31 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Problem using OSAtomic functions
by Herb Petschauer 31 Jan '06

31 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=L9IJidv01j3tMpRImjU368pMpRzIl5BARn6gb1DPp8fYTm2EsVJXrKMwzxDFPYfi8MSwLGGdAYm6w6+eVXoTyPQ/PjgOZIzAuE8U3BZk/jJYS37X3tlVWt/fNQxPmEUQBAeIP87mELzSmq7CIib9iOIeH1RVl0+9iL1Px/W5FwU= Taking a quick look at OSAtomic.h in both 10.4.3 and 10.3.9 I see no function called OSAtomicCompareAndSwap(), I do however see something called OSCompareAndSwap()... Just my two cents. -H. On 1/31/06, Carlos Silva <csilva(a)matrox.com> wrote: > > > > Did I post this on the right list? > > Has anyone ever used these functions at the kernel level? > > TIA > > > > ________________________________ > > > From: darwin-kernel-bounces+csilva=matrox.com(a)lists.apple.com > [mailto:darwin-kernel-bounces+csilva=matrox.com@lists.apple.com] > On Behalf Of Carlos Silva > Sent: Friday, January 27, 2006 4:04 PM > To: darwin-kernel(a)lists.apple.com > Subject: Problem using OSAtomic functions > > > > > Hello, > > > > I am writing a CoreAudio KEXT (based on the AudioReflectorDriver sample but > I also tried with the PhantomDriver sample) and I can not use any of the > functions included in <libkern/OSAtomic.h>.. > > I get a "was not declared in this scope" compilation error. > > Here's a code example: > > > > #include <libkern/OSAtomic.h> > > > > class foo > > { > > UInt32 Temp; > > > > void f() > > { > > OSAtomicCompareAndSwap(1, 0, &Temp); > > }; > > }; > > > > Any ideas? > > Carlos > > > > > > > _______________________________________________ > Do not post admin requests to the list. They will be ignored. > Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) > Help/Unsubscribe/Update your Subscription: > http://lists.apple.com/mailman/options/darwin-kernel/bogvardi%40gmail.com > > This email sent to bogvardi(a)gmail.com > > _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

RE: Problem using OSAtomic functions
by Carlos Silva 31 Jan '06

31 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Thread-index: AcYjhSK8EZFoQ69uTSKNRn5X2+Co0ADEBQRQ _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Re(5): kernel dump server
by Derek Kumar 31 Jan '06

31 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Hi Derek, I submitted rdar://4422761 a couple of days ago, and included an updated kdp_udp.c which solves this problem. Main issue was that there was no ARP request code there at all, and now there is. This also ARPs for the router_ip if that is defined in boot-args (this capability was mentioned in the code as a future enhancement). Thanks for filing a radar, I'll take a look. (It's likely that equivalent changes are awaiting release in this particular case, but contributions to the kernel are always welcome). Since it was in the same file, I also updated the code for creating the dump file name so that it now includes the xnu minor version. Notes in the code suggest that rdar://3735061 requested this capability. Great - as I recall, it was intended that we fix this by changing an internal submission script to stamp the kernel with the version number, rather than parsing the existing version string (which is created from OBJROOT (by config/newvers.pl), which includes the version number in an internal build environment). However, having the minor version numbers in crashdump filenames is definitely convenient. N.B.: You can also use the "showversion" kgmacro, which examines the magic address 0x501C - that holds a pointer to the version string (or "strings -" on the crashdump and grep that for the version pattern as a crude work-around if you don't have gdb handy). You can use that macro on live connections as well, to determine the version of the kernel you're connected to so you can find the right symbols. (An "implementation detail" - that macro is fine on PPC and live x86 connections, however it doesn't yet work on x86 crashdumps - the currently released x86 kernel doesn't have crashdump support, so it's not immediately relevant, but it will be eventually :-) Derek P.S.: From a quick look, if you're using this internally, your ARP request routine would have to change to allow control transfer to the debugger if a debugger packet arrives; also, the code to check for the crashdump protocol packets is probably unnecessary in your arp request routine. And apropos the comment about size checking, kdp_vm_read() is a debugger routine that checks for a valid translation for both source and destination pages in the currently active debugger pmap (virtual address space), and then does a physical mode copy, so it's a "safe read". We can get away with assuming the translation remains valid since the debugger runs with interrupts off on the active cpu, and the rest of the processors are signaled to spin on the debugger lock on entry. This email sent to site_archiver(a)lists.apple.com

1 0

Re(5): kernel dump server
by Peter Lovell 31 Jan '06

31 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Dec 27, 2005, at 10:20 AM, Derek Kumar wrote: If, for some reason, the ARP resolution broadcast for the server fails, the debugger will attempt to route the crashdump packets via the cached default route (the debugger's "network stack" is a fairly minimalist, polled implementation, and doesn't query routing tables and so on, which may or may not be valid post-panic), and if the router doesn't forward packets arriving at the internal interface addressed to internal nodes, you may see a timeout. You can either try and determine why the resolution fails, or reconfigure your router as appropriate. A future update should address/ameliorate these issues. Hi Derek, I submitted rdar://4422761 a couple of days ago, and included an updated kdp_udp.c which solves this problem. Main issue was that there was no ARP request code there at all, and now there is. This also ARPs for the router_ip if that is defined in boot-args (this capability was mentioned in the code as a future enhancement). Since it was in the same file, I also updated the code for creating the dump file name so that it now includes the xnu minor version. Notes in the code suggest that rdar://3735061 requested this capability. Regards.....Peter _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: kmem_alloc replacement in Tiger
by Andrew Gallatin 31 Jan '06

31 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Terry Lambert writes: > > On Jan 30, 2006, at 8:06 PM, Herbert wrote: > > I am trying to replace kmem_alloc with another equivalent call in > > Tiger. > > > > It was suggested that I used OSMalloc instead, but this won't work > > since it does not get it's memory from the same place. > > > > Basically, I have to allocate a huge chunk (4+ MB) for an array, so > > I need basic access. > > > > We are currently looking at IOBufferMemory etc., but we're not sure > > if it will work. > > > > Has anyone had to deal with this? > > > Unless you are creating your own submap, the memory all comes from the > same place: kernel_map and the underlying implementation is all the > same code. At least for non pageable memory, OSMalloc will allocate memory using kalloc. The kalloc() routine uses the kalloc_map submap, which is only 16MB in size. Drew _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: kmem_alloc replacement in Tiger
by Andrew Gallatin 31 Jan '06

31 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Herbert writes: > I am trying to replace kmem_alloc with another equivalent call in Tiger. <....> > Has anyone had to deal with this? Yes. I spent a day tracing through all the allocators available, and found that IOMallocAligned() from IOKit calls kernel_memory_allocate(), which works for us. > P.S. Bad Apple, for moving kmem_alloc to a private place, and > removing it's symbol!!! Agreed!! Drew _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: kmem_alloc replacement in Tiger
by Terry Lambert 31 Jan '06

31 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com <http://darwinsource.opendarwin.org/10.4/xnu-792/osfmk/kern/kalloc.c> <http://darwinsource.opendarwin.org/10.4/xnu-792/osfmk/vm/vm_kern.c> -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Jan 30, 2006, at 8:06 PM, Herbert wrote: I am trying to replace kmem_alloc with another equivalent call in Tiger. It was suggested that I used OSMalloc instead, but this won't work since it does not get it's memory from the same place. Basically, I have to allocate a huge chunk (4+ MB) for an array, so I need basic access. We are currently looking at IOBufferMemory etc., but we're not sure if it will work. Has anyone had to deal with this? Unless you are creating your own submap, the memory all comes from the same place: kernel_map and the underlying implementation is all the same code. P.S. Bad Apple, for moving kmem_alloc to a private place, and removing it's symbol!!! You can still get at it, if you are willing to link against the Unsupported symbol set or the System.6.0 symbol set. But if you use OSMalloc(), we'll have a descriptive tag so we know who to blame when things go wrong. Of course, not everyone views that as a feature... This email sent to site_archiver(a)lists.apple.com

1 0