June 2005 - Darwin-kernel - lists.apple.com

Re: 64-bit problem with VM statistics
by William Kucharski 06 Jun '05

06 Jun '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Monday, June 06, 2005, at 06:50AM, Andrew Gallatin <gallatin(a)cs.duke.edu> wrote: >Now, with all my whining and complaining aside, the IOMMU code in >tiger has gotten *much* better. With tiger, pinning memory is >nearly as fast as it is on a DAC machine. However, the address space >is still limited. There's nothing they can do about that. You don't have to constantly pin/unpin memory if your OS is sophisticated enough to be able to manage the process better. While Tiger can't currently do this, Solaris for one can. Though given today's Intel announcement, our discussion is largely moot at this point, as the entire Mac universe, and its performance characteristics, is about to change drastically. William Kucharski kucharsk(a)mac.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

tcplognke source files
by Andrey Babak 06 Jun '05

06 Jun '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Hello, Where can I get the TCPLogger example from? I've got a response from ADC to post the request in this mailing list. Thank you. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… It is mentioned in Tiger NKE documentation at http:// developer.apple.com/documentation/Darwin/Conceptual/NKEConceptual/ socket_nke/chapter_4_section_3.html This email sent to site_archiver(a)lists.apple.com

1 0

Re: 64-bit problem with VM statistics
by Andrew Gallatin 06 Jun '05

06 Jun '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com William Kucharski writes: > On Jun 5, 2005, at 9:26 AM, Andrew Gallatin wrote: > > > This is not really true. Many PCI devices these days are capable of > > addressing 64-bits of physical memory (eg, "DAC"). The DART enables > > address limited devices to address memory anywhere, at the price of > > artificially crippling DAC capable devices because of an oversight > > on IBM or Apple's part. > > Though due to the PCI bus' 32-bit address width, DAC needs two cycles > to specify the 64-bit address to be referenced, so depending upon > locality it's not necessarily any faster for a DAC device to address > memory than it is to do it via the DART or an IOMMU. One can easily > craft scenarios in which each method of access is faster. The time for an address cycle is of no consequence. I can show you DMA benchmarks comparing alpha, ia64 and opteron (all DAC) vs G5 (DART) if you don't beleive me. > > This is important to OS-bypass HPC network devices because it allows > > a large amount of memory to be pinned and available for network > > communication. Whereas on G5s (and IBM js20s), only 2GB of memory can > > be pinned at any one time. Allowing for the needs of other devices, > > this realistically means that only 1.5GB can be pinned at any one > > time. > > This is one of the handicaps the G5 platform faces in the HPC > > cluster market. > > I wouldn't consider the G5's performance in HPC clusters such as > Virginia Tech's > to exactly be "handicapped." Every system has trade-offs between > theoretical Take a look at the efficiency (peak vs max), and you see the handicap. DAC based machines can keep nearly all of RAM pinned, thereby avoiding any interaction with the host OS kernel during network communication. Eg, all the CPU cycles are spent on the application. However, IOMMU based machines either have to copy to/from a pinned region, or they keep pinning/unpinning memory when they need to transfer arrays whose size exceeds the amount of pinnable memory on the machine. Either way, CPU cycles are wasted either on copying, or pinning/unpinning, either of which increase both latency and overhead. This reduces the efficiency of the cluster. Now, with all my whining and complaining aside, the IOMMU code in tiger has gotten *much* better. With tiger, pinning memory is nearly as fast as it is on a DAC machine. However, the address space is still limited. There's nothing they can do about that. Drew _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: 64-bit problem with VM statistics
by William Kucharski 06 Jun '05

06 Jun '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Jun 5, 2005, at 9:26 AM, Andrew Gallatin wrote: This is not really true. Many PCI devices these days are capable of addressing 64-bits of physical memory (eg, "DAC"). The DART enables address limited devices to address memory anywhere, at the price of artificially crippling DAC capable devices because of an oversight on IBM or Apple's part. Though due to the PCI bus' 32-bit address width, DAC needs two cycles to specify the 64-bit address to be referenced, so depending upon locality it's not necessarily any faster for a DAC device to address memory than it is to do it via the DART or an IOMMU. One can easily craft scenarios in which each method of access is faster. Just my $.02 on the subject; now back to your regularly scheduled programming. :-) William Kucharski kuchars(a)mac.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This is important to OS-bypass HPC network devices because it allows a large amount of memory to be pinned and available for network communication. Whereas on G5s (and IBM js20s), only 2GB of memory can be pinned at any one time. Allowing for the needs of other devices, this realistically means that only 1.5GB can be pinned at any one time. This is one of the handicaps the G5 platform faces in the HPC cluster market. I wouldn't consider the G5's performance in HPC clusters such as Virginia Tech's to exactly be "handicapped." Every system has trade-offs between theoretical throughput and actual implementation, and while the DART can admittedly be a non-optimal solution, system software can work around the limitations in most cases, often by the use of "kernel cages" to partition memory for use by particular devices. This email sent to site_archiver(a)lists.apple.com

1 0

Re: 64-bit problem with VM statistics
by Andrew Gallatin 05 Jun '05

05 Jun '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com William Kucharski writes: > > If you think this is fun, PCI devices can only address a 32-bit > memory space, so when using PCI devices on a 64-bit architecture, > memory address translation must be done if you want to allow those > devices to access the entire memory space. This is what the DART is > all about, as described here: > > <http://developer.apple.com/documentation/Darwin/Conceptual/ > KernelProgramming/vm/chapter_7_section_8.html> > This is not really true. Many PCI devices these days are capable of addressing 64-bits of physical memory (eg, "DAC"). The DART enables address limited devices to address memory anywhere, at the price of artificially crippling DAC capable devices because of an oversight on IBM or Apple's part. Nearly 10 years ago, DEC Alpha PCI chipsets also had an IOMMU which made 64-bits of physical memory available to address limited devices. However, DEC wisely included the "monster window" as an escape hatch to allow 64-bit capable devices to bypass the IOMMU. The driver for a 64-bit capable device simply did the virt->phys translation, and set a very high bit to indicate to the IOMMU that it should pass the physical address unmolested. This high bit was larger than the biggest possible physical address, 1UL << 40 on older alphas and 1UL << 49 on newer alphas. So on alphas (as well as more modern 64-bit platforms, such as AMD64, EM64T, and ia64), 64-bit devices are capable of addressing any physical address without going through an IOMMU. This is important to OS-bypass HPC network devices because it allows a large amount of memory to be pinned and available for network communication. Whereas on G5s (and IBM js20s), only 2GB of memory can be pinned at any one time. Allowing for the needs of other devices, this realistically means that only 1.5GB can be pinned at any one time. This is one of the handicaps the G5 platform faces in the HPC cluster market. Drew _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: 64-bit problem with VM statistics
by William Kucharski 05 Jun '05

05 Jun '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Jun 4, 2005, at 12:27 PM, Tony Scaminaci wrote: PowerPC, as the Mac uses it, is a big endian platform. Thus say you have a memory buffer at location 0x1000. Make sense? e.g.: 0xfeedfacedeadbeefULL is represented in memory as: 0x1000: 0xfe 0x1001: 0xed 0x1002: 0xfa 0x1003: 0xce 0x1004: 0xde 0x1005: 0xad 0x1006: 0xbe 0x1007: 0xbf 0xc0edf00d as an unsigned int would be: 0x1000: 0xc0 0x1001: 0xed 0x1002: 0xf0 0x1003: 0x0d But in general, you want to use the 64-bit interfaces, which leads to: William Kucharski kucharsk(a)mac.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Why wouldn't it fill in the lower four bytes instead? I ask this because when I print out mac_pagesize, I do get 4096 as expected so the lower bytes must have the correct value. If the buffer is to be an unsigned int, the value goes into the four bytes starting at 0x1000, MSB at 0x1000. If the buffer is to be an unsigned long long, the value goes into the eight bytes starting at 0x1000, MSB at 0x1000. If the kernel thinks it's filling in an unsigned int at location 0x1000, it just writes those first four bytes, which happen to be the most significant four bytes of an unsigned long long as seen from that address. Why does it print correctly? Assuming you're just doing the printf with an unsigned specifier, the printf will only read those first four bytes as the value to be printed, so the printf will print the value 0x1000, but the system sees the value as the 64-bit quantity 0x100000000000ULL. This would defeat the purpose of running a 64-bit executable - to test more than 4 GB of memory. The memory address pointers have to be 64 bits if they're going to be able to access all the memory a user has installed. You're confusing virtual and physical addresses here; as long as the operating system can access physical memory with 64-bit addresses, a user could use a 32-bit address to access that memory, though not more than 4G at a time. Think of it as a sliding 32-bit window accessing the full 64-bit memory space. This is definitely an option if the 64-bit compiler will allow it. Even worse, I could just set the page size to 4096 and be done with it. Not a good idea in case it ever changes but it seems to be 4K on most modern OS's these days. There is "nothing wrong" with passing host_page_size the address of a 32-bit variable as long as the kernel isn't compiled with __ppc64__ set and as such won't try and write a 64-bit quantity to the 32-bit variable's address. As I said, this is somewhat of a hole in the current implementations, and Apple DTS should really be the ones to resolve it one way or the other. I'd consider host_page_size() to be part of libSystem if anything. If we can't address more than 4 GB of memory, what's the point of having a 64-bit OS and a PPC G5? At the very least, Tiger should be able to get at all of the user's installed memory assuming the user runs a 64-bit app. How can the kernel handle more than 4 GB of address space if it's not 64-bit? See above; you're not grasping the concept of a 32-bit virtual address addressing memory via a 64-bit physical address. This is how the kernel dealt with memory pre-Tiger; apps could access memory mapped with a 64-bit physical address space, an app just couldn't access more than 4GB at a time. There's got to be a simple workaround to this problem. I can't be the only person trying to use more than 4GB of memory at one time. Any application that bills itself as 64-bit has to address these issues. 32-bit applications CAN 'use' more than 4GB of memory at a time, but not via the standard malloc() interfaces in continuous virtually mapped buffers more than 4GB at a time. If you think this is fun, PCI devices can only address a 32-bit memory space, so when using PCI devices on a 64-bit architecture, memory address translation must be done if you want to allow those devices to access the entire memory space. This is what the DART is all about, as described here: <http://developer.apple.com/documentation/Darwin/Conceptual/ KernelProgramming/vm/chapter_7_section_8.html> So, bottom line, you're correct in wanting to compile your app in 64- bit mode, but there's a hole in this particular call that causes things to break due to Darwin's current mixed 32/64-bit nature. If Apple would move to a full 64-bit kernel, like Solaris or, as of 2.6, Linux, a lot of these issues would go away at the cost of compatibility with existing device drivers and some kernel code bloat as opcodes got wider. In theory IOKit should insulate even the driver level from the implementation details of the kernel itself, but in practice it's a lot more complex and MacOS X remains astraddle the 32/64-bit line... This email sent to site_archiver(a)lists.apple.com

1 0

Re: 64-bit problem with VM statistics
by Andrew Gallatin 04 Jun '05

04 Jun '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Tony Scaminaci writes: > There's got to be a simple workaround to this problem. I can't be the > only person trying to use more than 4GB of memory at one time. Any > application that bills itself as 64-bit has to address these issues. > Most scientfic programs, which I think Apple's 64-bit interfaces were designed for, simply malloc up a bunch of memory. Eventually people learn how large a problem they can run by trial and error. At least that's what my wife (a PhD statistician) does with her programs. Almost nobody but memory checkers would ask how much memory is available and try to grab all of it. I'm at least as irritated at the lack of full 64-bit support as you are. I maintain OS-bypass network drivers, and I have to wire memory from a anywhere in users address space. However, there's currently no Apple supported for a driver to do that. Cheers, Drew _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: 64-bit problem with VM statistics
by Tony Scaminaci 04 Jun '05

04 Jun '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Jun 1, 2005, at 4:03 PM, Peter Bierman wrote: What do you intend to do with the number when you have it? It is a meaningless number. Tony -pmb _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… At 2:28 PM -0500 5/31/05, Tony Scaminaci wrote: The following calculation of free memory (in bytes) No, it's used to allocate and lock a memory test buffer. That's what the application does, test huge chunks of memory, preferably close to all of the installed ram that the kernel will give up. In single-user mode, the kernel seems willing to give up about 97.2% of what vm_stat reports being available. Some amount larger than the "unused" RAM can usually be used without performance degradation. True, and I have experimented with grabbing some of the inactive memory as well. In Jaguar and Panther, grabbing too much inactive memory would cause the machine to stop responding to user input from the keyboard or mouse. What was going on, I have no idea. Either the OS was paging out huge chunks of application code or the OS just simply stopped responding. With a lot of experimentation, I found the safest thing to do was only lock free ram as reported by vm_stat. I didn't want to risk slowing a user's system down or creating a bunch of page files (which I also saw during my experiments). Some amount larger than the "phsyical" RAM can be usually used with a slight performance impact. This is not useful to my application. Memtest's purpose is to test the DIMMs and CPU caches, not VM. I specifically DON'T want this behavior so I have to take precautions to ensure that the memory I lock using mlock() is ONLY physical RAM. The absolute safe way to do this is to allocate and lock only the free memory reported by vm_stat. I could go slightly over that and cause some apps to page to disk but if I go to far into the inactive space, the machine becomes unusable. At least for the 30 minutes or so I was willing to wait. There are rarely hard values like "there is 45 MB of free RAM." At any given instant in time, what vm_stat returns is accurate and I use it immediately to allocate and lock memory. I don't know how Tiger is behaving with the issue of grabbing inactive memory space in multiuser mode. I haven't had any time to experiment with it yet. This email sent to site_archiver(a)lists.apple.com

1 0

Re: 64-bit problem with VM statistics
by Tony Scaminaci 04 Jun '05

04 Jun '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Jun 2, 2005, at 1:43 AM, William Kucharski wrote: I found your problem, and it's an interesting one. vm_size_t mac_pagesize; [ ... ] host_page_size(host_port, &mac_pagesize); #if defined(__ppc__) /* * For 32-bit PowerPC ABIs, the scalable types were * always based upon natural_t (unsigned int). * Because of potential legacy issues with name mangling, * we cannot use the stdint uintptr_t type. */ typedef natural_t vm_offset_t; typedef natural_t vm_size_t; #else /* __ppc64__ */ /* * For 64-bit PowerPC ABIs, we have no legacy name mangling * issues, so we use the stdint types for scaling these * types to the same size as a pointer. */ typedef uintptr_t vm_offset_t; typedef uintptr_t vm_size_t; #endif host_page_size(host_port, &mac_pagesize); the Darwin kernel does this: kern_return_t host_page_size( host_t host, vm_size_t *out_page_size) { if (host == HOST_NULL) return(KERN_INVALID_ARGUMENT); *out_page_size = PAGE_SIZE; return(KERN_SUCCESS); } Thus, two workarounds would be: William Kucharski kucharsk(a)mac.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… The problem lies here in your code and the fact that you have your project set to be compiled in 64-bit mode for G5s: The issue is that /usr/include/mach/ppc/vm_types.h defines vm_size_t as: What this means is if __ppc__ is defined, vm_size_t will be an unsigned int (size 4 bytes); if __ppc64__ is defined, vm_size_t will be a unintptr_t (size 4 bytes on PPC32, 8 bytes on PPC64.) The problem is that the kernel is NOT a 64-bit executable, and as such is not compiled with __ppc64__ set, so when you make this call from a 64-bit executable: I think you probably see the issue here; since the kernel thinks a vm_size_t is a four byte quantity but your 64-bit app thinks it's an eight byte quantity, the kernel will fill in the upper four bytes of the eight byte variable with the system page size and leave the rest of the eight byte variable untouched. Thus when the kernel tries to return 0x1000 (4096), your 64-bit app will instead see it as 0x100000000000, or 17592186044416 (!). Why wouldn't it fill in the lower four bytes instead? I ask this because when I print out mac_pagesize, I do get 4096 as expected so the lower bytes must have the correct value. Now your earlier wacky number of 281299054850211840 bytes doesn't really calculate out properly, as given this error you would then get 281299054850211840/17592186044416 = 15990 pages, or only 62M free, not the 4.4G or so you were theoretically expecting; if mac_pagesize were an automatic I'd blame garbage left over in the "low" four bytes of the variable, but since it's a global it was initialized as 0 so there IS no garbage in the low four bytes. 1) Compile 32-bit only This would defeat the purpose of running a 64-bit executable - to test more than 4 GB of memory. The memory address pointers have to be 64 bits if they're going to be able to access all the memory a user has installed. John's machine for example has 15 GB installed so the main pointer returned by malloc has to be 64 bits. The memory space gets divided into two buffers, each 8 GB, so both these pointers must also be 64 bit to address each buffer space. If I distribute the app as 32-bit only, then no one will be able to test more than 4GB on any machine. There seem to be quite a few people out there with 4.5 GB or more installed in their G5's and they're going to need to test it all for reliability with 64-bit apps. 2) Declare mac_pagesize as a natural_t or unsigned int rather than a vm_size_t. (Obviously this would mean that mac_pagesize wouldn't agree with the man pages, but you've got to do what works.) This is definitely an option if the 64-bit compiler will allow it. Even worse, I could just set the page size to 4096 and be done with it. Not a good idea in case it ever changes but it seems to be 4K on most modern OS's these days. This is why my earlier attempt worked fine; I didn't specify that gcc should specifically compile my app as a 64-bit application. Yep, I covered this with John Caldwell as well. He used gcc 3.3 which doesn't compile 64-bit. The question is, what SHOULD happen in this case, and should there be an ADC tech note warning of this issue? According to the Darwin 64-bit conversion guide (<http://developer.apple.com/documentation/Darwin/Conceptual/ 64bitPorting/index.html>), "Only libSystem and the Accelerate framework are available to 64-bit developers. Higher level frameworks are 32-bit only." But is a call to host_page_size() really considered a "higher level framework?" I'd consider host_page_size() to be part of libSystem if anything. If we can't address more than 4 GB of memory, what's the point of having a 64-bit OS and a PPC G5? At the very least, Tiger should be able to get at all of the user's installed memory assuming the user runs a 64-bit app. How can the kernel handle more than 4 GB of address space if it's not 64-bit? There's got to be a simple workaround to this problem. I can't be the only person trying to use more than 4GB of memory at one time. Any application that bills itself as 64-bit has to address these issues. I don't know the answer to that, so I'll leave it to the Apple folks who frequent this list to sort out (Quinn would probably be the expert here.) This email sent to site_archiver(a)lists.apple.com

1 0

Re: Increasing MAXSYMLINKS from 32
by Justin Walker 04 Jun '05

04 Jun '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Jun 4, 2005, at 0:35, Andre Pang wrote: What is your goal in this? Why isn't the current value sufficient? This is the right list. Regards, Justin -- Justin C. Walker, Curmudgeon-At-Large Institute for General Semantics -------- A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing on a mailing list? _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… I have a need for changing the value of MAXSYMLINKS in sys/param.h (the maximum depth for traversing symbolic links) from 32 to something higher. POSIX & limits.h apparently defines the maximum level of symbolic links to be 255. So, I have a couple of questions about this: * Is there any harm done in simply editing sys/param.h from 32 to something higher such as 64, 128 or 256? I'm not sure what you mean by 'harm'. Editing param.h is, in general, going to cause conflicts between newly-compiled programs, and those previously compiled and the kernel. * Given that this maximum-symlink-depth value appears to be a constant defined in an included kernel-related header file, is there anyway to increase this symlink limit to something higher in already-compiled programs? (Perhaps even via a kernel extension.) No; and even if there were, you would have to deal with the dozens of frameworks and libraries as well. * If there is no way to increase the symlink limit for already-compiled programs, would it be possible to raise the limit to 64/128/256 in future versions of Darwin and Mac OS X? Making this a sysctl would be even nicer ... If you think it makes sense, then by all means file an enhancement request/bug report (<http://bugreporter.apple.com>). Be aware that this list is not a channel to Apple, official or otherwise. Apple folk who participate in list conversations do so on their own, and can't do much based on list traffic. It's the bug list that drives the setting of priorities internally. (Sorry if this post is to the wrong list; I thought darwin-kernel was probably the best choice. Please feel free to move to the post to another list if you feel it's more appropriate.) This email sent to site_archiver(a)lists.apple.com

1 0