September 2009 - Darwin-dev

Re: Problem using .zerofill / -segaddr to create very large segments
by Jay Reynolds Freeman 22 Sep '09

22 Sep '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Terry Lambert gave good advice, including: > That said, unless you have a metric ton of memory, mapping 320GB > into your address space will take a lot of wired pages to represent > the pmap and other data structures needed to provide the mapping > for that many pages ... It is 160 GByte, not 320, I am sorry if something in my posts was off-by-two. Terry, I think you came in late on this: The problem is, I am doing mmapping, and I need a big block of memory at the *same* virtual address in each of several processes (because that block contains pointers to locations within itself). The solution I have been using -- which was recommended by Apple, actually -- is to use a .s file with .zerofills in it to create a large region of empty memory, and then use the -segaddr flag (to the linker) to tell the linker where to put it. That makes sense to me, and I think that kind of thing is what linkers are for. That has so far worked fine, but I seem to be up against a limit of 160 GByte to the size of my segment, which is independent of the address I provide with -segaddr. Any more memory and then, although the build and link run fine, nevertheless I get run-time errors about classes that cannot be loaded. I can make do with 160 GByte for a while, but I will certainly want more in the near future. At the very least, I would like some better error messages; I ought to get some warning that I am painting myself into a corner where a class won't load, and perhaps a clue what to do about it. -- Jay Reynolds Freeman --------------------- Jay_Reynolds_Freeman(a)mac.com http://web.mac.com/jay_reynolds_freeman (personal web site) _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… The processes are 64-bit applications that run in Snow Leopard. They can use of all the memory you can put in a Mac Pro, and then some. I have a parallel lisp system with a dual-memory scheme for garbage collection -- the memories are most of what goes in the shared segment. I want to allow each of those memories to be as large as the most RAM you can have in a Mac, which I believe is 64 GByte today. I am sure I can find plenty of room in any one process's address space, but the interprocess coordination required to make *certain* that all of the interested processes can find a block at the *same* address is a mess. I feel strongly that I should seek a simpler solution. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Problem using .zerofill / -segaddr to create very large segments
by Terry Lambert 22 Sep '09

22 Sep '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com > I assume, given the above, that the motivation for this is that > you're trying to set things up such that pointers in the shared > memory all have the same addresses in each of the processes, and > that means you want to reserve the same block of address space > no matter what. Kind regards, That's 2^16 * 2^32 = 64K *4GB = 256TB. -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Sep 22, 2009, at 3:35 AM, Alastair Houghton <alastair(a)alastairs-place.net > wrote: On 22 Sep 2009, at 11:19, Jay Reynolds Freeman wrote: Correct. Have you tried using the "vmmap" tool to investigate the virtual memory map of the process that you're having trouble with? That should enable you to investigate where the free regions of VM are likely to be at the correct point in your program's execution. I'm not sure what the precise virtual address space limits are for OS X---there's probably a tech. note somewhere, but someone here will know in any case---but it should be easy enough to locate a region that's large enough for your purposes. Alastair. There is an architectural limit of 48 bits on Intel processors, which is documented, well, pretty much everywhere anyone who is doing code generation is looking at Intel documentation. So I'm pretty sure your approach of finding a large hole in the middle of the available virtual address space and just using it for a fixed mapping will work fine. I definitely would NOT put the hole in the heap using static allocations to get mapping set up by the kernel image loader, whose job is really to get something (hopefully not a huge amount of that something) into memory rather crudely, then hand over control to dyld to do the rest. That said, unless you have a metric ton of memory, mapping 320GB into your address space will take a lot of wired pages to represent the pmap and other data structures needed to provide the mapping for that many pages, so I probably would not be trying this on a machine with a 32 bit kernel and a lot of physical RAM. 320GB / 1 page per 4MB stride for a 64 bit pmap works out to 80K pages of page mappings, or 320MB not counting other overhead: a good argument for mapping windows instead of direct mapping everything. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Problem using .zerofill / -segaddr to create very large segments
by Alastair Houghton 22 Sep '09

22 Sep '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha1; c=relaxed; d=alastairs-place.net; h= subject:mime-version:content-type:from:in-reply-to:date:cc: content-transfer-encoding:message-id:references:to; q=dns/txt; s= aug07; bh=H/N+dzEh6GBH9+ut9HcizvGWDos=; b=nEreoZkvzyI9/kICy+RSrw g6j5mVvfED7sHmHWkMTTosblXolo2HUjxG1jwLeHIthx7bRCdxJIpCm+hzUFMZ7i Ti2jANne4b6QBmdlHb5+xdld/n3zAZMuJUPRoH/SLcrsa1EMZ6PXJ9NHzzPykFEy NGQxB7bHbVmXPXHXDvxWE= On 22 Sep 2009, at 11:19, Jay Reynolds Freeman wrote: > I assume, given the above, that the motivation for this is that > you're trying to set things up such that pointers in the shared > memory all have the same addresses in each of the processes, and > that means you want to reserve the same block of address space > no matter what. Kind regards, Alastair. -- http://alastairs-place.net _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Replying privately since this isn't strictly Darwin any longer ... True to an extent, but I wanted to understand why you were doing what you were doing before making any suggestions, so I've bought it back on list :-) Correct. Have you tried using the "vmmap" tool to investigate the virtual memory map of the process that you're having trouble with? That should enable you to investigate where the free regions of VM are likely to be at the correct point in your program's execution. I'm not sure what the precise virtual address space limits are for OS X---there's probably a tech. note somewhere, but someone here will know in any case---but it should be easy enough to locate a region that's large enough for your purposes. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Problem using .zerofill / -segaddr to create very large segments
by Alastair Houghton 22 Sep '09

22 Sep '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha1; c=relaxed; d=alastairs-place.net; h= subject:mime-version:content-type:from:in-reply-to:date:cc: content-transfer-encoding:message-id:references:to; q=dns/txt; s= aug07; bh=9M39QW7AqgjVv/0LSUGliCM46JA=; b=Pupy4aRoMxE6CvzbY4zdoo JxEWUBtNskC6jD+n6ZdOlCJwicdL64SihkPUxba+Z9hmHPKY0uj78o/FTPBzz6cK ROJlCn0b3pY+ERdwMueCxUK/af3qyKYvXUaz8oF/eJlPDJHZIwENeOAGAe19dz9i nXSzBe7Aa2P3BX2MXc3kY= On 21 Sep 2009, at 22:57, Brian Mastenbrook wrote: That clarifies things considerably... Kind regards, Alastair. -- http://alastairs-place.net _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… The problem is that code, the malloc heap, etc. are already placed in memory at that point. Using a zero-fill segment in an object reserves the heap space first, which forces the linker to move any relocatable code and the malloc heap to be allocated elsewhere. This is useful in general for image-based programming languages that are natively compiled, as you can save an image and then re-map it in memory as it was originally represented. I assume, given the above, that the motivation for this is that you're trying to set things up such that pointers in the shared memory all have the same addresses in each of the processes, and that means you want to reserve the same block of address space no matter what. So I guess the next question is why not move the shared memory segment further away from the heap's initial location? Or alternatively, why not use offsets rather than pointers within the shared segment (of course, there will be a performance hit, but offsets might result in safer code anyway), in which case you don't need to use MAP_FIXED? This email sent to site_archiver(a)lists.apple.com

1 0

Re: Problem using .zerofill / -segaddr to create very large segments
by Joel Reymont 21 Sep '09

21 Sep '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:subject:mime-version :content-type:from:in-reply-to:date:cc:content-transfer-encoding :message-id:references:to:x-mailer; bh=J0KWZJijZaTCceOK0hAdxvcW1yRdZhq4btvO8EdChDE=; b=Kwt5a+FHRKD3F0g9wcG00Aw6K8TiWodsPWLf3dFCby8u0ae1jN2AaQs+qFRmsyQHfJ WxiniSqzkxFfEvYBl9Nd9NVFTfPRbwzk7GJZEMsOQNbxoKzoJODOfjGIt2nTkbQ+tIP9 YWXTOXjAQb+qIfnJezU16Vqar5Vhwzv54pKkY= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; b=tgUGnpCQa/xkpegKzz66j6t1qyQ2aHMotwlLrzkXwZxxb6PSt9860dXCKFBm4wmTCN AEqME18h3N95gs85MNl17Vk1tpcJ17Vy6VgzKTeVQpCkNqNRqzN2fNDGxubuY5cEAQS/ 1MrX6wGg4zrJioIrZafpByrk0O9dYvmlQds1Y= On Sep 21, 2009, at 8:24 PM, Brian Mastenbrook wrote: This is what I did to port a commercial Forth from Linux to the Mac: http://tinyco.de/2009/01/26/creating-mac-binaries-on-any-platform.html --- fastest mac firefox! http://wagerlabs.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I understand what it is you're trying to do here, since I've been through this battle myself. Long, long ago I implemented this solution for SBCL by creating a Mach-O object file directly; This email sent to site_archiver(a)lists.apple.com

1 0

Re: Problem using .zerofill / -segaddr to create very large segments
by Jay Reynolds Freeman 21 Sep '09

21 Sep '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com -- Jay Reynolds Freeman --------------------- Jay_Reynolds_Freeman(a)mac.com http://web.mac.com/jay_reynolds_freeman (personal web site) On Sep 21, 2009, at 2:50 PM, Nat! wrote: Am 21.09.2009 um 20:12 schrieb Jay Reynolds Freeman: Ciao Nat! --------------------------------------------------- Ansgar, Sie sind nicht disziplinierbar, und wir werden Ärger bekommen. Doch wenn Sie ihre Leistung bringen, stehe ich hinter Ihnen. -- B. Möhlmann _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/jay_reynolds_freeman%40ma… This email sent to jay_reynolds_freeman(a)mac.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… The purpose of this stuff -- the .zerofills and the use of -segaddr -- is indeed to set up to use mmap, but I get problems before my code gets to that point. I have mmap working for me, but somewhere between the use of the assembler and the completion of launching the application, something goes haywire. My problem concerns keeping a block of memory out of the clutches of the storage allocator (malloc, new, et cetera), so that I can use it for mmap. I don't understand the problem completely, but I'd be trying to use mmap + MAP_FIXED to map the region, just right after load. Or doesn't that work as expected ? This email sent to site_archiver(a)lists.apple.com

1 0

Re: Problem using .zerofill / -segaddr to create very large segments
by Brian Mastenbrook 21 Sep '09

21 Sep '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Sep 21, 2009, at 4:50 PM, Nat! wrote: Am 21.09.2009 um 20:12 schrieb Jay Reynolds Freeman: _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… My problem concerns keeping a block of memory out of the clutches of the storage allocator (malloc, new, et cetera), so that I can use it for mmap. I don't understand the problem completely, but I'd be trying to use mmap + MAP_FIXED to map the region, just right after load. Or doesn't that work as expected ? The problem is that code, the malloc heap, etc. are already placed in memory at that point. Using a zero-fill segment in an object reserves the heap space first, which forces the linker to move any relocatable code and the malloc heap to be allocated elsewhere. This is useful in general for image-based programming languages that are natively compiled, as you can save an image and then re-map it in memory as it was originally represented. -- Brian Mastenbrook brian(a)mastenbrook.net http://brian.mastenbrook.net/ This email sent to site_archiver(a)lists.apple.com

1 0

Re: Problem using .zerofill / -segaddr to create very large segments
by Nat! 21 Sep '09

21 Sep '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Am 21.09.2009 um 20:12 schrieb Jay Reynolds Freeman: Ciao Nat! --------------------------------------------------- Ansgar, Sie sind nicht disziplinierbar, und wir werden Ärger bekommen. Doch wenn Sie ihre Leistung bringen, stehe ich hinter Ihnen. -- B. Möhlmann _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… My problem concerns keeping a block of memory out of the clutches of the storage allocator (malloc, new, et cetera), so that I can use it for mmap. I don't understand the problem completely, but I'd be trying to use mmap + MAP_FIXED to map the region, just right after load. Or doesn't that work as expected ? This email sent to site_archiver(a)lists.apple.com

1 0

Re: Problem using .zerofill / -segaddr to create very large segments
by Jay Reynolds Freeman 21 Sep '09

21 Sep '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com -- Jay Reynolds Freeman --------------------- Jay_Reynolds_Freeman(a)mac.com http://web.mac.com/jay_reynolds_freeman (personal web site) _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Many thanks to Brian Mastenbrook for his recently-posted comment and link; I will try it posthaste, after dealing with some other fire drills, and will report further here. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Problem using .zerofill / -segaddr to create very large segments
by Brian Mastenbrook 21 Sep '09

21 Sep '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Jay, http://paste.lisp.org/display/87459 -- Brian On Sep 21, 2009, at 1:12 PM, Jay Reynolds Freeman wrote: I have a problem using the -segaddr linker flag in an Xcode build. Here are a few more details: .zerofill __WSS, __Wss000, _WraithSpace, 0x40000000 .zerofill __WSS, __Wss001, _zz001, 0x40000000 .zerofill __WSS, __Wss002, _zz002, 0x40000000 ... .zerofill __WSS, __Wss156, _zz156, 0x40000000 .zerofill __WSS, __Wss157, _zz157, 0x40000000 .zerofill __WSS, __Wss158, _zz158, 0x40000000 .zerofill __WSS, __Wss159, _zz159, 0x40000000 .zerofill __WSS, __Wss160, _zz160, 0x40000000 Thank you very much. -- Jay Reynolds Freeman --------------------- Jay_Reynolds_Freeman(a)mac.com http://web.mac.com/jay_reynolds_freeman (personal web site) _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/brian%40mastenbrook.net -- Brian Mastenbrook brian(a)mastenbrook.net http://brian.mastenbrook.net/ _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I understand what it is you're trying to do here, since I've been through this battle myself. Long, long ago I implemented this solution for SBCL by creating a Mach-O object file directly; at the time, the linker didn't even have a .zerofill directive if I remember correctly. This has since been removed from the main SBCL trunk (which I think was a mistake), but you can view the elements of the solution here: http://sbcl.cvs.sourceforge.net/viewvc/sbcl/sbcl/src/runtime/ppc-darwin-mkr… - creates a .o file with N zero-fill segments of zero size http://sbcl.cvs.sourceforge.net/viewvc/sbcl/sbcl/src/runtime/ppc-darwin-fix… - opens the SBCL executable, grovels through the sections, finds the zero-fill segments, and fixes the size and start address parameters This two-phase process was necessary because `nm' & friends had bugs with handling of zero-fill segments, and didn't understand that the size of the segment didn't indicate a number of bytes to be skipped in the object file. `nm' has gotten a little bit smarter about this these days; it still doesn't like seeing a size on a zero-fill segment, but it doesn't barf on it anymore. I've whipped up a quick adaptation of this for x86_64, which you can view here: I used this program to create an object with a 161GB reservation, and it seemed to work fine in a very cursory test. I have a problem which might be Xcode, Darwin, or even Cocoa; I ran this posting past the Xcode list a few days ago with no insightful developments, so my next thought was to ask here if anyone had a clue... I am building a 64-bit Macintosh application using Xcode 3.2 on MacOS 10.6.1, on an early-2008 model Mac Pro. The build targets architecture x86_64, uses the LLVM GCC 4.2 compiler, has a Mac OS X Deployment Target of Mac OS X 10.6, and if you would like to know any other build settings, let me hear from you. My problem concerns keeping a block of memory out of the clutches of the storage allocator (malloc, new, et cetera), so that I can use it for mmap. I do have a solution to do that, which DOES work, which is: 1) Include in my build a ".s" file that contains appropriate .zerofill directives to define a segment large enough for my purposes. 2) Tell the linker where to put that segment by putting, e.g, "- segaddr __WSS 0x10000000000" in the "Other Linker Flags" entry of the build rules pane of the Xcode information panel for the target for my application. This all works fine, and has been working for months, and I am not having any problem with mmap (knock on wood). My problem is that I need a *very* large segment -- at the moment I am using 160 Gigabytes (segment length #x2800000000 bytes, and remember, this is 64-bit code). I need to make the segment even larger for a forthcoming upgrade of the application, and I appear to be running up against some kind of limitation on the total size of segments created in this manner. It has been decades since I have written Intel x86 assembler in any form, and I have always regarded the linker as rather a black box and not tried to use unusual flags. So I hope I am merely making some embarrassingly simple mistake, but I have checked the man pages and the web, and I cannot find any useful clues. (By the way, since someone asked on the Xcode list, I will mention that I need a large shared segment for use in a parallel -- separate Unix processes -- Lisp system with shared Lisp main memory. I would be happy to talk about it off-list if anyone is interested.) I haven't been able to make the .zerofill directive understand 64- bit numbers -- #x1234567890LL and similar suffixes result in errors or incorrect linkages. So my ".s" file actually contains a lot of adjacent .zerofills, as follows: That is a total of 160 one-Gigabyte blocks, all in segment __WSS. That works fine, and has been working for months. The problem comes if I add even one more .zerofill block at the end of that file, creating a segment of 161 GByte instead of 160 GByte: In that case, the Xcode build completes (I do a clean and a complete fresh build) with no errors reported, BUT I get run-time errors when I launch the application, while the framework is still getting going, long before the actual mmap has taken place. Here is the first of many run-time errors, as reported in the Console Messages log. The error kind of makes sense, in that my application does indeed have the indicated class and should indeed be trying to load it at launch time (but why increasing the segment size has made it impossible to load the class is beyond me): 9/19/09 1:06:32 PM Wraith Scheme[35427] Unknown class JRF2NSTextView in Interface Builder file at path /Users/JayFreeman/Developer/Scheme/ WraithScheme.64/WraithCocoa/build/Release/Wraith Scheme.app/Contents/ Resources/English.lproj/MainMenu.nib. If I remove the extra .zerofill, re-clean and rebuild, all is well again. I have tried changing the address for __WSS that I provide via the - segaddr flag. I have also tried adding the additional reserved memory in a different segment (eg, .zerofill __MMS, __Mms000 _mm000, 0x40000000) with an additional -segaddr flag in "Other Linker Flags". Neither works. It looks like the system doesn't like to have the total size of reserved segments larger than about 160 GByte. (As you may have guessed, I learned about this limit by tripping over it; it is no accident that my present ".s" file defines precisely 160 GByte of reserved segment.) I am leaning toward thinking this is an Xcode problem, but I would be eager to know whether anyone can find any obvious error in my use of .zerofill and -segaddr, or whether anyone can think of anything in the Darwin body of code that might cause this phenomenon. Pointers to more documentation would also be very useful; I have spent a while searching, but I am such a stranger to x86 assembler and to the linker that I may very well be using the wrong search keywords, and my knowledge of Darwin _per_se_ is rather, er, unevolved ... This email sent to brian(a)mastenbrook.net This email sent to site_archiver(a)lists.apple.com

1 0