September 2009 - Darwin-dev

Re: Problem using .zerofill / -segaddr to create very large segments
by Jay Reynolds Freeman 23 Sep '09

23 Sep '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com And now I report a fascinating experiment: See my posting dated 10:12 PM PDT 22 September for context. I had been successfully running with a .s file containing quantity 160 of 1-GByte ".zerofill" lines, and failing when I added one more such line. So, I changed my file so that each .zerofill was for 1.5 GByte -- changed 0x40000000 to 0x60000000 in 160 places. It built and ran fine, and the output of vmmap shows a 240 GByte chunk properly mapped! That is enough memory for me for now. (That is just as well, because .zerofill appears to accept only SIGNED 32-bit numbers -- 0x80000000 causes an error about a negative section size...) Evidently the problem is not with the total size of the segment, but with the number of .zerofills. I suspect it is time for me to file some kind of bug report. With this information and the content of my previous post, can Terry or Greg suggest any more useful experiments, or any suggestions for what to file such a report against? I keep forgetting to thank you all for assistance; thanks very much. -- Jay Reynolds Freeman --------------------- Jay_Reynolds_Freeman(a)mac.com http://web.mac.com/jay_reynolds_freeman (personal web site) _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Problem using .zerofill / -segaddr to create very large segments
by Jay Reynolds Freeman 23 Sep '09

23 Sep '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Be patient with me, folks, I am not a Unix system expert and don't play one on Apple TV (tm), either. :-) I am just a garden-variety coder trying to link an application, so what if I need a few hundred GByte ... :-) Anyway ... I ran vmmap on my "good" process (160 GByte shared segment); I hope I have made the correct observations concerning its voluminous output (I have never used vmmap before): 1) There are no writeable or non-writeable regions anywhere near my mmapped chunk except a region called "__LINKEDIT" that starts immediately above the upper end of my chunk; my chunk is mapped properly to the range (hex) 0000010000000000-0000012800000000 2) In particular, there is a lot of framework, dylib and other system stuff way at the high end of (48-bit) memory; at addresses like 00007fffxxxxxxxx. All else (except __LINKEDIT and my chunk) is way low, addresses all lower than 00000001c0000000. It appears that my process's __TEXT and __DATA start at 0x100000000; there is lots of system __TEXT and system __DATA way high, above 0x00007fff00000000. 3) The vmmap summary for the "good" process looks like this: REGION TYPE [ VIRTUAL] =========== [ =======] CG backing stores [ 2480K] CG image [ 120K] CG raster data [ 64K] CG shared images [ 2260K] Carbon [ 2876K] CoreGraphics [ 16K] IOKit [ 256.0M] MALLOC [ 91.0M] Memory tag=240 [ 4K] Memory tag=242 [ 12K] Memory tag=243 [ 4K] Memory tag=249 [ 156K] Memory tag=251 [ 8K] STACK GUARD [ 56.1M] Stack [ 9232K] VM_ALLOCATE [ 16.3M] __DATA [ 9616K] __IMAGE [ 1240K] __LINKEDIT [ 30.8M] __TEXT [ 79.6M] __UNICODE [ 536K] __WSS [ 160.0G] <== My mmapped segment. mapped file [ 31.2M] shared memory [ 4396K] 4) The vmmap output from the 161-GByte version looks the same, except that the high end of my chunk is now at 0000012840000000 (and that is where __LINKEDIT now starts), and the vmmap summary shows that segment __WSS has size 161.0G. Both changes are consistent with one more 1-GByte .zerofill. I also ran "otool -l" on both executables; it also produced voluminous output. I am not sure quite what to look for -- I have never used otool before, either -- but: 5) In both executables, each of the 160 (resp. 161) sections of my big chunk -- each ".zerofill" -- has its own entry, which looks typically like this : Section sectname __Wss156 segname __WSS addr 0x00000109c0000000 size 0x0000000040000000 (past end of file) offset 0 align 2^0 (1) reloff 0 nreloc 0 flags 0x00000001 reserved1 0 reserved2 0 6) However, in each executable, there appears to be only one LC_SEGMENT_64 associated with the load of segment __WSS, which is this (in the "good" executable): Load command 3 cmd LC_SEGMENT_64 cmdsize 12872 segname __WSS vmaddr 0x0000010000000000 vmsize 0x0000002800000000 fileoff 606208 filesize 0 maxprot 0x00000007 initprot 0x00000003 nsects 160 flags 0x0 And is this in the "bad" one. Load command 3 cmd LC_SEGMENT_64 cmdsize 12952 segname __WSS vmaddr 0x0000010000000000 vmsize 0x0000002840000000 fileoff 606208 filesize 0 maxprot 0x00000007 initprot 0x00000003 nsects 161 flags 0x0 The vmaddrs, vmsizes, and nsects are in each case correct. The otool output shows 19 load commands for each of the two executables. -- Jay Reynolds Freeman --------------------- Jay_Reynolds_Freeman(a)mac.com http://web.mac.com/jay_reynolds_freeman (personal web site) _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Problem using .zerofill / -segaddr to create very large segments
by Terry Lambert 23 Sep '09

23 Sep '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com My simple-minded view of the linker is that it is supposed to reserve such segments as the user asks for, then arrange to put text, data and heap in what is left over, and complain if for some reason there is a conflict -- all this at link time. Similarly, I would hope for a useful run-time error message if for some reason either the loader or Apple's own code cannot make do with what the linker has done. What I actually get is an error-free build and link, and then run-time messages about not being able to load classes, with no hint of why, wherefore, or who is to blame. I believe I have a workaround -- Brian Mastenbrook posted code for a program that writes the requisite .o file, though I have not yet had time to try it. Yet bugs in the linker, loader, assembler, or run-time support stuff are worth chasing down; unless someone turns up an obvious error in what I am doing I will probably end up filing one or more bug reports on this stuff. osfmk/mach/i386/vm_param.h -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Sep 22, 2009, at 5:10 PM, Jay Reynolds Freeman wrote: It can't really do this _all_ at link-time, since it doesn't know the size of the shared libraries you will actually run against, only the ones you are currently linking against, and that can change in the future. It can complain at runtime if you run your static segment into the text of the program, the stack, or the heap, which may be the issue, or you could be running up against a limit in dyld (which Brian M.'s stuff might save you from, at least temporarily). If you'd used the environment variables I pointed it, it probably would have given you enough information to understand what's going on... You may or may not find yourself in the same boat; it depends on if you are running yourself into the heap or the default stack location, etc., which I really can't answer for you unless I was sitting in front of your machine and did the things I've been telling you you need to do to diagnose the issue. I expect you'll get an assembler bug out of it, to get the assembler to take 64 bit values, and you may or may not get a linker bug out of it, depending on why it can't load the class, and whether you are running into the stack, the text, or whatever with your huge segment. By the way, the 64 bit process address space layout (as far as the kernel is concerned) is documented in the published xnu sources, in the file: This email sent to site_archiver(a)lists.apple.com

1 0

Re: Problem using .zerofill / -segaddr to create very large segments
by Greg Parker 23 Sep '09

23 Sep '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com > The traditional way to pic-ify such data structures is to store > relative offsets instead of absolute pointers. Notable performance hit and serious addition of complexity: Replace *foo by *(something->getBaseAddress() + fooOffset) in about a zillion places. -- Greg Parker gparker(a)apple.com Runtime Wrangler _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Sep 22, 2009, at 5:50 PM, Jay Reynolds Freeman wrote: On Sep 22, 2009, at 5:42 PM, Greg Parker wrote: You are thinking relocatable binary files, I believe. Think "linked lists" and "structs with pointers to other structs", instead -- that is what is in this big memory-mapped block. The trick to save performance is to store an offset relative to the location where the offset is stored. Then `*foo` becomes `foo+*foo`. And you can save space by using 4-byte offsets if you know the complete data structure spans less than 4GB (though for you that appears not to be the case). Snow Leopard's Objective-C runtime uses both of those for the selector table in the dyld shared cache. Usually the shared cache loads at the same address everywhere, but this makes it work without modification otherwise. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Problem using .zerofill / -segaddr to create very large segments
by Jay Reynolds Freeman 23 Sep '09

23 Sep '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Sep 22, 2009, at 5:42 PM, Greg Parker wrote: > The traditional way to pic-ify such data structures is to store > relative offsets instead of absolute pointers. Notable performance hit and serious addition of complexity: Replace *foo by *(something->getBaseAddress() + fooOffset) in about a zillion places. > The linker and loader are not typically called upon to handle > hundreds of gigabytes of data. Welcome to the wonderful world of 64-bit applications! :-) More later ... -- Jay Reynolds Freeman --------------------- Jay_Reynolds_Freeman(a)mac.com http://web.mac.com/jay_reynolds_freeman (personal web site) _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… You are thinking relocatable binary files, I believe. Think "linked lists" and "structs with pointers to other structs", instead -- that is what is in this big memory-mapped block. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Problem using .zerofill / -segaddr to create very large segments
by Greg Parker 23 Sep '09

23 Sep '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com > Ideally, you'd fix the runtime to support PIC code generation so you > can relocate segments for locality of reference to keep the address > space access patterns non-fragmented over time (via feedback). That was for my .s file with 160 1-GByte sections. When I add another .zerofill, making 161 1-G-byte sections, I get a size -l output that appears to be exactly the same, except it correctly lists 161 sections in Segment __WSS, for a total segment size of 172872433664. What is the `vmmap` output of the bad and good versions? -- Greg Parker gparker(a)apple.com Runtime Wrangler _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Sep 22, 2009, at 5:29 PM, Jay Reynolds Freeman wrote: You are thinking relocatable binary files, I believe. Think "linked lists" and "structs with pointers to other structs", instead -- that is what is in this big memory-mapped block. The traditional way to pic-ify such data structures is to store relative offsets instead of absolute pointers. I believe you, but this is fundamentally a static-linking problem, and I keep thinking that is precisely what the linker is supposed to do! Using run-time operations to solve a static linking problem just does not sound like the right thing to do ... The linker and loader are not typically called upon to handle hundreds of gigabytes of data. You may have found a bug in one of them.Or you may have an error somewhere and not getting as much error reporting as you'd like. One alternative to using hundreds of .zerofill sections is to use one oversized __UNIXSTACK segment. Use -stack_addr and -stack_size to specify a __UNIXSTACK segment big enough to hold your main thread's stack plus your mmap data. The stack will start at the high address so you can mmap at the low address. Note that you'll lose stack-overflow protection unless you do more work. Be warned that -stack_addr may not take the value you expect. The first executable -- the one with 160 sections -- loads and runs fine. The second gives errors early in the load about not being able to load certain classes. This with *NO* other changes to anything. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Problem using .zerofill / -segaddr to create very large segments
by Jay Reynolds Freeman 23 Sep '09

23 Sep '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Sep 22, 2009, at 3:22 PM, Terry Lambert wrote: > You have a LISP runtime which you are using internal > compilation to generate code for language extensions and ... A little simpler, actually -- what is in my big memory-mapped block is just S-expressions, linked by pointers. No processor core's program counter is ever set to an address in that block. The Lisp system is an interpreter, and all code that actually runs is all in the normal text segments of the various processes that get to look at the shared block. Sometimes the content of an S-expression tells those processes what code to run, but the actual running code is all in their own text segments. > You need to grab a big chunk in the middle of your address space, > and not in your heap. It would help if I knew where the heap *was*. My impression is that the heap is relocatable, so presumably the linker or loader can simply put it in some likely location to avoid such other segments as I have claimed for my own use. This I try to do! > Ideally, you'd fix the runtime to support PIC code generation so you > can relocate segments for locality of reference to keep the address > space access patterns non-fragmented over time (via feedback). You are thinking relocatable binary files, I believe. Think "linked lists" and "structs with pointers to other structs", instead -- that is what is in this big memory-mapped block. > You really do not want to use all those .zerofill's. > I haven't seen the output of your otool -l, but I rather suspect that > each one might be getting it's own LC_SEGMENT_64. You may also be > generating LC_SEGMENT's and linking them in instead (which gets you > a 32 bit limit). Segment __PAGEZERO: 4294967296 (vmaddr 0x0 fileoff 0) Segment __TEXT: 528384 (vmaddr 0x100000000 fileoff 0) Section __text: 352852 (addr 0x100004a90 offset 19088) Section __symbol_stub1: 714 (addr 0x10005ace4 offset 371940) [...] total 509262 Segment __DATA: 163840 (vmaddr 0x100081000 fileoff 528384) Section __nl_symbol_ptr: 1152 (addr 0x100081000 offset 528384) Section __la_symbol_ptr: 952 (addr 0x100081480 offset 529536) [...] total 163017 Segment __WSS: 171798691840 (vmaddr 0x10000000000 fileoff 606208) Section __Wss000: 1073741824 (addr 0x10000000000 offset 0) Section __Wss004: 1073741824 (addr 0x10040000000 offset 0) Section __Wss008: 1073741824 (addr 0x10080000000 offset 0) [...] Section __Wss159: 1073741824 (addr 0x127c0000000 offset 0) total 171798691840 Segment __LINKEDIT: 131072 (vmaddr 0x12800000000 fileoff 606208) total 176094482432 <end of printout> That was for my .s file with 160 1-GByte sections. When I add another .zerofill, making 161 1-G-byte sections, I get a size -l output that appears to be exactly the same, except it correctly lists 161 sections in Segment __WSS, for a total segment size of 172872433664. -- Jay Reynolds Freeman --------------------- Jay_Reynolds_Freeman(a)mac.com http://web.mac.com/jay_reynolds_freeman (personal web site) _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… > If at all possible, you also need to arrange your code generation > such that your normal access pattern isn't sparse in the memory region > you allocate. > If you perform an mmap() very early on of zero-fill pages, then you > can do it as a .constructor or as part of an early constructor, or > very early in your runtime, and that will ensure that you get the common > mapping that you want. I believe you, but this is fundamentally a static-linking problem, and I keep thinking that is precisely what the linker is supposed to do! Using run-time operations to solve a static linking problem just does not sound like the right thing to do ... At the very least I am going to file a bug about .zerofill not accepting 64-bit section sizes. I suspect that was an oversight in going to 64- bit code. However, the .zerofills appear to be doing their job; keep reading. I ran size -l on the executable (inside the app wrapper) and get the following: I will spare you all the printout; in what follows, "__WSS" is the big segment that I create. The first executable -- the one with 160 sections -- loads and runs fine. The second gives errors early in the load about not being able to load certain classes. This with *NO* other changes to anything. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Problem using .zerofill / -segaddr to create very large segments
by Jay Reynolds Freeman 23 Sep '09

23 Sep '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com > Have you tried using the "vmmap" tool to investigate the > virtual memory map of the process that you're having trouble > with? That should enable you to investigate where the free > regions of VM are likely to be at the correct point in your > program's execution. I had previously used a run-time mechanism for figuring out what absolute address to use for the shared memory. I would like to get away from that because it is difficult, confusing, and error-prone to try to have multiple processes coordinate on this matter: If for any reason Apple's frameworks or system software are allocating in a different pattern for the different instances of the programs that need to share memory, finding a place to put the shared block is a mess. There certainly are ways to reserve memory at link time at specific addresses; assembly-language programmers have been doing that for decades. I just want to use the same techniques in my applications. That approach is straightforward and should be robust. My simple-minded view of the linker is that it is supposed to reserve such segments as the user asks for, then arrange to put text, data and heap in what is left over, and complain if for some reason there is a conflict -- all this at link time. Similarly, I would hope for a useful run-time error message if for some reason either the loader or Apple's own code cannot make do with what the linker has done. What I actually get is an error-free build and link, and then run-time messages about not being able to load classes, with no hint of why, wherefore, or who is to blame. I believe I have a workaround -- Brian Mastenbrook posted code for a program that writes the requisite .o file, though I have not yet had time to try it. Yet bugs in the linker, loader, assembler, or run-time support stuff are worth chasing down; unless someone turns up an obvious error in what I am doing I will probably end up filing one or more bug reports on this stuff. -- Jay Reynolds Freeman --------------------- Jay_Reynolds_Freeman(a)mac.com http://web.mac.com/jay_reynolds_freeman (personal web site) _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Problem using .zerofill / -segaddr to create very large segments
by Nat! 22 Sep '09

22 Sep '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Am 21.09.2009 um 23:57 schrieb Brian Mastenbrook: Ciao Nat! ------------------------------------------------------ Hence, if the present world doth go astray, In you the cause is, be it sought in you; And I therein will now be thy true spy. - Alighieri _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… The problem is that code, the malloc heap, etc. are already placed in memory at that point. Using a zero-fill segment in an object reserves the heap space first, which forces the linker to move any relocatable code and the malloc heap to be allocated elsewhere. This is useful in general for image-based programming languages that are natively compiled, as you can save an image and then re-map it in memory as it was originally represented. Maybe it's possible to setup a multiple stage-boot system. The basic app is very small basically just allocates the region with mmap, no linkage to the C runtime and then loads the rest as a bundle ? This email sent to site_archiver(a)lists.apple.com

1 0

Re: Problem using .zerofill / -segaddr to create very large segments
by Terry Lambert 22 Sep '09

22 Sep '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Sep 22, 2009, at 1:50 PM, Jay Reynolds Freeman wrote: Terry, I think you came in late on this: The problem is, I am doing mmapping, and I need a big block of memory at the *same* virtual address in each of several processes (because that block contains pointers to locations within itself). You have a LISP runtime which you are using internal compilation to generate code for language extensions and stuffing them into your heap area, and then l;ater reloading them as precompiled language extensions. The code is not PIC, so you want the code loaded at the same location so that the data structures and code can be found by the runtime. Right so far? You need to grab a big chunk in the middle of your address space, and not in your heap. If at all possible, you also need to arrange your code generation such that your normal access pattern isn't sparse in the memory region you allocate. Ideally, you'd fix the runtime to support PIC code generation so you can relocate segments for locality of reference to keep the address space access patterns non-fragmented over time (via feedback). The processes are 64-bit applications that run in Snow Leopard. They can use of all the memory you can put in a Mac Pro, and then some. I have a parallel lisp system with a dual-memory scheme for garbage collection -- the memories are most of what goes in the shared segment. I want to allow each of those memories to be as large as the most RAM you can have in a Mac, which I believe is 64 GByte today. I am sure I can find plenty of room in any one process's address space, but the interprocess coordination required to make *certain* that all of the interested processes can find a block at the *same* address is a mess. I feel strongly that I should seek a simpler solution. If you perform an mmap() very early on of zero-fill pages, then you can do it as a .constructor or as part of an early constructor, or very early in your runtime, and that will ensure that you get the common mapping that you want. You really do not want to use all those .zerofill's; I haven't seen the output of your otool -l, but I rather suspect that each one might be getting it's own LC_SEGMENT_64. You may also be generating LC_SEGMENT's and linking them in instead (which gets you a 32 bit limit). The output of vmmap on the running process (without the 161st .zerofill) would probably be informative. You may also want to look at the man pages for dyld, since there are a bunch of environment variables you can set to get more extensive diagnostics. This email sent to site_archiver(a)lists.apple.com

1 0