October 2009 - Darwin-dev - lists.apple.com

Re: Using mmap to extend an existing mapping?
by Jens Alfke 08 Oct '09

08 Oct '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Oct 8, 2009, at 11:14 AM, Terry Lambert wrote: —Jens _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… They're pretty basic, but not entirely obvious if you are used to owning the entire address space and having stacks grow down in it and heaps grow up [...] Modern address spaces are a lot more complicated than they were in the 1970s and early 80s. 8-). No, I understand that part; I was just expecting re-mmaping to act more like realloc and less like memcpy. Thanks for the advice. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Using mmap to extend an existing mapping?
by Terry Lambert 08 Oct '09

08 Oct '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Oct 8, 2009, at 9:08 AM, Jens Alfke <jens(a)mooseyard.com> wrote: On Oct 7, 2009, at 11:12 AM, Terry Lambert wrote: Typically you are expected to keep your mapped regions away from stack and heap allocations, which means either limiting yourself to a fraction of the address space and using a generally unused region, or by establishing your mapping followed immediately by an adjacent mapping of zero-fill pages that you can later choose to map over top of Thanks! This sounds like it requires knowing ahead of time where to put the mappings, and specifying fixed addresses. But how do you know? You are going down the wrong path; you just need to know the location of a hole large enough for your maximum eventual region size early in the life of your program. Hardcoding an address makes assumptions about how the OS lays out process address spaces (and this is code I'd like to make cross- platform), and runs the risk of colliding with other libraries that are making the same assumptions and trying to use the same addresses. Don't hard-code the address if the data contents you plan to put there don't require it through position dependence. But I don't know of any API that would let me figure this out dynamically, i.e. by iterating over the already-allocated address ranges. There are Mach APIs that can do this, but I would discourage their use, particularly in multithreaded programs. Preflighting in general is a bad idea (key Yoda: "Do or not do; there is no try"), and by the time you go to use your preflight answer, the answer may have changed on you because of the actions of another thread. Let's say my file is currently 100k but I want to allow for growth up to 10MB. If I call mmap with a range of (0,10MB) what happens? Does the kernel only allocate 100k, or do I get a 10MB region? If the latter, then if I append to the file does the new data appear in the region (assuming I used MAP_SHARED)? That would get me the behavior I want... If you map a file, you get however many pages are in the file, up to 10MB; if you map anonymous memory, you get 10MB always. So ask for 10MB of anonymous memory at any location, get it, and then map the file on top of it using MAP_FIXED and the base address you got back when you asked for the 10MB to make sure it maps on top of your 10MB reservation. However much of the file exists will replace the initial pages in the 10MB region, leaving the rest of the 10MB region reserved for future expansion of the file. Your disadvantage (one mapping overwrites another) has now become your advantage. Because you have the 10MB mapping already, the system won't break up your 10MB address region on you with its own mappings and allocations: it's effectively reserved exclusively for your use. (Apologies if these are obvious newbie questions. Feel free to point me to in-depth coverage of mmap; I've done some googling but not exhaustively.) They're pretty basic, but not entirely obvious if you are used to owning the entire address space and having stacks grow down in it and heaps grow up and the system not putting anything in the middle because you don't have thread stacks getting allocated and you don't have fragmentation of the address space from allocations and deallocations because you don't have a garbage collector and you don't dynamically load/unload code and data and you either don't use object oriented languages, or all your objects are globally scoped, etc.. Modern address spaces are a lot more complicated than they were in the 1970s and early 80s. 8-). This email sent to site_archiver(a)lists.apple.com

1 0

Re: Using mmap to extend an existing mapping?
by Jens Alfke 08 Oct '09

08 Oct '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Oct 7, 2009, at 11:12 AM, Terry Lambert wrote: —Jens _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Typically you are expected to keep your mapped regions away from stack and heap allocations, which means either limiting yourself to a fraction of the address space and using a generally unused region, or by establishing your mapping followed immediately by an adjacent mapping of zero-fill pages that you can later choose to map over top of Thanks! This sounds like it requires knowing ahead of time where to put the mappings, and specifying fixed addresses. But how do you know? Hardcoding an address makes assumptions about how the OS lays out process address spaces (and this is code I'd like to make cross- platform), and runs the risk of colliding with other libraries that are making the same assumptions and trying to use the same addresses. But I don't know of any API that would let me figure this out dynamically, i.e. by iterating over the already-allocated address ranges. Let's say my file is currently 100k but I want to allow for growth up to 10MB. If I call mmap with a range of (0,10MB) what happens? Does the kernel only allocate 100k, or do I get a 10MB region? If the latter, then if I append to the file does the new data appear in the region (assuming I used MAP_SHARED)? That would get me the behavior I want... (Apologies if these are obvious newbie questions. Feel free to point me to in-depth coverage of mmap; I've done some googling but not exhaustively.) This email sent to site_archiver(a)lists.apple.com

1 0

Re: getting library information wrt address.
by Arnab Ganguly 08 Oct '09

08 Oct '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=+yc7Q44nma8eZ+aNyeD72r9Pd1TviMz5kfddGGzOctg=; b=MPEkA4hY6cYExu+gu3uZwe2dM/AIXnikhzvdW5ts1HUjQ6FDkcz25vbweZHRQcntTc yIIY+oB8bMdWUxy+i95lta4rVPJOhcWSVMGKfdvcDtUA4Pfx+cXNL+vTb0gi8VFoAQnd lgvUJhxAiRUJq6snTHuOrP0rhZopHY0g2V2Vw= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=r+t1l80bjJu8iWwtaqGWHULNnzfdqyew7v8Y3Hr70zUST5Lj77652+Cvp6FiYk+o4H yaItghlsn9tExhFgOMaaV7rnJOvJF6cr6fgtFMSoANuNWVINFYOWF1vXfAzJxn6D/gUp cKuMHSL9om+pU2k1AIQe80etiuBp/Z1Zy7Aic= _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: getting library information wrt address.
by Mo McRoberts 08 Oct '09

08 Oct '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Thu, Oct 8, 2009 at 15:38, Arnab Ganguly <aganguly01(a)gmail.com> wrote: > Hi All, > Is it possible to get the library name wrt to address ? I am using dladdr > system call and with respect to address it the field dli_fname gives the > framework name, how to achieve the same for shared libraries? > I don't want to use atos. If some system call is available it would be > great. To the best of my knowledge, a framework _is_ a shared library, just one which happens to lack a .dylib extension, a lib prefix and is bundled into an old-style bundle directory structure. Once it’s actually loaded into your process, I’m pretty sure there’s no actual difference between it and a .dylib except for the name. M. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

getting library information wrt address.
by Arnab Ganguly 08 Oct '09

08 Oct '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=Pnc9WFM+Ga0i9WNf3nPWmIfHMydwR2Q2+82A9mX6LJo=; b=On2Mhp4WVDvoEtkXCDJkk/CSItYRPcs0tmO0keKJ846igDIGg9TtAcuBOJIDF5vTA6 ionF1Ur6yCBt7jybgoOpoFJNUx000HInBBI2G0Ns5Zhu9x1/HGNZjSK10BKBaTdkPMJs 8I4LRv9f6BL7wmqYW4QP1Qa6DzwXwYkU1rzsQ= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=xLSbBex1ahWcPfHx/641f7PUX0XFs6WjTGwgIHk3Bhy/8a1oXT2csPgHu1hcQhuqh5 H17S9nWR4WKWJFKa91VAnnprpFCVfl6BmmZk9TXE6AWpwrgtjRZzg4Kayn5oVO2gAfYO 5b5x8POeStuV2oFYOhUqXdrosglVuzUhvp9f4= _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: How to work on case-sensitive projects?
by Steve Byan 08 Oct '09

08 Oct '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Oct 7, 2009, at 1:44 PM, Alastair Houghton wrote: On 7 Oct 2009, at 18:29, Allen Curtis wrote: -- Steve Byan <stevebyan(a)me.com> Littleton, MA 01460 _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… BTW: This will be a problem for any Linux kernel developer wishing to use their Mac for development. The problem is the stupid decision on the part of the Linux kernel people to have filenames that differ only by case in their source tree. True, but it is something that needs to be dealt with..... Ideally by the Linux kernel maintainers fixing their source tree (though doubtless they don't think it's a problem), but in the meantime there is an easy workaround. I believe the Linux kernel developers introduced this purposefully in order to sabotage people trying to use Linux code on Windows. It's a feature, not a bug. The Linux kernel similarly sabotages people wanting to write kernel extensions in C++ by purposefully using C++ keywords such as "class" as names of kernel structures. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Crash reporter starting stack trace
by Arnab Ganguly 08 Oct '09

08 Oct '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=8GvsWtrq9XjoL/GCDErwfbLrWjmWhv6cYzBmQ87u/XQ=; b=kHgTr/Bwl5dFxKFcSXmhs182JQ/2ua1O9ELPi181YCNXq8Oz2sQoXwio5wfQrOcnuM 5W/TgFV1taHaIy/bDMrXYoiAR7OlJyliChD1YnyU2eCQN22Luts6ZM29Q0WHaQW0x3mf yRyqSUQ/B7hBVc0D9O8YzPJCvPL0sdVJUdKVI= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=DkvDL3WTv887hiE6Je2wfn2ZT7h2bJAZyNS+R9+G765G97GPT6Ga2WWeb79n8xLS3P WWHxISOEdE2Nz6z1AvpthgMyYz8cXrV+gZl7zCh73yB4utjdjrb3KlOpvmdjkwRZQAwo aoBggZLIjeXAA6Il080g8A3MXhERd6OMyCeeg= I told you this before... -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com Hi All, Thanks for all your inputs.I have checked the Xcode project settings.But there generate Position Dependent Code is unchecked.Is there anything else do I need to check ? My further observation looks like the system dynamic libraries, system frameworks names are coming incorrect but the frameworks developed by us names comes correctly. Any inputs would be very much appreciated. -A On Oct 6, 2009, at 10:25 AM, Arnab Ganguly <email@hidden> wrote: Hi All, Some inputs required. When an application crashes, what is the starting stack address Crashreporter gets as? Whatever is in the %esp/%rbp within the thread state struucture for each thread at the time the mach crash exception is thrown. We developed a similar crash app but when we compare the stack trace with the Crashreporter's there is a mismatch with initial stack addresses but after some traces it matches. Specially the traces very much internal to the system dynamic libraries doesn't match.Addresses related to Application it matches. Any inputs or pointers would be great help. That's because libraries are loaded at different locations on different machines and you still are not taking this into account with your crash report architecture, where you send the data back to your server for symbolication. Unless you save off specific task address space mapping locations for each, as part of the information you send back to your server, then your server will have different base addresses for the libraries and frameworks and therefore be unable to decode them properly. PS: It works for your main app only because you have not enabled ASLR for your app; if it is a network client or server, your app is likely vulnerable to buffer overflow and other attacks based on the fixed location of your code. Such an attack would work cookie-cutter fashion everywhere your software is deployed. By varying the address where it's loaded, you make it so an attack that works against one machine won't (statistically speaking) work against others. Admittedly, varying the location where system libraries are loaded is only protection if your app doesn't always load a vtable pointing at framework entry points into the same location on every machine, so that the same call sites exist everywhere (it kind of defeats the purpose if you do that in fact).

1 0

Re: Using mmap to extend an existing mapping?
by Terry Lambert 07 Oct '09

07 Oct '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Oct 7, 2009, at 10:32 AM, Jens Alfke <jens(a)mooseyard.com> wrote: I'm calling mmap to create a new mapping of a file. Later on I append data to the file (just using regular write calls) and then I want to map in the new data. If possible I want to grow the existing mapping in place, to avoid invalidating all the existing pointers. Based on the mmap(2) man page, it sounded as though I could do this by passing in the base address of the old mapping and the MAP_FIXED flag; and it would replace the mapping with a larger one if possible, or return MAP_FAILED if there wasn't room. ::mmap(_start, newLength, PROT_READ, MAP_SHARED | MAP_FIXED, _fd, _position); Instead, what I see happening (by running vmmap(1) before and after the call) is that the mmap call blows away any other following VM regions to make room for my request. Since these regions are often things like stacks, object code or malloc heaps, this is immediately fatal. :-( Is there any way to get the behavior I want, where mmap will extend the mapping but will fail rather than deleting any other mappings? Generally no. These are required semantics according to both historical behaviour and the standards, both of which assume the programmer knows better than the API. Typically you are expected to keep your mapped regions away from stack and heap allocations, which means either limiting yourself to a fraction of the address space and using a generally unused region, or by establishing your mapping followed immediately by an adjacent mapping of zero-fill pages that you can later choose to map over top of (and generally it's a good idea to map these pages initially as no read or write to have them serve a dual function as guard pages). There are Mach APIs that would let you check for collisions, but I'm not sure that the check and allocation if not taken can be made atomic for a multithreaded application. You could potentially request an mmap() flag/option to enforce this in some later Mac OS X major release by filing an enhancement request bug report, but if there is no way to do it in the underlying Mach APIs... This email sent to site_archiver(a)lists.apple.com

1 0

Re: How to work on case-sensitive projects?
by Terry Lambert 07 Oct '09

07 Oct '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Oct 7, 2009, at 10:09 AM, Allen Curtis <ajcurtis84(a)gmail.com> wrote: On 6 Oct 2009, at 20:08, Jamison Hope wrote: Use a case sensitive file system. Have a disk partition formatted as UFS, or a UFS-formatted disk image and do your svn checkout there. I don't think Disk Utility can create a UFS disk image, but hdiutil can. http://codesnippets.joyent.com/posts/show/2084 I'd strongly recommend against using UFS, actually. Case-sensitive HFS+ is a much better option. Well, I am still a little confused about which is case-sensitive vs. case-preserving. In my case, it must be case-sensitive. There are two types of file system lookup: directory iteration, and lookup of individual files. For directory iteration, you go to the directory and ask it to tell you its contents; for individual file lookup, you ask it to open a specific name. A file system can be case insensitive on storage, meaning it stores in the directory monocase names, or it can be case preserving on storage, meaning it stores in the directory names as they were given to it by the software creating the name. On iteration, a file system can return case insensitive names in monocase - MS-DOS does this; such names may be returned as all uppercase because they were stored that way, all lowercase, or, on some systems, with implied transforms, such as lowercase with the first letter upper case, the file extension all upper case, the file extension hidden, etc.. Or it can give you back exactly what you gave it. On lookup of individual names, such as in an open(2), chdir(2), chdir (3), etc., the FS can either be case sensitive on lookup, meaning an open will only match the exact name stored (or the exact name, after transformation), or it can be case insensitive on lookup, meaning it ignores the case for the purposes of finding what it considers an exact match. HFS+ by default is case sensitive on storage, case insensitive on lookup. Since a create operation involves a lookup, this generally means that if you expand an archive that contains "foo.c" and "FOO.c" and "FOO.C" on such a system, whichever is first in the archive determines the case of the stored name, and whichever is last in the archive determines the contents. You can turn on case sensitivity on lookup as well to avoid this, but it requires a rebuild of catalog indices to make the lookups case sensitive, since a balanced btree implies ordering based on intended sensitivity on lookups. BTW: This will be a problem for any Linux kernel developer wishing to use their Mac for development. It's unfortunate that your source code is structured in such a way as to assume so much about FS capabilities and semantics. Such code tends to reduce its portability to new environments, particulary cross- compilation environments, by making such assumptions. It makes it very hard to bootstrap such code to new platforms using cross-compilation with native tools; one could hope the code would be reorganized to be more platform independent in the future. In any case, you can make an HFS+ disk image with the necessary properties and mount it as an FS, or use the server install option to create the root volume case sensitive by default, or create a secondary volume on another disk partition or external drive, or NFS mount the source code directly from another system that exports it case sensitively, or any of a lot of other options, such as reorganizing your source code, etc.. All of these would work around your source organizations assumptions about its compilation environment. This email sent to site_archiver(a)lists.apple.com

1 0