January 2015 - Darwin-kernel

Re: Build 10.4/10.5 kext on 10.10
by Regis Duchesne 12 Jan '15

12 Jan '15

Benjamin, that is exactly what I intended to do. And you even included the wrapper script. Awesome! Thanks a lot, that will save me some time! -- hpreg > On Jan 10, 2015, at 1:06 PM, Benjamin Fleischer <fleiben(a)gmail.com> wrote: > > Hi All, > > I ran into the same issue when trying to build osxfuse on OS X 10.10 and created a small library that provides a simple map_fd implementation based on mmap and can be injected into ld_classic. So far I have not run into any issues using this approach. > > The source is available on GitHub https://github.com/bfleischer/map_fd. The library can be downloaded from https://github.com/bfleischer/map_fd/releases. > > Regards, > Benjamin > > Am 07.01.2015 um 17:17 schrieb Christopher Chandler <chrisjechandler(a)me.com>: > >> Hi Regis, >> >> You are welcome. >> >> One final thought: if you go the DYLD_LIBRARY_ROUTE, you’ll have to find a way to make the existing binaries look in your new .dylib rather than in libSystem.dylib. The dyld(1) man page https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManP… suggests that it might be possible using DYLD_INSERT_LIBRARIES but I haven’t investigated this. >> >> Good luck. >> >> Regards, >> Chris >> >> >> >>> On 7 Jan 2015, at 11:10 am, Regis Duchesne <hpreg(a)vmware.com> wrote: >>> >>> Thanks for the additional info, Christopher! >>> -- >>> Regis "HPReg" Duchesne >>> Technical Lead for Fusion & Mac OS Guest >>> VMware, Inc. >>> >>> >>>> On Jan 6, 2015, at 7:47 PM, Christopher Chandler <chrisjechandler(a)me.com> wrote: >>>> >>>> Hi Regis, >>>> >>>> Replying to your comments: >>>> >>>> 1) SDKs and compilers: >>>> Yes. Basically I try to use the correct compiler version(s) with the correct SDKs. The downside of this is that of course you lose bug fixes that have been made in newer compiler versions, and if you’re trying to use newer language features (etc.) you need to partition your code into libraries etc. The easiest way I’ve found of doing this is to use an Xcode config file, which allows you to specify the particular SDK, compiler etc. Then you can swap in and out tools depending on the build target, and over-ride the new defaults in the .config file (if necessary) in specific targets >>>> >>>> 2) DYLD_LIBRARY_PATH: >>>> That’s definitely neater than hack-rebuilding the code. To some extent, I was investigating how hard it would be to do it, and as I said, I needed something quick & dirty quickly. As I’m sure you’re aware, sometimes making DYLD_LIBRARY_PATH work can be a bit awkward especially as some applications watch this for security threats etc. >>>> >>>> 3) Source code changes. I grep-ed the source and found quite a few instances of map_fd() (and mmap() strangely - it’s as though Apple started using mmap() in the newer features but never went back and fixed the older code). In order to link ld.NEW for my purposes, I only needed to fix the ones in ld.c and pass1.c. In all cases, map_fd() is used to read in an existing file opened read-only into memory in one go. No changes are written back to the file. The memory needs to be writable, presumably so that the linker can update fields etc. >>>> >>>> 4) Replace map_fd() with mmap(). >>>> Here’s the map_fd() prototype, according to http://www.cs.cmu.edu/afs/cs/project/mach/public/doc/published/mapfiles87.ps >>>> map_fd(fd, offset, addr, find_space, numbytes) >>>> int fd; >>>> >>>> vm_offset_t offset; >>>> vm_offset_t *addr; >>>> boolean_t find_space; >>>> vm_size_t numbytes; >>>> >>>> For example: /Darwin/ADC/Common/Xcode-3_1_4/cctools-698.1/ld/ld.c, line 1020 (or so - ) >>>> >>>> if((fd = open(argv[i+3], O_RDONLY, 0)) == -1) >>>> system_fatal("Can't open: %s for %s %s %s", >>>> argv[i+3], argv[i], argv[i+1], argv[i+2]); >>>> if(fstat(fd, &stat_buf) == -1) >>>> system_fatal("Can't stat file: %s for %s %s %s", >>>> argv[i+3], argv[i], argv[i+1], argv[i+2]); >>>> /* >>>> * For some reason mapping files with zero size fails >>>> * so it has to be handled specially. >>>> */ >>>> if(stat_buf.st_size != 0){ >>>> #if 0 /* CJEC, 24-Dec-14: map_fd() has been deprecated in OS X 10.10.x */ >>>> if((r = map_fd((int)fd, (vm_offset_t)0, >>>> (vm_offset_t *)&(sect_spec->file_addr), >>>> (boolean_t)TRUE, (vm_size_t)stat_buf.st_size) >>>> ) != KERN_SUCCESS) >>>> mach_fatal(r, "can't map file: %s for %s %s %s", >>>> argv[i+3], argv[i], argv[i+1], argv[i+2]); >>>> #else >>>> sect_spec -> file_addr = mmap (NULL, stat_buf.st_size, PROT_READ | PROT_WRITE, MAP_FILE | MAP_PRIVATE, fd, 0); >>>> if (sect_spec -> file_addr == NULL) >>>> system_fatal ("%s %s{%u}: Can't mmap file: %s for %s %s %s", __PRETTY_FUNCTION__, __FILE__, __LINE__, argv [i+3], argv [i], argv [i+1], argv [i+2]); >>>> #endif >>>> } >>>> else{ >>>> sect_spec->file_addr = NULL; >>>> } >>>> close(fd); >>>> >>>> For your solution, the good news is that map_fd() seems to be always used in the same way. So you could implement a specific solution that satisfies this particular set of map_fd() options (addr == NULL, and find_space == true), and throw some kind of fatal error if it map_fd() is called in any way differently. >>>> >>>> Good luck - I’m sure it will work. >>>> >>>> Regards, >>>> Chris >>>> >>>> >>>>> On 5 Jan 2015, at 12:04 pm, Regis Duchesne <hpreg(a)vmware.com> wrote: >>>>> >>>>> Christopher, >>>>> >>>>>> For those of you who are still trying to support older OS X releases, you might like to know that I’ve investigated this further. The good news is that Xcode 6 still works with the older plugin formats, build tool configuration files etc. so follow the same approach with Xcode 6 as I indicated with Xcode 5 previously http://lists.apple.com/archives/darwin-kernel/2014/May/msg00017.html My environment now incudes Xcode 6.1.1 on top of 5.0.3, 4.6.2, and 3.2.6, and I can build on all SDKs and platforms from OS X 10.4 >>>>> >>>>> Since Apple does not ship earlier SDKs with later Xcode versions, I'm afraid this technique may not work in the long run: newer compilers may require you to edit older SDKs in order to compile. >>>>> >>>>> Instead I simply install multiple versions of Xcode on the same box. But that only works well because I don't use Xcode projects, I use the compilers/linkers directly within make files (SCons files actually). >>>>> >>>>>> There is one major new gotcha. It seems that Apple has removed the deprecated map_fd() system call from OS X 10.10.1 (and probably 10.10.0 but I have not tested it) and so /Xcode-3_2_6/usr/bin/ld_classic, the 32 bit old linker, crashes when you attempt to use it to link a 10.4/10.5 compatible kernel extension. I managed to work around this by downloading the Xcode 3.1.4 cc-tools source from http://www.opensource.apple.com/release/developer-tools-314/ and hacking the environment and makefiles enough to be able to build ld.NEW under OS X 10.6.8, 32 bit mode and Xcode 3.2.6. >>>>> >>>>> I have been thinking about doing this a different way: create a new map_fd.dylib that only provides the map_fd() symbol, implemented in terms of a mmap() call. Then invoke the Apple binary by setting DYLD_LIBRARY_PATH and/or DYLD_INSERT_LIBRARIES to make use of that new map_fd.dylib. >>>>> >>>>> The advantage is that there is nothing to recompile from open source: you keep using the official Apple ld_classic binary. And you can use the same map_fd.dylib for other programs that need the map_fd symbol. >>>>> >>>>>> (Virtual machines are a wonderful thing!) >>>>> >>>>> Yes, and it is precisely to keep virtual machines (really VMware Tools) running that I need to be able to build 32-bit kexts on a 10.10 host! >>>>> >>>>>> Once I was able to build the ld_build target in cctools-698.1/ld/Makefile, I modified ld.c and pass1.c replacing the map_fd() calls with mmap (…PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_FILE..,). This successfully links a 10.4 compatible kernel extension under OS X 10.10 using Xcode 6/5/4/3 together. >>>>> >>>>> To implement my own map_fd() function, I would be interested in the exact modification you did (that will save me a bit of time). I.e. what was the exact map_fd() call code you replaced, and with what exact mmap() call did you replace it. >>>>> >>>>> Thanks, >>>>> -- >>>>> Regis "HPReg" Duchesne >>>>> Technical Lead for Fusion & Mac OS Guest >>>>> VMware, Inc. >>>> >>>> >>>> >>>> >>> >> >> >> >> >> >> >> >> >> _______________________________________________ >> Do not post admin requests to the list. They will be ignored. >> Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) >> Help/Unsubscribe/Update your Subscription: >> https://lists.apple.com/mailman/options/darwin-kernel/fleiben%40gmail.com >> >> This email sent to fleiben(a)gmail.com > _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Build 10.4/10.5 kext on 10.10
by Benjamin Fleischer 10 Jan '15

10 Jan '15

_______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Build 10.4/10.5 kext on 10.10
by Christopher Chandler 07 Jan '15

07 Jan '15

_______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Build 10.4/10.5 kext on 10.10
by Christopher Chandler 06 Jan '15

06 Jan '15

map_fd(fd, offset, addr, find_space, numbytes) int fd; vm_offset_t offset; vm_offset_t *addr; boolean_t find_space; vm_size_t numbytes; _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Build 10.4/10.5 kext on 10.10
by Regis Duchesne 05 Jan '15

05 Jan '15

Christopher, > For those of you who are still trying to support older OS X releases, you might like to know that I’ve investigated this further. The good news is that Xcode 6 still works with the older plugin formats, build tool configuration files etc. so follow the same approach with Xcode 6 as I indicated with Xcode 5 previously http://lists.apple.com/archives/darwin-kernel/2014/May/msg00017.html My environment now incudes Xcode 6.1.1 on top of 5.0.3, 4.6.2, and 3.2.6, and I can build on all SDKs and platforms from OS X 10.4 Since Apple does not ship earlier SDKs with later Xcode versions, I'm afraid this technique may not work in the long run: newer compilers may require you to edit older SDKs in order to compile. Instead I simply install multiple versions of Xcode on the same box. But that only works well because I don't use Xcode projects, I use the compilers/linkers directly within make files (SCons files actually). > There is one major new gotcha. It seems that Apple has removed the deprecated map_fd() system call from OS X 10.10.1 (and probably 10.10.0 but I have not tested it) and so /Xcode-3_2_6/usr/bin/ld_classic, the 32 bit old linker, crashes when you attempt to use it to link a 10.4/10.5 compatible kernel extension. I managed to work around this by downloading the Xcode 3.1.4 cc-tools source from http://www.opensource.apple.com/release/developer-tools-314/ and hacking the environment and makefiles enough to be able to build ld.NEW under OS X 10.6.8, 32 bit mode and Xcode 3.2.6. I have been thinking about doing this a different way: create a new map_fd.dylib that only provides the map_fd() symbol, implemented in terms of a mmap() call. Then invoke the Apple binary by setting DYLD_LIBRARY_PATH and/or DYLD_INSERT_LIBRARIES to make use of that new map_fd.dylib. The advantage is that there is nothing to recompile from open source: you keep using the official Apple ld_classic binary. And you can use the same map_fd.dylib for other programs that need the map_fd symbol. > (Virtual machines are a wonderful thing!) Yes, and it is precisely to keep virtual machines (really VMware Tools) running that I need to be able to build 32-bit kexts on a 10.10 host! > Once I was able to build the ld_build target in cctools-698.1/ld/Makefile, I modified ld.c and pass1.c replacing the map_fd() calls with mmap (…PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_FILE..,). This successfully links a 10.4 compatible kernel extension under OS X 10.10 using Xcode 6/5/4/3 together. To implement my own map_fd() function, I would be interested in the exact modification you did (that will save me a bit of time). I.e. what was the exact map_fd() call code you replaced, and with what exact mmap() call did you replace it. Thanks, -- Regis "HPReg" Duchesne Technical Lead for Fusion & Mac OS Guest VMware, Inc. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists… This email sent to site_archiver(a)lists.apple.com

1 0