January 2006 - Darwin-dev - lists.apple.com

Re: intercepting system calls?
by Mike Smith 23 Jan '06

23 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Jan 22, 2006, at 12:02 PM, darwin-dev-request(a)lists.apple.com wrote: I want to be able to capture the I/O calls of an application and then ship them back over the network to be handled by another machine. = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… There are plenty of examples of this in the system already; they're called "network filesystems". One of them is even called "the Network File System". It also seems like Apple could keep the system call interface stable between versions - the great thing about system calls is if you need to change the meaning of one, you can just create another system call with a different name. You can leave the old one stable, and just change the framework to call the new one - if someone has found a way to call the actual system call, the can keep using it. This is a naive assumption. It is much easier to version at the shared library interface; locking down the system call layer introduces a lot of problems, especially if you guarantee the interface as ABI. This email sent to site_archiver(a)lists.apple.com

1 0

Re: intercepting system calls?
by Erik Paulson 22 Jan '06

22 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com User-agent: Mutt/1.4.2.1i On Sat, Jan 21, 2006 at 01:10:13PM -0800, Kevin Van Vechten wrote: > If you're not trying to corral a hostile environment, a simple way to > intercept syscalls is to override them at the Libsystem layer. > Define your own functions in a dylib, and then se the > DYLD_INSERT_LIBRARIES and DYLD_FORCE_FLAT_NAMESPACE environment > variables. > What I'm trying to do is create something like this: http://www.cse.nd.edu/~ccl/software/parrot/ I want to be able to capture the I/O calls of an application and then ship them back over the network to be handled by another machine. I work on a distributed computing system (similiar to XGrid) and we want to able to run jobs on machines that don't have access to the same shared filesystem, so we trap system calls, send them back to the "submitting machine" and execute them there. The remote job, as far as it can tell, has access to all of the files as it would on the submitting machine (yes, obviously the idea is that the compute time to system call ratio is high, because remote system calls are many many times slower, and yes we take care of security) The problem with replacing at the library level is that the libraries may have private interfaces - finding all of the call points that I need to interpose on can be hard, and if I miss just one the game is over. We've been doing this for years on Linux, and for the longest time we did interpose at the library level - and every time a new Linux distro came out and it used a new libc, we'd have to figure out what new interfaces it was using - besides open, we needed to catch __open, __open2, __open3OnlyUsedInOnePlace, and __somethingCalledFooButReallyOpen Because those calls are all at user level, there was no way for us to find out if one of them got called by some code path we didn't catch. The system call interface was much narrower, so there are far fewer things we need to trap. Even better, because it's a trap, we could capture ALL of them, and anything we didn't know what to do with we could detect at runtime. It's much slower, but we're dominated by the network latency anyway so we didn't care. We want to stay out of the other processes address space as much as possible, and we certainly want to avoid executing code in the other processes address space (I took a look at APE - as an application writer it made my skin crawl :) Similarly, I want to avoid trapping in with a KEXT - I want to convince other people to use my software at low-risk to themselves, and asking them to install a KEXT doesn't jive with that. It would also mean that I'd see every system call on the machine, which I don't want. The ideal system lets my tracing process capture the system calls of another process running at the same privilege level - that way, if I'm running as use 'batchsystem', the only processes I can capture are 'batchsystem' processes. It also seems like Apple could keep the system call interface stable between versions - the great thing about system calls is if you need to change the meaning of one, you can just create another system call with a different name. You can leave the old one stable, and just change the framework to call the new one - if someone has found a way to call the actual system call, the can keep using it. Thanks, -Erik _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Loading a shared library directly from inside of a ZIP file
by Amanda Walker 22 Jan '06

22 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Apple has some sample code at http://developer.apple.com/samplecode/MemoryBasedBundle/listing3.html that may be helpful. Amanda Walker _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Jan 22, 2006, at 3:05 AM, Peter O'Gorman wrote: I'm not sure how well NSCreateObjectFileImageFromMemory works, never tried it. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Loading a shared library directly from inside of a ZIP file
by Peter O'Gorman 22 Jan '06

22 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com User-agent: Mozilla Thunderbird 1.0.2 (Macintosh/20050317) -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 John Davidorff Pell wrote: | How would you decompress the library? Is it in the ZIP file | uncompressed? the dlopen() family of calls are *extremely* simple, so | "hacking" them would not be a bad idea since they are "hacks" to begin | with. They're just wrappers around native calls anyway. Check out the | dlcompat library (which Tiger includes). Tiger does not include dlcompat :-) dlopen code was much simplified and put into the new dyld. man NSObjectFileImage, look for NSCreateObjectFileImageFromMemory(), also look at man NSModule for NSLinkModule. I'm not sure how well NSCreateObjectFileImageFromMemory works, never tried it. Peter -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (Darwin) iQCVAwUBQ9M8ybiDAg3OZTLPAQIZrwQAjacSFzH6V4ggLmk5nT1oiU86DdDvpY7M JjBEEp2bMJ1GEt7j397tHdzXEuhIi2DgaxWGY2yxdG2/a99U0+BwpcXIf1JiuoNk D8kpsUeDl8oK8fLftXd20SetS2ZgwC6zb1tghQC71Fj2rpqeA8Hw8MktU3FVsqLz aO8j1hW0ZN0= =nB8z -----END PGP SIGNATURE----- _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Loading a shared library directly from inside of a ZIP file
by John Davidorff Pell 22 Jan '06

22 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com JP On 20 Jan 2006, at 07:45, Martin Knapp-Cordes wrote: Hi, Is there a 'public' interface (with a different name) that will replace the interface: void *dlopen(const char *filename, int flag); by say: void *dlopen_new(int fd, void *start, size_t length, int flag); This is motivated by the mmap interface: The idea is that you are given a triplet: (name,offset,length) where name is file descriptor for the ZIP file (int fd), and the offset (void *start) is the start of the shared library in the ZIP file, and length (size_t length) is the length of the shared library. Note that dlopen() under the hood does a mmap. If there isn't, is the only recourse, to 'hack' the source or are there some other ideas or published solutions. The solution of making a copy into the filesystem first and using the old interface is NOT acceptable. Sincerely, Martin Knapp-Cordes --------------------------------------------------------------- | Martin Knapp-Cordes martin(a)mathworks.com | | The MathWorks, Inc. http://www.mathworks.com | | 3 Apple Hill Drive Natick, MA 01760 | | Tel: (508) 647-7321 Fax: (508) 647-7015 | --------------------------------------------------------------- -- John Davidorff Pell johnpell(a)mac.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… How would you decompress the library? Is it in the ZIP file uncompressed? the dlopen() family of calls are *extremely* simple, so "hacking" them would not be a bad idea since they are "hacks" to begin with. They're just wrappers around native calls anyway. Check out the dlcompat library (which Tiger includes). void * mmap(void *start, size_t length, int prot , int flags, int fd, off_t offset); _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/jpell.lists% 40mac.com This email sent to jpell.lists(a)mac.com This email sent to site_archiver(a)lists.apple.com

1 0

Re: How to noify the framework when application closes
by John Davidorff Pell 22 Jan '06

22 Jan '06

site_archiver(a)lists.apple.com Delivered-To: Darwin-dev(a)lists.apple.com JP On 16 Jan 2006, at 01:58, rohit dhamija wrote: Hi All, Please suggest. -- Rohit Dhamija(M) 9818446545 _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Where are you allocating the memory? In the kernel? If its inside user space (which it would be if you just malloc() it in the framework, then it will be disposed of upon any termination (abnormal or otherwise). I have build an framework for accessing the usb bulk storage device using iokit.framework. Multiple applications can access the device using this framework. For each application accessing the device, the framework allocates some memory. Now, I need to free this memory when ever the application: a. Closes decently b. Abnornal termination So is there any way that I can notify the framework that the application has been closed ? Is giving notification to the framework through signals the right way ? If yes, then how ? _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/jpell.lists% 40mac.com This email sent to jpell.lists(a)mac.com -- "... was it a dream where you see yourself standing in sort-of Sun- God robes, on a pyramid, with a thousand naked women screaming and throwing little pickles at you? ... Why am I the only one who has that dream?" This email sent to site_archiver(a)lists.apple.com

1 0

Re: intercepting system calls?
by Kevin Van Vechten 21 Jan '06

21 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com See the dyld(1) man page for details on the environment variables. - Kevin On Jan 20, 2006, at 9:01 PM, Mike Smith wrote: PT_SYSCALL is, as you have probably noticed, not implemented for the Darwin ptrace. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… If you're not trying to corral a hostile environment, a simple way to intercept syscalls is to override them at the Libsystem layer. Define your own functions in a dylib, and then se the DYLD_INSERT_LIBRARIES and DYLD_FORCE_FLAT_NAMESPACE environment variables. Here is a sample dylib that overrides some common syscalls: <http://cvs.opendarwin.org/cgi-bin/cvsweb.cgi/projects/darwinbuild/ darwintrace/> On Jan 20, 2006, at 12:02 PM, darwin-dev-request(a)lists.apple.com wrote: I'd like to catch the system calls that an process is making, so I can examine the system call and optionally handle it myself. Under Linux and Solaris, I can use ptrace() with the PTRACE_SYSCALL command, which will let the process run until it enters or exits a system call. Is there an equivalent for Darwin? Using a KEXT to patch the system call table is really not an option... Remember that the binary compatibility layer guarantee made by Mac OS X is at the framework boundary, *not* at the syscall layer. Direct reference to system calls is avoided wherever possible in user space; they are in effect just a convenient way of implementing what the frameworks offer and can move around quite a bit from version to version. As such, if you want to interpose and affect the outcome of application calls to system services, you should look to the dyld interpositioning mechanisms. For the most part you only need to worry about libSystem; other frameworks call it to make system calls. You may, however, also need to consider intercepting Mach IPC depending on exactly what it is that you're trying to do. = Mike This email sent to site_archiver(a)lists.apple.com

1 0

Re: intercepting system calls?
by Mike Smith 21 Jan '06

21 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Jan 20, 2006, at 12:02 PM, darwin-dev-request(a)lists.apple.com wrote: PT_SYSCALL is, as you have probably noticed, not implemented for the Darwin ptrace. = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I'd like to catch the system calls that an process is making, so I can examine the system call and optionally handle it myself. Under Linux and Solaris, I can use ptrace() with the PTRACE_SYSCALL command, which will let the process run until it enters or exits a system call. Is there an equivalent for Darwin? Using a KEXT to patch the system call table is really not an option... Remember that the binary compatibility layer guarantee made by Mac OS X is at the framework boundary, *not* at the syscall layer. Direct reference to system calls is avoided wherever possible in user space; they are in effect just a convenient way of implementing what the frameworks offer and can move around quite a bit from version to version. As such, if you want to interpose and affect the outcome of application calls to system services, you should look to the dyld interpositioning mechanisms. For the most part you only need to worry about libSystem; other frameworks call it to make system calls. You may, however, also need to consider intercepting Mach IPC depending on exactly what it is that you're trying to do. This email sent to site_archiver(a)lists.apple.com

1 0

Re: IONotificationPortRef cleanup
by Steve Checkoway 20 Jan '06

20 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Jan 20, 2006, at 6:58 AM, Jim Magee wrote: On Jan 10, 2006, at 2:44 PM, Steve Checkoway wrote: - Steve _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com On Jan 10, 2006, at 2:35 PM, Chris Sarcone wrote: To be completely clear about this, these leaked ports are not reclaimed when the application exits? Mach will clean them up for you at task termination. However, whenever you don't need the resource, you should do the proper thing to release it. Thank you for the clarification. It seems to me that the example should be changed to simply _exit(2) since the ports will be cleaned up. The problem is that someone will copy the example into a library routine and replace the _exit() with a return. Then they wonder why they leak - since Apple provided the example and it should have managed its resources. That's a good point. Maybe the signal handler could be changed (and maybe catch SIGQUIT as well) to stop the run loop and then clean up after the run loop has terminated. But then, that might just confuse the issue. Actually, a comment just after the run loop would probably be best saying that in actual use the clean up code needs to be called. smime.p7s

1 0

Re: MoreBacktrace for i386?
by Lassi A.Tuura 20 Jan '06

20 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Hi, (Yes, I filed a radar feature request, oh, some years ago :-) Lassi -- When the situation is desperate, it is too late to be serious. Be playful. --Edward Abbey _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… It might be better at this point to use dwarf2 eh tables to get a backtrace than it is from x86's stack frame because if you don't have a frame pointer, stack frames are "messed" up. Ah but the interesting bit is getting through signal frames and other bits of code without supporting eh tables ;-) Otherwise the compiler-bundled _Unwind_* stuff would have been sufficient already on PPC. Real support in system library for backtrace() (like linux), or any similar call found on other operating systems would be very welcome. Doesn't have to be standard across platforms, as long as it understands and goes through signal frames and other odd bits of code. This email sent to site_archiver(a)lists.apple.com

1 0