August 2010 - Darwin-kernel

Re: inetdCompatibility
by Dreamcat4 19 Aug '10

19 Aug '10

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:mime-version:received:in-reply-to :references:from:date:message-id:subject:to:content-type; bh=AedLEGNZrrUkaO9zjIfHwIEORCC4efRzwHZJYqY8pT8=; b=tN1a/WjsUGif/NKNX1eSXYd4OFyuhzix8mPwCQjch9P12PMeO/ShDq5FZHYNu83o7n ZZnWHA2HL8Zd+SFL8ECPLHlcOqCcRamuCHc4+J2Y/+T/ES+3VlNBU/Lb2aWAV6y7k5pO cTS9U1xrFFXpbCfwJz35sfQ7aqhmTM5e89PdY= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; b=fr044sXCF/cz4B6l8fJKMRygeciQQgeZlfl53rxZJ4YsrfNeLICVG+t/IRWDsGA1uM obVUaJbUj2D+CVt8rtlGLYUPm5vl+Va6pig+XwPALxX/VOInF2dTNc0ujkqeXAC+qnG6 Y5TDnB5spDAqnuEtKsDDNOylgs7ABo0nP/QuQ= On Thu, Aug 19, 2010 at 11:48 AM, Dreamcat4 <dreamcat4(a)gmail.com> wrote: > I seem to be confused about what this feature actually does. > SSH works because it has a special inetd compatibility flag that will > accept stream based socket connections from stdin (as per telnet). > > Wheras Nginx is a more streamlined and optimized kind of webserver, > which would be too slow for operating in that inetd kind of way. Nginx > just wants to be launched and be kept running after the first > incoming connection. > > However once launchd loads a job with its Sockets listeners > definition, it will remain bound to that incoming TCP port on behalf > of that launchd job and not let it go. > > Im just wondering if its possible to get around all this with a > starter script that can trigger nginx to load (on the first incoming > connection). The starter script would also have to immediately unload > itself from launchd. So as to get launchd to free up the incoming TCP > port(s) for nginx. So the very first http connection might have > to be dropped, however thats not really such a big problem. > > Nginx's launchd plist can be configured to start with KeepAlive => > OtherJobEnabled => false. So to say "start up when the triggering job > unloads itself". > > The triggering job would have to stay unloaded for the remainder of > the previous OS session. But can a launchd job be configured to load > itself again during the next boot? > > It seems the only way to get the above scenario to work. Short of > reopening the launchd source code for new functionality / features. Well, that was easy enough http://gist.github.com/537897 When I do this, launchd does unload the starter job as requested. However it also report the following error to console: Aug 19 13:39:44 janitors-Mac-mini com.apple.launchd.peruser.502[852] (unload): Bug: launchd_core_logic.c:8873 (24108):0 Another thing is that when loading a User-Level job (from ~/Library/LaunchAgents), and specifying: <key>SockServiceName</key> <string>http-alt</string> Launchd reports another error: janitors-Mac-mini:~ id$ launchctl load -w ~/Library/LaunchAgents/unload.plist bind(): Permission denied However "http-alt" is actually port 8080, well above 1024. That doesnt seem right b/c the launchd.plist documentation refers to "Daemon/Agent writers" in the Sockets description. So Launch Agents should be supported. > On Wed, Aug 18, 2010 at 2:59 PM, Dreamcat4 <dreamcat4(a)gmail.com> wrote: >> Hi, >> >> 1. >> There is this patch on the Apple Support forums. Are exporting those >> environment variables the right way to pass the remote ip information >> to the spawned process? >> >> http://discussions.apple.com/thread.jspa?messageID=9360848&tstart=0 >> >> 2. >> (unrelated to the above patch) >> Im having trouble with getting the inetdCompatibility feature to work. >> >> The popular example >> /System/Library/LaunchDaemons/ssh.plist >> >> Which was followed for nginx pretty much verbatem: >> >> http://gist.github.com/534777 >> >> >> Regardless, Launchd isn't starting nginx when a connection is made to >> http://localhost:8080. The 'http-alt' service is specified in >> "/etc/services" to point to 8080. So as far as I can see, this should >> be no different to Launchd than for triggering a start of sshd. >> >> The behaviour is confounding as Apple sshd will work just fine. >> Flicking inetdCompatibility to Wait => true has no effect either. >> Nothing. >> >> >> >> dreamcat4 >> dreamcat4(a)gmail.com >> > _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Synchronous communication for Antivirus application.
by Peter Lovell 19 Aug '10

19 Aug '10

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Aug 18, 2010, at 1:48 AM, Deena Awate wrote: > I am working on an antivirus project for MAC. We have taken vnode level file system activity monitor code from MAC OS X internals book as our guide. What we are trying to achieve is monitoring on the vnode scope. When a file operation takes place at the vnode level the kernel will pass on the filename to usermode application for scanning. Meanwhile the kernel needs to wait for the scanning to complete and look at the return value. The user mode app will wake up the kernel once scanning is done and pass the scan result down to the kernel. Based on the scan result the kernel then either allows or denies the request. So basically we want to achieve synchronous communication between the kernel extension and the user mode application. We are newbies on MAC. We read documentation but is confusing as to how we can create event in user mode and pass it down to kernel on which the kernel will wait. If you could point us to a sample piece of code or clear documentation on how we can achieve our objective it will be great. Which are the synchronisation objects to be used in user mode and which ones to use in user mode. Is this the right design ? I have gone through posts on the forum but did not get how to achieve our objective. > Thanks, > Deena The code is old now but you will find some useful info in the work done for SEdarwin. It's available at www.sedarwin.org Cheers.....Peter _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Synchronous communication for Antivirus application.
by Michael Smith 19 Aug '10

19 Aug '10

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Synchronous communication for Antivirus application.
by Terry Lambert 18 Aug '10

18 Aug '10

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Aug 17, 2010, at 10:48 PM, Deena Awate wrote: > I am working on an antivirus project for MAC. We have taken vnode level file system activity monitor code from MAC OS X internals book as our guide. What we are trying to achieve is monitoring on the vnode scope. When a file operation takes place at the vnode level the kernel will pass on the filename to usermode application for scanning. Meanwhile the kernel needs to wait for the scanning to complete and look at the return value. The user mode app will wake up the kernel once scanning is done and pass the scan result down to the kernel. Based on the scan result the kernel then either allows or denies the request. So basically we want to achieve synchronous communication between the kernel extension and the user mode application. We are newbies on MAC. We read documentation but is confusing as to how we can create event in user mode and pass it down to kernel on which the kernel will wait. If you could point us to a sample piece of code or clear documentation on how we can achieve our objective it will be great. Which are the synchronisation objects to be used in user mode and which ones to use in user mode. Is this the right design ? I have gone through posts on the forum but did not get how to achieve our objective. > Thanks, > Deena In general: Do Not Make The Kernel Wait (tm). My first suggestion is that you should push the smallest amount of information necessary to answer the kernels question(s) down from user space to kernel space through one of the allowed mechanisms (and store the information in the kernel, in pageable memory). And then not depend on user space, unless it wants to push you more data, e.g. a virus pattern file update happens. If you can't do it in a small amount of space, then your next option is to load it into pageable memory in the user space process so that the user space process can answer any kernel queries *immediately*, so as not to make the kernel wait. If you do this, you should make your own process immune from triggering up-calls from the kernel to user space in order to avoid deadly embrace deadlocks (e.g. as a result of paging). You should also consider making 'root' owned processes immune as well. You should minimize the interfaces your user space process uses, in order to avoid using a framework which under the covers talks to another process which triggers an up-call. The general recommendation in this case is to limit yourself to POSIX file I/O, and do no networking (networking requires name resolution which talks to mDNSResponder which runs as non-root which can trigger up-calls). You never know what a framework is going to do to try and service your request -- it could be a huge amount of work involving a dozen different processes. If you do up-calls at all, you need to provide a mechanism to time out the kernel's requests. This mechanism should be implemented in the kernel, and if the user space process doesn't respond in a reasonable period of time (in general: 30 seconds or less is reasonable, since that's 10 seconds of delay and 20 seconds of you causing an application to SPOD), you should revoke its authority to answer your questions. SPOD any longer than that, and the human hooked up to the keyboard is going to start force-quitting things. The user space process should get an error when it tries next time, which it should interpret as you (in the kernel) telling it to kill itself and come back when it can answer your questions in a reasonable period of time. Obviously, this is not a synchronous interface -- and it's not intended to be. If the kernel becomes unresponsive, your entire system locks up, so you have to avoid that happening because of a KEXT you write. If you want to see an example of a time-out up-call interface that you should probably *NOT* copy faithfully, see the functions identitysvc() and kauth_resolver_submit() in the xnu source file bsd/kern/kern_credential.c (you shouldn't try to copy it because you shouldn't make the kernel wait, and because you need to audit all the code paths and be sure you don't rely on other processes, and because you can't implement a system call like identitysvc()). But it will give you the overall picture. The general methods to use for user/kernel communication are (in my personal highest-to-lowest preference order): o Don't need to do it in the first place o IOUserClient o Control socket o BSD device file o sysctl Documentation on these and others is available from <http://developer.apple.com/>, along with source code examples. -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Synchronous communication for Antivirus application.
by Deena Awate 18 Aug '10

18 Aug '10

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1282110517; bh=Uc+NkHEIMgkxrwQS34avSurrPht3X7Hvm3dbPcicLiA=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=RmL6s5omi2gZZ4lHOA4hLWoS0MEAIGI/Ze8uRoaZ/R5G/OMD+g6O9/kGGoV2lY1BhVhxU94XRLN2yVtREIPKQ7N9JLy1pbQ9MHIFA0TBqd+r8FQ/3Z3sE2cCHlKHU6LVu9LmeBHMrO/YQ742qXF09SQCSf9JgLv7FCENBpw2jiQ= Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=0QK5O8/ueX1gcJXLLWbjDPKVwNLxqCUr8XbGbAhs1GfNY0AtvhOxbs0UhzPOOOO2WAvf/bItYhQ7JeAVMV0QG0C1VTZOAWvRwqWbTNwtmNnWrSpzsu4LWQq3PpdFZmpqQcDmBn3mlGNd7gdixGC/b5uTWTUdFhfkVyAYSgo1Ks8=; _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: close() hangs when closing a kqueue in 10.5 only
by Terry Lambert 17 Aug '10

17 Aug '10

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Aug 16, 2010, at 12:56 PM, Justin C. Walker wrote: > On Aug 16, 2010, at 11:41 , Jerry Krinock wrote: >> A Cocoa app I've written uses a kqueue to watch its document file. It works fine in Mac OS 10.6, but when run in Mac OS 10.5.8, when I close the kqueue like this: >> >> close (kqueueFileDescriptor) ; >> >> The function close() hangs forever. The argument kqueueFileDescriptor is the integer I got previously from: >> >> NSInteger kqueueFileDescriptor = kqueue() ; >> >> Does anyone know what might be going on here? Documentation says that read() can block until data is ready, but there is no mention of this in the documentation for close(). >> >> I suppose I could just leave the kqueue open if running in 10.5, because a typical application run only creates one of them. >> >> Is close() open-source? I can't find it in macosforge.org or opensource.apple.com. > > As mentioned, this system call is in the open-source part of Mac OS X. The tricky part is that very low-level calls like this may not have an implementation as you might expect to see it. I believe that close() is implemented by table-manipulation tom-foolery, during the build of the kernel. > > The close call in libsystem probably invokes a generic "system call" trap directly, which gets "vectored" to a dispatch table in the kernel. I think the actual code is in > bsd/kern/kern_descrip.c > called closef_locked(). The logic here is kind of twisty because the system has to account for a variety of different file descriptor types and behaviors, and in particular, detect "last close" issues. The implementation is in the struct fileops in the xnu file bsd/kern/kern_event.c. <http://fxr.watson.org/fxr/source/bsd/kern/kern_event.c?v=xnu-1228> Specifically, it's going to be blocking in kqueue_dealloc(), probably because you either have a lock on the proc elsewhere (see the stackshot output to find out who you're blocked against), or you're adding events faster than they can be discarded, or you have another thread camping out on the fd in kevent(). If it's events being added faster than they can be removed as part of the close, you want to disable the events you have outstanding before doing the close if that's the cause. If it's a thread, you should pthread_kill() it to interrupt the kevent call (meaning you need to have it's thread ID saved off when it starts or as a result of calling pthread_self() and stashing the result in a volatile global. Otherwise, it's someone in the way of the lock, and you need to find out who has the lock and why. You can also add the kmem boot argument if it's not old enough to have a /dev/kmem, and get read-only gdb access to the currently running kernel, without having to set up two machine debugging. You're probably going to need this to dump the lock structure contents for whoever is currently holding the lock in your way. The lock will have a thread id in the structure which will tell you who's holding it. Typically, if your application is mutithreaded, you could be doing something on another thread which holds the lock. You could also be racing two attempts to close the same fd, which could (possible) deadlock with a deadly embrace as it's trying to reacquire the proc_fdlock(). You might also have a long-standing sysctl going on. This is typically something people who want to add system calls via KEXT use, but you actually don't want to block in the sysctl, since it still holds the global funnel in older versions of the OS. The rule with sysctl is you get in, you do your work, and you get out, quickly as possible. -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: close() hangs when closing a kqueue in 10.5 only
by Justin C. Walker 16 Aug '10

16 Aug '10

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Aug 16, 2010, at 11:41 , Jerry Krinock wrote: close (kqueueFileDescriptor) ; NSInteger kqueueFileDescriptor = kqueue() ; HTH Justin -- Justin C. Walker, Curmudgeon at Large Institute for the Absorption of Federal Funds ----------- Like the ski resort full of girls hunting for husbands and husbands hunting for girls, the situation is not as symmetrical as it might seem. - Alan MacKay -- _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… A Cocoa app I've written uses a kqueue to watch its document file. It works fine in Mac OS 10.6, but when run in Mac OS 10.5.8, when I close the kqueue like this: The function close() hangs forever. The argument kqueueFileDescriptor is the integer I got previously from: Does anyone know what might be going on here? Documentation says that read() can block until data is ready, but there is no mention of this in the documentation for close(). I suppose I could just leave the kqueue open if running in 10.5, because a typical application run only creates one of them. Is close() open-source? I can't find it in macosforge.org or opensource.apple.com. As mentioned, this system call is in the open-source part of Mac OS X. The tricky part is that very low-level calls like this may not have an implementation as you might expect to see it. I believe that close() is implemented by table-manipulation tom-foolery, during the build of the kernel. The close call in libsystem probably invokes a generic "system call" trap directly, which gets "vectored" to a dispatch table in the kernel. I think the actual code is in bsd/kern/kern_descrip.c called closef_locked(). The logic here is kind of twisty because the system has to account for a variety of different file descriptor types and behaviors, and in particular, detect "last close" issues. This email sent to site_archiver(a)lists.apple.com

1 0

Re: close() hangs when closing a kqueue in 10.5 only
by Ken Hornstein 16 Aug '10

16 Aug '10

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com >You can use <x-man-page://1/stackshot> with the "-i" argument to get >backtraces of the kernel without having to mess around with the kernel >debugger. Huh, didn't know about that one. Very cool! I agree, that would be MUCH easier! --Ken _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: close() hangs when closing a kqueue in 10.5 only
by Quinn "The Eskimo!" 16 Aug '10

16 Aug '10

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On 16 Aug 2010, at 20:31, Ken Hornstein wrote: > If it was up to me > ... well, I might consider attaching a kernel debugger to the system in > question and seeing what the process was hanging on, but I can understand > why you might not want to go to that much trouble. You can use <x-man-page://1/stackshot> with the "-i" argument to get backtraces of the kernel without having to mess around with the kernel debugger. S+E -- Quinn "The Eskimo!" <http://www.apple.com/developer/> Apple Developer Relations, Developer Technical Support, Core OS/Hardware _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: close() hangs when closing a kqueue in 10.5 only
by Ken Hornstein 16 Aug '10

16 Aug '10

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com >Thank you, Graham. But I've been clicking and clacking for 10 minutes >but still can't find the source code for close() in libsystem or xnu. >If anyone could take my hand for the first couple steps in finding this >I'd appreciate it. At least in 10.6.x, you want ${XNU}/bsd/kern/kern_descrip.c. Although I think you might find that there are enough levels involved that tracking down the core issue might be difficult. If it was up to me ... well, I might consider attaching a kernel debugger to the system in question and seeing what the process was hanging on, but I can understand why you might not want to go to that much trouble. --Ken _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0