April 2009 - Darwin-kernel - lists.apple.com

Re: testing the current bootstrap context?
by Bill Janssen 09 Apr '09

09 Apr '09

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Michael Smith <drivers(a)mu.org> wrote: > > On Apr 8, 2009, at 4:50 PM, Bill Janssen wrote: > > > The only way to do this in Leopard seems to be with launchd, which > > seems > > to do the right thing. The main problem with using launchd here is > > that > > users cannot manipulate their own server; you can either use a root > > LaunchDaemon which only root can unload, or a user LaunchDaemon > > which is > > again fatally bound to the login bootstrap context. > > It's usually straightforward for the daemon to vend the appropriate > self-manipulation functionality; frequently you want to gate this with > application-specific logic anyway. Well, the daemon does a lot of things, but what I'm concerned with is stop, start, and restart. "restart" is easy; I just have a LaunchDaemon that specifies RunOnLoad=true, UserName=foo, KeepAlive=true. So if the user kills it (with SIGTERM, say, which a non-root user can send to the daemon process, and the daemon can handle) it just gets restarted by launchd. Assuming I set the ThrottleInterval and ExitTimeOut properly, it should be fine. It's "stop" (and "start") that bugs me. I essentially need to do a "launchctl unload -w" as root to do this. Otherwise the KeepAlive kicks in and it's running again before you can say, "Aw, shucks". And I need to be able to do this with a non-admin user. Same thinking for "start". Terry pointed out that I could always write *another* daemon that ran as root in the startup context and just responded to requests to manage my first daemon. Yes, I could, but launchd is just sitting there... Perhaps the simplest change would be to allow non-root launchctl invocations to control (unload -w) daemons running with a UserName equivalent to the euid of the invoking process, provided the plist file is also owned by that uid. Bill _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: testing the current bootstrap context?
by Michael Smith 09 Apr '09

09 Apr '09

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: testing the current bootstrap context?
by Bill Janssen 09 Apr '09

09 Apr '09

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Terry Lambert <tlambert(a)apple.com> wrote: > On Apr 8, 2009, at 4:50 PM, Bill Janssen wrote: > > The only way to do this in Leopard seems to be with launchd, which > > seems > > to do the right thing. The main problem with using launchd here is > > that > > users cannot manipulate their own server; you can either use a root > > LaunchDaemon which only root can unload, or a user LaunchDaemon > > which is > > again fatally bound to the login bootstrap context. > > Actually, Damien pointed out that you can call bootstrap_subset(), but > I imagine you might consider this to count as "fatally bound", with it > being an inferior member of the hierarchy of that context. Right. The problem is that the user eventually logs out, so anything that's inherited from the login bootstrap context will start experiencing failures. We noticed this some months ago when we couldn't resolve DNS addresses in the daemon, but couldn't figure out at the time just why. > You could always have a root LaunchDaemon that owned the context via > bootstrap_subset(), and it could voluntarily respond to outside proxy > requests to manipulate it from non-root users, which is effectively > how loginWindow handles it, and how the per-user launchd handles it. > The model is a little more complicated, but it would work. Yes, thanks for the idea. Though I'm guessing that at this point more developers would turn to setuid scripts -- which is perhaps another reason to provide a more comprehensive strategy. Bill _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: testing the current bootstrap context?
by Terry Lambert 09 Apr '09

09 Apr '09

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Apr 8, 2009, at 4:50 PM, Bill Janssen wrote: The only way to do this in Leopard seems to be with launchd, which seems to do the right thing. The main problem with using launchd here is that users cannot manipulate their own server; you can either use a root LaunchDaemon which only root can unload, or a user LaunchDaemon which is again fatally bound to the login bootstrap context. Actually, Damien pointed out that you can call bootstrap_subset(), but I imagine you might consider this to count as "fatally bound", with it being an inferior member of the hierarchy of that context. You could always have a root LaunchDaemon that owned the context via bootstrap_subset(), and it could voluntarily respond to outside proxy requests to manipulate it from non-root users, which is effectively how loginWindow handles it, and how the per-user launchd handles it. The model is a little more complicated, but it would work. This email sent to site_archiver(a)lists.apple.com

1 0

Re: testing the current bootstrap context?
by Bill Janssen 08 Apr '09

08 Apr '09

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Thanks, I've read those Quinn <eskimo1(a)apple.com> wrote: > <http://developer.apple.com/samplecode/BootstrapDump/index.html> Yes, I've run it :-). I've mainly been using launchctl bslist, though, which I think is less useful. > Check out the "MAP" option. Yes, I should have looked harder at this. > Most problems like this are caused by folks starting their program in > the wrong bootstrap namespace. TN2083 describes this whole process > in gory detail. Specifically, check out the "Mach Bootstrap Basics", > "Starting a Daemon", and "Bootstrap Namespaces: One More Thing" > sections. > > <http://developer.apple.com/technotes/tn2005/tn2083.html> Right. What I didn't understand was that even the conventional UNIX daemonization dance of fork, setsid, fork, exec would not remove the resulting daemon from the login bootstrap context. For some reason I figured setsid() would put the resultant process group in the startup context. The only way to do this in Leopard seems to be with launchd, which seems to do the right thing. The main problem with using launchd here is that users cannot manipulate their own server; you can either use a root LaunchDaemon which only root can unload, or a user LaunchDaemon which is again fatally bound to the login bootstrap context. For the delectation of darwin-kernel readers, here's more on this: Subject: Re: should setsid() create new bootstrap context? From: Damien Sorresso <dsorresso(a)apple.com> Date: Fri, 3 Apr 2009 18:21:17 PDT To: Terry Lambert <tlambert(a)apple.com> Cc: Bill Janssen <janssen(a)parc.com>, darwin-dev(a)lists.apple.com On Apr 3, 2009, at 3:36 PM, Terry Lambert wrote: > On Apr 3, 2009, at 12:42 PM, Bill Janssen wrote: >> Damien Sorresso <dsorresso(a)apple.com> wrote: >>> On Apr 3, 2009, at 11:32 AM, Bill Janssen wrote: >>>> I think the root of my recent trouble with Darwin daemons is that I >>>> expected a call to setsid() to create a new bootstrap context. >>>> However, >>>> it doesn't seem to do that. Is that a bug in Darwin? >>> >>> Why would you expect this? >> >> Well, perhaps I'm just confused about the mapping of POSIX "session" >> onto Darwin. I thought the whole point of setsid() is to create a >> new >> process group that was independent of the pre-setsid process group. >> I'd >> expect it to create a new bootstrap context that would also be >> independent of the bootstrap context existing before setsid() was >> called, so that that the process group in that parent session could >> be >> terminated without affecting processes in the child session. >> >> As it is, when I call >> >> fork() ==> process B >> setsid() ==> process group B >> fork() ==> process C >> exec() >> >> as a non-root user from a bootstrap context which is then terminated >> (say, an SSH login), I wind up with process C having a bad bootstrap >> context. There are two possible reasons for this that I can see: >> >> 1. setsid() doesn't create a new bootstrap context for process >> group B, >> >> or >> >> 2. setsid() does create a new BC, but process B is the group leader, >> and the termination of B destroys the new context, so C winds up >> with an invalid BC. >> >> I'd appreciate more insight into this... > > POSIX sessions and launchd bootstrap sessions and Kerberos sessions > and windowServer sessions and loginWindow sessions and securityd > sessions and Common Criteria Auditing sessions, ..., are all > orthogonal things. Expanding on this, we at Apple are just as aware of and frustrated with the utter confusion that these different notions of "sessions" generate as our external developers are. You're not alone. :) So, as a public service announcement, in the currently shipping Mac OS X (Leopard), the existing notions of sessions are as follows. launchd bootstrap. This session type exists in a hierarchical structure. The system (i.e. PID 1) launchd provides the root bootstrap. Per-user launchd's provide sub-bootstraps for each user. Within each per-user bootstrap can be several sub-bootstraps for different entry points into the system (i.e. an "Aqua" bootstrap for graphical logins and a "StandardIO" bootstrap for SSH logins). Some processes may also call bootstrap_subset() to create sub-bootstraps for their own purposes. WindowServer session. Very closely tied to a launchd bootstrap. WindowServer sessions are expressed by the presence of the "com.apple.WindowServer.session" MachService in a launchd bootstrap. When a client sends a message to that service port, WindowServer knows which port in its portset the message came from and assigns a session based on that. securityd session. Very closely tied to a launchd bootstrap. securityd sessions are expressed by the presence of the "com.apple.SecurityServer" MachService in a launchd bootstrap. When a client sends a message to that service port, securityd knows which port in its portset the message came from and assigns a session based on that. loginwindow session. This could also be described as a "graphical console session". loginwindow does a funny dance. It begins its life as a daemon in the system bootstrap, but when a user logs in, it packs up and moves itself (and its family) to the new user bootstrap that is created. So when loginwindow shows its UI, it's part of a "LoginWindow" bootstrap. After a user logs in, it becomes part of that new (per-user) bootstrap. If there are multiple users logged in at the console (i.e. through fast user-switching), there will be one loginwindow instance per console session to manage it. So if there are no users logged in, there is one loginwindow instance. If N users are logged in, there are N loginwindow instances. There are also Common Criteria auditing sessions, but those are only applicable on systems which have auditing turned on. Hope this helps. And again, sorry for all the confusion. We really do want to change it. -- Damien Sorresso BSD Engineering Apple Inc. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: testing the current bootstrap context?
by Quinn 08 Apr '09

08 Apr '09

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Which is a shame, because I don't read darwin-development. I created a tool for debugging this sort of thing. <http://developer.apple.com/samplecode/BootstrapDump/index.html> Check out the "MAP" option. <http://developer.apple.com/technotes/tn2005/tn2083.html> S+E -- Quinn "The Eskimo!" <http://www.apple.com/developer/> Apple Developer Relations, Developer Technical Support, Core OS/Hardware _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… At 09:21 -0700 2/4/09, Bill Janssen wrote: Sorry, I meant to send this to darwin-development, not darwin-kernel. Most problems like this are caused by folks starting their program in the wrong bootstrap namespace. TN2083 describes this whole process in gory detail. Specifically, check out the "Mach Bootstrap Basics", "Starting a Daemon", and "Bootstrap Namespaces: One More Thing" sections. This email sent to site_archiver(a)lists.apple.com

1 0

Re: What if process crashes while holding a locked semaphore?
by Jerry Krinock 02 Apr '09

02 Apr '09

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On 2009 Apr 01, at 19:33, Quinn wrote: _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… When using Mach ports like this, you have to be very careful about bootstrap namespaces. Each process inherits a reference to a bootstrap namespace, meaning that which namespace it registers in depends on how it was launched. A port registration will only fail if the conflicting process is in the same namespace. This may or may not be what you want. TN2083 has all the gory details. Thanks, Quinn. I believe I'm OK since my mutually-exclusive processes are all current-user agents launched by launchd to fulfill tasks in ~/ Library/LaunchAgents. So, I am obeying the "rules to remember" in TN2083, and they should all be in the same namespace. I've added your comments to my code for future reference. Maybe when needed I'll be able to find a nice Cocoa interface to Unix Domain Sockets. This email sent to site_archiver(a)lists.apple.com

1 0

Re: testing the current bootstrap context?
by Bill Janssen 02 Apr '09

02 Apr '09

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Sorry, I meant to send this to darwin-development, not darwin-kernel. Bill Bill Janssen <janssen(a)parc.com> wrote: > I seem to have a situation where the bootstrap context of a long-running > daemon process sometimes "goes bad". Not sure just what's happening, > but it can't seem to use any Mach ports. Can't connect to the > windowServer, can't do DNS lookups, etc. When I use 'launchctl bslist' > to look at the BC, I get an odd output (which I don't have written down > right now). > > Is there a function I could call from a process to check the current > bootstrap context? If the process could detect, in some systematic way, > whether or not its bootstrap context is still good, it could restart > when it needs to. > > Bill > _______________________________________________ > Do not post admin requests to the list. They will be ignored. > Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) > Help/Unsubscribe/Update your Subscription: > http://lists.apple.com/mailman/options/darwin-kernel/janssen%40parc.com > > This email sent to janssen(a)parc.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

testing the current bootstrap context?
by Bill Janssen 02 Apr '09

02 Apr '09

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com I seem to have a situation where the bootstrap context of a long-running daemon process sometimes "goes bad". Not sure just what's happening, but it can't seem to use any Mach ports. Can't connect to the windowServer, can't do DNS lookups, etc. When I use 'launchctl bslist' to look at the BC, I get an odd output (which I don't have written down right now). Is there a function I could call from a process to check the current bootstrap context? If the process could detect, in some systematic way, whether or not its bootstrap context is still good, it could restart when it needs to. Bill _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: What if process crashes while holding a locked semaphore?
by Steve Checkoway 02 Apr '09

02 Apr '09

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Apr 2, 2009, at 2:03 AM, Terry Lambert wrote: Of course. Thanks. The engineering perspective was what I was interested in. -- Steve Checkoway "Anyone who says that the solution is to educate the users hasn't ever met an actual user." -- Bruce Schneier _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com On Apr 1, 2009, at 9:31 PM, Steve Checkoway <s(a)pahtak.org> wrote: Is there a technical reason that Darwin doesn't support POSIX unnamed semaphores or it's just not a priority since they're not necessary? I can give you technical information about why I haven't personally done it, but realize I am not speaking officially in any capacity by doing so. With that in mind, I'll try to answer your questions from an engineering perspective, rather than putting it as the more pragmatic "if Apple implemented POSIX unnamed semaphores, how many X Serves would you buy?"... They aren't required for UNIX conformance because they are not mandatory interfaces per Appendix 9 of the Single UNIX Specification, and there are lots of alternatives, most of them with better overall semantics, which this tread has amply demonstrated. Most portable and historical software uses System V semaphores, since POSIX semaphores are relatively new, so there's little software portability incentive. The software which does like to use them is typically based on GNU autoconf/automake which typically uses linkage tests to find interfaces and blows it by doing that and ignoring <unistd.h> contents, since historical conformance with older versions of the specification permitted stub functions which errored out, and the linkage tests only check linkability rather than functionality. So basically the software that wants them usually fails to conform to the standards which would have allowed them to be used safely and reliably in the first place. Then there is the little problem of binary backward compatibility for POSIX named semaphores, if the error or success returns don't happen to match the standard once testing of a full implementation became possible. As a Mac OS X implementation detail, sem_t is the required data type for both POSIX named semaphores and POSIX unnamed semaphores according to the standard. The sem_t definition has historically been a 32 bit value containing an fd cast to a pointer, which is problematic for maintaining binary and source backward compatibility (hint: think symbol decoration) for named semaphores while at the same time permitting typical usage of unnamed semaphores. Specifically, typical usage of unnamed semaphores is to use them as an IPC synchronization mechanism between otherwise unrelated programs by allocating a shared memory region shared between them of sizeof(sem_t)*number_of_semaphores_desired, and the casting the base address of that memory region to a (sem_t *) and indexing off that to get a semaphore index in the range 0.. (number_of_semaphores_desired - 1). The implementation problems should now be obvious. They are not insurmountable; I sometimes pose how to resolve the conflicting goals involved as an interview question. 8-). But there isn't really a very obvious fix I'd call elegant, either. After reading your response and the notes for fp_getfsem, I believe I see the issues. Thanks for taking the time to explain. Maybe someone will surprise me someday, or maybe an outside developer will attach elegant code capable of balancing the conflicting goals to a bug report the next time they request the feature, or maybe someone will contract to buy a bunch of X Serves. I'm afraid I neither need the feature nor have the money to buy a bunch of Xserves. I was just curious about its absence. smime.p7s

1 0