January 2006 - Darwin-kernel

Re: Kernel NKE projects user space daemon.
by Stephane Sudre 05 Jan '06

05 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On jeudi, janvier 5, 2006, at 06:07 PM, Mike Smith wrote: On Jan 5, 2006, at 2:26 AM, Stephane Sudre wrote: An admin may choose to remove ps from the standard location. Well, in that case, Windows is better than OS X :-) _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… [...] An extreme case could also be that the ps tool is not installed on the computer. Then the system is not Mac OS X. Would you also consider than when grep is not installed, it's not Mac OS X? The fact is that it was not part of the standard install before 10.3. Anyway, the main problem IMHO is that OS X/Darwin is lacking either a true and trustful API to get this information or a proc folder (even though I think I saw such a folder in one case (maybe it was on OS X Server)). There is no such interface because the information you want cannot be reliably returned; an interface that returns wrong information is worse than no interface at all. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Kernel NKE projects user space daemon.
by Mike Smith 05 Jan '06

05 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Jan 5, 2006, at 2:26 AM, Stephane Sudre wrote: Then the system is not Mac OS X. = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… If you use the output of "ps" with appropriate field selection arguments to get only the data you want, then we can change things out from under you, and your application will still work, since "ps" is revised in lockstep with the kernel itself. As Mike also pointed out in another message, "ps" is OK for this particular case, since the issue only comes up when specific network traffic is being allowed/denied via a user interaction. The "ps" program itself won't trigger this in the normal case, since it's run from local disks, and has no network dependencies of its own. Yet the problem with using ps is that you are relying on an external tool which may change in the future and to which you do not have access as changes are being made. Example: the ps source code for Tiger was not available until the release of Tiger. You don't need the ps source code; you just invoke ps directly. The interface to ps-the-program is much less likely to change than the kernel's internal data structures. An extreme case could also be that the ps tool is not installed on the computer. Anyway, the main problem IMHO is that OS X/Darwin is lacking either a true and trustful API to get this information or a proc folder (even though I think I saw such a folder in one case (maybe it was on OS X Server)). There is no such interface because the information you want cannot be reliably returned; an interface that returns wrong information is worse than no interface at all. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Kernel NKE projects user space daemon.
by Stephane Sudre 05 Jan '06

05 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On 04 Jan 2006, at 22:47, Terry Lambert wrote: _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Jan 4, 2006, at 9:34 AM, Stephane wrote: On Jan 4, 2006, at 5:58 PM, Mike Smith wrote: On Jan 4, 2006, at 2:58 AM, Stephane Sudre wrote: ps can still be wrong, but it is more likely to both be right and to be maintained in such a fashion that it prints reasonable output. I would say that ps is _always_ wrong when the process has been launched using a relative path since a relative path can match multiple absolute paths. In this case, ps is no more wrong than anything else. The KAuth API would give an absolute path AFAIK. The approach I described using a kauth listener to get the path information depends on the directory lookup name cache information. The problem with this approach, which I think Mike is alluding to, is that between the time you cache the information in the user space daemon, and he time you use it to present to the user a choice, the target of the path that was cached may have changed. Even if you specifically go out of your way in user space, and also pass dev_t and ino_t information, verifying it immediately before presentation in the UI -- there's no real way to programatically lock a path so that its components are not permitted to change, without seriously obstructing the ability of the system to get actual work done. So it's basically a snapshot, however you approach it. Affirmative, but I would rather get an absolute snapshot than a relative one. Using the output of the "ps" program is actually the recommended method of getting information which is the least prone to breakage in future versions of the OS going forward. For the most part, I personally hate the KERN_* sysctl's - they are data interfaces, and therefore subject to becoming potentially stale. I agree that the sysctl thing is a problem since for instance the ps code using sysctl from Panther is not working on Tiger. If you use the output of "ps" with appropriate field selection arguments to get only the data you want, then we can change things out from under you, and your application will still work, since "ps" is revised in lockstep with the kernel itself. As Mike also pointed out in another message, "ps" is OK for this particular case, since the issue only comes up when specific network traffic is being allowed/denied via a user interaction. The "ps" program itself won't trigger this in the normal case, since it's run from local disks, and has no network dependencies of its own. Yet the problem with using ps is that you are relying on an external tool which may change in the future and to which you do not have access as changes are being made. Example: the ps source code for Tiger was not available until the release of Tiger. An extreme case could also be that the ps tool is not installed on the computer. Anyway, the main problem IMHO is that OS X/Darwin is lacking either a true and trustful API to get this information or a proc folder (even though I think I saw such a folder in one case (maybe it was on OS X Server)). This can be done from a bootstrap session. You just have to launch a helper tool for the UI. If the UI does not need to be really complex, you can also use a CFUserNotification dialog. We discourage this - "...can't be done *legitimately*..." [emphasis mine] - because you never know that what happens to work right now will continue to work the same way in some far off future. There's a big difference between "can be done this way" and "should be done this way". I'm not sure to see where the illegitimate part lies in the CFUserNotification API or launching a helper tool. Can you shed some light on this? This email sent to site_archiver(a)lists.apple.com

1 0

Re: Kernel NKE projects user space daemon.
by Mike Smith 05 Jan '06

05 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Jan 4, 2006, at 9:34 AM, Stephane wrote: On Jan 4, 2006, at 5:58 PM, Mike Smith wrote: On Jan 4, 2006, at 2:58 AM, Stephane Sudre wrote: In this case, ps is no more wrong than anything else. The KAuth API would give an absolute path AFAIK. = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… ps can still be wrong, but it is more likely to both be right and to be maintained in such a fashion that it prints reasonable output. I would say that ps is _always_ wrong when the process has been launched using a relative path since a relative path can match multiple absolute paths. The KAUTH API is useless here; there is no association between the authorisation check for executability on a vnode (where you would be able to fetch the looked-up path for the vnode) and a process which just *happens* to be associated with a task which just *happens* to have the text segment from said file mapped in a fashion consistent with a binary being executed. There is no guarantee that a process is a result of launching an executable. There is no guarantee that any file mapped into the task address space associated with a process has anything to do with the path given to the exec(2) system call. There is no guarantee that there even *was* an exec(2) system call. On Jan 4, 2006, at 1:47 PM, Terry Lambert wrote: The problem with this approach, which I think Mike is alluding to, is that between the time you cache the information in the user space daemon, and he time you use it to present to the user a choice, the target of the path that was cached may have changed. Even if you specifically go out of your way in user space, and also pass dev_t and ino_t information, verifying it immediately before presentation in the UI -- there's no real way to programatically lock a path so that its components are not permitted to change, without seriously obstructing the ability of the system to get actual work done. This is one facet of the issue; it's a specific instance of a generic problem. As Mike also pointed out in another message, "ps" is OK for this particular case, since the issue only comes up when specific network traffic is being allowed/denied via a user interaction. The "ps" program itself won't trigger this in the normal case, since it's run from local disks, and has no network dependencies of its own. I was actually referring to the comparatively low overhead of running ps vs. displaying UI, but the above is also good point. Note that it's important to specify the minimal set of fields for ps to display, as some symbolic information may result in nested network traffic leading to deadlock. This email sent to site_archiver(a)lists.apple.com

1 0

Re: OSMalloc Fails (returns 0)
by Terry Lambert 05 Jan '06

05 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Jan 4, 2006, at 2:48 PM, Russ Seehafer wrote: On Jan 2, 2006, at 8:34 PM, Terry Lambert wrote: On Dec 25, 2005, at 2:22 PM, Russ Seehafer wrote: I have a socket filter nke based on the tcplognke sample. I make a call to OSMalloc with the gOSMallocTag to malloc some space for a struct inside the nke's tl_attach_fn function. Usually this works great, but every now and then the malloc fails, usually after the system has been running for a while. Anyone know why this would happen? I couldn't find any documentation on it. In general, this shouldn't happen unless you are leaking memory (e.g. not freeing the memory on detach, attaching too many times, etc.). You can use the command "zprint" to display memory usage by zone on the system; I suspect you will see one of the kalloc.# zones that is the size of your structure (or larger, if your structure is not an even power of 2 in size) has hit its maximum number of elements. If you need a lot of memory and are willing to manage it yourself, you should alloc a page at a time; then you will also allocate pageable memory instead of wired memory (internally OSMalloc will call either kalloc() or kmem_alloc_pageable()). -- Terry OK, I have went through my mallocs and frees and I can't find any leaks. I did notice that the structure that causes the malloc to fail is bigger than I thought, a lot bigger. It is about 220000 bytes. Yes you read that right. What are my options for getting this much memory so frequently in the nke? Im not exactly clear on what you mean by getting memory a page at a time and managing it myself. Thanks so much for your input. Your only options are pretty much to grab it up front, and manage it yourself, or find a more memory efficient data structure for what you're trying to do. Better to avoid the memory fragmentation issue entirely, by not allocating and freeing big chunks like that. To elaborate: if you grab the first chunk that large you can, end up freeing it later, and even one allocation happens in the freed space, you are looking for the next place in memory that can hold a 220K contiguous object; do that enough times, and there won't be a place in memory your allocation is capable of being allocated, and you get your 0 return. You will also *definitely* want to guarantee that the size you request to allocate is aligned to an even multiple of the page size, for something that large. This email sent to site_archiver(a)lists.apple.com

1 0

Re: OSMalloc Fails (returns 0)
by Russ Seehafer 04 Jan '06

04 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:in-reply-to:references:mime-version:content-type:message-id:cc:content-transfer-encoding:from:subject:date:to:x-mailer:sender; b=VPOH9xD3fQ7xI/rQ03iQ47xIKhUnk2uQXba4wGm867L4NQBFgCFHvtc1f+KbUwBlLqjePzfObKqNdFOqpj/R0u2WSTeGJqGm2D+aJu/1dYaV6dgQkg0jftoBgrOQR9MTndYsW6W1kL30dwLC15X0T2kVgdD70GJFjhEJFtip6lc= On Jan 2, 2006, at 8:34 PM, Terry Lambert wrote: -Russ _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Dec 25, 2005, at 2:22 PM, Russ Seehafer wrote: I have a socket filter nke based on the tcplognke sample. I make a call to OSMalloc with the gOSMallocTag to malloc some space for a struct inside the nke's tl_attach_fn function. Usually this works great, but every now and then the malloc fails, usually after the system has been running for a while. Anyone know why this would happen? I couldn't find any documentation on it. In general, this shouldn't happen unless you are leaking memory (e.g. not freeing the memory on detach, attaching too many times, etc.). You can use the command "zprint" to display memory usage by zone on the system; I suspect you will see one of the kalloc.# zones that is the size of your structure (or larger, if your structure is not an even power of 2 in size) has hit its maximum number of elements. If you need a lot of memory and are willing to manage it yourself, you should alloc a page at a time; then you will also allocate pageable memory instead of wired memory (internally OSMalloc will call either kalloc() or kmem_alloc_pageable()). -- Terry OK, I have went through my mallocs and frees and I can't find any leaks. I did notice that the structure that causes the malloc to fail is bigger than I thought, a lot bigger. It is about 220000 bytes. Yes you read that right. What are my options for getting this much memory so frequently in the nke? Im not exactly clear on what you mean by getting memory a page at a time and managing it myself. Thanks so much for your input. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Kernel NKE projects user space daemon.
by Terry Lambert 04 Jan '06

04 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com So it's basically a snapshot, however you approach it. -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Jan 4, 2006, at 9:34 AM, Stephane wrote: On Jan 4, 2006, at 5:58 PM, Mike Smith wrote: On Jan 4, 2006, at 2:58 AM, Stephane Sudre wrote: ps can still be wrong, but it is more likely to both be right and to be maintained in such a fashion that it prints reasonable output. I would say that ps is _always_ wrong when the process has been launched using a relative path since a relative path can match multiple absolute paths. In this case, ps is no more wrong than anything else. The KAuth API would give an absolute path AFAIK. The approach I described using a kauth listener to get the path information depends on the directory lookup name cache information. The problem with this approach, which I think Mike is alluding to, is that between the time you cache the information in the user space daemon, and he time you use it to present to the user a choice, the target of the path that was cached may have changed. Even if you specifically go out of your way in user space, and also pass dev_t and ino_t information, verifying it immediately before presentation in the UI -- there's no real way to programatically lock a path so that its components are not permitted to change, without seriously obstructing the ability of the system to get actual work done. Using the output of the "ps" program is actually the recommended method of getting information which is the least prone to breakage in future versions of the OS going forward. For the most part, I personally hate the KERN_* sysctl's - they are data interfaces, and therefore subject to becoming potentially stale. If you use the output of "ps" with appropriate field selection arguments to get only the data you want, then we can change things out from under you, and your application will still work, since "ps" is revised in lockstep with the kernel itself. As Mike also pointed out in another message, "ps" is OK for this particular case, since the issue only comes up when specific network traffic is being allowed/denied via a user interaction. The "ps" program itself won't trigger this in the normal case, since it's run from local disks, and has no network dependencies of its own. As for the suid thing, why not just used a StartupItem or a launchd thing? Bootstrap context is usually the nirvana for daemon needing to communicate with a kernel extension. As Matt clearly pointed out, he needs to present UI to the user; this can't be done legitimately from the bootstrap session. And it doesn't help solve this problem. This can be done from a bootstrap session. You just have to launch a helper tool for the UI. If the UI does not need to be really complex, you can also use a CFUserNotification dialog. We discourage this - "...can't be done *legitimately*..." [emphasis mine] - because you never know that what happens to work right now will continue to work the same way in some far off future. There's a big difference between "can be done this way" and "should be done this way". This email sent to site_archiver(a)lists.apple.com

1 0

Re: Kernel NKE projects user space daemon.
by Stephane 04 Jan '06

04 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Jan 4, 2006, at 5:58 PM, Mike Smith wrote: On Jan 4, 2006, at 2:58 AM, Stephane Sudre wrote: The KAuth API would give an absolute path AFAIK. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… ps can still be wrong, but it is more likely to both be right and to be maintained in such a fashion that it prints reasonable output. I would say that ps is _always_ wrong when the process has been launched using a relative path since a relative path can match multiple absolute paths. In this case, ps is no more wrong than anything else. As for the suid thing, why not just used a StartupItem or a launchd thing? Bootstrap context is usually the nirvana for daemon needing to communicate with a kernel extension. As Matt clearly pointed out, he needs to present UI to the user; this can't be done legitimately from the bootstrap session. And it doesn't help solve this problem. This can be done from a bootstrap session. You just have to launch a helper tool for the UI. If the UI does not need to be really complex, you can also use a CFUserNotification dialog. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Kernel NKE projects user space daemon.
by Mike Smith 04 Jan '06

04 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Jan 4, 2006, at 2:58 AM, Stephane Sudre wrote: In this case, ps is no more wrong than anything else. = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… ps can still be wrong, but it is more likely to both be right and to be maintained in such a fashion that it prints reasonable output. I would say that ps is _always_ wrong when the process has been launched using a relative path since a relative path can match multiple absolute paths. As for the suid thing, why not just used a StartupItem or a launchd thing? Bootstrap context is usually the nirvana for daemon needing to communicate with a kernel extension. As Matt clearly pointed out, he needs to present UI to the user; this can't be done legitimately from the bootstrap session. And it doesn't help solve this problem. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Kernel NKE projects user space daemon.
by Stephane Sudre 04 Jan '06

04 Jan '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Le 04 Jan 2006, à 08:27, Mike Smith a écrit : On Jan 3, 2006, at 12:14 PM, matt jaffa wrote: _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… So my NKE gets the pid of the process sending information out. I send this pid to a daemon that will lookup that process path based on the pid using the sysctl KERN_PROC methods. But this sysctl will fail if it is looking up the path on a process owned by root. So my daemon has to be running as root to get this info, and I want to be able to have a daemon for each logged in user so the UI can be displayed to the current active user. I could spawn a process with popen ( ps ) and gather the path that way, but ps will be spawned everytime I get a new pid my NKE doesn't know about. Is this a bad thing? Spawning a lot of little "ps" just to get this information. Don't use popen, as it invokes the shell and that puts you at risk (the shell may be invalid, or its startup files may have been coopted). However, given that you're about to draw UI and block waiting for the user, running ps (use a custom output format to override any defaults, and invoke ps via an explicit path) is not going to cause any observable overhead. Even if your application supports saved rules, running ps once per application is not a major hit. ps can still be wrong, but it is more likely to both be right and to be maintained in such a fashion that it prints reasonable output. I would say that ps is _always_ wrong when the process has been launched using a relative path since a relative path can match multiple absolute paths. As for the suid thing, why not just used a StartupItem or a launchd thing? Bootstrap context is usually the nirvana for daemon needing to communicate with a kernel extension. This email sent to site_archiver(a)lists.apple.com

1 0