March 2004 - Darwin-kernel - lists.apple.com

Re: Determine port holder
by Matt Jaffa 03 Mar '04

03 Mar '04

Brian, Actually I am using a NKE with protocol filters, using proc *p = current_proc() Actually gets me who is doing it, the only reason I bring up this question, Is I am trying to find a way to distinguish Classic apps, That is very hard since they share the same pid, which is TruBlueEnvironment, well they all go through that pid, and that is what current_proc returns, My program runs fine if I care less about Classic traffic, I started out patching the syscall table, but was informed this was bad practice and I switched along time ago. I just figure now that I am going to give up on trying to intercept classic traffic, too hard to distinguish which app (Within Classic) made the call. Thanks, Matt On Mar 3, 2004, at 10:29 AM, Brian Bergstrand wrote: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I think Matt, wants something similar if functionality to ObDev's Little Snitch. And to do that you have to do what they are doing, and patch the kernel syscall table. There is no reliable way to get a PID from a NKE filter. The way little snitch does this, is by patching the kernel syscalls for socket opens (sosend, socreate, others?). Since these calls exist in the context of the BSD kernel the PID of the calling process is readily available and Little Snitch can pull the PID (and check the dest addr at the same time). If the check fails, it calls out to it's userland daemon to handle the failure. Once in userland, you can find the process name (or something close to it) from the process table. Of course none of this is guaranteed to work, it just happens to at the moment. In fact, Little Snitch ran into some panic problems with the upgrade to 10.3 because it does patch the syscall table. Patching is a very evil thing (as any 0S 9 developer and/or user can attest to). Little Snitch panic (that was eventually fixed) : <http://forums.macnn.com/showthread.php? s=&threadid=195232&highlight=little+snitch> HTH. On Mar 3, 2004, at 1:14 AM, Justin Walker wrote: On Tuesday, March 2, 2004, at 10:45 PM, Matt Jaffa wrote: Hi, Is there a way to determine which app that is running has control over a certain port. This is dealing with sockets, I have intercepted a raw_packet going out, and using the src_port I would like to know which app is listening, or owns it. Is there a way to do this within the Kernel, and /or within a daemon inside user space? The concept of "control" over a port does not exist. Ports in certain ranges are "allocated" by IANA to certain functions/services, but that is convention, and is locally dictated by things like "/etc/services" (if you decide to change that usage, of course, you end up breaking a lot of functionality, but that's a separate issue). In addition, an open socket can be handed to another process, and another UID, in a couple of ways: - parent forks, child and parent both have 'control' of an open socket - application execs a new executable (same process, different code) - process A can establish a local socket communication with process B, and then pass another open socket (say, INET4) to B - the UID of a process can change through system calls and the SETUID bit in the executable. Any number of processes, each running different code, can have the same socket open and be actively using it. Remember that a socket is a kernel structure that is treated like a file structure and shared among processes. Each proc table has an array of 'descriptors' that point to such things as socket structures, so the socket structure itself can't be "owned" by any one process. This has been discussed before, and you might benefit from trying to search the archives. For this kind of question, the ones to look at are darwin-kernel, darwin-development, and possibly the opendarwin lists. Depending on where your extension lives, you may be able to determine the identity of one of the users of the socket at the time it is accessed for read or write. That's the best you can hope for. Regards, Justin -- Justin C. Walker, Curmudgeon-At-Large * Institute for General Semantics | Some people have a mental | horizon of radius zero, and | call it their point of view. | -- David Hilbert *-------------------------------------- *-------------------------------* _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored. Brian Bergstrand <http://www.bergstrand.org/brian/>, AIM: triryche206 PGP Key: <http://www.bergstrand.org/brian/misc/public_key.txt> Conscience is the inner voice that warns us that someone might be looking. - H. L. Mencken As of 11:16:51 AM, iTunes is playing "The Shortest Straw" from "...And Justice For All" by "Metallica" -----BEGIN PGP SIGNATURE----- Version: PGP 8.0.3 iQA/AwUBQEYH3nnR2Fu2x7aiEQIaPQCgvX4NLw4iXGw44weSHTEWBXnIwrgAnA8k G0KUbNgGbs5QShJwYgfyB52M =ESqs -----END PGP SIGNATURE----- _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Determine port holder
by Brian Bergstrand 03 Mar '04

03 Mar '04

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I think Matt, wants something similar if functionality to ObDev's Little Snitch. And to do that you have to do what they are doing, and patch the kernel syscall table. There is no reliable way to get a PID from a NKE filter. The way little snitch does this, is by patching the kernel syscalls for socket opens (sosend, socreate, others?). Since these calls exist in the context of the BSD kernel the PID of the calling process is readily available and Little Snitch can pull the PID (and check the dest addr at the same time). If the check fails, it calls out to it's userland daemon to handle the failure. Once in userland, you can find the process name (or something close to it) from the process table. Of course none of this is guaranteed to work, it just happens to at the moment. In fact, Little Snitch ran into some panic problems with the upgrade to 10.3 because it does patch the syscall table. Patching is a very evil thing (as any 0S 9 developer and/or user can attest to). Little Snitch panic (that was eventually fixed) : <http://forums.macnn.com/showthread.php? s=&threadid=195232&highlight=little+snitch> HTH. On Mar 3, 2004, at 1:14 AM, Justin Walker wrote: > On Tuesday, March 2, 2004, at 10:45 PM, Matt Jaffa wrote: > >> Hi, >> >> Is there a way to determine which app that is running has control >> over a certain port. >> >> This is dealing with sockets, I have intercepted a raw_packet going >> out, and using the >> src_port I would like to know which app is listening, or owns it. >> >> Is there a way to do this within the Kernel, and /or within a daemon >> inside user space? > > The concept of "control" over a port does not exist. Ports in certain > ranges are "allocated" by IANA to certain functions/services, but that > is convention, and is locally dictated by things like "/etc/services" > (if you decide to change that usage, of course, you end up breaking a > lot of functionality, but that's a separate issue). > > In addition, an open socket can be handed to another process, and > another UID, in a couple of ways: > - parent forks, child and parent both have 'control' of an open > socket > - application execs a new executable (same process, different code) > - process A can establish a local socket communication with process > B, > and then pass another open socket (say, INET4) to B > - the UID of a process can change through system calls and the > SETUID bit > in the executable. > > Any number of processes, each running different code, can have the > same socket open and be actively using it. > > Remember that a socket is a kernel structure that is treated like a > file structure and shared among processes. Each proc table has an > array of 'descriptors' that point to such things as socket structures, > so the socket structure itself can't be "owned" by any one process. > > This has been discussed before, and you might benefit from trying to > search the archives. For this kind of question, the ones to look at > are darwin-kernel, darwin-development, and possibly the opendarwin > lists. > > Depending on where your extension lives, you may be able to determine > the identity of one of the users of the socket at the time it is > accessed for read or write. That's the best you can hope for. > > Regards, > > Justin > > -- > Justin C. Walker, Curmudgeon-At-Large * > Institute for General Semantics | Some people have a mental > | horizon of radius zero, and > | call it their point of view. > | -- David Hilbert > *-------------------------------------- > *-------------------------------* > _______________________________________________ > darwin-kernel mailing list | darwin-kernel(a)lists.apple.com > Help/Unsubscribe/Archives: > http://www.lists.apple.com/mailman/listinfo/darwin-kernel > Do not post admin requests to the list. They will be ignored. > > > Brian Bergstrand <http://www.bergstrand.org/brian/>, AIM: triryche206 PGP Key: <http://www.bergstrand.org/brian/misc/public_key.txt> Conscience is the inner voice that warns us that someone might be looking. - H. L. Mencken As of 11:16:51 AM, iTunes is playing "The Shortest Straw" from "...And Justice For All" by "Metallica" -----BEGIN PGP SIGNATURE----- Version: PGP 8.0.3 iQA/AwUBQEYH3nnR2Fu2x7aiEQIaPQCgvX4NLw4iXGw44weSHTEWBXnIwrgAnA8k G0KUbNgGbs5QShJwYgfyB52M =ESqs -----END PGP SIGNATURE----- _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

auditing support
by John C. Daub 03 Mar '04

03 Mar '04

Hello. I'm looking at the auditing support that was added to the kernel in Panther. I'm figuring out some things from headers, source, and Google, but it's not enough. Just wondering if anyone knows of any documentation and/or sample code pertaining to Darwin's kernel auditing support. Thanx. -- John C. Daub }:-)>= <mailto:hsoi@hsoi.com> <http://www.hsoi.com/> "Life is good, a rock sled utopia, a self-imposed myopia. You gotta love this pace." - Dave Wyndorf _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Stacked file systems - why not and what instead?
by Quinn 03 Mar '04

03 Mar '04

At 5:55 +0100 3/3/04, Ragnar Sundblad wrote: Apple Q&A 1242 says: Marek did a fine job of answering your questions but, seeing as I wrote the Q&A in question, I feel obliged to follow up with more information. My questions: Could this be implemented at all without using a stacked file system, and what would be the apple recommended way of doing it? Yes, implement it the way WebDAV works. This isn't a full blown stacked file system, but it does palm a lot of the work off to the actual file system that holds the cache. ... it's not possible to create a stacking VFS plug-in that: 1. works reliably, Darwin inherited a VFS design from 4.4BSD that included stacking. However, Apple made significant changes to the VFS architecture (specifically, UBC and volfs) that affect the viability of stacking. I believe that these changes break stacking in the general case, and no one in Apple file system engineering has the resources to create an example stacked file system that proves me wrong. btw For information about volfs, see DTS Q&A 1113 "The '/.vol' directory and 'volfs'". <http://developer.apple.com/qa/qa2001/qa1113.html> Regardless of the in-kernel problems there are also serious management problems for stacked file systems. Off the top of my head I can think of the following. o How do you get stacked on top of all mounted volumes? The disk mounting process on Mac OS X is managed by Disk Arbitration, which has no support for stacking. o Imagine you're writing a compression stacked file system. How do you ensure that your file system is stacked above any encrypting file systems? [Compressing after encryption is pointless.] In some cases you can work around these issues by constraining the design or UI of your stacked file system, but that's not true in the general case. 2. does not severely impinge on system performance Marek explained the vnode problem in his response. This isn't an issue for a prototype system (just increase the size of the vnode pool), but it is a problem if Apple was to support stacking as a general purpose solution for a consumer OS like Mac OS X. Imagine if the user had 4 file systems stacked on top of each vnode (not implausible: file system notification, anti-virus, compression, encryption). Suddenly the vnode pool is 20% smaller than it should be. Even if it was acceptable to gobble that much wired memory, managing the growth of the vnode pool is tricky without global knowledge of what stacked file systems are installed and what vnodes they are using. The problem with the 4.4BSD stacking architecture IMHO is that it uses one solution (stacking vnodes) to cover a spectrum of problems. Not all stacked file systems need a full vnode at their layer. For example, an anti-virus stacked file system just wants to intercept file system opens; it doesn't need its own copy of the file data (which is what the vnode represents). OTOH, an encrypting stacked file system would probably benefit from having a vnode at its level (the stacked vnode holds the decrypted UBC data while the underlying vnode holds the encrypted UBC data). Unfortunately, our current architecture doesn't support this distinction. And regarding this one: " 3. has any hope of binary (or even source-level) compatibility with future systems. " Would a stacking file system have (significantly) more compatibility problems than a non stacking file system? You're right, a stacked file system is not substantially less binary compatible than a non-stacked file system (although that could change in the future). They both have serious release-to-release compatibility problems. However, we chose to support leaf file systems for the following reasons. o More developers work on leaf file systems than on stacked file systems. o Leaf file systems are more important to our market than stacked file systems. o There are up-to-date, production-quality leaf file systems in Darwin. You you can use these as sample code. This isn't true for stacked file systems. In fact, I know of no up-to-date, production-quality open source stacked file system for Mac OS X. o When we do have a kernel binary compatibility story (see below), it will likely address leaf file systems as a priority. This is because Apple has internal leaf file systems that we have to port to it. Stacked file systems may be supported by these interfaces, or may not be, depending on the availability of engineering resources. Wasn't there some ideas about a cleaner and more static file system api for darwin? We introduced the concept of Kernel Programming Interfaces (KPI) at WWDC 2003. This introduction was more of a architecture sketch than a detailed specification. Since then we've been working hard to flesh out the KPI story for VFS plug-ins and NKEs. I'm hoping that we'll cover this in depth at WWDC 2004. However, the session details haven't been made public yet so I can't commit to anything at this time. If you're interested in hearing the latest news about KPIs, keep an eye on the WWDC web site. The session details should be published soon. <http://developer.apple.com/wwdc/> If you wish to be notified of WWDC-related topics, you should subscribe to ADC News. <http://developer.apple.com/devnews/index.html> * * * Finally, I'd like to reiterate the comments I made in the Q&A. The fact that Apple doesn't encourage or support stacking doesn't mean that you can't do it. It just means that a) you can't get official support from DTS, and b) you will probably suffer more binary compatibility problems than if you were developing a supported product. However, we have no secrets--the entire kernel source is available via Darwin--and there's a variety of unofficial support resources (such as this list). Marek is even using this technology in a commercial product and he hasn't (yet) been abducted by the Apple Kernel Purity Police (tm) (-: S+E -- Quinn "The Eskimo!" <http://www.apple.com/developer/> Apple Developer Technical Support * Networking, Communications, Hardware _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Determine port holder
by Justin Walker 03 Mar '04

03 Mar '04

On Tuesday, March 2, 2004, at 10:45 PM, Matt Jaffa wrote: Hi, Is there a way to determine which app that is running has control over a certain port. This is dealing with sockets, I have intercepted a raw_packet going out, and using the src_port I would like to know which app is listening, or owns it. Is there a way to do this within the Kernel, and /or within a daemon inside user space? The concept of "control" over a port does not exist. Ports in certain ranges are "allocated" by IANA to certain functions/services, but that is convention, and is locally dictated by things like "/etc/services" (if you decide to change that usage, of course, you end up breaking a lot of functionality, but that's a separate issue). In addition, an open socket can be handed to another process, and another UID, in a couple of ways: - parent forks, child and parent both have 'control' of an open socket - application execs a new executable (same process, different code) - process A can establish a local socket communication with process B, and then pass another open socket (say, INET4) to B - the UID of a process can change through system calls and the SETUID bit in the executable. Any number of processes, each running different code, can have the same socket open and be actively using it. Remember that a socket is a kernel structure that is treated like a file structure and shared among processes. Each proc table has an array of 'descriptors' that point to such things as socket structures, so the socket structure itself can't be "owned" by any one process. This has been discussed before, and you might benefit from trying to search the archives. For this kind of question, the ones to look at are darwin-kernel, darwin-development, and possibly the opendarwin lists. Depending on where your extension lives, you may be able to determine the identity of one of the users of the socket at the time it is accessed for read or write. That's the best you can hope for. Regards, Justin -- Justin C. Walker, Curmudgeon-At-Large * Institute for General Semantics | Some people have a mental | horizon of radius zero, and | call it their point of view. | -- David Hilbert *--------------------------------------*-------------------------------* _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Determine port holder
by Matt Jaffa 03 Mar '04

03 Mar '04

Hi, Is there a way to determine which app that is running has control over a certain port. This is dealing with sockets, I have intercepted a raw_packet going out, and using the src_port I would like to know which app is listening, or owns it. Is there a way to do this within the Kernel, and /or within a daemon inside user space? Thanks, Matt _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Stacked file systems - why not and what instead?
by Marek Kozubal 03 Mar '04

03 Mar '04

On Wed, 3 Mar 2004, Ragnar Sundblad wrote: > Background: > We are depending on the file system AFS in our environment > of different unix flavours. The AFS client is, as I > understand it, very much a stacked file system. What you would do, is create a brand new file system, that's not layered, and use UNIX sockets to do all file IO through a userland daemon, ala webdav. This would remove any need for a layered file system. > My questions: > Could this be implemented at all without using a stacked > file system, and what would be the apple recommended way > of doing it? (Of course one could use a separate cache See above... > I'd be happy if someone could explain both of these statements > from Q&A 1242: > " > ... it's not possible to create a stacking VFS plug-in that: > 1. works reliably, Writing file systems is never a fun task, there are a lot of locking issues involved and any minor mess up here and you'll be KP'ing some place or another. They can also be *very* hard to track down. > 2. does not severely impinge on system performance In a layered file system, you consume resources more quickly. The kernel keeps N vnodes alive at any given time, if you're doing a layered file system then each vnode really now uses 2 vnodes, so you've effectively changed N to N/2. > 3. has any hope of binary (or even source-level) compatibility > with future systems. Apple makes no promises not to completely change how all this stuff works at some future date. The only Kernel interface that is not going to change drastically I believe is the IOKit. > Would a stacking file system have (significantly) more > compatibility problems than a non stacking file system? I've written a stacking file system in Mac OS X, we've not really seen any more issues than I think I'd see w/a non-stacked file system, but I must just have been lucky so far... > Wasn't there some ideas about a cleaner and more static > file system api for darwin? It would be nice if Apple decided how they wanted it to work, and then implelemnted it. Though I really doubt we'll see anything for 10.4. --- Marek Kozubal marek(a)portents.com _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Stacked file systems - why not and what instead?
by Ragnar Sundblad 03 Mar '04

03 Mar '04

Hello all, First, excuse me for not knowing very much about vfs and asking maybe trivial questions, I am trying to learn about this stuff. Apple Q&A 1242 says: ... Stacking File Systems -- Apple does not support the development of stacking VFS plug-ins on Mac OS X. We've taken this position because, in our opinion, it's not possible to create a stacking VFS plug-in that: 1. works reliably, 2. does not severely impinge on system performance, and 3. has any hope of binary (or even source-level) compatibility with future systems. ... If you have a problem that can only be solved via a stacking file system, please let us know by mailing DTS. ... Background: We are depending on the file system AFS in our environment of different unix flavours. The AFS client is, as I understand it, very much a stacked file system. AFS works more or less like many other file systems except that the client caches files on local disk. To keep the client caches in sync, the client get callbacks from the server when the server need to change a file that the client has cached. (Very simplfied, it also has some other nice and pretty unique properties that I skip here.) We have for some years had the privilege of having not just one but two working AFS clients for mosx, Arla and OpenAFS. Now with 10.3/darwin 7 both these clients have different problems. OpenAFS seems to have some general vnode problem causing the kernel to panic sooner or later typically with "panic(cpu 0): vnreclaim: v_usecount", "panic(cpu 0): non UBC vnode on inactive list" or "panic(cpu 1): vinactive: vnode still on list", Arla has some other issues. Arla has a userland daemon that handles the communication with the servers and a kernel module that does as little as possible, I think it has many similarities to the webdav client. OpenAFS does (almost?) everything in the kernel. Both use file caches on local disk with ordinary files which live in a special cache directory. My questions: Could this be implemented at all without using a stacked file system, and what would be the apple recommended way of doing it? (Of course one could use a separate cache partition with a completely custom file system, but that, for a novice as me, sounds like much more work and much more bug prone than using an existing file system for the cache, so I don't like that idea.) I'd be happy if someone could explain both of these statements from Q&A 1242: " ... it's not possible to create a stacking VFS plug-in that: 1. works reliably, 2. does not severely impinge on system performance " And regarding this one: " 3. has any hope of binary (or even source-level) compatibility with future systems. " Would a stacking file system have (significantly) more compatibility problems than a non stacking file system? Wasn't there some ideas about a cleaner and more static file system api for darwin? Thanks for any help or information! /ragge _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: what's the replacement of page
by Quinn 02 Mar '04

02 Mar '04

At 4:30 -0800 2/3/04, Ramble Leon wrote: struct page in linux is used very wide, most in I/O operations, such as file read/write from device, sometimes it's also used in network communicatin. At the same time, page is the basic memory unit while using virtual memory. So could you please give me some suggest about that? PS: I'm not working in the xnu, just add a kext, so what I need is the struct can be used in kext. Hmmm, I don't think there is any direct equivalent in Darwin. It really depends on what type of KEXT you're implementing. If you're working in the I/O Kit space, you need to look at IOMemoryDescriptor. If you're working in the BSD or Mach space, you should look at UPLs (Universal Page Lists). It'd really be helpful if you could describe what you're trying to do at a higher level. Is there any facility in mach provide functions like schedule_timeout(time), which means current thread will giveup processor and not be scheduled until time out? I think thread_switch(...) can do this, but seems the thread calling thread_switch(...) cannot be resumed before the time is exhausted. But it's very important for me to resume the thread at sometime before the time is exhausted. Again, this depends on what area of the kernel you're working in. BSD, Mach, and I/O Kit all have support for stuff like this, and they are all different. In general, you want to use the support from the subsystem in which you're operating. S+E -- Quinn "The Eskimo!" <http://www.apple.com/developer/> Apple Developer Technical Support * Networking, Communications, Hardware _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: what's the replacement of page
by Ramble Leon 02 Mar '04

02 Mar '04

struct page in linux is used very wide, most in I/O operations, such as file read/write from device, sometimes it's also used in network communicatin. At the same time, page is the basic memory unit while using virtual memory. So could you please give me some suggest about that? PS: I'm not working in the xnu, just add a kext, so what I need is the struct can be used in kext. I have another questions. Is there any facility in mach provide functions like schedule_timeout(time), which means current thread will giveup processor and not be scheduled until time out? I think thread_switch(...) can do this, but seems the thread calling thread_switch(...) cannot be resumed before the time is exhausted. But it's very important for me to resume the thread at sometime before the time is exhausted. __________________________________ Do you Yahoo!? Yahoo! Search - Find what youre looking for faster http://search.yahoo.com _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0