November 2007 - Darwin-kernel

Re: pthread_cancel and cancelation points still broken in Mac OS X 10.5 Leopard?
by Terry Lambert 25 Nov '07

25 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com -- Terry On Nov 25, 2007, at 4:42 AM, Per Mildner <Per.Mildner(a)sics.se> wrote: On Nov 25, 2007, at 2:52 AM, Terry Lambert wrote: ... Regards, _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Thanks, Per. Looks like we will need to fix something. I'll see if I can route the radar three doors down the hall from me faster. As far as the current subject is concerned, the default cancel type is PTHREAD_CANCEL_DEFERRED. It would be handy if Per were to be a little more specific about his execution environment, given that if stdin is not the slave side of a pty, he is potentially in an uninterruptible state blocked in a buggy device driver, rather than being interruptible in the tty code. It would also be handy if he set the type to async and tried it again, so that the person who will be catching that radar doesn't have to chase all those things down. I have updated the bug report with code that reads from a pipe (created with pipe()) and tried both with deferred and asynchronous cancellation type. The behavior is identical, the process hangs forever. This email sent to site_archiver(a)lists.apple.com

1 0

Re: pthread_cancel and cancelation points still broken in Mac OS X 10.5 Leopard?
by Per Mildner 25 Nov '07

25 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Nov 25, 2007, at 2:52 AM, Terry Lambert wrote: ... Regards, _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… As far as the current subject is concerned, the default cancel type is PTHREAD_CANCEL_DEFERRED. It would be handy if Per were to be a little more specific about his execution environment, given that if stdin is not the slave side of a pty, he is potentially in an uninterruptible state blocked in a buggy device driver, rather than being interruptible in the tty code. It would also be handy if he set the type to async and tried it again, so that the person who will be catching that radar doesn't have to chase all those things down. I have updated the bug report with code that reads from a pipe (created with pipe()) and tried both with deferred and asynchronous cancellation type. The behavior is identical, the process hangs forever. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Blocking file access within KAUTH
by Terry Lambert 25 Nov '07

25 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Nov 25, 2007, at 12:30 AM, Damir Dezeljin <mac(a)dezo.org> wrote: Terry, Michael: thanks, your posts were really helpful even if my mail was hard to understand -> was writing just before going to bed :) Anyway I'll try to clarify my use case a bit better: Lets say I'm implementing a virus scanner and I want performing some actions only when (before) the file data is being accessed. So, ideally I would skip all directory listing or chmod-s, renames and other operations not requiring file data. As I understood it is not possible to distinguish between all such operations, but I have to deal with VNODE action bit mask and do my decision depending on results (well, first I have to check if the VNODE is a file ...). As far as good -> I'm going to do some more testing and I guess I'll return with additional questions later on. BTW: I don't really care about the path I get from the VNODE. The only important thing is I will be able to open the resolved path in user space and scan the file. I've been considering a "open vnode vp in process q" function for a while now. There are a number of uses for something like this. If you could file an enhancement request and link it to a real performance issue for you, It'd help considerably. Make sure you indicate that it's a "kernel KPI enhancement request" in the first sentence, as that will get it routed faster. Depending on results, I'll block the file access. I'm concerned about performance and results of my approach for cases Michael described: ---- there isn't an operation actually in progress, but someone wants to know whether it would be permitted. You may want to ignore operations with this bit set for performance reasons, but be aware that it may give odd results. ---- I some not directly related questions to the subject: - I have just one MAC available right now. I'm using it for developing and testing too. This makes it hard tracking down kernel (my KEXT) deadlocks as I had to reset it lot of times already :) I'm wondering if it is possible to debug the kernel remotely using a Linux / Windows box? Possible, but there are no cookboos for it. We generally expect you to have a bunch of macs. 8-). Or is it possible to run Darwin on a normal PC (not MAC) - I'm pretty sure it is, but I don't know if it is possible compiling or just loading KEXT-s on such installations. This works. You have to have all your symbol files etc. over there, but gdb is pretty much gdb, and you don't need GUI code for it. - I'm still confused by means for obtaining certain functions documentations. E.g. I used sources as well, list posts and a PDF to understand what msleep() actually do. Is there anything I could install to get 'man msleep' working Not really, no. We don't have a man(9) section for KPIs. We do have developer documentation at <http://developer.apple.com> as well as tech notes. I did a huge amount of documenting in block comments in the source code for Leopard (think Amit got early access to at least the UBC documentation for his book, since the only thing that really changed there was the per thread creds), but obviously, didn't get to everything. It's likely to be an iterative process. That should help some, particulary for credentials, kauth, and the like. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Blocking file access within KAUTH
by Damir Dezeljin 25 Nov '07

25 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:in-reply-to:mime-version:content-type:references:x-google-sender-auth; bh=YZRwmrXZNZGKcVbn8kNBMgupX0rTL34qxcOgE6wsGNw=; b=t6iyUyyWIENwwoW1mo1CYwjoZp9zHy8S8MYksxeorfdL/2QitIDBHhYetmxpFpKA9A68mDgKIxHlQxv6BtqjJC1TpiM5aYHR61cy2CW4wIpSyIimC/Ufl0Jow1gF3bzVePZyK9DCa3YKfkYdG7S27OdInINc0twERT4QG/w44OQ= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=received:message-id:date:from:sender:to:subject:in-reply-to:mime-version:content-type:references:x-google-sender-auth; b=BRxOfFzhsThI1KXZ8leYP3q+iUnYDKhrTiRCjAuOyZ7p8aFVCQNErPPf8ouS6xzxC6kuCJdogYsd1uh1bFnWpnnz45ZabaOg/sWOdknPtUg2jR3psptzjlTnV/OPDBSAMDCHeS8dS5qPlaD6MmqY4qHYpqRBOeocCZOh308S4EA= _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Blocking file access within KAUTH
by Damir Dezeljin 25 Nov '07

25 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:in-reply-to:mime-version:content-type:references:x-google-sender-auth; bh=DZeeuhjb6dCvvtt7RouJOlS88+16N5gD8GCLWN+7yRE=; b=iJsIB4qJjhd/+RS7IzJhKfPiHGeqYji4mM1s1qIahubqVv+vB+tB17gygIqNSqlLKlxfM5eyjWOUEeL9xCJoJvTWMLjJITQPvExEBYitRXyC0XFbWzXPJ0BuiOlku65KboIZrgmyF28sC3Sgidh1XqlK/5vQa9XdhvzQgR2hNNY= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=received:message-id:date:from:sender:to:subject:in-reply-to:mime-version:content-type:references:x-google-sender-auth; b=V6xjmovg12xa+fKw2GerfAdyGxrfM5o8P8nplL6Xkgt4j6/DhOy9R/BXYwYJHEc24s8E/ZRMcLZbI7KbgaSgjsXh9QCHE//nEp8HJeFYsgZ+J0rjFcCzkInritKUMOuGXB07KjcYVQo5isoDR+K3nQ6sWHWeK3JeW11qg98Ok5g= _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Blocking file access within KAUTH
by Michael Smith 25 Nov '07

25 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com I gave you some bad advice here: On Nov 24, 2007, at 8:23 PM, Michael Smith wrote: = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Do not assume that the path to a file will remain the same over its lifetime. Open files move, and you must be prepared to handle this (you will be notified post-facto via KAUTH_FILEOP_RENAME). You will only receive this notification if the node itself is renamed; if a parent component is renamed (changing the path to the node) you will *not* receive this notification for the node, only for the parent component. Unless you feel like checking the paths to all of the nodes you are currently tracking for this parent, this notification will not help you. Maintaining a shadow copy of part or all of the filesystem namespace is a losing game. I cannot recommend it. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Blocking file access within KAUTH
by Michael Smith 25 Nov '07

25 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Nov 24, 2007, at 3:32 PM, Damir Dezeljin wrote: You want to register in KAUTH_SCOPE_FILEOP. It takes the following arguments: ctx = (vfs_context_t)arg0; vp = (vnode_t)arg1; dvp = (vnode_t)arg2; HTH = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… So it seems I have to use the KAUTH_SCOPE_VNODE scope. However, as I know, it is impossible to 'filter' only open operations (access) in the VNODE scope. Am I right? I'm not sure I understand what you are asking here (did you mean "as far as I know"?). Authorisation for file operations is computed at file open time by the handlers in the KAUTH_SCOPE_VNODE; they are asked "do you want to allow this". KAUTH_SCOPE_FILEOP handlers are told "this has already happened". We do not support revocation of granted authorisation for files, so there is no interface on a per-operation basis (operations are coalesced for a variety of reasons that would make revocation difficult to implement, and applications would deal very badly with revocation in general, so it is not a path that is worth pursuing). Well, my concern is I want to minimize the kernel <-> user space trafic and so I would like to request user-space processing only once per file open lifetime (until the close() ). If you mean that you only want your kernel<->user conversation to happen when an attempt is made to open a file, then you can do this fairly easily. Please, can anyone give me a suggestion how to address this issue? Without knowing what your policy is, I can't tell you how to decide whether you should communicate with your user-space component or not. Here are a few things to consider: - You may have more than one right being requested at a given point in time. Think carefully about whether you care about specific rights. - Check what the vnode you're being passed is, consider vode_isreg(), vnode_isdir(), etc. Think carefully about what will happen when you are passed vnodes of any type (including types you don't know or care about). - Check the path to the vnode. Be prepared to deal with vnodes that don't have paths you understand, or don't have paths at all. - The KAUTH_VNODE_ACCESS bit tells you that the request is advisory - there isn't an operation actually in progress, but someone wants to know whether it would be permitted. You may want to ignore operations with this bit set for performance reasons, but be aware that it may give odd results. - If you don't have an opinion on an operation, return KAUTH_RESULT_DEFER. Do not return KAUTH_RESULT_ALLOW unless you have a specific opinion. Note that the kernel default policy handler for KAUTH_SCOPE_VNODE always has an opinion, and you cannot override it when it decides to deny an operation. Another thing I'm looking for is the posibility to attach some kind of 'file context' to the file on open. This would help a lot as I would be able to do the user-space processing on KAUTH_SCOPE_FILEOP and attach results for all subsequential VNODE callbacks related to the mentioned file. However; I guess this is not possible, but please correct me if I'm wrong. No, this is not possible. There are a number of things that the kernel reserves the right to do with vnodes that would ruin most of the assumptions that you might want to make about vnode pointers passed to scope handlers. If you need to maintain persistent state about open files on a per-process basis, I encourage you to consider using library interposing and doing your work in user space entirely. It may not be sufficient for your needs, but the guarantees associated with open files there are somewhat more solid. Additionally - is there any way how to avoid calling vn_getpath() for every vnode callback (e.g. if it was already called for the opened file)? The namecache makes vn_getpath() quite cheap; don't worry too much about it for KAUTH_SCOPE_VNODE. Note that KAUTH_SCOPE_FILEOP handlers are more performance-sensitive, and the scope handler is given a pointer to the path already. Do not assume that the path to a file will remain the same over its lifetime. Open files move, and you must be prepared to handle this (you will be notified post-facto via KAUTH_FILEOP_RENAME). And the last question for today -> I found two or three different VNODE callback parameters explanations on the internet. Where can I get the 'official' (the correct) one? The definitive source of information is the source code; the system registers a KAUTH_SCOPE_VNODE handler to implement the normal file access checking. This is currently vnode_authorize_callback() in xnu/ bsd/vfs/vfs_subr.c static int vnode_authorize_callback(kauth_cred_t cred, void *idata, kauth_action_t action, uintptr_t arg0, uintptr_t arg1, uintptr_t arg2, uintptr_t arg3) { vfs_context_t ctx; vnode_t vp, dvp; ctx is the vfs_context_t for the operation. This is where you should fetch the process and credential associated with the operation - do not use current_proc() as the request may have come from another task. Likewise, do not use the cred argument - fetch the credential from the vfs_context_t. vp is the vnode pointer for the object against which the operation is being authorised. dvp is the parent directory, for operations (specifically KAUTH_VNODE_DELETE) where rights have to be considered vs. both against the object and its parent. This is often NULL; if you need the parent directory you should use vnode_getparent(). Note that vnodes may have multiple parents; you should be prepared to be given a parent vnode that is not the directory in which your client looked up the file. Likewise there can be multiple paths to a node, so vnode_getpath() may also give you unexpected results. action is a bitmask of KAUTH_VNODE_* rights. For an operation to be permitted, ALL of these rights must be granted. Also, if you feel that the technotes could be improved on, please file a bug with specific comments. Quinn and the DTS guys do a great job of maintaining these, but they can always use your feedback. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Blocking file access within KAUTH
by Terry Lambert 25 Nov '07

25 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com I believe you are looking for the KAUTH_VNODE_ACCESS bit not being set. -- Terry On Nov 24, 2007, at 3:32 PM, Damir Dezeljin <mac(a)dezo.org> wrote: Hi, Thanks and best regards, Damir _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/tlambert%40apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… For your per vnode data: sorry, you have to maintain your own table. On the other hand, you could add a MAC label, but be aware that that's not KPI, you would have to link against the entire kernel each time a software update happens, and it could change out from under you. For vn_getpath(), it really depends on why you think you need the file name. It's best to check which bits are set and just prefilter requests. If you can't do that, then when you get one in, you could always just look it up in your table. For your last question, parameters vary by callback. If we didn't document it, it's probably because you can't rely on them for all calls. By reading the TN2127 and list posts I managed to develop a KEXT 'KAUTH_SCOPE_FILEOP' on OS X 10.4 that send file paths on open to user space and blocks the execution until when the reply is received. This is fine; however, I would need to block certain access to files. As I know (TN2127 states it) this is not possible withing KAUTH_SCOPE_FILEOP as the return value is ignored. So it seems I have to use the KAUTH_SCOPE_VNODE scope. However, as I know, it is impossible to 'filter' only open operations (access) in the VNODE scope. Am I right? Well, my concern is I want to minimize the kernel <-> user space trafic and so I would like to request user- space processing only once per file open lifetime (until the close () ). Please, can anyone give me a suggestion how to address this issue? Another thing I'm looking for is the posibility to attach some kind of 'file context' to the file on open. This would help a lot as I would be able to do the user-space processing on KAUTH_SCOPE_FILEOP and attach results for all subsequential VNODE callbacks related to the mentioned file. However; I guess this is not possible, but please correct me if I'm wrong. Additionally - is there any way how to avoid calling vn_getpath() for every vnode callback (e.g. if it was already called for the opened file)? And the last question for today -> I found two or three different VNODE callback parameters explanations on the internet. Where can I get the 'official' (the correct) one? This email sent to tlambert(a)apple.com This email sent to site_archiver(a)lists.apple.com

1 0

Re: pthread_cancel and cancelation points still broken in Mac OS X 10.5 Leopard?
by Terry Lambert 25 Nov '07

25 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Nov 23, 2007, at 2:00 AM, Per Mildner wrote: Cancellation passes VSTH test suite conformance testing, and is therefore compliant with the letter of the specification. - -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Nov 23, 2007, at 4:00 AM, Steve Checkoway <s(a)pahtak.org> wrote: I do not think any tests could verify conformance, for the same reason no tests can verify the absence of bugs. This was pointed out last time Terry mentioned this, not that I've ever read the standard. =) <http://lists.apple.com/archives/Darwin-dev/2007/Jul/msg00068.html> <http://lists.apple.com/archives/Darwin-dev/2007/Jul/msg00071.html> That conversation was specifically about changing mtime on a file where the metadata and file contents remain unchanged as a result of specifying the O_TRUNC flag to an open call. You have anecdotal support for your contention that we should not short circuit the optimization of not calling the VNOP, and I haven't seen a failing test case show up in either VSU or VSX to make me change my mind about the optimizations correctness (yet). I still think it's pretty weird for the build system you are using for your Linux kernel depends on this behaviour, since it means there are some FS types where this can never work. But if it were failing a test case, it would get fixed more or less instantly. As far as the current subject is concerned, the default cancel type is PTHREAD_CANCEL_DEFERRED. Assuming that stdio is attached to the slave side of a pty rather than a serial console on an XServe or some third party dongle or whatever, the the read is going to be blocked on the bsd wait channel "ttin", with the thread in an interruptible state. I think a strict interpretation of section 2.9.5 of System Interfaces chapter 2 would permit, in the deferred case, the behaviour Per is seeing. I agree, however, that in the async case, the thread should wake up as if pthead_kill() had been called on it. Unfortunately for me, I am on vacation, and as you can probably tell by the spelling "corrections", I am using my phone to access email. That means I can't fix the code by calling pthread_setcanceltype() to asnc to see if the behaviour changes to the one Per wants. If not, then it's a genuine bug in my book, but if so, I expect that we will not change the behaviour of the deferred case. Otherwise, if a fix was needed, we _may_ change the behaviour of the deferred case in fixing the async case. It would be handy if Per were to be a little more specific about his execution environment, given that if stdin is not the slave side of a pty, he is potentially in an uninterruptible state blocked in a buggy device driver, rather than being interruptible in the tty code. It would also be handy if he set the type to async and tried it again, so that the person who will be catching that radar doesn't have to chase all those things down. Given this is a POSIX issue, and I pretty much own kernel "POSIXness", but not threads, I'm pretty sure to be consulted on this when the bug report gets out of DTS and over to the kernel team. Doing the checks ahead of time and putting the data in the radar will keep it from bouncing for more information. This email sent to site_archiver(a)lists.apple.com

1 0

Blocking file access within KAUTH
by Damir Dezeljin 24 Nov '07

24 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:mime-version:content-type:x-google-sender-auth; bh=OzhQvWu7ljnwbNrbBqhoQ1unCLSz+U7bFIRVAj18U4Y=; b=U7xuqHVIJD+HNK0+bTVmo2Iy5s8aG2/I6ycIfIqWEyaxyc3Uvx9qX1kgh+G8O0rgkcQAWSxoM98Lc4mz0O91R/XSNah9P20Wv9lq/2hHZNYEwRc8UUoAqt1nhpLeEJKq7yGZXG2+dKphA5os9R5a95jPfBE4irOzqR5HoH38M1w= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=received:message-id:date:from:sender:to:subject:mime-version:content-type:x-google-sender-auth; b=umeWlDToMeby4XH07eUCY6XjvC7UDfV5mKugscDXvCIG+DYOrpbqdmFRA5UMwi++ZN4gup3Wcxrq8KPLhoCcYh0VP+J94ryLpXGrdIHEVkCxB1mIPRWp1lBQiViKnxVsQvQGHwhahre8cI4L3DM77r4eNG/X1vlPYWHyWaMsk/4= _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0