March 2010 - Darwin-kernel - lists.apple.com

PackageMaker 3.0.4 and Installing Kexts
by Jonathan Lott 04 Mar '10

04 Mar '10

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Thanks _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… For some reason when I create a package to install a kext, when testing it out in the command line I get the following message: The System extension was installed improperly and cannot be used. Please try ireinstalling it, I have made sure that the permissions were set properly in the packagemaker doc, but it seems that when it is installed, these priveleges aren't being set during the install. Can someone tell me what to do. I did see a radar about this http://openradar.appspot.com/7253630 , but I need some answers as to how to work around this issue. Is packagemaker just not up to it? This email sent to site_archiver(a)lists.apple.com

1 0

Re: Memory barriers & the kernel
by Dave Keck 04 Mar '10

04 Mar '10

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :from:date:message-id:subject:to:content-type :content-transfer-encoding; bh=t4Hs8oAzzG20WKuJzhQQhwbvFJ0zrZJRrwAGWSqrWEY=; b=U2ycZP/ef3bJdNRKOXKRJyZ4STpPMqAFbrgOOTx3Agm65whiyzbdQnT0Ub96kLKqgk sHYBeAKbscPgN6Dk46hr3TrL+cg8aEH2WdWjZ24QU8nLcHGntDAubcPrxFa+TsUZwq+3 iZ32dbgHx77yioGXaeGymo1/edtRkvSdfvrHQ= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:content-transfer-encoding; b=k+ioX8U9Z0cvUZ8XUZWKwox9HjNekeYshkfWL+4/BkLwtknoH35hWxpvuPgx/Yv6Ha JW9nf4pHZFB22li2WObNqaRqV1pCJC39dLFRK8MAB4dnPih5Eo+aubJ0GijltZu/xHAo AOiZIjaPYbJNeuqmuHYXuPoYbE6zq/ch5DJoY= After further digging, I believe I've found the answers to my questions. I'm documenting my findings here for the sake of the archives. 1. The kernel's mutex facilities do indeed include memory barriers (at least the one I've looked at anyway - lck_mtx_t.) The implementation of lck_mtx_lock() can be seen here: http://opensource.apple.com/source/xnu/xnu-1456.1.26/osfmk/i386/i386_lock.s The core mechanism is the lock/cmpxchg instruction. As stated by Intel's 64 and IA-32 Architectures Software Developer’s Manual, Volume 3A, section 8.1.2.2 (http://www.intel.com/Assets/PDF/manual/253668.pdf) "Locked instructions can be used to synchronize data written by one processor and read by another processor. For the P6 family processors, locked operations serialize all outstanding load and store operations (that is, wait for them to complete)." 2. The OSAtomic functions in kernel-space also include a barrier, which is mentioned in the 10.6 version of OSAtomic.h. (I was reading the 10.5 header in my original email.) The kernel-space atomic functions are defined here: http://opensource.apple.com/source/xnu/xnu-1456.1.26/libkern/i386/OSAtomic.s Note the lock/cmpxchgl CAS instructions. Furthermore, I found that both the barrier and non-barrier OSAtomic functions in user-space actually both use a barrier. The user-space atomic functions are defined here: http://opensource.apple.com/source/Libc/Libc-583/i386/sys/OSAtomic.s (Note the DECLARE macro at the beginning, which is the mechanism for combining the barrier and non-barrier versions of the various atomic functions.) This file references functions in the commpage: http://opensource.apple.com/source/xnu/xnu-1456.1.26/osfmk/i386/commpage/at… ... and once again we find our lock/cmpxchgl. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Upgrade Kext from 10.5 to 10.6 - mac_vnop and KUNC
by Terry Lambert 03 Mar '10

03 Mar '10

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com I am trying to get a Kext that I had working on 10.5 to work on 10.6. It's a research system to detect flow from processes to files in the system using KAUTH to detect events, extended attribtutes to mark files, and KUNC to notify the users on certain events (sometimes preventing execution before it occurs). It works on 10.5.8 how I want it to, never crashes, etc. When trying to move it onto 10.6 it is failing to load and I am getting these errors when I run "kextibs -all-symbols" on it - 3 symbols not found in any library _KUNCUserNotificationDisplayAlert _mac_vnop_getxattr _mac_vnop_setxattr I know that none of these things are actually supported, but since this is just for limited use, is there any way around this problem? -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Mar 3, 2010, at 12:32 PM, Chris Gates wrote: There are no errors when building. I imagine that you were previously linking your KEXT vs. the entire kernel to get at all the kernel symbols, which is no longer supported. My guess is that you are now working with a 64 bit kernel? KUNCUserNotificationDisplayAlert is exported by System6.0.exports and Unsupported.exports You can get to this by declaring appropriate dependencies for your KEXT (OSBundleLibraries / com.apple.kpi.unsupported - String - 10,0,0d2). Note that "Unsupported" is code for "this is going away in a future release, so please stop using this API and build your own user agent daemon to talk to your driver instead". mac_vnop_getxattr and mac_vnop_setxattr are exported by MACFramework.exports You will not be able to get to this; the MACF KPI is private and currently unpublished, and subject to change or removal in future releases. Your best bet is to build your own kernel, including your code in the kernel proper so that it can access private symbols like this. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Upgrade Kext from 10.5 to 10.6 - mac_vnop and KUNC
by Nik Gervae 03 Mar '10

03 Mar '10

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com man kextlibs: -unsupported Search unsupported library kexts for symbols (by default they are not searched). That will get you your unsupported kext libraries, if they still define those symbols. You can also use kextfind's -defines-symbol (-dsym) predicate: kextfind -defines-symbol _mac_vnop_getxattr Nik Gervae Kext Management Engineer Apple Inc. On 03 Mar, 2010, at 12:32 , Chris Gates wrote: > I am trying to get a Kext that I had working on 10.5 to work on 10.6. > > It's a research system to detect flow from processes to files in the > system using KAUTH to detect events, extended attribtutes to mark > files, and KUNC to notify the users on certain events (sometimes > preventing execution before it occurs). It works on 10.5.8 how I want > it to, never crashes, etc. > > When trying to move it onto 10.6 it is failing to load and I am > getting these errors when I run "kextibs -all-symbols" on it - > 3 symbols not found in any library > _KUNCUserNotificationDisplayAlert > _mac_vnop_getxattr > _mac_vnop_setxattr > > I know that none of these things are actually supported, but since > this is just for limited use, is there any way around this problem? > > There are no errors when building. > > Thanks for any info - > Chris > _______________________________________________ > Do not post admin requests to the list. They will be ignored. > Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) > Help/Unsubscribe/Update your Subscription: > http://lists.apple.com/mailman/options/darwin-kernel/ngervae%40apple.com > > This email sent to ngervae(a)apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Upgrade Kext from 10.5 to 10.6 - mac_vnop and KUNC
by Chris Gates 03 Mar '10

03 Mar '10

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=twWbRzt0Ydk6n383pChiwiqscjliH5Y8zt+nLPFhf/8=; b=gs6mkyzjbMb2ijpQeQ0tRqmOE2B3JMIAYh+U4sjZeE/1sW6qFB9gqOwKY7xgivSMsp CaGECtNvftTIuicdL6DzaFbfWpy9OnXWvBWECf6XkfkDUdEbyWtwrxLgt5GAM3ldvAPI mSkSSliPCbtJok6+cCogVZiIS884/YTPnWmRM= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=lQLchhdtLLAszY4feNlbGXF1YFPneQ8bEoaUU/PrfkqoKVm8E/uSKTGGsRPfV7oyIl A/bPDaKd3ZO4rTsXpxqeNr2I5QJKAtZJXbYAxD47L8HKaAU6h4YHaq/xERSME/p1pmlD xzlQEEs7IlSpQUAp22Nv9XJ9J5UzAcrdvW2A8= I am trying to get a Kext that I had working on 10.5 to work on 10.6. It's a research system to detect flow from processes to files in the system using KAUTH to detect events, extended attribtutes to mark files, and KUNC to notify the users on certain events (sometimes preventing execution before it occurs). It works on 10.5.8 how I want it to, never crashes, etc. When trying to move it onto 10.6 it is failing to load and I am getting these errors when I run "kextibs -all-symbols" on it - 3 symbols not found in any library _KUNCUserNotificationDisplayAlert _mac_vnop_getxattr _mac_vnop_setxattr I know that none of these things are actually supported, but since this is just for limited use, is there any way around this problem? There are no errors when building. Thanks for any info - Chris _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Memory barriers & the kernel
by Dave Keck 03 Mar '10

03 Mar '10

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:from:date:message-id :subject:to:content-type; bh=KjSMV3qbLwNyn2Z+DC7nLTTNcWQMrzlkPit8AfKWDhU=; b=Tuv2Jt5iXfmfTnfrTTjt66vg6Hl8QC7oFTCDp6klHlvmBClQIEpgMGcZLm+vEKdj9s Uu0FqF3Rt5AtQWtstD28mZ/s5zQKMiVNzF84BUWT2ONcdrXwJge2eTgqQTHhVq96nibO JsQZB6qQvnL1Qk1eEeBkbDDg4UOLb0v2w/zkw= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:from:date:message-id:subject:to:content-type; b=wPs+/5QiUonHISHDo3xjfn+MDr9FgVhrAXYpSnpP1zVXBtf+1KbdZH7XaFCBp6A/9k uE0/ZNhjkUuW9wgwFJaRaMgm/cpgrIqqJRTHOQF5EEOgG6rcEZg3Vu+8F7H49u1DvYBN 8CpCqUeb9Crlpo9maS3twHQfc9+gWv5zFDLn0= Hey list, Coming from userland I'm accustomed to the implicit memory barriers across mutexes as well as the OSAtomic functions. In kernel documentation though, there are hardly any mentions of barriers. (Particularly confusing is their omission from the kernel's version of OSAtomic.h, while they exist in the userspace version.) Do kernel-space mutexes and OSAtomic functions employ barriers? If not, isn't that unsafe? Thanks for your help! David _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Possible bug with nanosleep()?
by William Kucharski 02 Mar '10

02 Mar '10

site_archiver(a)lists.apple.com Delivered-To: Darwin-kernel(a)lists.apple.com > I tried to log into bugreport.apple.com with my apple ID, but I get this message: > > You do not have access to the Apple Bug Reporter (aka RadarWeb). > Please contact the IS&T HelpLine. > Dial 4-7777 from any Cupertino campus phone. > Dial 7777 from any Sacramento campus phone. > Dial 4-HELP (4357) from any Austin campus phone. > Dial 1-800-800-6272 from anywhere in the U.S. > Dial 1-408-974-7777 from SCV or outside the U.S. You need to create a (free) ADC Online account to use Apple Bug Reporter: <http://developer.apple.com/products/membership.html> William Kucharski _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Possible bug with nanosleep()?
by Chris Wilson 02 Mar '10

02 Mar '10

site_archiver(a)lists.apple.com Delivered-To: Darwin-kernel(a)lists.apple.com User-agent: Alpine 1.00 (DEB 882 2007-12-20) Hi Terry, On Tue, 2 Mar 2010, Terry Lambert wrote: (A) Have to ask for a very short sleep You do not have access to the Apple Bug Reporter (aka RadarWeb). Please contact the IS&T HelpLine. Dial 4-7777 from any Cupertino campus phone. Dial 7777 from any Sacramento campus phone. Dial 4-HELP (4357) from any Austin campus phone. Dial 1-800-800-6272 from anywhere in the U.S. Dial 1-408-974-7777 from SCV or outside the U.S. Cheers, Chris. -- _ ___ __ _ / __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Any ideas why I'm seeing (1<<32)-1 in tr.tv_sec after the call, when the call finishes late? This is the actual problem that I'm having. The base problem causing the issue is the signal handler taking a very long time to run in the test code. Specifically, the idea of the 'remain' in the libc function in the nanosleep() implementation is a result of a call to clock_get_time(), which occurs after the SEMWAIT_SIGNAL(). If an EINTR happens there as the result of the signal, the EINTR will not return from the semwait until after the signal trampoline has been run, which will take however long the signal trampoline takes to run. Since the current time used to calculate the remainder time is polled non-atomically with regard to the semwait, using the separate clock_get_time, the remaining delta can be off by the amount of time that it took to run the handler. If this number is larger than the initial timespec time request, then you can "go backwards". So as your code is currently written, you: (B) Have a registered signal handler for the interrupting signal which takes longer than the (remaining) very short sleep to run. Consequently, the SUB_MACH_TIMESPEC() subtracts 1 from the second as a borrow for the tv_nsec, and underflows. I can't see anything in the spec that forbids tv_sec from having a negative value, and its type in Darwin is long, which is signed. So it's not necessarily a bug per se that it would contain a negative value in this case. The bug is that it does not; it contains (1<<32)-1 which is a very large positive value. The usual suggestion for (A) is to not loop so tightly, or, if you are doing the work in another thread than the one you are observing the timing on, by injecting a thread via the test harness, then block all signals on it. I don't think I am using threads where this bug is triggered, but I do have a signal handler installed (which just sets a flag and does nothing else, so should not qualify for B). And the minimum sleep supported by my safe_sleep function is 1 second. But if the nanosleep call returned just before expiry, with a very small remaining time, then I would sleep again for a very short time. Did you manage to reproduce the issue in the end, or do I need to continue work on a better, self-contained test case using nanosleep and signals (not just the integer overflow demonstration)? For now, I would suggest you look at the BAD_MACH_TIMESPEC() macro, and post-test the remainder from the EINTR. IIf it comes up bad, then it means that you've actually spent more time than requested in the nanosleep() plus the signal handler(s), and you can just pretend it returned normally... I'd prefer not to do anything non-portable here, so I think I might just call gettimeofday() instead if that's OK. Meanwhile, you should file a bug, and include this conversation as part of your description. The component it needs to be filed against is Libc. It's not clear to me where or how to file a bug with Apple, as I'm not a customer and I don't have a support contract. People are lending me remote access to their machines to test the software on and fix bugs, so I don't have the serial numbers of their machines. I tried to log into bugreport.apple.com with my apple ID, but I get this message: This email sent to site_archiver(a)lists.apple.com

1 0

Re: Possible bug with nanosleep()?
by Terry Lambert 02 Mar '10

02 Mar '10

site_archiver(a)lists.apple.com Delivered-To: Darwin-kernel(a)lists.apple.com <http://www.opengroup.org/onlinepubs/009695399/functions/nanosleep.html> -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Mar 2, 2010, at 12:59 AM, Chris Wilson wrote: I agree, this is not a real-time application. What I meant is that I'd really prefer it if my nanosleep was not interrupted by signals, and if it is, I just want to sleep again until the appointed time. I don't want to mess about with signal masks as nanosleep might (now or in future) be internally implemented using signals on some platforms (I've seen this with sleep()). FWIW: To this specific question, POSIX requires that nanosleep() be non-intrusive of the signals function, so you are guaranteed that it will not be internally implemented that way on any POSIX compliant platform; see paragraph 2 of the Rationale section at: Oh, and please report the bug (per my previous email); can't fix it if it isn't reported! This email sent to site_archiver(a)lists.apple.com

1 0

Re: Possible bug with nanosleep()?
by Terry Lambert 02 Mar '10

02 Mar '10

site_archiver(a)lists.apple.com Delivered-To: Darwin-kernel(a)lists.apple.com Hi Terry and all, [ ... code elided ... ] int32_t secs = (int32_t) ts.tv_sec; Very interesting issue. 8-). (A) Have to ask for a very short sleep Hope that helps, -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Mar 2, 2010, at 12:26 AM, Chris Wilson wrote: On Mon, 1 Mar 2010, Terry Lambert wrote: You need to be looking at tr, not ts, or you need to do the structure assign immediately after the EINTR, before your trace statements. The tr contains the remainder time, the ts structure contents are irrelevant after the nanosleep() call. You're BOX_ macros appear to indicate you wanted the remainder time. Sorry, my mistake, my code does that but I edited it somewhat while writing the email and forgot to update that part of the sample that I pasted. For reference, this is the code I have in the application now: And if I change both int32_t to long (or __darwin_time_t, the type used in the structure) on this line: then it hangs forever on OSX. OK, there is a possible scenario which will cause it to drift, and my modification of your original test function to put it into a working test harness wouldn't have (didn't) trigger it. I've modified my initial test harness to cause the problem to reproduce (moral: always include fully working example code when noting a problem to a mailing list). The comment in the Libc code about the layout isn't strictly correct. Specifically, the ADD_MACH_TIMESPEC() and SUB_MACH_TIMESPEC() are macros, and the marcros only really care about having a corresponding field name for tv_nsec and tv_usec in the structures, so the difference in element size in the structure isn't going to matter, at least until we hit Y2038. By that time I expect that the mach_timespec_t will be using a 64 bit tv_sec value (unless you are running your clock forward to do Y2038 compliance testing, in which case, nanosleep() isn't currently Y2038 compliant because clock_get_time() isn't Y2038 compliant, even when doing 64 bit programs). Actually I originally wrote the code to pass the same structure for both, and was wondering whether that was allowed (it's not documented as supported or unsupported) so I wanted to separate the two to make sure that people wouldn't claim that I was using the function inappropriately and thus ignore the rest of my message. Even then, this is probably a bad use of signals, since multiple signals being sent won't necessarily result in multiple notifications. I'm not interested at all in signals in this code, I wish they wouldn't happen, I just want to sleep for the appointed time and nothing else. Any ideas why I'm seeing (1<<32)-1 in tr.tv_sec after the call, when the call finishes late? This is the actual problem that I'm having. The base problem causing the issue is the signal handler taking a very long time to run in the test code. Specifically, the idea of the 'remain' in the libc function in the nanosleep() implementation is a result of a call to clock_get_time(), which occurs after the SEMWAIT_SIGNAL(). If an EINTR happens there as the result of the signal, the EINTR will not return from the semwait until after the signal trampoline has been run, which will take however long the signal trampoline takes to run. Since the current time used to calculate the remainder time is polled non-atomically with regard to the semwait, using the separate clock_get_time, the remaining delta can be off by the amount of time that it took to run the handler. If this number is larger than the initial timespec time request, then you can "go backwards". So as your code is currently written, you: (B) Have a registered signal handler for the interrupting signal which takes longer than the (remaining) very short sleep to run. Consequently, the SUB_MACH_TIMESPEC() subtracts 1 from the second as a borrow for the tv_nsec, and underflows. The usual suggestion for (A) is to not loop so tightly, or, if you are doing the work in another thread than the one you are observing the timing on, by injecting a thread via the test harness, then block all signals on it. Asking for a really tight loop, though, would still leave you racing up to the expiration time because of (B). The usual suggestion for (B) is to only set a volatile variable in any signal handler, and then examine it in the main thread, instead of doing the work on the signal handler. It would be a very tiny window then, but given the code, you'd still have one, since you're not going to hit the trampoline in 0 instructions. POSIX specifically states "The suspension time may be longer than requested" for resolution and scheduling reasons, so that's not entriely incorrrect. For now, I would suggest you look at the BAD_MACH_TIMESPEC() macro, and post-test the remainder from the EINTR. IIf it comes up bad, then it means that you've actually spent more time than requested in the nanosleep() plus the signal handler(s), and you can just pretend it returned normally, without the EINTR. I'd caution you that if a signal handler is taking long enough to run that it's causing you to see this, then it's probably also throwing off any elapsed time you are measuring. If all you are doing is delaying "for at least this long", then that's not an issue for you. Meanwhile, you should file a bug, and include this conversation as part of your description. The component it needs to be filed against is Libc. This email sent to site_archiver(a)lists.apple.com

1 0