May 2010 - Darwin-kernel - lists.apple.com

Re: Using OSKextRequestResouce from kext init routine?
by Ken Hornstein 18 May '10

18 May '10

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com >If you find any documentation wanting in detail, please file a radar. >There are limits to what we can cram into HeaderDoc, which is the only >documentation I contribute directly to, but I spent the first half of my >career as a technical writer so I take documentation very seriously and >want to make sure it covers the bases. Any question you have that the >doc doesn't address is a failure that needs to be corrected. You >shouldn't have to experiment with our APIs to figure out how to use >them. Okay ... how do I do that, exactly? (I had some vague impression that radar is the Apple internal bug-tracking system). Just FYI, the two big thins documentation would have been nice for were the aforementioned issue with the callback not being called until after the start routine had exited, and the amazingly difficult question of "Where exactly ARE resource files located inside of a kext?" (The answer: ${KEXTDIR}/Contents/Resources. Hoo boy, that was a tough one to find, and if that is documented somewhere, please let me know, because I couldn't find it!). --Ken _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

changing thread affinity after a thread is started?
by Bill Janssen 18 May '10

18 May '10

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Is it possible to change the affinity of a thread after it's been started? When would such a change take effect? The next time the scheduler looked at it? Bill _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Using OSKextRequestResouce from kext init routine?
by Nik Gervae 17 May '10

17 May '10

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On 14 May, 2010, at 18:52 , Ken Hornstein wrote: >> There should be headerdoc for the function, but I just tried to search >> for in Xcode and got zilch. The comments are in the header, though. > > Right, I read those. But it's a little ... sparse? I mean, yeah, > I worked out how to call it, but some of the details are a little > vague. And I did read the release notes that mentioned this function > so I understand why it was put into place ... and that I'm abusing > it. If you find any documentation wanting in detail, please file a radar. There are limits to what we can cram into HeaderDoc, which is the only documentation I contribute directly to, but I spent the first half of my career as a technical writer so I take documentation very seriously and want to make sure it covers the bases. Any question you have that the doc doesn't address is a failure that needs to be corrected. You shouldn't have to experiment with our APIs to figure out how to use them. >> If you have a BSD kext that requirest a single moderately-sized resource >> to start, it isn't so bad to just include the data in your executable. > > Sigh. If only it was that simple. > > I'm in a weird spot. I'm porting a Linux network filesystem to > MacOS X. Yes, that is almost insane as it sounds. [...] Michael Smith is providing better advice than I could on this particular problem. Nik _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Using OSKextRequestResouce from kext init routine?
by Ken Hornstein 16 May '10

16 May '10

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com >The general reason for not doing file I/O from inside the kernel is that >it's just not designed to work that way. File I/O is something that >user processes do, or more specifically that parts of the kernel do on >behalf of user processes. Trying to do it from other parts of the >kernel can lead to surprising (the bad kind) results. It's quite >difficult to do right, and there is almost always a better way that >moves the file I/O into userspace. You're presuming that I'm not doing that already, and you'd be wrong; these kexts already write directly to files. And the fact that Apple already provides a supported series of kpi's to do exactly that really makes me think that Apple realized that it can occasionally be useful. I can tell you from direct experience that it's not difficult to do it right: it's actually pretty straightforward, and I have not run into any problems. I will agree that it's not ideal, but the alternative would involve writing a lot of new code that I am, quite frankly, not that interested in writing. > - Have all of the 15 kexts start routines just check in with a >'manager' of some sort, and have the 'manager' handle fetching your >properties and sequencing the 'real' startup routines correctly. > > - Link all 15 kexts together into one mondo kext. Have a 'true' start >routine that handles property fetches and sequencing of the >initialisation of the other kexts. Yeah ... I'm not going to do either of those things. Why? Well, it's a lot of new code I would have to write. The only gain I can see to writing all of this new could would be that I would avoid having to write to and from files directly from my kext. Since I already know how to do that, just reading in the module options seems like the simplest option. I agree that it's not ideal, but sometimes thats just the way it works out. --Ken _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Using OSKextRequestResouce from kext init routine?
by Michael Smith 16 May '10

16 May '10

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Using OSKextRequestResouce from kext init routine?
by Ken Hornstein 15 May '10

15 May '10

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com >> It occurs to me, though, that maybe I'm making this harder than I need to. >>I mean, I could simply call the appropriate vnode routines to read in >> the file and parse it myself. In a perfect world I could parse a prefs >> XML file from the kernel (or have it parsed for me by a userspace daemon), >> but somehow I don't see that happening anytime soon :-/ > >That's very bad, don't do that. How big are your preferences? Can you >make an I/O Kit kext for your filesystem (match on IOBSD) and get your >prefs via the kext's info.plist? Um, why is that bad, actually? All of the routines to read a file from a kext are exported, supported routines (vnode_open() and vn_rdwr() are part of the BSD kpi). I suppose there might be some architectural purity reasons why I shouldn't, but honestly ... I'm already so far off of the reservation here, architectural purity is the least of my concerns. I have no objection to making an I/O Kit kext, but it's not clear to me how exactly I do that (nor is it clear to me how doing that means I get access to the kext's Info.plist). Unless .... are you talking about putting the stuff under IOKitPersonalities. Phew ... that seems like a lot of code. (I haven't really looked at I/O Kit very much, since it's all focused on hardware). Unfortunately, the documentation about I/O Kit seems large, and unless there's a good example I don't think I have the energy to look through it all, understand it, and make a lot of changes when it's for a relatively small section of code that I need. >Is there any reason you can't defer your registration, so use >OSKextRequestResource but use the completion context rather than the >context that called your start routine to complete your startup? I actually tried doing that ... but it's complicated. The main issue here is kext ordering. Right or wrong, the filesystem doesn't consist of one kext ... there are actually 15. I can't take the blame for that one; that's just how it is organized. Certain kext's need to load _and start_ before others. Right now that's handled by the dependencies in the kext's Info.plist. But when I tried running the kext initializations from the OSKextRequestResource callback, I started getting kernel panics, and I traced those down to the kext's trying to use uninitialized resources. I suppose I could do some dependency tracking myself ... but that seems like a lot of work, again, for relatively little gain. --Ken _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Using OSKextRequestResouce from kext init routine?
by Michael Smith 15 May '10

15 May '10

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Using OSKextRequestResouce from kext init routine?
by Ken Hornstein 15 May '10

15 May '10

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com >There should be headerdoc for the function, but I just tried to search >for in Xcode and got zilch. The comments are in the header, though. Right, I read those. But it's a little ... sparse? I mean, yeah, I worked out how to call it, but some of the details are a little vague. And I did read the release notes that mentioned this function so I understand why it was put into place ... and that I'm abusing it. >If you have a BSD kext that requirest a single moderately-sized resource >to start, it isn't so bad to just include the data in your executable. Sigh. If only it was that simple. I'm in a weird spot. I'm porting a Linux network filesystem to MacOS X. Yes, that is almost insane as it sounds. I actually have something working, but I'm going back and cleaning up some of the details that I skipped. One of those details is that the modules that make up this filesystem make use of the Linux kernel module parameter feature, where you can declare module parameters in your source code and then set those parameters in files in /etc/modprobe.d. I will spare everyone the details of how that's actually implemented in Linux; suffice it to say that my solution involves a perl script that processes all of the source files. Now, I've read the archives, and I understand that everyone is saying that you should _not_ have a preferences file for kexts; a userspace program should set those parameters at startup time, probably via sysctl. But ... one of my goals is to actually get these changes push back into the original source tree. If those changes involve restructuring whole piles of the original source code ... well, the chances of those changes actually making it back in would be approximately nil. (It's not that the authors of the filesystem in question don't care about portability; they've made some strides in the portability department, but the more complex the change, the harder it is to get it pushed into their tree). So, I'm kinda stuck when it comes to large restructuring; if MacOS X does something a whole lot different than Linux, it's my job to make it damn well work with the minimum changes possible to the core code. And to top it all off, there is a bunch of initialization that happens at module startup time; some of the initialization depends on key options that need to be set before the initialization routines get run (some options can be set at runtime, and I already support sysctl access for those). Obviously I have reasonable defaults, but sometimes those aren't good enough. If I was writing this from scratch, yeah, it would be done completely differently ... but that simply isn't in the cards. It occurs to me, though, that maybe I'm making this harder than I need to. I mean, I could simply call the appropriate vnode routines to read in the file and parse it myself. In a perfect world I could parse a prefs XML file from the kernel (or have it parsed for me by a userspace daemon), but somehow I don't see that happening anytime soon :-/ --Ken _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Using OSKextRequestResouce from kext init routine?
by Nik Gervae 14 May '10

14 May '10

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com There should be headerdoc for the function, but I just tried to search for in Xcode and got zilch. The comments are in the header, though. As for your problem, OSKextRequestResource() is asynchronous (as any up-call from the kernel pretty much has to be). All it does is queue a request for kextd to fulfill. However, kext start & stop routines are synchrounous with regard to kext load and unload, so if your kext is being loaded by kextd, kextd won't even see the resource request until your kext has returned from its start routine. It's perfectly fine to call OSKextRequestResource() from your kext start routine, but as you noted, you cannot expect the request to be fulfilled within the scope of that call, so you can't base success/failure of the start routine on the availability of a resource (which I am guessing is what you want to do). You can log an error from the resource callback, and even arrange for your kext to be autounloaded, but you cannot cause the load itself to fail on an unavailable resource. Even if we modified kextd to be all multithreaded and able to take simultaneous requests from the kernel, we couldn't guarantee execution time on any requests received. (More-than-you-probably-care-to-know alert.) This feature was designed to handle the issue of drivers with huge firmware blobs in their personalities, which consume kernel wired memory whether the drivers are loaded or not (and worse, get copied for each instance of the driver class created if they do get loaded). A fully general, synchronous-capable solution that worked during early boot (before kextd) as well as later would have been prohibitively expensive, requiring kexts to name which resource files might be needed during a load, the booter to look for those resources and copy them into memory, mkext & kernelcache to be built including those resource, the kernel to find them and catalog them in case a kext should need one, and kext load operations to also grab them and send them down as part of the load operation. The code that would be involved and regressions in boot time and (albeit temporary) wired memory made that simply infeasible. (Whew!) If you have a BSD kext that requirest a single moderately-sized resource to start, it isn't so bad to just include the data in your executable. Nik Gervae The Guy Who Wrote That On 14 May, 2010, at 11:48 , Ken Hornstein wrote: > I'm having an issue with using /(), specifically from > my kext initialization routine. > > This function is unfortunately not well documented (hey, if there's > documentation for it, point me to it!), but ... it seems like you can't > use it from a kext initialization routine. More specifically, you > can't expect the callback to be called until after your initialization > routine exits (actually, it's worse than that, from what I can tell; I > have a bunch of kexts that depend on each other, and none of the > callbacks are called until after they all finish loading and are > started). Is my understanding correct? (I would guess that's more of > a limitation in kextd). > > --Ken > _______________________________________________ > Do not post admin requests to the list. They will be ignored. > Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) > Help/Unsubscribe/Update your Subscription: > http://lists.apple.com/mailman/options/darwin-kernel/ngervae%40apple.com > > This email sent to ngervae(a)apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: problems with modifying payload in iff_input_func
by Brendan Creane 14 May '10

14 May '10

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:date:message-id :subject:from:to:content-type:content-transfer-encoding; bh=u2YeCv2OQ3BHYJxncZjEfxdX0c2MNvjhr7iC96b9Vik=; b=t33DWbIOIqSpXHTwSzWdYFBZDpE3epMK2ERS38VySnmpCh5ImSUOtUzTh/Lq1HZGn4 j8xCcg2PD7zZaeabq0rA1ttLq+IRfmoZ3mI4gtpnRHnGkqgaTFNYxGRRHc7uYRKU4w/E hbcMvm2xtWl/KjO3h8vtvzOh3yB3So9WDwSgw= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type :content-transfer-encoding; b=oHPbjXEtl1QD1ovnvemRE0svNRE5iAiCCcPVv84+XWRKENsHMxSgyhnOsZSNeebk5D ew7IiRFMtUluaE8D1yCsXeFJXi6hUTJ1z7O9vAM2KOOGBNIhd0Tw/n40xLzLamPJK65z rBssGSqd89yK/mMeXnJzoAE4PhvOy2T0s2ezk= I believe mbuf_copyback() will return a partial mbuf list if it fails to allocate enough buffer space for the modified packet. Since you're not passing in MBUF_WAITOK, I'm guessing you're leaking mbufs rapidly. Try calling mbuf_freem() if copyback fails. You can also get some insight into the failure using two machine debugging via ethernet or firewire. regards, Brendan Creane On Fri, May 14, 2010 at 12:02 PM, <darwin-kernel-request(a)lists.apple.com> wrote: > Send Darwin-kernel mailing list submissions to > darwin-kernel(a)lists.apple.com > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.apple.com/mailman/listinfo/darwin-kernel > or, via email, send a message with subject or body 'help' to > darwin-kernel-request(a)lists.apple.com > > You can reach the person managing the list at > darwin-kernel-owner(a)lists.apple.com > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Darwin-kernel digest..." > > Today's Topics: > > 1. Re: processor_info x86 perf. counters (Wade Tregaskis) > 2. problems with modifying payload in iff_input_func (Andy Huang) > 3. Using OSKextRequestResouce from kext init routine? (Ken Hornstein) > > > ---------- Forwarded message ---------- > From: Wade Tregaskis <wadeslists(a)mac.com> > To: ray dhurandhar <raydhurandhar(a)gmail.com> > Date: Thu, 13 May 2010 12:09:49 -0700 > Subject: Re: processor_info x86 perf. counters >> What's the best way to access the performance counters programmatically ? > > It's not exactly programmatic, or at least not necessarily in the way you mean, but, Shark. Note however that Shark will not work on the open source kernel. > > Wade > > > > ---------- Forwarded message ---------- > From: Andy Huang <andy.y.huang(a)gmail.com> > To: darwin-kernel(a)lists.apple.com > Date: Thu, 13 May 2010 14:35:07 -0500 > Subject: problems with modifying payload in iff_input_func > Hi, > > I am trying to modify enetlognke NKE, and in it I try to modify the payload inside enet_input_func(iff_input_ > func). Here is my code: > static errno_t > enet_input_func(void* cookie, ifnet_t interface, protocol_family_t protocol, > mbuf_t *data, char **frame_ptr) > { > > mbuf_t m = *data; > u_int32_t mbufs = 0, bytes = 0, ip_hdr_len = 0, ip_pkt_len = 0; > errno_t err; > struct ip ip_hdr, *pIpHdr; > char buf[BUF_SIZE]; > > /* check whether we have seen this packet previously */ > if (CheckTag(*data, INBOUND_DONE)) > { > /* we have processed this packet previously since the INBOUND_DONE bit was set. > bail on further processing > */ > return 0; > } > > if (protocol != AF_INET) > { > //printf("protocol = %d, don't process it\n", protocol); > return 0; > } > > err = SetTag(*data, INBOUND_DONE); > if (err != 0) > { > printf("Error - mbuf_tag_allocate returned an error %d\n", err); > return err; // stops processing and will cause the mbuf_t to be released > } > > do > { > mbufs++; > bytes += mbuf_len(m); > m = mbuf_next(m); > } while (m != NULL); > > if (bytes < BUF_SIZE) // BUF_SIZE is the size of 4096 > { > printf("bytes(%d) < BUF_SIZE, don't process it\n", bytes); > return 0; > } > > err = mbuf_copydata(*data, 0, bytes, buf); > if (err != 0) > { > printf("Error - mbuf_copydata failed\n"); > return 0; > } > > // modify the payload in the buf > .......... > > // copy back > err = mbuf_copyback(*data, 0, bytes, buf, MBUF_DONTWAIT); > if (err != 0) > { > printf("Error - mbuf_copyback for %d bytes returned an error %d\n", bytes, err); > return err; > } > > return 0; > } > > The code works most of the time. But when the mac (10.6.3) goes to sleep and then wake up, after a while, it goes to either freeze or panic. > Any help will be greatly appreciated. > > ---------- Forwarded message ---------- > From: Ken Hornstein <kenh(a)pobox.com> > To: darwin-kernel(a)lists.apple.com > Date: Fri, 14 May 2010 14:48:47 -0400 > Subject: Using OSKextRequestResouce from kext init routine? > I'm having an issue with using OSKextRequestResouce(), specifically from > my kext initialization routine. > > This function is unfortunately not well documented (hey, if there's > documentation for it, point me to it!), but ... it seems like you can't > use it from a kext initialization routine. More specifically, you > can't expect the callback to be called until after your initialization > routine exits (actually, it's worse than that, from what I can tell; I > have a bunch of kexts that depend on each other, and none of the > callbacks are called until after they all finish loading and are > started). Is my understanding correct? (I would guess that's more of > a limitation in kextd). > > --Ken > > > _______________________________________________ > Darwin-kernel mailing list > Darwin-kernel(a)lists.apple.com > http://lists.apple.com/mailman/listinfo/darwin-kernel > _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0