August 2007 - Darwin-kernel

Re: mbuf_pullup practically useless
by Platon Fomichev 27 Aug '07

27 Aug '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Thanks Vincent Best regards, Stauff__ On Aug 27, 2007, at 8:40 AM, Vincent Lubet wrote: Platon, Thanks, Vincent On Aug 26, 2007, at 2:19 PM, Platon Fomichev wrote: That's my lament is like this: _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… I will surely do this. Btw as a side note questions - are there any plans to include OpenBSD Packet Filter (quite a good piece of software imho) into Mac OS? I am thinking of doing some initial investigations on possibility of this project if no-one is already on this. As another side-note I am constantly in question why does ./configure like scripts take an exceptionally big amount of time on even most powerful Mac Pro's compared to FreeBSD? Is this related to fork()/exec () or sh port itself or something else? Anyone ever did some research on this? As the last one note (possibly offtopic sorry) - I am curious why ASMx86 SHA1/MD5, etc optimizations are not present in OpenSSL in Mac - they are considered to be rock solid and give a tremendous amount of improvements on any x86 macintosh. We -- Apple engineers lurking on the darwin mailing list -- really do appreciate suggestions like yours. The best way to make sure your enhancement request does fall through the crack is to file a problem report at http:// bugreporter.apple.com/ so that it track into Apple's bug tracking database. To ensure the problem report is routed to the correct component you can always send a note to the mailing list with the problem ID number. This is more or less a generic lamentation not a question. I've been doing quite a lot of KEXT coding recently (NKE mostly interface filter) and thus I had to analyze packets. One of the functions that I've used was mbuf_pullup. Here's its description: Move the next len bytes in to mbuf from other mbufs in the chain. This is commonly used to get the IP and TCP or UDP header contiguous in the first mbuf. If mbuf_pullup fails, the entire mbuf chain will be freed. And I used it exactly for that - to get IP/TCP(UDP) header contiguous in the first mbuf. To my surprise it turned out that about 5-10% mbufs can't be pulled-up as they have something like <--160 bytes leading space-->data-ptr<10 bytes used><2 bytes free>. In such case pulling up will not fail but will not be correct either - I even can't get a complete IP header (not speaking about TCP/UDP)! I checked kernel sources and it seems that developers think exactly the same - they do not pullup - instead they create a copy of the header with the help of mbuf_copydata and stack allocated data. May be mbuf_pullup can be enhanced with some flag to actually move data_ptr to the beginning of leading space, copy data and only after that do a pullup? May be an additional mbuf API can be added that can consume leading bytes and move data to the mbuf beginning? Also I am interested in smth. like this - if for example I will write such a routines - can I submit them to Apple? If they will accept them will they appear in next OS release or at some fix level? I also have some additional thoughts on NKE implementation based on the NDIS driver development for Win32 and now Mac - anyone from Apple interested to hear? I am new to this list, so I don't know the politics - whether Apple kernel devs at least browse it, but judging from some mails recently I guess they do.. This email sent to site_archiver(a)lists.apple.com

1 0

Re: mbuf_pullup practically useless
by itojun＠itojun.org 27 Aug '07

27 Aug '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com > > This is more or less a generic lamentation not a question. I've > > been doing quite a lot of KEXT coding recently (NKE mostly > > interface filter) and thus I had to analyze packets. One of the > > functions that I've used was mbuf_pullup. Here's its description: > > > > Move the next len bytes in to mbuf from other mbufs in the chain. > > This is commonly used to get the IP and TCP or UDP header > > contiguous in the first mbuf. If mbuf_pullup fails, the entire mbuf > > chain will be freed. haven't checked about darwin-local stuff, but mbuf_pullup() seems to me that an equivalent of BSD m_pullup(). there's a little bit extended variant that should be inside darwin - called m_pulldown(). the detail is here, and it is available on all *BSDs. http://www.usenix.org/publications/library/proceedings/usenix2000/freenix/h… itojun _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: mbuf_pullup practically useless
by Vincent Lubet 27 Aug '07

27 Aug '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Platon, Thanks, Vincent On Aug 26, 2007, at 2:19 PM, Platon Fomichev wrote: That's my lament is like this: _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… We -- Apple engineers lurking on the darwin mailing list -- really do appreciate suggestions like yours. The best way to make sure your enhancement request does fall through the crack is to file a problem report at http:// bugreporter.apple.com/ so that it track into Apple's bug tracking database. To ensure the problem report is routed to the correct component you can always send a note to the mailing list with the problem ID number. This is more or less a generic lamentation not a question. I've been doing quite a lot of KEXT coding recently (NKE mostly interface filter) and thus I had to analyze packets. One of the functions that I've used was mbuf_pullup. Here's its description: Move the next len bytes in to mbuf from other mbufs in the chain. This is commonly used to get the IP and TCP or UDP header contiguous in the first mbuf. If mbuf_pullup fails, the entire mbuf chain will be freed. And I used it exactly for that - to get IP/TCP(UDP) header contiguous in the first mbuf. To my surprise it turned out that about 5-10% mbufs can't be pulled-up as they have something like <--160 bytes leading space-->data-ptr<10 bytes used><2 bytes free>. In such case pulling up will not fail but will not be correct either - I even can't get a complete IP header (not speaking about TCP/UDP)! I checked kernel sources and it seems that developers think exactly the same - they do not pullup - instead they create a copy of the header with the help of mbuf_copydata and stack allocated data. May be mbuf_pullup can be enhanced with some flag to actually move data_ptr to the beginning of leading space, copy data and only after that do a pullup? May be an additional mbuf API can be added that can consume leading bytes and move data to the mbuf beginning? Also I am interested in smth. like this - if for example I will write such a routines - can I submit them to Apple? If they will accept them will they appear in next OS release or at some fix level? I also have some additional thoughts on NKE implementation based on the NDIS driver development for Win32 and now Mac - anyone from Apple interested to hear? I am new to this list, so I don't know the politics - whether Apple kernel devs at least browse it, but judging from some mails recently I guess they do.. This email sent to site_archiver(a)lists.apple.com

1 0

mbuf_pullup practically useless
by Platon Fomichev 26 Aug '07

26 Aug '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Hello Everyone That's my lament is like this: This is more or less a generic lamentation not a question. I've been doing quite a lot of KEXT coding recently (NKE mostly interface filter) and thus I had to analyze packets. One of the functions that I've used was mbuf_pullup. Here's its description: Move the next len bytes in to mbuf from other mbufs in the chain. This is commonly used to get the IP and TCP or UDP header contiguous in the first mbuf. If mbuf_pullup fails, the entire mbuf chain will be freed. And I used it exactly for that - to get IP/TCP(UDP) header contiguous in the first mbuf. To my surprise it turned out that about 5-10% mbufs can't be pulled-up as they have something like <--160 bytes leading space-->data-ptr<10 bytes used><2 bytes free>. In such case pulling up will not fail but will not be correct either - I even can't get a complete IP header (not speaking about TCP/UDP)! I checked kernel sources and it seems that developers think exactly the same - they do not pullup - instead they create a copy of the header with the help of mbuf_copydata and stack allocated data. May be mbuf_pullup can be enhanced with some flag to actually move data_ptr to the beginning of leading space, copy data and only after that do a pullup? May be an additional mbuf API can be added that can consume leading bytes and move data to the mbuf beginning? Also I am interested in smth. like this - if for example I will write such a routines - can I submit them to Apple? If they will accept them will they appear in next OS release or at some fix level? I also have some additional thoughts on NKE implementation based on the NDIS driver development for Win32 and now Mac - anyone from Apple interested to hear? I am new to this list, so I don't know the politics - whether Apple kernel devs at least browse it, but judging from some mails recently I guess they do.. Best regards, Stauff__ _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: printf logs are not getting printed properly on system.log
by Terry Lambert 25 Aug '07

25 Aug '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com HI Rodney, Thanks for the initial response. -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Aug 23, 2007, at 1:27 AM, JanakiRam wrote: I'm using the printf function defined at kernel. Is there any similar way for printf ( similar to IOLog). Can i use the IOLog in Generic Kernel Extension ? Rodney's answer is more or less correct; the standard kernel console log operations are written to a ring buffer which is serviced by syslogd, and there's nothing their to ensure that they are synchronous; it's expected that if you are debugging that type of problem, you would set ncpu=1 in the boot-args, and it wouldn't ever be an issue. By using the IOLog facility, you get locked access (but this doesn't stop anyone else from using printf and stomping in the middle of your IOLog). Either way, it gets into the same ring buffer, and you get to hope that syslogd in user space gets enough time to run to log the data reliably to a user space log file. The reason this facility isn't reliable is because (1) we didn't want kernel logging to be very verybose, (2) we didn't want people using it rather than the standard IPC mechanism, and (3) it's unacceptable to have a user space process (syslogd) potentially hang the kernel. Basically, if you want reliable communication from the kernel to user space, don't use logging, use one of the IPC facilities (sockets, IOKitUser, etc.) instead. If this is for debugging, and it's critical you don't lose data, then use the firewire debugging KEXT instead. This email sent to site_archiver(a)lists.apple.com

1 0

Wait-Queue Panic while loading the Kext
by JanakiRam 25 Aug '07

25 Aug '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type; b=ox169hO8hEmU2Pa+SuQ0vwTKCJdF4XKNZVDbP+6VraXxhO4IHxFNqF1mjCNtj+krgaWopLHCL5GjabecW1jLxG2nntLFhRgBQOaIdeQGpSKpesMQG303/MPIqMc7OEk4N9CmpS84Yz+cyY9K2vcB6oapaq449DtUq1iUDwFAzLo= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:mime-version:content-type; b=GJl/+kwgZvMGhCFottGDFiCS1sBCteqDygqiWcE9H1asu0g+drsCsbpiczknT0+oNhJdGZ2KUlu1aIojfRpdcZAI5qaRt9Rr0YaMk5L08G20xIXGg4UavgDAVgzab2HSr9rxStxSTVJr7zpAgT+rSTGWb/So0SiSxWDm6nwuNhI= _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: printf logs are not getting printed properly on system.log
by Draichis 25 Aug '07

25 Aug '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding; b=Km/vvKxE/SDLo9nw3/ZVTs7K7n8rjjuIQwA5ZbZWjLq07UP8KwCjaMjZ7oGZeztHIJ8jucJPsPLEdSwJGspezm0iWac+JK3UL8WzaX88jp2+Xjo1f5pMRbnSIIIqiUt827o5QSOGuw+QGqOVa5tlZOZwRfLRb024HjBgY10BaOQ= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding; b=aMufLnKfw2zrP5hOwOdXUqxMcpLsFpEl8tKuWSt7uuVPhah7cPsmtexO/ObxnAYkJSLYkTo3CDLUWtCpocA2ulA+Sc/NMmKzsomUYX8r/ThH4CH38D2DoNxIML0OmImj3JRLG2wDdLTfzWRUg5M+3O1q7ZVzRxXLg5lA/8ZuV/E= User-agent: Thunderbird 1.5.0.10 (Macintosh/20070221) On 23 Aug 2007, at 20:04, Draichis wrote: Amazing. <sys/systm.h> (printf can be used with this) <sys/buf.h> (buffer cache functions like bread) <sys/dirent.h> (directory entries) <sys/errno.h> (Error numbers) <sys.kernel.h> (kernel special variables) <sys/lock.h> (spinlocks) <sys/malloc.h> (malloc) <sys/queue.h> (queues) <sys/rand.h> (random generation numbers) <sys/systm.h> (bzero, bcopy.. etc) <sys/system.h> (timeout, untimeout) <sys/time.h> (time functions) <sys/types.h> <mach/mach_types.h> (standard type declarations) <sys/ucred.h> (user credentials) <sys/utsname.h> (os and system information) Thanks for this. -- --- Draichis draichis(a)gmail.com draichis(a)craplandia.org http://draichis.craplandia.org "I know that fewer people are won over by the written word than by the spoken word and that every great movement on this earth owes its growth to great speakers and not to great writers." _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Jeremy Pereira wrote: One of the most common problems faced when programming in the kernel is use of “standard” functions—things like printf or bcopy. Many commonly used standard C library functions are implemented in the kernel. In order to use them, however, you need to include the appropriate prototypes, which may be different from the user space prototypes for those functions, and which generally have different names when included from kernel code. In general, any non–I/O Kit header that you can safely include in the kernel is located in xnu/bsd/sys or xnu/osfmk/mach, although there are a few specialized headers in other places like libkern and libsa. Normal headers (those in /usr/include) cannot be used in the kernel (or rather, cannot be used 'properly'). If the standard C function you are trying to use is not in one of the following files, chances are the function is not supported for use within the kernel, and you need to implement your code in another way: For developing kernel extensions, surely it is better to be guided by the header files that are in Kernel.framework in the SDK which are also the ones that should be used - at least they are the ones that XCode uses when it creates a new kext project. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Offloading checksums to H/W
by Platon Fomichev 24 Aug '07

24 Aug '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Let's say .8 is my Mac: Thus in order to use the h/w for checksum calc one should: Have fun, Stauff On Aug 24, 2007, at 9:40 PM, mm w wrote: Hi platon, struct packet { struct mbuf *buf; struct iphdr header; int sum; tcp_seq seq; int size; ... /* what you want */ }; @+ Hey mm Yup exactly, that's what is written in Apple's docs but as we all have access to the kernel and overall - why on earth we should need to do this? Of course there are cases - when I change destination, metric, etc, etc. But if f.ex. I just changed packet length? Imho it's not the case for re-injection. And it actually works as expected on my test-cases - I can do whatever I want as long as I keep its destination, etc. But my case is simple I need to increase data size for say 20 bytes. I actually managed to get some weird checksum (wrong) from H/w - I will keep you informed about my progress.. Best regards, Platon On Aug 24, 2007, at 7:54 PM, mm w wrote: Hi Platon, Cheers Hi Everybody Thus some questions: Best regards, Platon. On Aug 23, 2007, at 9:05 AM, JanakiRam wrote: openspecies%40gmail.com This email sent to openspecies(a)gmail.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Ok I sorted this out - it actually works as expected. The main issue is that H/W sometimes can calculate only pure TCP header & data checksum without calculating pseudo-header checksum which is complex for h/w to do. Thus stack should do it itself. If you look at your Mac tcpdump trace you may see smth like this for TCP/UDP: 00:08:52.813229 IP (tos 0x0, ttl 64, id 41585, offset 0, flags [none], proto: UDP (17), length: 70) 10.0.44.8.49382 > 10.0.44.1.domain: [bad udp cksum 754f!] 12935+ PTR? 21.24.12.64.in- addr.arpa. (42) As you see TCPDump considers cksum as incorrect! But this is a pseudo- header cksum! The other half will be computed by the driver later before sending packet. 1) Make sure that h/w actually can do _TCP and _UDP checksum offloading, see ifnet_offload 2) If you correct packet and you absolutely sure you will not re- inject it correct TCP/UDP checksum in their respective headers to contain a correct pseudo-header checksum. 3) There is a slim chance that h/w can do full (without the necessity to do pseudo-checksums) TCP/UDP checksumming (in XServes imho) but I can't find out how to figure this out ;) as far i know, it isn't implemented i understand well your problem, the problem is, the check-sum is applied to the whole of packet, if you change the size, you change it On 8/24/07, Platon Fomichev <pfomichev(a)alarity.com> wrote: If your modifications involve changes to packet header information you must swallow the packet entirely and reinject it. On 8/24/07, Platon Fomichev <pfomichev(a)alarity.com> wrote: It seems I can't figure how to correctly offload my TCP/UDP packet to the H/W network checksumming routines. It seems rather inconvenient that I have to add all typical checksumming code into the driver - because neither of mbuf API can actually re-checksum the packet. As for H/W support - I can see from the flags that my H/W supports TCP offload but if I set tcp checksum f.ex. to zero in my Interface filter the packet will not be checksummed later in the hardware. 1) Seems that there is no kernel API to do IP checksumming in Mac - is it true, if not how to do it? mbuf finalize func does not do it. 2) Seems that hardware TCP/UDP checksum offload is actually missing from Mac at all? At least it is not present on MacBook/MacBook pro. If yes why does it advertise itself using ifnet API as capable to do it? If I am wrong how can I check that my network adapter is actually doing its work and how can I benefit from this in my code (I am actively changing the packet) This email sent to site_archiver(a)lists.apple.com

1 0

Re: Offloading checksums to H/W
by Platon Fomichev 24 Aug '07

24 Aug '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Hey mm Best regards, Platon On Aug 24, 2007, at 7:54 PM, mm w wrote: Hi Platon, If your modifications involve changes to packet header information you must swallow the packet entirely and reinject it. Cheers Hi Everybody It seems I can't figure how to correctly offload my TCP/UDP packet to the H/W network checksumming routines. It seems rather inconvenient that I have to add all typical checksumming code into the driver - because neither of mbuf API can actually re-checksum the packet. As for H/W support - I can see from the flags that my H/W supports TCP offload but if I set tcp checksum f.ex. to zero in my Interface filter the packet will not be checksummed later in the hardware. Thus some questions: 1) Seems that there is no kernel API to do IP checksumming in Mac - is it true, if not how to do it? mbuf finalize func does not do it. 2) Seems that hardware TCP/UDP checksum offload is actually missing from Mac at all? At least it is not present on MacBook/MacBook pro. If yes why does it advertise itself using ifnet API as capable to do it? If I am wrong how can I check that my network adapter is actually doing its work and how can I benefit from this in my code (I am actively changing the packet) Best regards, Platon. On Aug 23, 2007, at 9:05 AM, JanakiRam wrote: openspecies%40gmail.com This email sent to openspecies(a)gmail.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Yup exactly, that's what is written in Apple's docs but as we all have access to the kernel and overall - why on earth we should need to do this? Of course there are cases - when I change destination, metric, etc, etc. But if f.ex. I just changed packet length? Imho it's not the case for re-injection. And it actually works as expected on my test-cases - I can do whatever I want as long as I keep its destination, etc. But my case is simple I need to increase data size for say 20 bytes. I actually managed to get some weird checksum (wrong) from H/w - I will keep you informed about my progress.. On 8/24/07, Platon Fomichev <pfomichev(a)alarity.com> wrote: This email sent to site_archiver(a)lists.apple.com

1 0

Offloading checksums to H/W
by Platon Fomichev 24 Aug '07

24 Aug '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Hi Everybody Thus some questions: Best regards, Platon. On Aug 23, 2007, at 9:05 AM, JanakiRam wrote: Hi ALL, Please help me to solve this. Thanks in Advance. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… It seems I can't figure how to correctly offload my TCP/UDP packet to the H/W network checksumming routines. It seems rather inconvenient that I have to add all typical checksumming code into the driver - because neither of mbuf API can actually re-checksum the packet. As for H/W support - I can see from the flags that my H/W supports TCP offload but if I set tcp checksum f.ex. to zero in my Interface filter the packet will not be checksummed later in the hardware. 1) Seems that there is no kernel API to do IP checksumming in Mac - is it true, if not how to do it? mbuf finalize func does not do it. 2) Seems that hardware TCP/UDP checksum offload is actually missing from Mac at all? At least it is not present on MacBook/MacBook pro. If yes why does it advertise itself using ifnet API as capable to do it? If I am wrong how can I check that my network adapter is actually doing its work and how can I benefit from this in my code (I am actively changing the packet) I'm newbie to this kernel prorgamming. Please clarify my following query. I'm using printf for logging messages from my kext. One observation is all my logs are not getting printed when i start my kext using kextload. Its not consitent , how many logs are going to get printed every time i load the kext. -JanakiRam. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/pfomichev% 40alarity.com This email sent to pfomichev(a)alarity.com This email sent to site_archiver(a)lists.apple.com

1 0