July 2005 - Darwin-kernel - lists.apple.com

Re: sock_receive_internal() and its (ab)use of uio
by Josh Graessley 26 Jul '05

26 Jul '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Hi Sam, Thanks, -josh On Jul 25, 2005, at 4:51 PM, Sam Vaughan wrote: Hi Josh, _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com You don't need to allocate another uio. You just need to allocate some iovecs. The uio is handy for you because you happened to have a uio available for other reasons. When we looked at what most kexts were doing, we found that 90+% of them had to allocate and fill out a uio just to send some data. We wanted to make it easier by dealing with the uio so the kext developer didn't have to worry about that. Your request is the first we have received for a uio variant. There are probably more developers frustrated that they have to fill out an iovec array to describe their data instead of passing a pointer and length. We will not be able to add a uio variant in a software update (10.4.x). This means you would need to wait for the next major release at the earliest for such a function. Since you will have to work around this problem to support 10.4.x, I'm not sure it makes a lot of sense to add this function. If we hear more requests for a uio variant of the send and receive functions, I will look in to getting those in. Thanks for the speedy reply. I'm only interested in sending/ receiving to/from kernel memory here so that's not a concern right now. Don't you agree however that in moving from soreceive() to sock_receive(), the interface has dropped in usefulness? The idea of allocating yet another uio, walking the iovecs in my own one and adding new iovecs with the appropriate addresses and lengths to the new one makes me shudder. That'd be three uios allocated, prepared and freed just to do one I/O operation - one in our platform-common code, one in the 10.4 wrapper around sock_receive(), and another in sock_receive_internal(). One more entry point that takes a uio and passes it straight to soreceive would be a real bonus, for elegance, performance AND sanity ;o) Sam smime.p7s

1 0

Re: sock_receive_internal() and its (ab)use of uio
by Sam Vaughan 25 Jul '05

25 Jul '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Hi Josh, Sam On 26/07/2005, at 4:03 AM, Josh Graessley wrote: Hi Sam, -josh On Jul 25, 2005, at 12:58 AM, Sam Vaughan wrote: soreceive(so, NULL, uiop, NULL, NULL, flagsp); My first attempt at a KPI replacement looked like this: size_t bytes = uio->uio_resid; struct msghdr msg = {0}; msg.msg_iov = uiop->uio_iov; msg.msg_iovlen = uiop->uio_iovcnt; error = sock_receive(so, &msg, 0, &bytes); if (!error) { uiop->uio_offset += bytes; uiop->uio_resid -= bytes; ASSERT(uiop->uio_resid >= 0); } This email sent to jgraessley(a)apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Thanks for the speedy reply. I'm only interested in sending/ receiving to/from kernel memory here so that's not a concern right now. Don't you agree however that in moving from soreceive() to sock_receive(), the interface has dropped in usefulness? The idea of allocating yet another uio, walking the iovecs in my own one and adding new iovecs with the appropriate addresses and lengths to the new one makes me shudder. That'd be three uios allocated, prepared and freed just to do one I/O operation - one in our platform-common code, one in the 10.4 wrapper around sock_receive(), and another in sock_receive_internal(). One more entry point that takes a uio and passes it straight to soreceive would be a real bonus, for elegance, performance AND sanity ;o) The socket KPI was written to simplify the use of sockets in the kernel. The new socket KPI is based more on the user space APIs for interacting with sockets. sock_send is very much like sendmsg, sock_receive is like recvmsg. You may create a new iovec array to point to the buffers you're interested in copying in to. One other thing to be aware of is that the pointers in the iovec array must point to data that resides in the kernel's address space. If you need to receive in to a buffer in another process you will need to copy the data yourself. To avoid copying the data twice, you can use the sock_receivembuf variant which will receive the mbuf chain without performing a copy. I'm trying to clean up the areas in our code where I'm using symbols from mach_kernel instead of the 10.4 KPI. One of these is the use of soreceive() and sosend(). I avoided using the KPI on these two initially because the new interface that uses struct msghdr looked confusing. Now that I've delved into it a bit more it looks pretty much unworkable. Let me use sock_receive() here as an example. When linking directly to the kernel, I can simply pass a uio pointer to soreceive() as follows: I can do this safe in the knowledge that it will use uio_move() to transfer the data it reads into my uio, using and updating my uio_offset and uio_resid fields as appropriate. But it's clearly deficient because the uio_offset isn't being passed into sock_receive(). It's also not clear that it will use my uio_resid value. Sure enough when you run it up, this code asserts pretty quickly, having read more than I wanted it to. The problem as I see it is that the new socket KPI doesn't allow you to specify an offset and a resid along with your iovecs. It's therefore not possible to read into a range of memory somewhere in the middle of the buffers represented by your iovecs. If you use sock_receivembuf(), the data argument to sock_receive_internal() won't be NULL and the value in recvdlen will get used as the uio's resid. This solves one of the problems. Unfortunately the call to uio_createwithbuffer() will still specify 0 for the initial offset of the uio, so this is no good if you want to read into somewhere in the middle of your buffers. The only solution I can see at the moment is to create yet another uio structure before calling sock_receive(). This uio would be given iovecs with data addresses and lengths that represent the region of the original buffers to be received into. This is clearly not an optimal solution, given the extra overhead and unnecessary code to write and debug. It seems odd to me that the shiny new uio KPI wasn't used in the sock_receive() and sock_send() interfaces, since this is exactly the functionality that uios are designed (and were previously used) for. I've raised a radar bug requesting sock_receive_uio() and sock_send_uio() to be added, (#4191646) but I'd be interested to hear other people's opinions on this. Sam _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/jgraessley% 40apple.com This email sent to site_archiver(a)lists.apple.com

1 0

Re: Mac OS 10.4 (Tiger) compilation errors ( GCC v3.3 ) with X Code 2.0 tool
by Josh Graessley 25 Jul '05

25 Jul '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com What are you trying to accomplish with this? -josh On Jul 25, 2005, at 5:31 AM, Karunakar Reddy G wrote: Hello, Any body knows how to set checksums , In my previous code I set as follows on Mac 10.1.3 pkt->m_pkthdr.csum_flags = 0x0020; pkt->m_pkthdr.csum_data = CRC32_INITIAL; How to set on 10.4 using mbuf_pkthdr ? thanx in advance.. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Regards, Karunakar _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/jgraessley% 40apple.com This email sent to jgraessley(a)apple.com This email sent to site_archiver(a)lists.apple.com

1 0

Re: sock_receive_internal() and its (ab)use of uio
by Josh Graessley 25 Jul '05

25 Jul '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Hi Sam, -josh On Jul 25, 2005, at 12:58 AM, Sam Vaughan wrote: soreceive(so, NULL, uiop, NULL, NULL, flagsp); My first attempt at a KPI replacement looked like this: size_t bytes = uio->uio_resid; struct msghdr msg = {0}; msg.msg_iov = uiop->uio_iov; msg.msg_iovlen = uiop->uio_iovcnt; error = sock_receive(so, &msg, 0, &bytes); if (!error) { uiop->uio_offset += bytes; uiop->uio_resid -= bytes; ASSERT(uiop->uio_resid >= 0); } This email sent to jgraessley(a)apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… The socket KPI was written to simplify the use of sockets in the kernel. The new socket KPI is based more on the user space APIs for interacting with sockets. sock_send is very much like sendmsg, sock_receive is like recvmsg. You may create a new iovec array to point to the buffers you're interested in copying in to. One other thing to be aware of is that the pointers in the iovec array must point to data that resides in the kernel's address space. If you need to receive in to a buffer in another process you will need to copy the data yourself. To avoid copying the data twice, you can use the sock_receivembuf variant which will receive the mbuf chain without performing a copy. I'm trying to clean up the areas in our code where I'm using symbols from mach_kernel instead of the 10.4 KPI. One of these is the use of soreceive() and sosend(). I avoided using the KPI on these two initially because the new interface that uses struct msghdr looked confusing. Now that I've delved into it a bit more it looks pretty much unworkable. Let me use sock_receive() here as an example. When linking directly to the kernel, I can simply pass a uio pointer to soreceive() as follows: I can do this safe in the knowledge that it will use uio_move() to transfer the data it reads into my uio, using and updating my uio_offset and uio_resid fields as appropriate. But it's clearly deficient because the uio_offset isn't being passed into sock_receive(). It's also not clear that it will use my uio_resid value. Sure enough when you run it up, this code asserts pretty quickly, having read more than I wanted it to. The problem as I see it is that the new socket KPI doesn't allow you to specify an offset and a resid along with your iovecs. It's therefore not possible to read into a range of memory somewhere in the middle of the buffers represented by your iovecs. If you use sock_receivembuf(), the data argument to sock_receive_internal() won't be NULL and the value in recvdlen will get used as the uio's resid. This solves one of the problems. Unfortunately the call to uio_createwithbuffer() will still specify 0 for the initial offset of the uio, so this is no good if you want to read into somewhere in the middle of your buffers. The only solution I can see at the moment is to create yet another uio structure before calling sock_receive(). This uio would be given iovecs with data addresses and lengths that represent the region of the original buffers to be received into. This is clearly not an optimal solution, given the extra overhead and unnecessary code to write and debug. It seems odd to me that the shiny new uio KPI wasn't used in the sock_receive() and sock_send() interfaces, since this is exactly the functionality that uios are designed (and were previously used) for. I've raised a radar bug requesting sock_receive_uio() and sock_send_uio() to be added, (#4191646) but I'd be interested to hear other people's opinions on this. Sam _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/jgraessley% 40apple.com This email sent to site_archiver(a)lists.apple.com

1 0

Re: Mac OS 10.4 (Tiger) compilation errors ( GCC v3.3 ) with X Code 2.0 tool
by chas＠cmf.nrl.navy.mil 25 Jul '05

25 Jul '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com In message <70821652050725053146a006a2(a)mail.gmail.com>,Karunakar Reddy G writes >pkt->m_pkthdr.csum_flags =3D 0x0020; >pkt->m_pkthdr.csum_data =3D CRC32_INITIAL; > >How to set on 10.4 using mbuf_pkthdr ? mbuf_set_csum_performed() perhaps? take a look at kpi_mbuf.h _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Mac OS 10.4 (Tiger) compilation errors ( GCC v3.3 ) with X Code 2.0 tool
by Karunakar Reddy G 25 Jul '05

25 Jul '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:in-reply-to:mime-version:content-type:references; b=KS6lxwSm+pmmii0quJ9MuAjqjgleqd14TD1kXDF7RZVd+Ygb+m+h95d4vbicxlGN03xDYXE83pyPPmLDtL/wtRvXwY+Ym2rl+zSKnkhPExVpZGGCyo93e9wlJap4GwjLg3wXdnpOmvkPuhDd3tIKeDfN7YtEg9wHM5MshuYg4ig= _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

sock_receive_internal() and its (ab)use of uio
by Sam Vaughan 25 Jul '05

25 Jul '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com soreceive(so, NULL, uiop, NULL, NULL, flagsp); My first attempt at a KPI replacement looked like this: size_t bytes = uio->uio_resid; struct msghdr msg = {0}; msg.msg_iov = uiop->uio_iov; msg.msg_iovlen = uiop->uio_iovcnt; error = sock_receive(so, &msg, 0, &bytes); if (!error) { uiop->uio_offset += bytes; uiop->uio_resid -= bytes; ASSERT(uiop->uio_resid >= 0); } Sam _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… I'm trying to clean up the areas in our code where I'm using symbols from mach_kernel instead of the 10.4 KPI. One of these is the use of soreceive() and sosend(). I avoided using the KPI on these two initially because the new interface that uses struct msghdr looked confusing. Now that I've delved into it a bit more it looks pretty much unworkable. Let me use sock_receive() here as an example. When linking directly to the kernel, I can simply pass a uio pointer to soreceive() as follows: I can do this safe in the knowledge that it will use uio_move() to transfer the data it reads into my uio, using and updating my uio_offset and uio_resid fields as appropriate. But it's clearly deficient because the uio_offset isn't being passed into sock_receive(). It's also not clear that it will use my uio_resid value. Sure enough when you run it up, this code asserts pretty quickly, having read more than I wanted it to. The problem as I see it is that the new socket KPI doesn't allow you to specify an offset and a resid along with your iovecs. It's therefore not possible to read into a range of memory somewhere in the middle of the buffers represented by your iovecs. If you use sock_receivembuf(), the data argument to sock_receive_internal() won't be NULL and the value in recvdlen will get used as the uio's resid. This solves one of the problems. Unfortunately the call to uio_createwithbuffer() will still specify 0 for the initial offset of the uio, so this is no good if you want to read into somewhere in the middle of your buffers. The only solution I can see at the moment is to create yet another uio structure before calling sock_receive(). This uio would be given iovecs with data addresses and lengths that represent the region of the original buffers to be received into. This is clearly not an optimal solution, given the extra overhead and unnecessary code to write and debug. It seems odd to me that the shiny new uio KPI wasn't used in the sock_receive() and sock_send() interfaces, since this is exactly the functionality that uios are designed (and were previously used) for. I've raised a radar bug requesting sock_receive_uio() and sock_send_uio() to be added, (#4191646) but I'd be interested to hear other people's opinions on this. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Where is sysent structure hiding in 10.4.2?
by Karunakar Reddy G 25 Jul '05

25 Jul '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:in-reply-to:mime-version:content-type:references; b=VHIWsqfqmQo4QrGZ/g1TKhRaxq+LGi/vYks8HmA0OMfNUEfpziMVOK/ZsUkTW8TNUKoG2qjfAusH3OTg7E/Jer4dxtatQinlEqPlsjqd2hhm1s3b6Eo0luJnl9w+ZdUfk7Tsh/BDrd6qP7nGOjbgQ+U7+29APZq1d6XrqCfjfMQ= _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: SUM16 question
by Josh Graessley 22 Jul '05

22 Jul '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Hello Chris, -josh On Jul 22, 2005, at 9:03 AM, Chris Vitale wrote: What is the secret to the "Weird Apple checksum" bit? This email sent to jgraessley(a)apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… The special bit tells the ethernet driver to tell the chip to perform a checksum in hardware. The hardware has a very brain dead hardware checksum. It requires that we stash extra information in to the csum_value field. I would guess that if you aren't clearing these flags, when the packet goes out, the checksum is being calculated again in hardware which leads to an invalid checksum or the data stashed in the csum_value field is no longer valid after the changes you make. With the KPIs on 10.4.x there is a function called mbuf_finalize_packet or something like that. It should look at the flags, calculate any checksums, then clear the flags. There was a bug that prevented it from handling the weird apple checksum bit. That has since been fixed. I've got an NKE protocol filter for panther that touches ip packets and thus recalculates the checksums. TCP packets that travel through my wireless card (which has no HW assist) are fine, so I believe my routines are sound. TCP packets over my copper gig nic have bad tcp checksums. They become bad after I set correct the checksum. The are still bad if I don't do anything to the checksum. I'm pretty sure it has something to do with SUM16 (0x1000) bit that is set in the mbuf's csum_flags. This is the only bit set in csum_flags. vitale _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/jgraessley% 40apple.com This email sent to site_archiver(a)lists.apple.com

1 0

SUM16 question
by Chris Vitale 22 Jul '05

22 Jul '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com I've got an NKE protocol filter for panther that touches ip packets and thus recalculates the checksums. TCP packets that travel through my wireless card (which has no HW assist) are fine, so I believe my routines are sound. TCP packets over my copper gig nic have bad tcp checksums. They become bad after I set correct the checksum. The are still bad if I don't do anything to the checksum. I'm pretty sure it has something to do with SUM16 (0x1000) bit that is set in the mbuf's csum_flags. This is the only bit set in csum_flags. What is the secret to the "Weird Apple checksum" bit? vitale _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0