RE: Asynchronous sock_sendmbuf
RE: Asynchronous sock_sendmbuf
- Subject: RE: Asynchronous sock_sendmbuf
- From: "Eddy Quicksall" <email@hidden>
- Date: Thu, 22 May 2008 22:10:37 -0400
Maybe the efficiency would still be fairly good but one thing is four sure.
It would take lots more memory to have 1000 or so threads just to handle
what can be handled with a single thread.
Regarding upper level recovery ... any networking software must be aware
that the connection can be lost and any time. It is not the responsibility
of the lower layers to recover from a lost connection. For good upper layer
protocols, this is built into them. For example, iSCSI has lots of
mechanisms to deal with this.
I'll look into sototcpcb(so). Thanks for that tip.
Regarding " not published API", if I don't want to use extra threads and I
don't want to poll and I don't want to check every few ms, how would you
suggest that I implement non-blocking socket calls?
Eddy
-----Original Message-----
From: Terry Lambert [mailto:email@hidden]
Sent: Thursday, May 22, 2008 7:36 PM
To: Eddy Quicksall
Cc: 'Igor Mikushkin'; email@hidden
Subject: Re: Asynchronous sock_sendmbuf
I don't think so; I was responding to the efficiency claim of not
using a writer thread with a blocking socket instead of using a non-
blocking socket and upcalls to try to avoid the need for a writer
thread. Whether it's more efficient depends on your specific use case.
If your queue is full, you will get EWOULDBLOCK, given that the socket
is non-blocking. If you don't get EWOULDBLOCK, all that means to me
is that you have not exceeded your send queue depth, not that you
don't have to check for it. It just means that your pool retention
time is short enough that the pool never overflows.
This is an important distinction, since this conversation is being
archived and will end up indexed by Google, and someone with a lot of
data to send might take your statement to mean that they don't have to
worry about checking for it.
The same argument applies to the assumption that there would be upper
level recovery (and why I referenced the application layer -- I don't
actually agree that "everyone knows" would include someone who had to
Google for the information in the first place).
Also, instead of:
sotoinpcb_hdr(so)->inph_ppcb))->snd_una
which is a WIDE-based BSD-ism in in_pcb_hdr.h, which Darwin doesn't
have, I think you meant to say:
sototcpcb(so)
...however, this is not published API and is considered kernel private
(like it says with the #ifdef in the Darwin sources in tcp_var.h), it
is an internal implementation detail, and you are therefore not
permitted to introspect the structure directly this way in order to
get this information, since doing so could cause your code to break in
each and every software update.
-- Terry
On May 22, 2008, at 2:08 PM, Eddy Quicksall wrote:
> You must have misunderstood something. I don't poll. I use the
> upcall. And I
> don't get EWOULDBLOCK.
>
> Wouldn't the mbufs be freed as the tcp/ack's arrive? It seems that
> ((struct
> tcpcb*)(sotoinpcb_hdr(so)->inph_ppcb))->snd_una) would tell when
> space is
> becoming available. Then if another so_send is done then data would
> be moved
> to the available mbuf's and everything would proceed nicely. So you
> don't
> need a guarantee of space, just a hint that there may be more space.
>
> Everyone knows that there is no guarantee that the application
> actually got
> the data just because you get an ack. But that is what upper level
> recovery
> is for.
>
> Eddy
>
> -----Original Message-----
> From: Terry Lambert [mailto:email@hidden]
> Sent: Thursday, May 22, 2008 3:51 PM
> To: Eddy Quicksall
> Cc: Igor Mikushkin; email@hidden
> Subject: Re: Asynchronous sock_sendmbuf
>
> Thread switches not involving a protection domain or address space
> crossing are essentially free.
>
> Marshalling send data to a blocking send thread would require
> synchronization, which could result in stall barriers, but this
> overhead has to be traded against that of retrying the same
> outstanding internal send request multiple times and getting
> EWOULDBLOCK, plus the polling requirement given a lack up a send
> specific upcall. So the efficiency claim may not be valid.
>
> In general, no send specific upcall that existed could guarantee send
> buffer space in any case, given the possibility that your mbufs are
> ehausted and/or multiple writers competing for send space.
>
> Also, as Mike pointed out, data not in your send buffer any more does
> not imply data received at the other end. Even an ack-based upcall
> based on data acknowledged, assuming a direct link and no intermediate
> bridging or stateful firewall, would only mean that the data had
> gotten to the targets receive buffer, not that the target application
> had receive it. The only way to guarantee this would be an
> application layer acknowledgement, probably in-band.
>
> -- Terry
>
> On May 22, 2008, at 10:51 AM, Eddy Quicksall <email@hidden
>> wrote:
>
>> I'm using non-blocking / upcall because (1) it is more efficient than
>> switching threads and (2) I only need one thread for all connections
>> (and 3,
>> it is an embedded application and not much memory is available for
>> lots of
>> threads).
>>
>> Eddy
>>
>> -----Original Message-----
>> From: darwin-kernel-bounces+quicksall_temp=email@hidden
>>
[mailto:darwin-kernel-bounces+quicksall_temp=email@hidden
>
>> ]
>> On Behalf Of Terry Lambert
>> Sent: Wednesday, May 21, 2008 7:31 PM
>> To: Igor Mikushkin
>> Cc: email@hidden
>> Subject: Re: Asynchronous sock_sendmbuf
>>
>> On May 21, 2008, at 3:30 AM, Igor Mikushkin wrote:
>>> Network Kernel Extensions Programming Guide says here:
>>> If your code is in a performance-critical part of the kernel (as
>>> opposed to a call from user space), you should generally perform
>>> socket I/O asynchronously.
>>
>> This is to prevent you from blocking on a write, and having
>> everything
>> pile up behind you waiting for you. For example, if your KEXT is a
>> block device driver that does network based logging, then blocking on
>> logging is going to prevent anything talking to your block device
>> driver from talking to the media while you are busy waiting for the
>> logging to occur.
>>
>> It's perfectly reasonable to marshall the I/O off to another thread,
>> and have it make a blocking call on your behalf, so long as you don't
>> depend on the call having been made for subsequent operations to be
>> successful
>>
>> -
>>
>> Maybe we can step back a second...
>>
>> What are you trying to accomplish here that you think getting an
>> upcall would accomplish for you?
>>
>> -- Terry
>> _______________________________________________
>> Do not post admin requests to the list. They will be ignored.
>> Darwin-kernel mailing list (email@hidden)
>> Help/Unsubscribe/Update your Subscription:
>>
>
>> uth.net
>>
>> This email sent to email@hidden
>>
>
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Darwin-kernel mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden