Re: Optimal read buffer size?
Re: Optimal read buffer size?
- Subject: Re: Optimal read buffer size?
- From: Peter Bierman <email@hidden>
- Date: Wed, 18 May 2005 13:14:11 -0700
At 12:56 PM -0600 5/18/05, Chaz McGarvey wrote:
I find that when I write network code, I never really know how big
I should make the read buffer. Some protocols have only short
packets so I usually have a pretty small buffer, or only read in
one packet at a time, but if I'm reading in a very big packet like
a mult-megabyte resource download that I'm writing to disk, is
there a sweet spot that requires the fewest read calls and doesn't
waste (much) memory?
My question involves getting the data from the kernel into my own
buffer so it can be written to disk. I would want to do this as
fast as possible,
While it's difficult to make an impedance match with the network, you
can try and make your code as efficient as possible, so that the
bottlenecks are somewhere else.
The best way to do that is to focus on eliminating as many of the
buffers as you can, instead of calculating good sizes for them. :-)
The best way to do that on OS X is to memory map your destination
file and pass that address to the read() calls so that the kernel is
copying the data directly from the network buffers to the address
space that you've told the VM system to connect to your file.
Then the data should go from the networking hardware, to the
networking stack buffers, to the unified buffer cache, where it will
be flushed to disk by the VM system as necessary.
If you know the size of the data you're expecting from the server,
you can allocate a single buffer of exactly that size (up to nearly 4
GB!), and loop on read() with the address incrementing (and the size
decreasing) by the amount received each call.
-pmb
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Macnetworkprog mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden