Re: Speed-optimized file copy
Re: Speed-optimized file copy
- Subject: Re: Speed-optimized file copy
- From: Wim Lewis <email@hidden>
- Date: Wed, 20 Jul 2011 12:28:52 -0700
On 20 Jul 2011, at 3:55 AM, Quinn The Eskimo! wrote:
> On 19 Jul 2011, at 18:38, Wim Lewis wrote:
>> I haven't tested on Darwin, but on some systems you can get a significant efficiency improvement using the aio_* interface (POSIX asynchronous IO).
>
> AIO is unlikely to help you here; if you look at the Darwin implementation, you'll find that it just uses a bunch of kernel threads to run the I/O via the normal mechanism.
In any case, I'd expect AIO to behave similarly to the pool-of-user-threads approach, in terms of keeping the disk busy (the VM-system stuff I mentioned seems unlikely to have much of an effect here).
> The notion of passing lots of requests to the kernel and have it deal with optimising them is an interesting one. This works really well in some circumstances, but it's probably not the best choice here. Specifically, this approach works well when you're doing lots of scattered I/O, or where the I/O is coming from multiple unrelated processes.
There are still some advantages even if you're reading linearly through one big file. It makes sure the disk is always busy. With single, blocking read()s, the disk has nothing to do after getting the bits off the platter, until they make their way across the bus, into user space, wake up the process, give it enough time to issue the next read(), and that request makes its way down and across the bus again to the disk controller.
Now that I think about it, all of these optimizations amount to implementing readahead in user space. The question then is whether Darwin's readahead is operational even when F_NOCACHE is in effect, and whether it's aggressive enough to keep up with a disk-to-disk copy operation, or whether it can be made to be so (with F_RDADVISE or F_RDAHEAD?). If the answers are yes and yes, then perhaps the best thing to do is to drop all this userspace cleverness and rely on the kernel to DTRT.
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden