Re: Speed-optimized file copy
Re: Speed-optimized file copy
- Subject: Re: Speed-optimized file copy
- From: Stevo Brock <email@hidden>
- Date: Tue, 19 Jul 2011 14:25:11 -0700
Hi Quinn,
Thanks so much for the feedback.
-I had heard recently on one of the developer lists (I think that's where it was) that valloc eventually called down to vm_allocate so I was just taking a short cut. I've gone back to now using valloc.
-The buffer size and read size (and write size) is currently 1MB. Obviously testing can illuminate, but in your experience, is this a good ballpark number? I don't want it to be too small to be inefficient, and also not too big to cause thrashing, like you said.
-All caching has been disabled - or rather fcntl(fd, F_NOCACHE, 1) is called after open() in each case.
-Optimizing for seeks. Yes, this is the tricky one. I can tell you that the environment that this application will be used in is always going from a source drive to a number of (usually 1 or 2) destination drives. Is it even possible to know what spindle a file is being written to? And it seems like a file could cross spindles, unless there's something in the OS and drive that tries to avoid this.
-Stevo
On Jul 19, 2011, at 1:08 AM, Quinn The Eskimo! wrote:
>
> On 18 Jul 2011, at 21:25, Stevo Brock wrote:
>
>> What are ways that we could improve the efficiency of this system?
>
> It's hard to optimise stuff like this without actually looking at the code in action. The issue is that there can non-obvious interactions between the components that you'll only discover by profiling. However, my standard hints for this are:
>
> o use page aligned buffers (I'd recommend <x-man-page://3/valloc> in preference to vm_allocate, btw)
>
> o use read offsets and sizes that are a multiple of the page size
>
> o disable caching on both source and destination
>
> o optimise for seeks
>
> The last part is the tricky one. You typically want to have one 'thread' of execution per spindle. For example, if you're copying between two spindles, you want one producer 'thread' for the source spindle and one consumer 'thread' for the destination. In contrast, if the source and destination are on the same drive, you want to alternate between producing and consuming, with a large buffer in between. The tricky parts of this are:
>
> o what's a spindle -- It's very hard to determine this in the general case.
>
> o large buffer -- You want the buffer to be large enough to amortise the seek cost between producing and consuming, but not too big as to start the system thrashing.
>
> S+E
> --
> Quinn "The Eskimo!" <http://www.apple.com/developer/>
> Apple Developer Relations, Developer Technical Support, Core OS/Hardware
>
>
> _______________________________________________
> Do not post admin requests to the list. They will be ignored.
> Filesystem-dev mailing list (email@hidden)
> Help/Unsubscribe/Update your Subscription:
>
> This email sent to email@hidden
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden