Re: Speed-optimized file copy
Re: Speed-optimized file copy
- Subject: Re: Speed-optimized file copy
- From: Stevo Brock <email@hidden>
- Date: Tue, 19 Jul 2011 14:58:28 -0700
Hi Jim,
I had not considered this or any other single file to single destination file copy API, as in the runtime environment for the app, the sources can be multi GB in size and there can be multiple destinations. So I've been trying to do a single read and multiple writes.
However, the f_iosize from statfs looks interesting and will investigate that...
-Stevo
On Jul 19, 2011, at 2:41 PM, Jim Luther wrote:
> Have you considered using/trying copyfile(3) to see if it is fast enough? It uses f_iosize (from statfs) as it's buffer size.
>
> - Jim
>
> On Jul 18, 2011, at 1:25 PM, Stevo Brock wrote:
>
>> Hi all,
>>
>> We're developing an app to copy files and perform MD5 checksum verification on the destination files and are looking for further ways to optimize the system. Here's what we have implemented and working so far:
>>
>> The files to copy may be many or few, but will be spread out over a folder hierarchy and will total many gigabytes and may be going to multiple destinations.
>>
>> The whole workflow is managed by an NSOperationQueue that can have multiple concurrent operations as specified by the user (current testing is using 4 concurrent operations)
>>
>> An operation essentially does this for the copy:
>> -Allocate buffer - file transfer buffer size currently set to 1MB:
>> kern_return_t err = vm_allocate((vm_map_t) mach_task_self(), (vm_address_t*) &buffer,
>> kFileTransferBufferSize, VM_FLAGS_ANYWHERE);
>>
>> -Open source file:
>> int sourceFD = open([path UTF8String], O_RDONLY);
>>
>> -Create and open destination files:
>> destFileFDs[i] = open([path UTF8String], O_WRONLY | O_CREAT, destFileMode);
>>
>> -Loop while still bytes to read.
>> -Read bytes:
>> byteCount = read(sourceFD, buffer, byteCount);
>>
>> -Compute checksum in memory
>> -Loop on destinations
>> -Write bytes:
>> writeByteCount = write(destFileFDs[i], buffer, byteCount);
>>
>> When the copy is finished, the operation continues to do the verification:
>> -Loop on destinations
>> fd = open([path UTF8String], O_RDONLY);
>>
>> -Loop while still bytes to read:
>> byteCount = read(fd, buffer, byteCount);
>>
>> -Compute checksum in memory
>>
>> Then the buffer is deallocated:
>> vm_deallocate((vm_map_t) mach_task_self(), (vm_address_t) buffer, kFileTransferBufferSize);
>>
>>
>> The idea is that with 4 concurrent operations executing, the OS is getting fed a lot of disk operation requests such that it can reorder the requests to maximize file system efficiency. The deployment is OS X 10.6 minimum so we can use any tech available.
>>
>> What are ways that we could improve the efficiency of this system?
>>
>> -Stevo
>>
>>
>> _______________________________________________
>> Do not post admin requests to the list. They will be ignored.
>> Filesystem-dev mailing list (email@hidden)
>> Help/Unsubscribe/Update your Subscription:
>>
>> This email sent to email@hidden
>
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden