Speed-optimized file copy
Speed-optimized file copy
- Subject: Speed-optimized file copy
- From: Stevo Brock <email@hidden>
- Date: Mon, 18 Jul 2011 13:25:23 -0700
Hi all,
We're developing an app to copy files and perform MD5 checksum verification on the destination files and are looking for further ways to optimize the system. Here's what we have implemented and working so far:
The files to copy may be many or few, but will be spread out over a folder hierarchy and will total many gigabytes and may be going to multiple destinations.
The whole workflow is managed by an NSOperationQueue that can have multiple concurrent operations as specified by the user (current testing is using 4 concurrent operations)
An operation essentially does this for the copy:
-Allocate buffer - file transfer buffer size currently set to 1MB:
kern_return_t err = vm_allocate((vm_map_t) mach_task_self(), (vm_address_t*) &buffer,
kFileTransferBufferSize, VM_FLAGS_ANYWHERE);
-Open source file:
int sourceFD = open([path UTF8String], O_RDONLY);
-Create and open destination files:
destFileFDs[i] = open([path UTF8String], O_WRONLY | O_CREAT, destFileMode);
-Loop while still bytes to read.
-Read bytes:
byteCount = read(sourceFD, buffer, byteCount);
-Compute checksum in memory
-Loop on destinations
-Write bytes:
writeByteCount = write(destFileFDs[i], buffer, byteCount);
When the copy is finished, the operation continues to do the verification:
-Loop on destinations
fd = open([path UTF8String], O_RDONLY);
-Loop while still bytes to read:
byteCount = read(fd, buffer, byteCount);
-Compute checksum in memory
Then the buffer is deallocated:
vm_deallocate((vm_map_t) mach_task_self(), (vm_address_t) buffer, kFileTransferBufferSize);
The idea is that with 4 concurrent operations executing, the OS is getting fed a lot of disk operation requests such that it can reorder the requests to maximize file system efficiency. The deployment is OS X 10.6 minimum so we can use any tech available.
What are ways that we could improve the efficiency of this system?
-Stevo
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden