Re: [NSPipe pipe] returning nil (running out of filehandles?)
Re: [NSPipe pipe] returning nil (running out of filehandles?)
- Subject: Re: [NSPipe pipe] returning nil (running out of filehandles?)
- From: Martin Redington <email@hidden>
- Date: Thu, 3 Apr 2008 18:15:27 +0100
Just thought I'd add some hard numbers to this thread.
My original fixed NSTask based implementation ran in 8 minutes.
Using the code below, with a 1k buffer size, runs in 6:15 minutes, so
the speedup is about 25%.
A 10k buffer took about the same. Boosting the buffer to 1MB actually
ran slower (9:25)
cheers,
m.
========
#import <openssl/md5.h>
////
MD5_CTX md5_context;
MD5_Init(&md5_context);
size_t bytes_read;
while((bytes_read = read(fd, buffer, buf_size)) > 0)
{
MD5_Update(&md5_context, buffer, bytes_read);
}
unsigned char checksum[MD5_DIGEST_LENGTH];
if(bytes_read == 0)
{
MD5_Final(checksum, &md5_context);
}
On 2 Apr 2008, at 00:53, Martin Redington wrote:
On 2 Apr 2008, at 00:33, Jim Correia wrote:
On Apr 1, 2008, at 7:19 PM, Martin Redington wrote:
I'm running /sbin/md5 via an NSTask on a number of files (3,000
or so, with a new NSTask each time).
I know the "unix way" is to string together small purpose built
tools via pipes.
But it strikes me that NSTask is the wrong hammer for this nail.
Computing an md5 programatically yourself is probably the better
solution, regardless of the leaky file descriptor problem you are
having.
Well, I think I've solved the leak.
[md5Task waitUntilExit];
before
[md5Task release];
does the trick. No fd leaks, and it seems a lot faster generally.
You'd kind of hope that NSTask would clean up in its dealloc
method, but apparently not.
Jim - any particular reason why you'd not use NSTask?
I guess I'm paying the fork/exec overhead, and some interprocess
piping and (a very small bit of) extra I/O for each NSTask. I will
actually be caching a lot of the output, so I expect the numbers to
be much smaller on subsequent scans, although even once off, 3,000
spawns does seem a bit gratuitous. Performance was much better than
I expected though - about 12 minutes for the md5 phase.
I'm still in pre-optimisation mode right now, and I'll probably
migrate to the openssl MD5 method in any case, but I'm interested
to hear if there's any other reason apart from the above.
_______________________________________________
Cocoa-dev mailing list (email@hidden)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
40ucl.ac.uk
This email sent to email@hidden
_______________________________________________
Cocoa-dev mailing list (email@hidden)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden