Re: Multithreaded file read
Re: Multithreaded file read
- Subject: Re: Multithreaded file read
- From: "Michael Ash" <email@hidden>
- Date: Wed, 30 Jul 2008 06:50:43 -0400
On Wed, Jul 30, 2008 at 2:06 AM, Roger Herikstad
<email@hidden> wrote:
> Hi,
> I was wondering if anyone on the list has a solution for my problem.
> I'm keeping a number of files containing indices into a big (>2GB)
> data file. I have a cocoa program that will read these index files and
> fetch the appropriate data from the big file, plot it and store as an
> image. I would like to run these print operations in parallel, but I'm
> concerned that a conflict will arise if two threads are reading into
> the same file. The index files are not disjoint, so a particular data
> vector could be requested by several threads. Is there a way to handle
> this in cocoa? The big file resides on a local network server and is
> shared over nfs. Thanks!
There are (at least) three reasonable options. I shall let you decide
which fits best:
1) The POSIX pread() function does not have the concept of a "cursor".
Instead, you tell it where to read from the file with each call. It is
safe to share a single file descriptor among multiple threads which
are all calling pread() on it.
2) Nothing says you can't open a single file more than once. Open it
in each thread, then read. Each time you open the file, you'll get an
independent cursor into it, so the individual threads won't interfere
with each other. However, be aware that open file descriptors are a
relatively scarce resource (hundreds to thousands) so depending on
your design this could cause problems. If you have, say, four threads
then this is not really a concern. You can do this with low-level
POSIX functions, with C stdio, or with NSFileHandle or NSStream.
3) Memory map the file, then read the mapped memory. Cocoa makes this
really easy using NSData. However, with a 2+GB file, you'll have
trouble ensuring that it fits into your address space unless you're
building exclusively for 64-bit. It is also difficult to gracefully
recover from errors when using memory mapping; if your NFS share
disappears and you're using one of the previous techniques then your
reads will return errors, but when using memory mapping you'll just (I
believe) segfault.
Mike
_______________________________________________
Cocoa-dev mailing list (email@hidden)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden