Re: Read lines from very large text file
Re: Read lines from very large text file
- Subject: Re: Read lines from very large text file
- From: Michael Ash <email@hidden>
- Date: Fri, 6 Feb 2009 21:20:01 -0500
On Fri, Feb 6, 2009 at 6:44 PM, Matt Neuburg <email@hidden> wrote:
> On Tue, 03 Feb 2009 00:42:07 +1100, Jacob Rhoden <email@hidden>
> said:
>>I am wondering what the best way to read a text file, line by line, when
>>the file size is much larger than available memory.
>
> Would there be a way to do this with dataWithContentsOfMappedFile? I've long
> wondered about that... m.
Yes and no. +dataWithContentsOfMappedFile: can be used to do this kind
of efficient parsing, as memory mapping of files means that the file
is, essentially, read piecemeal, lazily by the operating system on
demand.
The trouble is that on a modern Mac, "much larger than available
memory" almost always also means "much larger than a 32-bit address
space can hold". A 32-bit process only gets 4GB of address space, and
realistically you can only allocate about 2GB of that contiguously
(the rest is broken up or occupied by system frameworks and the like),
and even that you can only do one at a time. Even though the file is
read lazily, the address space has to be allocated up-front, and if
you don't have it, you can't do it.
Realistically, if your files are going to be more than a few dozen
megabytes, memory mapping them is only practical if you strictly
control how many get opened at once, or if you only run in 64-bit
mode.
Mike
_______________________________________________
Cocoa-dev mailing list (email@hidden)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden