Re: Working with large files and Memory
Re: Working with large files and Memory
- Subject: Re: Working with large files and Memory
- From: "Clark Cox" <email@hidden>
- Date: Tue, 11 Mar 2008 10:10:58 -0700
On Tue, Mar 11, 2008 at 9:54 AM, Carl E. McIntosh <email@hidden> wrote:
> Can you please give advice about handling large data files with memory
> management techniques? I am attempting to read three large files (1
> GB, 208 MB, 725 MB) sequentially and place the data into arrays for
> processing. Here is my psuedocode:
>
> 1) Import a file into NSString.
> NSString *aFileString = [NSString stringWithContentsOfFile:
> fileLocation]; // Convert file at path to myFileString text holder;
Depending on the encoding of the file itself, this could end up
allocating a block of memory twice the size of the file.
> 2) Use NSScanner pull out integers and floats
> NSScanner *aFileScanner = [[NSScanner alloc] initWithString:
> aFileString];
This has the potential of copying the passed in string (not likely to
be a problem, as you string is immutable, but is still something to
keep in mind).
>
> 3) Store values into arrays.
> float myFloats [100000][2000]; or
> float myInts [100000][2000];
Again, *another* huge block of memory.
> 4) repeat three times with 3 different files.
Depending on how this is done (how tight your autorelease pools are,
etc.), the old string may not have been deallocated yet.
> This algorithm works for smaller files but chokes on the larger files
> and I get malloc errors. I've attempted to use NSZone's to the same
> failure.
>
> Can you please give advice about handling large data files with memory
> management techniques?
You'll have to use less memory. Remember, that on a 32-bit machine,
you will never be able to allocate more than 4GB of memory in your
address space. On top of that, much of that memory is already used by
the system's frameworks (usually between 1 and 2 GB).
> I have 4 GB ram and can hog off 2 - 3 GBs for the process. I don't
> know how to explicitly allocate real memory. I'd rather not use
> virtual memory. Any references or examples would be appreciated.
Memory doesn't work like that (i.e you cannot allocate "real" memory).
The closest that you can get is to allocate memory and to "wire" it
down; however, this is a horrible idea for allocations this large
(you'll still run out of address space as before). Your best bet is to
change your algorithm to not have to load the entire file at once
(perhaps load it a line at a time, or in blocks of 4K or so).
--
Clark S. Cox III
email@hidden
_______________________________________________
Cocoa-dev mailing list (email@hidden)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden