Re: Maximum size of an NSMutableData object?
Re: Maximum size of an NSMutableData object?
- Subject: Re: Maximum size of an NSMutableData object?
- From: Bill Bumgarner <email@hidden>
- Date: Tue, 3 Dec 2002 23:25:55 -0500
Short answer: each process on your machine has an absolute maximum of
about 4GB of address space that it can futz with as a part of its heap;
this is a limitation of the current processor architecture -- 2**32 ==
4GB.
In reality, your app has less than that 4GB to play with in that the
system will reserve some of it for whatever mysterious reasons it may
have and the various frameworks/dylibs/resources that are mapped into
the address space of your app will consume chunks of the address space.
Keep in mind that address space does not equal allocated memory -- more
on that interspersed with your original message...
On Tuesday, December 3, 2002, at 09:54 PM,
email@hidden wrote:
I have a question as to whether there is any relationship between the
amount of physical RAM installed in a machine and the maximum size of
an
NSMutableData object that can be allocated.
No relationship; the virtual memory subsystem will ensure that the
appropriate hunks of memory are paged out to disk as your app consumes
[not demands] more memory.
Demand == App says "I want X KB/MB/GB of memory from kernel" -- kernel
ensures that app has that amount of memory available within its address
space. In this case, memory does not mean physical RAM.
Consume == App actually writes to a page of memory that has been
demanded by the app. In this case, the kernel ensures that a physical
page of memory [a hunk o' burnin' RAM] is assigned to the App. If
necessary, this may cause some other app's (or the same app, for that
matter) page(s) of memory to be paged out to disk [or tossed in the
case of mapped read only files].
Until an app consumes the memory that has been reserved for it, the
kernel doesn't actually do anything-- there doesn't have to be a hunk
of physical RAM associated with the memory.
.... demo code deleted -- good stuff ....
When I run this program, I see the following results:
2002-12-04 12:55:31.832 MemoryTest[1131] Physical RAM installed =
536870912
bytes
2002-12-04 12:55:31.832 MemoryTest[1131] About to allocate 1073741823
bytes
2002-12-04 12:55:56.942 MemoryTest[1131] NSMutableData size =
1073741823 bytes
2002-12-04 12:55:57.248 MemoryTest[1131] About to allocate 1073741824
bytes
2002-12-04 12:55:57.292 MemoryTest[1131] *** Uncaught exception:
<NSInvalidArgumentException> *** -[NSConcreteMutableData
initWithCapacity:]: absurd capacity: 1073741824
MemoryTest has exited due to signal 5 (SIGTRAP).
Looks like a limitation in NSData that CFData does not suffer from.
Given that NSData and CFData are interchangeable via the no cost
bridge, I suggest you do something like:
int main (int argc, const char * argv[]) {
NSAutoreleasePool * pool = [[NSAutoreleasePool alloc] init];
CFIndex size = (((unsigned)1073741824) * 2);
CFMutableDataRef d = CFDataCreateMutable(NULL, size);
CFDataSetLength(d, size);
NSLog(@"l 0x%x - %u", d, CFDataGetLength(d));
[pool release];
return 0;
}
Note, that if you go to large -- say:
....
CFIndex size = (((unsigned)1073741824) * 2) + 1073741000;
....
You will see an error like:
*** malloc: vm_allocate(size=3221225472) failed with 3
*** malloc[20057]: error: Can't allocate region
[note: CFData called malloc() to reserve a hunk of memory... malloc()
discovered that the process did not have that many bytes reserved in
its address space and, as such, called vm_allocate() to request more
addresses [not actual bytes of physical ram] from the kernel. That
failed because the virtual memory infrastructure either could not add
an additional 3221225472 bytes to the process or because it could not
do so continguosly. 3 is KERN_NO_SPACE error...]
Now, technically I should have 3221225472 bytes in the address space
for my application. However, I very likely do not have that many
contiguous bytes! Because the frameworks, the various resources, and
the application executable are all mapped into memory, each consumes
addresses in the address space-- more than you might think. Each
framework is assigned its own address into which it is mapped -- this
occurs at COMPILE time, but a framework will be moved [thus disabling
prebinding] if two framework's asssigned addresses collide at runtime.
See...
file://localhost/Developer/Documentation/ReleaseNotes/Prebinding.html
... for more info.
In any case, a framework that is only 100K on disk may occupy 1MB + of
address space. Even if it doesn't, if the various frameworks are
spread out in memory, it reduces the size of the largest contiguous
chunk of available memory and, hence, the maximum sized NSData or
CFData you can allocate (though it does appear that NSData has an
arbitrary hardwired limit as you discovered.
---
Finally -- note that your example code and the above code run really,
really fast even though-- in my case-- I'm effectively allocated over
4x the amount of physical RAM that is on my machine (512MB TiBook).
Now, if the app actually consumed that memory upon allocation, I'd be
sitting here for a damned long time waiting for the very slow hard
drive [4,200 RPM laptop drive] to lay down *at least* 1.5GB of pages to
the disk.
But that isn't what happens.
Mach's virtual memory system is smarter than that; simply because an
app *requests* the memory, does not mean it uses it. Instead, when
the app attempts to write to an address of memory that has been
reserved, but not physically allocated to the app, a kernel interrupt
is raised indicating to the kernel that the app is trying to diddle a
page that needs to be made available to the app. The kernel then frees
up a physical page of memory in RAM-- maybe by paging something out--
and fills that page with all zeros before assigning it to the
appropriate base address that will include the address the app is
trying to write to.
(Fills a security hole -- an app can't allocate, say, 2GB then go
traipsing through the pages looking for passwords, CC#s, etc.. that
some other app may have left behind as its pages were shifted out to
disk.)
So-- if you allocate 2GB and write 5 bytes at, say, location
1,234,567,890, your app will only have used *one* page of physical RAM!
The rest of the pages won't fault into place until you attempt to
write to 'em.
Of course-- if you touch all, say, 2.5GB of allocated pages on a 512MB
machine, life is going to be *painfully* slow for a while.
(A lot of this has been simplified for the sake of brevity and to focus
it at a higher level-- closer to the Cocoa layer than to the mach/vm
layer... some accuracy was sacrificed, but the basic ideas should be
correct.)
b.bum
[Anyone need assistance with Cocoa or WebObjects development? CodeFab
can help.]
_______________________________________________
cocoa-dev mailing list | email@hidden
Help/Unsubscribe/Archives:
http://www.lists.apple.com/mailman/listinfo/cocoa-dev
Do not post admin requests to the list. They will be ignored.