Re: memory management question
Re: memory management question
- Subject: Re: memory management question
- From: Shawn Erickson <email@hidden>
- Date: Mon, 22 Mar 2004 08:36:29 -0800
On Mar 20, 2004, at 8:05 AM, matt neuburg wrote:
On Fri, 19 Mar 2004 14:21:29 -0800, Shawn Erickson
<email@hidden>
said:
On Mar 18, 2004, at 3:36 PM, matt neuburg wrote:
My app reads lots of TIFFs from disk. As it does this, I
can see (with MemoryStick) that the amount of the
computer's "inactive" memory goes way up - that is, more
and more RAM is being allocated to "inactive" memory,
until RAM fills up (the amount of "free" memory drops to
near zero). When the app is done, this memory is not
given back.
This is normal, it is the UBC (universal buffer cache)
attempting to cache data that may possibly be used again
in an attempt to avoid disk or network I/O.
What I'd like to do is give back the memory after I'm
done with each file. I have tried all the Cocoa tricks I
can think of (autorelease pools everywhere, explicit
init and release wherever possible, and so forth) but
nothing helps.
You really don't need to give it "back" because it isn't
really taken. The UBC is simply using available pages in
physical RAM to hold onto information. If someone needs a
page and a free one isn't available then an inactive one
will get used (since it is caching file data no page out
is needed, it just zeros it and returns the page like it
would if it was a free page).
That is a lovely theory, and perhaps some varieties of Unix work that
way,
but on Mac OS X that is not in fact what happens. What happens is that
the
"inactive" stuff in memory accumulates and overwhelms physical RAM and
pageouts start to happen, and swapfiles are created. On my machine, a
single
run of my program, reading, say, a dozen large TIFFs, can easily cause
two
additional swapfiles to be created. This is *not* due to a leak in my
program! The amount of "wired" and "active" RAM in use does not
increase.
This is all due to "inactive" RAM swelling. The mere act of reading
large
TIFFs from disk seems to cause the problem.
What problem does this page-out cause the user? What OS are you seeing
this on? If panther then two additional swap files likely amount to
around 384 MB, is that what you see?
Anyway I am curious on what is being paged out. Could you provide a
copy of your application so one could monitor things?
It is likely what is being paged out is some type data that is not
backed by a file and is tagged such that it is considered page-able
even in the case of caching in the UBC.
The problem is alleviated somewhat if the user happens to have
extremely
large amounts of RAM, but that is not really acceptable. The user
really
should not need more RAM to run my program than is necessary to hold in
memory two images: the TIFF currently being read, and the image I am
creating (I'm making a thumbnail of each successive TIFF and "pasting"
the
thumbnail into my image).
Is it really a problem, as in a performance issue, or is it something
you would just like to avoid?
Once a TIFF has been read from disk, it will not be read again. The
system
is wrong to cache it; this is a false optimization.
If that is the case then tell the OS this fact, it cannot know your
usage intent up front. In most situations a once used file is likely to
be used again so that is the default optimization in most cases (almost
all OSes today have this type of default caching mode).
Thus, I am looking for a
way to get the system to "give back" the RAM occupied by each TIFF
after I
am done with it. m.
It looks like Michael pointed out the fcntl to use, I could not recall
of the top of my head when I sent my prior message. If the pattern is
as you described I would recommend that you avoid the UBC so as to not
pollute it with items that are likely only to be read once.
If you are writing out new TIFFs are you doing that in one shot or
writing out data in pieces? If that later you may want to set the file
size up front to avoid file growth overhead and better avoid
fragmentation of the file.
-Shawn
_______________________________________________
cocoa-dev mailing list | email@hidden
Help/Unsubscribe/Archives:
http://www.lists.apple.com/mailman/listinfo/cocoa-dev
Do not post admin requests to the list. They will be ignored.