Re: Resident Memory, Private Memory, Shared Memory (top, Activity Monitor and vmmap)
Re: Resident Memory, Private Memory, Shared Memory (top, Activity Monitor and vmmap)
- Subject: Re: Resident Memory, Private Memory, Shared Memory (top, Activity Monitor and vmmap)
- From: Markus Hanauska <email@hidden>
- Date: Fri, 30 Nov 2007 13:48:48 +0100
Hi Michael,
I'm sorry that I have ignored your posts so far, but I considered
your posts little helpful (compared to what Jim posted, which was
really giving me some insight into the things behind the scene) and
also kind of "harsh" and "derogative" (What? You don't know this?
Geez, how pitiful). Might also be a misinterpretation on my side, not
being a native English speaker, I may read the wrong cadence out of
your postings or you just had a bad day or something. Experience told
me, if a post of someone makes you feel uncomfortable, don't reply to
him, that is the road down to flaming and frustration, and it's
usually of no use to the other readers of a mailing list. Still you
seem to have a lot of valuable knowledge and if you'd share some more
of it, you can probably help a lot of coders to understand the nasty
details behind the wonderful world of a modern OS. So I'm pushing my
luck...
On Tue, 2007-11-27, at 22:53, Michael Smith wrote:
This is not true; I have myself authored at least a dozen replies
to variations on this basic question in multiple forums.
I like your stile of quoting and commenting, it's good, old style
comment below stile.
I have only searched Apple's mailing list archive and I have not
found anything; Apple has too many ifdef __APPLE__ in the kernel to
say for sure that xnu behaves like any other BSD kernel.
It's also nowhere documented.
This is, techically speaking, not true; it is documented by the
(freely available) source code and the activity of your system.
Well, source code is no real documentation. The difference is, it
took me about 1-2 hours to figure out how top calculates private and
shared memory, something a documentation had told me in 2 sentences
in 3 minutes.
1. The Resident Memory Size (aka Real Memory)
2. The Private Memory Size
3. The Shared Memory Size
So far, all these values seems self explaining... but they are not at
all for one single reason:
How comes 1 + 2 != 3???
What makes you think that they would be?
Because all memory a process "has" (all pages in a process address
space) are either shared or private. So the real memory a process
used should be the sum of both. That the private and shared are
calculated in such a unfavourable fashion is another thing.
"resident memory" is an old OS term, and you will find it defined
in many OS textbooks. If you have reached this point, it's more or
less assumed that you will have read Tanenbaum or something more
recent, and the concept should be self-explanatory.
Resident according to top's own explanation is:
'This column reflects the amount of physical memory currently
allocated to each process. This is also known as the "resident set
size" or RSS. A process can have a large amount of virtual memory
allocated (as indicated by the SIZE column) but still be using very
little physical memory.'
I think you'll find that memory mapped MAP_ANON is accounted as
private
Which makes perfectly sense, since how could it be not private? ANON
memory can't be shared.
MAP_PRIVATE is handled as COW
Which already makes a bit less sense, but it's no issue, since COW
memory with only one reference is still accounted towards private
memory by top (and private memory will only have one reference).
and MAP_SHARED is erroneously always accounted as shared even if
nobody else has that range of the object mapped.
That is a bit stupid, since it does not reflect reality. OTOH drawing
buffers of Cocoa apps are in fact really shared (between the
application and the window server, that's what I saw on some
technote) and still saying "they are shared" seems awkward, as it's
technically correct, but these buffers only exist because of the
application and without it they will also vanish from the window
server, so they could as well be accounted towards that application
only.
It's kind of you to call shared libraries "clever", but see above
inre: OS textbooks. They've been around a long time. 8)
"clever" was not referring to shared libraries themselves or how they
work, it was referring to the fact how top looks at the reference
count of their pages to decide if they are accounted towards shared
or private memory.
In the case of shared submaps or shared pmap regions, pages may be
resident for a task that has never referenced them. Clustered
pageins can also cause this behaviour.
Well, you are all focusing way too much onto the system libraries and
their 256 MB area. It's not that I care a lot for these or that these
make me worry a lot. The system libraries are around in the system
most likely regardless if I run a specific process or not, since
almost every process uses them in one way or another. I rather care
for our own libraries or non standard system libraries.
Top is not telling you, except in the crudest sense, how much real
memory your process uses.
Then what is it telling me? If the values are so useless for
everything, then why displaying them in the first place?
Note that Activity Monitor (which displays the same values as top,
but I have no source to back this up, I can only compare them) is no
developer tool. It is used by normal, simple minded users, that have
no idea what "virtual memory" actually means. And this utility
displays real, private and shared memory to the user. What do you
expect will a simple minded user think, when seeing these values?
Simple minded users are using these values to determine memory
consumption of a process. Now if these values are so far away from
reality, they are useless and should not even be displayed to simple
minded users. Instead, this all should be replaced with a single
value, that really can give the user a rough estimation of how much
memory this process needs.
Understanding the real memory usage of your task is better handled
with different tools.
Now it gets interesting. That would be which tools?
I would encourage you to tinker with Shark and the X-ray
instruments to see if you can't get a better understanding of what
your process is doing and what that costs.
These tools basically tell me how much memory I have "wasted" using
malloc. Even in the best case, they might take my stack into account.
But what they won't take into account is the memory lost by caching
code pages, the memory lost by loaded libraries and their code pages,
and so on. And in rarest cases they will even distinguish private and
shared memory at all.
So if I know my processes uses malloc to allocate about 800 kb of
memory during runtime, this could not be any further away from the
real number of memory the system loses by running my process as it
is. This is an approximation of memory usage, that is far, far away
from reality and a much worse approximation than all the values
mentioned before.
The number I'm looking for is:
How much memory does the system need, to keep my process running,
assuming it was not allow to swap anything of it (everything needs to
be in memory) and here it should take any memory into account that
belongs to my process directly or indirectly, which includes malloc,
stack, code, static memory, code of libraries and static memory of
libraries, coded needed by the dyld - and that clearly distinguishes
between shared and unshared memory.
What would this number be good for? Well, it tells me "even if you
had no swap space at all, you could run about X instances of my
process on a system with 512 MB RAM, assuming that no other process
needs memory and the kernel operates within its normal parameters".
Or I could say "if you start my process, the system will need at
least X MB free memory to handle my process needs, otherwise it will
swap". If I take the malloc value here, it's nowhere near reality. My
process may only malloc 800 kb of memory, but if your system has only
800 kb of memory spare, it will not be able to run my process without
swapping (either to swap file or by permanently reloading code pages
with the vnode-pager, which is not much different to a page-in from
swap-file).
--
Best Regards,
Markus Hanauska
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Darwin-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden