site_archiver@lists.apple.com Delivered-To: darwin-dev@lists.apple.com Hi Michael, On Tue, 2007-11-27, at 22:53, Michael Smith wrote: 1. The Resident Memory Size (aka Real Memory) 2. The Private Memory Size 3. The Shared Memory Size So far, all these values seems self explaining... but they are not at all for one single reason: Resident according to top's own explanation is: Now it gets interesting. That would be which tools? The number I'm looking for is: -- Best Regards, Markus Hanauska _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.appl... I'm sorry that I have ignored your posts so far, but I considered your posts little helpful (compared to what Jim posted, which was really giving me some insight into the things behind the scene) and also kind of "harsh" and "derogative" (What? You don't know this? Geez, how pitiful). Might also be a misinterpretation on my side, not being a native English speaker, I may read the wrong cadence out of your postings or you just had a bad day or something. Experience told me, if a post of someone makes you feel uncomfortable, don't reply to him, that is the road down to flaming and frustration, and it's usually of no use to the other readers of a mailing list. Still you seem to have a lot of valuable knowledge and if you'd share some more of it, you can probably help a lot of coders to understand the nasty details behind the wonderful world of a modern OS. So I'm pushing my luck... This is not true; I have myself authored at least a dozen replies to variations on this basic question in multiple forums. I like your stile of quoting and commenting, it's good, old style comment below stile. I have only searched Apple's mailing list archive and I have not found anything; Apple has too many ifdef __APPLE__ in the kernel to say for sure that xnu behaves like any other BSD kernel. It's also nowhere documented. This is, techically speaking, not true; it is documented by the (freely available) source code and the activity of your system. Well, source code is no real documentation. The difference is, it took me about 1-2 hours to figure out how top calculates private and shared memory, something a documentation had told me in 2 sentences in 3 minutes. How comes 1 + 2 != 3??? What makes you think that they would be? Because all memory a process "has" (all pages in a process address space) are either shared or private. So the real memory a process used should be the sum of both. That the private and shared are calculated in such a unfavourable fashion is another thing. "resident memory" is an old OS term, and you will find it defined in many OS textbooks. If you have reached this point, it's more or less assumed that you will have read Tanenbaum or something more recent, and the concept should be self-explanatory. 'This column reflects the amount of physical memory currently allocated to each process. This is also known as the "resident set size" or RSS. A process can have a large amount of virtual memory allocated (as indicated by the SIZE column) but still be using very little physical memory.' I think you'll find that memory mapped MAP_ANON is accounted as private Which makes perfectly sense, since how could it be not private? ANON memory can't be shared. MAP_PRIVATE is handled as COW Which already makes a bit less sense, but it's no issue, since COW memory with only one reference is still accounted towards private memory by top (and private memory will only have one reference). and MAP_SHARED is erroneously always accounted as shared even if nobody else has that range of the object mapped. That is a bit stupid, since it does not reflect reality. OTOH drawing buffers of Cocoa apps are in fact really shared (between the application and the window server, that's what I saw on some technote) and still saying "they are shared" seems awkward, as it's technically correct, but these buffers only exist because of the application and without it they will also vanish from the window server, so they could as well be accounted towards that application only. It's kind of you to call shared libraries "clever", but see above inre: OS textbooks. They've been around a long time. 8) "clever" was not referring to shared libraries themselves or how they work, it was referring to the fact how top looks at the reference count of their pages to decide if they are accounted towards shared or private memory. In the case of shared submaps or shared pmap regions, pages may be resident for a task that has never referenced them. Clustered pageins can also cause this behaviour. Well, you are all focusing way too much onto the system libraries and their 256 MB area. It's not that I care a lot for these or that these make me worry a lot. The system libraries are around in the system most likely regardless if I run a specific process or not, since almost every process uses them in one way or another. I rather care for our own libraries or non standard system libraries. Top is not telling you, except in the crudest sense, how much real memory your process uses. Then what is it telling me? If the values are so useless for everything, then why displaying them in the first place? Note that Activity Monitor (which displays the same values as top, but I have no source to back this up, I can only compare them) is no developer tool. It is used by normal, simple minded users, that have no idea what "virtual memory" actually means. And this utility displays real, private and shared memory to the user. What do you expect will a simple minded user think, when seeing these values? Simple minded users are using these values to determine memory consumption of a process. Now if these values are so far away from reality, they are useless and should not even be displayed to simple minded users. Instead, this all should be replaced with a single value, that really can give the user a rough estimation of how much memory this process needs. Understanding the real memory usage of your task is better handled with different tools. I would encourage you to tinker with Shark and the X-ray instruments to see if you can't get a better understanding of what your process is doing and what that costs. These tools basically tell me how much memory I have "wasted" using malloc. Even in the best case, they might take my stack into account. But what they won't take into account is the memory lost by caching code pages, the memory lost by loaded libraries and their code pages, and so on. And in rarest cases they will even distinguish private and shared memory at all. So if I know my processes uses malloc to allocate about 800 kb of memory during runtime, this could not be any further away from the real number of memory the system loses by running my process as it is. This is an approximation of memory usage, that is far, far away from reality and a much worse approximation than all the values mentioned before. How much memory does the system need, to keep my process running, assuming it was not allow to swap anything of it (everything needs to be in memory) and here it should take any memory into account that belongs to my process directly or indirectly, which includes malloc, stack, code, static memory, code of libraries and static memory of libraries, coded needed by the dyld - and that clearly distinguishes between shared and unshared memory. What would this number be good for? Well, it tells me "even if you had no swap space at all, you could run about X instances of my process on a system with 512 MB RAM, assuming that no other process needs memory and the kernel operates within its normal parameters". Or I could say "if you start my process, the system will need at least X MB free memory to handle my process needs, otherwise it will swap". If I take the malloc value here, it's nowhere near reality. My process may only malloc 800 kb of memory, but if your system has only 800 kb of memory spare, it will not be able to run my process without swapping (either to swap file or by permanently reloading code pages with the vnode-pager, which is not much different to a page-in from swap-file). This email sent to site_archiver@lists.apple.com