November 2007 - Darwin-dev - lists.apple.com

Resident Memory, Private Memory, Shared Memory (top, Activity Monitor and vmmap)
by Markus Hanauska 27 Nov '07

27 Nov '07

site_archiver(a)lists.apple.com Delivered-To: Darwin-dev(a)lists.apple.com Hi! 1. The Resident Memory Size (aka Real Memory) 2. The Private Memory Size 3. The Shared Memory Size How comes 1 + 2 != 3??? Is this really so hard to answer? Okay, your turn, please share you wisdom :) -- Best Regards, Markus Hanauska _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I'd like to ask a simple question. This question has been asked about ten times on various mailing lists here in the past and whenever somebody asked this question directly or indirectly, he never received a reply. Either the question was ignored in the reply or the post never received any reply in the first place. This either means nobody wants to answer it or nobody can answer it... maybe nobody knows it, which would be very, very sad, as that is one of the most fundamental things one should know. It's also nowhere documented. I even searched many MacOS internals books for an answer, but I had no success. It's simply not explained anywhere. It can't be that we have to file a support incident just to receive a reply to this very simple question. So I try it again and I will try it over and over again, till I find someone who can answer it. When you use top or Activity Monitor, does not matter, you receive three interesting memory results per process. The VM stats are not interesting (no matter if VM private or VM total). The three values of interest are: So far, all these values seems self explaining... but they are not at all for one single reason: There are processes, where 1 + 2 is much bigger than 3. How comes? Where is the memory lost? Which pages count towards 1 or 2, but are ignored for 3? For other processes, 1 + 2 is much smaller than 3. How comes? Which memory is ignored for 3, but is accounted towards 1 or 2? I looked at the source of top, here's what I found out (some people might find this useful): A) The resident memory is obtained using the call task_info(). Since it is nowhere documented what this call will count towards memory of the task and what not in the field resident_size, it's basically magic and I can't investigate this value any further. B) The other two values are obtained by getting every memory region of a process using mach_vm_region() (same function vmmap uses) and show some interesting behavior. Basically all private memory is summed up as private memory. Real shared memory (e.g. mmap memory) is summed up as shared memory. This is expected behavior. Memory that is in theory share-able (Copy on write) is treated differently depending on whether it is actually shared. E.g. code of the process or libraries used by the process is counted towards private memory if it is only referenced once (it could be shared, but currently it isn't). That means libraries shared with other processes are counted towards shared memory. This is indeed very clever. If I start a second instance of my process, my code regions are shared between both processes, so the private memory decreases and the shared one raises, which reflects what really goes on within the system. Same with some library code. E.g. libraries that only my process uses are first counted towards my private memory. With a second instance using them, they are counted towards shared memory. You see, pretty clever and desired behavior. C) Only "touched" memory is accounted at all. This is also very clever. Just because I load a library does not mean, that all library code is in memory. Only space for all is reserved. Unless I ever access any page of that library, the page is not really in memory. Same for my code or malloc'ed memory. This is also desired behavior. D) Still top overestimates the real memory my process uses. E.g. I use a library that another process uses, too. I only need one function of the library. If I was the only one using it, only the page containing this function would be real memory, that means only one page would be accounted towards my process (the rest stayed virtual). Now the other process uses all of the library. Because of this, all pages of the library are loaded to real memory. Since I use the library, too, now all the loaded pages are accounted to my process as shared memory, just as they are accounted to the other process. In fact, only one of the pages is in memory because of my process and in use by my process. The rest is only in memory because of the other process. So more shared memory is accounted towards my process than my process actually uses. But since it is accounted towards shared memory, this is something I can live with (forcing the kernel to remember because of which process which virtual page has been mapped into physical memory would be pretty much overkill and bloat all memory structures enormously). After reading the top source, I understand pretty much how the values Shared and Private memory are calculated (the source basically is self explaining in this aspect), what they actually mean and how to interpret them. But I have no idea what resident memory is, what it means, where it comes from and why it is so much different than the sum of private and shared for many processes. Also playing around with vmmap could not really answer that. BTW, vmmap is pretty much useless to build a tool to determine the real memory use of a process. This is because a section, that is COW (Copy on Write) will stay COW, even if some pages within it have already been copied. These pages would now count towards private, but vmmap won't tell me how many pages have already been shadowed like this and whether a page is shared between many processes or already private towards one of the processes makes a huge different. What I'm missing is a tool that works just like vmmap, but reports statistics about every single *PAGE* of memory of a process space, not just every larger region and that takes into account if this page has been copied or not for a COW page. Then one could really make a tool that prints out the real shared and private memory amounts of my process, except for issue (D), where everything of the library that has physical memory is counted towards my shared memory (to correct that, I had to eliminate all other processes using the library and then create the values). This email sent to site_archiver(a)lists.apple.com

1 0

Re: Why no crash reporter if I install some signal handler on Leopard
by Jonas Maebe 27 Nov '07

27 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On 27 Nov 2007, at 17:28, Steve Sisak wrote: Jonas _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Nov 26, 2007, at 1:23 AM, zhanglin <zhanglin.zhou(a)gmail.com> wrote: If I want to install the signal handler and make the crash reporter dialog to be shown too, what should I do? Pass the exception in signal hander? How? Is there a best practice for generating a crash report for a signal that one partially handles? Install the signal handler as SA_RESETHAND and let the signal be triggered again by simply returning from the signal handler after you've done what you want (should work at least for SIGFPE, SIGSEGV and SIGBUS as long as you don't start modifying the signal context). This email sent to site_archiver(a)lists.apple.com

1 0

Re: Why no crash reporter if I install some signal handler on Leopard
by Steve Sisak 27 Nov '07

27 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com The OP understands that. -Steve _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… At 1:19 PM -0800 11/26/07, Terry Lambert wrote: A consequence of this is that caught signals that would otherwise be fatal no longer trigger crash reports. He asks what to do if he wants to do something in response to the signal (and therefore needs to install an handler), but still wants a crash report generated: On Nov 26, 2007, at 1:23 AM, zhanglin <zhanglin.zhou(a)gmail.com> wrote: If I want to install the signal handler and make the crash reporter dialog to be shown too, what should I do? Pass the exception in signal hander? How? Is there a best practice for generating a crash report for a signal that one partially handles? This email sent to site_archiver(a)lists.apple.com

1 0

Re: Daemon/launchd crashes with EXC_BREAKPOINT (SIGTRAP)
by Eric Gouriou 27 Nov '07

27 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Is launchd a RAM addict?
by Stéphane 27 Nov '07

27 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Lowering Thread Priority
by Duane Murphy 27 Nov '07

27 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com I would like to lower the priority of a background thread in my kext. I'm using thread_policy_set() with the THREAD_PRECEDENCE_POLICY. At best I am making a guess at what the importance number should be for this thread. I'd have a better understanding of what to use for importance if I could call thread_policy_get() except that thread_policy_get() is not exported <sigh>. This one thread is a background thread while most of the other threads are IO requests from userland, so I'm assuming I really shouldn't be changing any priorities there. "Using Mac Scheduling From User Applications" says that an importance number of 0 is idle, described as "threads set to IDLE_PRI will only execute when no other thread is scheduled to execute." This seems too low for my purposes; that is i'd like it to run sometimes, just not as high priority as the IO request threads. Another part of "Using Mac Scheduling From User Applications" says "While this is a signed 32-bit value, the minimum legal value is zero (IDLE_PRI)." However, some reading in past emails use negative values for importance. (I can't even find a definition for IDLE_PRI). How can I determine an appropriate value for importance? Or is there a better way to lower the priority of this thread? Some background on this thread. This thread does not run all the time. It is called upon to do a lot of work for a period of time and then stop. The work is in competition with the IO requests, but should have a lower priority to ensure that the IO requests get through in a reasonable amount of time. Thanks for any tips on how to lower the priority of this background thread. ...Duane _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Why no crash reporter if I install some signal handler on Leopard
by Terry Lambert 26 Nov '07

26 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com -- Terry On Nov 26, 2007, at 1:23 AM, zhanglin <zhanglin.zhou(a)gmail.com> wrote: Hi, all, Best Regargs zhanglin _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/tlambert%40apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… If you catch and handle the signal, then by definition you haven't crashed. Leopard crash handling changed to issue a specific exception on crashes, so there are fewer spurious reports. A consequence of this is that caught signals that would otherwise be fatal no longer trigger crash reports. Does any one know why no crash reporter if I install some signal handler on Leopard? On Tiger, even tough I install the handler, I can get the crash reporter too. If I want to install the signal handler and make the crash reporter dialog to be shown too, what should I do? Pass the exception in signal hander? How? This email sent to tlambert(a)apple.com This email sent to site_archiver(a)lists.apple.com

1 0

Re: Why no crash reporter if I install some signal handler on Leopard
by Justin C. Walker 26 Nov '07

26 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Nov 26, 2007, at 01:23 , zhanglin wrote: Hi, all, This seems to work fine for me. Justin -- Justin C. Walker, Curmudgeon-At-Large Institute for the Absorption of Federal Funds -------- Men are from Earth. Women are from Earth. Deal with it. -------- _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Does any one know why no crash reporter if I install some signal handler on Leopard? On Tiger, even tough I install the handler, I can get the crash reporter too. If I want to install the signal handler and make the crash reporter dialog to be shown too, what should I do? Pass the exception in signal hander? How? I'm not sure what you mean. Are you trying to catch signals in your code (using signal(3))? This email sent to site_archiver(a)lists.apple.com

1 0

Why no crash reporter if I install some signal handler on Leopard
by zhanglin 26 Nov '07

26 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:from:to:content-type:content-transfer-encoding:mime-version:subject:date:x-mailer; bh=RtaOo5h4dlZMLHBP6+jJBYfu80PMpCvDScEEQFqXmQ8=; b=w5SZk/LQbo/NMpVVESZu23u1AHEOT+dOGwwtdIYzElqP5w8Nq91CUA3h0O/0TUw3PrMVNO23OA2tpzBZL54Q7Eu8p8kQ0i0JsCUAaPfP7ZJboYyUTgrhpl25b5YkE2/gpOE80iDFpsCc3tAV7JdkFTZzd2b+gYtREfwkw9Oo4kg= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=received:message-id:from:to:content-type:content-transfer-encoding:mime-version:subject:date:x-mailer; b=M/UH/Fchst31IMDjEUZYx7++vQZGt5i7PyuRJh1MzYw+YcBCCiunNML4S4c120i9dlrVWJtDjnqdxD2mHmMRWnNY1+M5NPJ7nKa99hNrn4v2tSM5L3pz4SPj/+hM6TH2veCEAQuWl9MFIvlFYmx+5DhKlzTqhPJNbWlBVH57VvI= Hi, all, Best Regargs zhanglin _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Does any one know why no crash reporter if I install some signal handler on Leopard? On Tiger, even tough I install the handler, I can get the crash reporter too. If I want to install the signal handler and make the crash reporter dialog to be shown too, what should I do? Pass the exception in signal hander? How? This email sent to site_archiver(a)lists.apple.com

1 0

Re: Daemon/launchd crashes with EXC_BREAKPOINT (SIGTRAP)
by Elango C 26 Nov '07

26 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; bh=cNVb9AQcLgdFe5VmR0CFuYHsCboaTw3XYumsQCaMYY0=; b=OSeasKYQMTF+Y4hZ5n9W0Ves1WrcNC/IU/o68T10Vl0AnsKrEVpFUE8gXPiITTuRRaFmZEJP7eAxNn2cnxvFlsBuGBDxgK6MbH7RaomNvbPWsmrticHqDkDTl7NwwdxBz2XjSjfWS/l0FygRJFPhfper0dtnc6ctDzTdPFuLLgs= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=Rv6FQ6Qxw+WIe35zI/3Pc8BCeZ4ehnl+/6cKvYek+oarKxZHyWx9948IjPhbIoP4ep0DlaL8tdxwUHt8nzEsAPFOr0HL3k88lxco3dS4x2INvcCPfcFtNMonCs6LUsWbAwzs+HJQ3IRp0EsqDhdIOPyu/nnTJjewmCSKM8RKpQ8= _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0