November 2007 - Darwin-dev - lists.apple.com

Re: pthread_cancel and C++ destructors
by Terry Lambert 30 Nov '07

30 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Missing libstdc++.dylib symbolic link in 10.4u SDK
by Pierre Baillargeon 30 Nov '07

30 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On 29-Nov-07, at 16:34 , Terry Lambert wrote: On 10.5, both in the system and in its 10.5 SDK, it is indeed a copy. -- Pierre Baillargeon _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Nov 29, 2007, at 11:18 AM, Pierre Baillargeon wrote: I've checked on a 10.4 system, and under the /usr/...etc.../4.0.1/ there is indeed a symbolic link back to the proper dylib three levels up. I think you broke something on your machine; this works for me (it's a copy, not a symlink on my machine). On 10.4, both in the system and its 10.4u SDK, as sen on a co-worker's machine, it's a symlink. This email sent to site_archiver(a)lists.apple.com

1 0

Leopard and kvm_open*()
by Steve Thompson 30 Nov '07

30 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Hi all, Steve _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I conducted some dialog with this list on this subject back in January 2007, but now that Leopard is here I have some additional questions. I have written a batch processing engine. Imagine the situation where a daemon launches a shell script on behalf of another user. We have no control over what that shell does; all we know is the name of the shell and its script content. However, we need to know periodically, while the top-level shell is still running, what its CPU total usage is so that we can report back to the user on the progress of the job, and can perhaps enforce a CPU time limit. All I know about the running shell is the PID of the top-level process; by doing a sysctl(CTL_KERN, KERN_PROC, KERN_PROC_ALL) I can trace through the process relationships and work out the total resource consumption, but only if I can get the CPU used by terminated children of an arbitrary PID via the proc->p_stats->p_cru fields. Right now (on Tiger) I am doing a kvm_read() to get this information, but in Leopard this is no longer possible I assume. I'd appreciate some pointers about how I might get at the p_cru info on Leopard. TIA, This email sent to site_archiver(a)lists.apple.com

1 0

Re: pthread_cancel and C++ destructors
by Ethan Tira-Thompson 30 Nov '07

30 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com solution.cc

1 0

Re: Leo prebinding...
by Jeshua Lacock 30 Nov '07

30 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Nov 29, 2007, at 11:37 PM, Eric Gouriou wrote: ...on the 10.5 system, and does not hang on the same message on 10.4. No, they were built with the Mac OS X 10.4.x dev tools. As far as I know there is no prebinding done in Leopard. Thanks for the tips, I will do as you suggest. Thanks again, Jeshua Lacock, Owner <http://OpenOSX.com> phone: 877.240.1364 _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Our build of AbiWord launches in about 2 seconds on Mac OS X 10.4, while it takes about 60 seconds on Mac OS X 10.5 (on the same machine), with similar results for our other applications built on the same GNOME stack including Gnumeric, Gnucash, etc. I set the "DYLD_PREBIND_DEBUG" variable, and the same list of files stating the image is not prebound is presented on both systems. The only difference is that it hangs on the message: dyld: image not prebound, so could not use prebinding in /tmp/ OpenOSX/office/bin/AbiWord-2.4 Can any recommend something to try or point me towards some documentation? Were those binaries built with the Leopard tools ? In the absence of other feedback I'd recommend 1) Profiling the launch (via sample and shark) to see if a hotspot appears 2) File a bug report, providing the profile information 3) Provide the bug number on the list, and also a highlight of what samples and shark reported. This might be enough information to point to the issue This email sent to site_archiver(a)lists.apple.com

1 0

Re: Resident Memory, Private Memory, Shared Memory (top, Activity Monitor and vmmap)
by Markus Hanauska 30 Nov '07

30 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Hi Michael, On Tue, 2007-11-27, at 22:53, Michael Smith wrote: 1. The Resident Memory Size (aka Real Memory) 2. The Private Memory Size 3. The Shared Memory Size So far, all these values seems self explaining... but they are not at all for one single reason: Resident according to top's own explanation is: Now it gets interesting. That would be which tools? The number I'm looking for is: -- Best Regards, Markus Hanauska _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I'm sorry that I have ignored your posts so far, but I considered your posts little helpful (compared to what Jim posted, which was really giving me some insight into the things behind the scene) and also kind of "harsh" and "derogative" (What? You don't know this? Geez, how pitiful). Might also be a misinterpretation on my side, not being a native English speaker, I may read the wrong cadence out of your postings or you just had a bad day or something. Experience told me, if a post of someone makes you feel uncomfortable, don't reply to him, that is the road down to flaming and frustration, and it's usually of no use to the other readers of a mailing list. Still you seem to have a lot of valuable knowledge and if you'd share some more of it, you can probably help a lot of coders to understand the nasty details behind the wonderful world of a modern OS. So I'm pushing my luck... This is not true; I have myself authored at least a dozen replies to variations on this basic question in multiple forums. I like your stile of quoting and commenting, it's good, old style comment below stile. I have only searched Apple's mailing list archive and I have not found anything; Apple has too many ifdef __APPLE__ in the kernel to say for sure that xnu behaves like any other BSD kernel. It's also nowhere documented. This is, techically speaking, not true; it is documented by the (freely available) source code and the activity of your system. Well, source code is no real documentation. The difference is, it took me about 1-2 hours to figure out how top calculates private and shared memory, something a documentation had told me in 2 sentences in 3 minutes. How comes 1 + 2 != 3??? What makes you think that they would be? Because all memory a process "has" (all pages in a process address space) are either shared or private. So the real memory a process used should be the sum of both. That the private and shared are calculated in such a unfavourable fashion is another thing. "resident memory" is an old OS term, and you will find it defined in many OS textbooks. If you have reached this point, it's more or less assumed that you will have read Tanenbaum or something more recent, and the concept should be self-explanatory. 'This column reflects the amount of physical memory currently allocated to each process. This is also known as the "resident set size" or RSS. A process can have a large amount of virtual memory allocated (as indicated by the SIZE column) but still be using very little physical memory.' I think you'll find that memory mapped MAP_ANON is accounted as private Which makes perfectly sense, since how could it be not private? ANON memory can't be shared. MAP_PRIVATE is handled as COW Which already makes a bit less sense, but it's no issue, since COW memory with only one reference is still accounted towards private memory by top (and private memory will only have one reference). and MAP_SHARED is erroneously always accounted as shared even if nobody else has that range of the object mapped. That is a bit stupid, since it does not reflect reality. OTOH drawing buffers of Cocoa apps are in fact really shared (between the application and the window server, that's what I saw on some technote) and still saying "they are shared" seems awkward, as it's technically correct, but these buffers only exist because of the application and without it they will also vanish from the window server, so they could as well be accounted towards that application only. It's kind of you to call shared libraries "clever", but see above inre: OS textbooks. They've been around a long time. 8) "clever" was not referring to shared libraries themselves or how they work, it was referring to the fact how top looks at the reference count of their pages to decide if they are accounted towards shared or private memory. In the case of shared submaps or shared pmap regions, pages may be resident for a task that has never referenced them. Clustered pageins can also cause this behaviour. Well, you are all focusing way too much onto the system libraries and their 256 MB area. It's not that I care a lot for these or that these make me worry a lot. The system libraries are around in the system most likely regardless if I run a specific process or not, since almost every process uses them in one way or another. I rather care for our own libraries or non standard system libraries. Top is not telling you, except in the crudest sense, how much real memory your process uses. Then what is it telling me? If the values are so useless for everything, then why displaying them in the first place? Note that Activity Monitor (which displays the same values as top, but I have no source to back this up, I can only compare them) is no developer tool. It is used by normal, simple minded users, that have no idea what "virtual memory" actually means. And this utility displays real, private and shared memory to the user. What do you expect will a simple minded user think, when seeing these values? Simple minded users are using these values to determine memory consumption of a process. Now if these values are so far away from reality, they are useless and should not even be displayed to simple minded users. Instead, this all should be replaced with a single value, that really can give the user a rough estimation of how much memory this process needs. Understanding the real memory usage of your task is better handled with different tools. I would encourage you to tinker with Shark and the X-ray instruments to see if you can't get a better understanding of what your process is doing and what that costs. These tools basically tell me how much memory I have "wasted" using malloc. Even in the best case, they might take my stack into account. But what they won't take into account is the memory lost by caching code pages, the memory lost by loaded libraries and their code pages, and so on. And in rarest cases they will even distinguish private and shared memory at all. So if I know my processes uses malloc to allocate about 800 kb of memory during runtime, this could not be any further away from the real number of memory the system loses by running my process as it is. This is an approximation of memory usage, that is far, far away from reality and a much worse approximation than all the values mentioned before. How much memory does the system need, to keep my process running, assuming it was not allow to swap anything of it (everything needs to be in memory) and here it should take any memory into account that belongs to my process directly or indirectly, which includes malloc, stack, code, static memory, code of libraries and static memory of libraries, coded needed by the dyld - and that clearly distinguishes between shared and unshared memory. What would this number be good for? Well, it tells me "even if you had no swap space at all, you could run about X instances of my process on a system with 512 MB RAM, assuming that no other process needs memory and the kernel operates within its normal parameters". Or I could say "if you start my process, the system will need at least X MB free memory to handle my process needs, otherwise it will swap". If I take the malloc value here, it's nowhere near reality. My process may only malloc 800 kb of memory, but if your system has only 800 kb of memory spare, it will not be able to run my process without swapping (either to swap file or by permanently reloading code pages with the vnode-pager, which is not much different to a page-in from swap-file). This email sent to site_archiver(a)lists.apple.com

1 0

Re: pthread_cancel and C++ destructors
by Lassi Tuura 30 Nov '07

30 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Importance: Normal User-agent: SquirrelMail/1.4.6 Hi, > 2. read() cannot be made to return by interruption with a signal, > although the man pages suggests a return code for that condition. (I > was sending a SIGALRM with my own handler installed is there a > particular signal I need to send, perhaps without installing my own > handler?) Honestly it sounds like you'd be better of working on a different level of abstraction. iostream really isn't designed for stuff like this. And at the other level you can avoid a lot of these issues. Someone already mentioned but you need to check that SA_RESTART is off if you want read() not to resume; that might get the EINTR to propagate. However that might just get you into another swamp with std::iostream, but well, if that's your choice of technology, ... > 5. I'm basically recreating the asynchronous cancel type.... because > it's apparently not supported at all on OS X (even ignores explicit > pthread_testcancel calls, much less exit from busy loop!) This might be suggesting that you might want to consider a design that simply does not require this feature. Your description sounds like these features just are not mature enough for real production use yet? Lassi _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Leo prebinding...
by Eric Gouriou 30 Nov '07

30 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com ...on the 10.5 system, and does not hang on the same message on 10.4. Eric _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Nov 29, 2007, at 11:34 AM, Jeshua Lacock wrote: Our build of AbiWord launches in about 2 seconds on Mac OS X 10.4, while it takes about 60 seconds on Mac OS X 10.5 (on the same machine), with similar results for our other applications built on the same GNOME stack including Gnumeric, Gnucash, etc. I set the "DYLD_PREBIND_DEBUG" variable, and the same list of files stating the image is not prebound is presented on both systems. The only difference is that it hangs on the message: dyld: image not prebound, so could not use prebinding in /tmp/ OpenOSX/office/bin/AbiWord-2.4 Can any recommend something to try or point me towards some documentation? Were those binaries built with the Leopard tools ? As far as I know there is no prebinding done in Leopard. In the absence of other feedback I'd recommend 1) Profiling the launch (via sample and shark) to see if a hotspot appears 2) File a bug report, providing the profile information 3) Provide the bug number on the list, and also a highlight of what samples and shark reported. This might be enough information to point to the issue This email sent to site_archiver(a)lists.apple.com

1 0

Re: pthread_mutex_unlock not permitted in signal handler?
by Eric Gouriou 30 Nov '07

30 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Eric _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Nov 29, 2007, at 8:37 PM, Ethan Tira-Thompson wrote: I did realize however, that read() has a EINTR error return code, so instead of trying to unlock within the signal handler itself, perhaps I could just make the handler a no-op, and then test for the error code following the read(). However, in its infinite wisdom, the system resumes the read and doesn't bat an eye at the signal coming and going, so this idea never actually interrupts the read(), nor causes the unlock and exit to occur. If you are using sigaction() to establish the signal handler, it's time to double-check that SA_RESTART is _not_ being set. --- If a signal is caught during the system calls listed below, the call may be forced to terminate with the error EINTR, the call may return with a data transfer shorter than requested, or the call may be restarted. Restart of pending calls is requested by setting the SA_RESTART bit in sa_flags. The affected system calls include open(2), read(2), write(2), sendto(2), recvfrom(2), sendmsg(2) and recvmsg(2) on a communications channel or a slow device (such as a terminal, but not a regular file) and during a wait(2) or ioctl(2). However, calls that have already committed are not restarted, but instead return a partial success (for example, a short read count). --- However the EINTR behavior is specified for read(2), I don't know if one can get iostream::read() to avoid calling read(2) again upon receiving that errno. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Resident Memory, Private Memory, Shared Memory (top, Activity Monitor and vmmap
by Michael Smith 30 Nov '07

30 Nov '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Nov 28, 2007, at 8:46 AM, darwin-dev-request(a)lists.apple.com wrote: On Wed, 2007-11-28, at 00:08, Jim Magee wrote: This value as returned by task_info() indicates how many page-level pmap mappings are currently present in the task's address space. In other words, how many valid page table entries there are for this processes current use of its virtual address space. This also means this value is the most uninteresting one for me as developer. But if the objects are bigger than what we have mapped, we may not be able to make dead accurate accounting. Not sure what you are trying to say with that sentence. If an object is 200 pages, and 20 pages are in fact present in physical memory (180 are not), then the accounting is perfectly correct when it says 20 * page_size is the used memory of this object. It may not be my process causing these 20 pages to be in physical memory, but since this object is mapped into my address space, I keep the memory object itself alive and thus I keep the 20 pages alive and mapped to memory, don't I? No. Private and shared memory sizes are an even greater approximation than the resident memory size. But they return different info. The private and shared memory counts say "how many pages of the objects I have mapped are cached in memory", regardless of whether I've accessed them yet/lately, where the resident memory size says "how many page-table-entries does my process already have to quickly access those cached pages". Okay, maybe I asked the wrong question here. Maybe the question should rather be how to interpret these values. E.g. what does it mean if a process has very little resident memory, but huge shared and huge private memory? Without other contextual information, not a lot. This means I have memory objects mapped into my address space, large ones, that have many physical memory pages assigned to them and that are either private or shared, however, I have not touched many of these pages lately or at all, right? Not necessarily. So my process is not really using many of the pages of these objects, still, I keep these objects alive and since they are alive, a lot of pages are lost of memory as these objects need them to keep process-private or process-shared data in memory. A real live example, I load a big library, a huge one, that is in use by many other processes, too. The code segment of that library will be shared (COW). The text segment for a shared library is normally mapped read-only. Since some other process uses this library heavily, many of these COW code pages are resident in physical memory (cached so to say, if you see all physical RAM just as a cache for the swap file or other mapped binary files). This will raise my shared memory value to a huge value. However, if I never access any of the pages of the code segment of the library, all this memory is not accounted towards my resident size. Is this interpretation more or less accurate? I think I understood the case above, but how to interpret a process that has a huge resident memory size, but shared and private memory together are much less? What would be a real live example for that? Why would this process have so many physical pages mapped into its virtual address space, if all the objects it has mapped there have so little physical pages? What kind of pages would that be? Doesn't every page mapped into my process space needs to belong to some memory object in the kernel? One side question: A virtual page is just virtual. The opposite of a virtual page is... a physical one? There is no "opposite" for a virtual page. If I run vmmap with -resident, it will print the virtual size of every object and the physical size of every object (resident size). For the resident size, does it play a role whether the page is currently really in memory or swapped? See above inre: the definition of "resident". = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… As I already attempted to point out (you did read to the end of my last message, right?), these numbers are not going to be interesting to you. Apple makes tools that will tell you much more useful and interesting things about your application. You should use them, rather than complaining that numbers that do other things aren't doing what you want. We can try to explain some of the finer details here, but really if you're not going to take advice, or do the background reading that's necessary for you to understand what's going on, there's a limit to how much this is going to help... You have mappings that touch objects with large resident counts, both shared and private mappings. You don't have many pages currently mapped, either because you haven't touched many, or because they've been stolen, or evicted. No. The fact that you have mappings against these objects implies nothing (reliable) about the residency status of pages outside the ranges you have mapped. If this is a system framework, you may inherit a shared sub-pmap, and the residency stats for the shared pmap are common to all of the address spaces containing it. In the case of a shared pmap, the pages are technically resident regardless of whether you've touched them or not. For non-system libraries (which are much less frequently shared) this is a more reasonable approximation. It's posible (and common) to allocate process virtual space that isn't backed by a mapped object that's accounted for in those numbers. As has been previously noted in this thread, the accounting for these numbers is an approximation. This email sent to site_archiver(a)lists.apple.com

1 0