December 2008 - Darwin-dev - lists.apple.com

Re: what is expected malloc behavior re. limits/insufficient resources?
by IainS 27 Dec '08

27 Dec '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com 6467883: 10.4/10.5 setrlimits are not honored by memory allocators. 6467884: 10.X systems are not robust when paging space is exceeded There are attached programs to both bugs that demonstrate the effects, ... and cross-references to the get/setrlimits manpages and this thread. cheers, Iain _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On 25 Dec 2008, at 11:00, IainS wrote: I suggest filing a problem report. I'll do this and copy the radar number here This email sent to site_archiver(a)lists.apple.com

1 0

Re: Shared library & install names
by Martin Costabel 27 Dec '08

27 Dec '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com User-agent: Thunderbird 2.0.0.18 (Macintosh/20081105) For this kind of things, filebyproc.d is less overwhelming than dtruss. -- Martin _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Terry Lambert wrote: On Dec 26, 2008, at 7:09 PM, Ryan Govostes <rgovostes(a)gmail.com> wrote: I discovered my misconception; I thought @loader_path would be the path to etree.so, but in fact it is the path to its parent directory. Things now seem to be working. It would be nice if there was a DYLD_ debug variable to show where it's looking for libraries, as this would have saved me a considerable amount of time. The dtruss command will tell you about attempts to look up and get descriptors for shared objects. Alternately, you could write a script that only prints pat resolution attempts, which would make the failures to find things and where it's looking when it fails pretty obvious. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Shared library & install names
by Terry Lambert 27 Dec '08

27 Dec '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Dec 26, 2008, at 7:09 PM, Ryan Govostes <rgovostes(a)gmail.com> wrote: I discovered my misconception; I thought @loader_path would be the path to etree.so, but in fact it is the path to its parent directory. Things now seem to be working. It would be nice if there was a DYLD_ debug variable to show where it's looking for libraries, as this would have saved me a considerable amount of time. The dtruss command will tell you about attempts to look up and get descriptors for shared objects. Alternately, you could write a script that only prints pat resolution attempts, which would make the failures to find things and where it's looking when it fails pretty obvious. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Shared library & install names
by Ryan Govostes 27 Dec '08

27 Dec '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:content-type :content-transfer-encoding:references:subject:message-id:in-reply-to :to:date:mime-version:x-mailer; bh=vXpCf8o++NoZUotDbnN/y/YMhT808j+cSYY0ZiTP1Tc=; b=cJlsG+L6IVYEztFxmGmYsuV5uaWMWrlsK9XmBPMCzdWSRSw5wCY/qpS7b/ei+bEktq yNn9Jinh2Lhpxl4mzJT70wHUJa090DUJiwHrj2sOn5Zjw4qWoEIwnR8hsC0RuDza9ouL k89bwSV6PyWwwzJopnjG0plTeGhHkKLa0/g3o= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:content-type:content-transfer-encoding:references:subject :message-id:in-reply-to:to:date:mime-version:x-mailer; b=xvsHzg6AkjFiqqIK1DacCYinPHMAa7MD1TCfPQ7C5N4rGCBcszbFzgp31HMOH59vkJ 5OZpbXd2D0H9oBJsbukpfrXCS00ydkX4VZXwy5c0Rf4vXa/Ow38ybFTPjohS5w/7DKJT h12R7v6S7Nu/xvfopjEkFVX6tw1NwSYumjAjs= Cheers, Ryan Govostes _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I discovered my misconception; I thought @loader_path would be the path to etree.so, but in fact it is the path to its parent directory. Things now seem to be working. It would be nice if there was a DYLD_ debug variable to show where it's looking for libraries, as this would have saved me a considerable amount of time. This email sent to site_archiver(a)lists.apple.com

1 0

Shared library & install names
by Ryan Govostes 27 Dec '08

27 Dec '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:content-type :content-transfer-encoding:mime-version:subject:message-id:to:date :x-mailer; bh=emu5EnPscWh2ZbsB9prbx8cM0d0F6lN7xZBxAcUnDvc=; b=oKBmSMNz4/2qvso/Zkt0PJfDjlpFflSmGa+Y0U+ce7gQZ1gUTOyymPBERbCazG+Ovb I9EDWVdlg2lZ77F+9e4M57+LDt4s8TAnlDSLFWxd65NnGqggN5iGZ6/nRUGxlRAP4Vxw ykCWkWoqqt9/bDVbq4bcqW611o/RjK+HGuYhc= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:content-type:content-transfer-encoding:mime-version:subject :message-id:to:date:x-mailer; b=UQtjF0DZ8YOnxxYGJJidqIKVLxE1BvphnrGiJkItCpJeZO7Kk7g3EU5EWQM9l0udBy y+ntcDD/s66k0NhAEzNLaT93elFItzISjVMcoSCpJskTmhhsvBl+C3fqwXpZuVcrpzen JDnJ+QXD1w09kLjxkJEcq4s8EG4vHxS5pbE4A= Hi list, dyld: loaded: ~/.python-eggs/pystuff.egg-tmp/pystuff/shared.so dyld: loaded: /usr/lib/libxml2.2.dylib Regards, Ryan Govostes _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I'm packaging up a Python .egg for distribution. The Python code calls functions in a shared library, which in turn imports from libxml2. I need a newer version of libxml2 than what ships with OS X, so I've built my own, and want to add this to the .egg. However, I'm having a hard time figuring out the install names for the shared library (shared.so) and my custom built libxml2.dylib. For instance, I tried setting the install path of libxml2 to @loader_path/../libxml2.dylib as follows: ~/.python-eggs/pystuff.egg-tmp/pystuff/shared.so: @loader_path/../libxml2.dylib (compatibility version 10.0.0, current version 10.2.0) /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.3) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 117.0.0) /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0) /~.python-eggs/pystuff.egg-tmp/pystuff/libxml2.dylib: @loader_path/../libxml2.dylib (compatibility version 10.0.0, current version 10.2.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 117.0.0) /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.3) /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0) /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0) However, DYLD_PRINT_LIBRARIES reveals it's loading /usr/lib/ libxml2.2.dylib despite my intentions: This suggests to me that it's not "finding" my libxml2 despite being in the same directory as shared.so. I take it that I misunderstood what I've read about install names. What should I look into doing? This email sent to site_archiver(a)lists.apple.com

1 0

Re: what is expected malloc behavior re. limits/insufficient resources?
by Terry Lambert 26 Dec '08

26 Dec '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com In a situation where the malloc request: is < MALLOC_ABSOLUTE_MAX_SIZE but > [total RAM + total free swap partition space] or > ulimit -v , or ulimit -d (regardless of soft/hard) Cheers, -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Dec 25, 2008, at 4:58 PM, Jack Howarth <howarth(a)bromo.med.uc.edu> wrote: it seems that malloc() returns a non-NULL pointer (i.e. without error). It returns a pointer into a page mapping in the process virtual address space. No physical pages are allocated until you touch them. This can exceed total RAM plus swap +because of memory overcommit, which has pretty much been SOP on unix since the DEC VAX version introduced virtual memory. Terry, Why does linux treat this situation as a malloc error? Does UNIX conformance imply acceptance of situations like this that have been addressed by the linux community? Jack (NB: I can talk about what administrative limits mean in the context of UNIX conformance, if you want, but that doesn't seem to be the issue you want to talk about here) Assuming, for the sake of your argument, things work the way you have indicated on all Linux distributions with every malloc() implementation, I wouldn't so much say Linux has "addressed" the "situation" as that they've made different design decisions than us based on implementation details and how they expect their product to be used. I've previously pointed out one difference in implementation detail that bears strongly on this particular issue, which is static allocation of swap partition space on Linux systems. Given a static bounding on that, if they ignore competing processes, they can make decisions based on how much swap will be available in the future, no matter what else happens; we can't, since creating/extending (non- swap) files detracts from the actual amount for us. On the other hand, on Linux, if you run out of swap because you have two processes that unwittingly cooperate to run you out, game over. If the same thing happens on Mac OS X (or Windows), and there's free disk space available, more space is allocated automatically, and you merrily chug along. Unless you have badly broken application code or an already full disk, that's going to be a much better user experience than "time to repartition your disk and reinstall your OS". If you think one of Linux's decisions is wrong, you can give the maintainers of whatever distribution code that implements things a different way, and Alan Cox or one of the other gatekeepers can make a decision to accept or reject your code. If you have the same problem with Mac OS X, you can do the same thing via a problem report (or even file the report without including code; you don't have to be a computer programmer to report what you think is a problem). At this point, I suggest you file a problem report; please include your use case that demonstrates why it's a problem, since that will give the engineer it lands on more ammunition to make the change you want if they can say it's a demonstrable customer-facing issue. This email sent to site_archiver(a)lists.apple.com

1 0

Re: what is expected malloc behavior re. limits/insufficient resources?
by Jack Howarth 26 Dec '08

26 Dec '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com User-agent: Mutt/1.5.18 (2008-05-17) >> In a situation where the malloc request: >> >> is < MALLOC_ABSOLUTE_MAX_SIZE >> >> but > [total RAM + total free swap partition space] >> >> or > ulimit -v , or ulimit -d (regardless of soft/hard) >> >> it seems that malloc() returns a non-NULL pointer (i.e. without error). > > It returns a pointer into a page mapping in the process virtual address space. No physical pages are allocated until you touch them. This can exceed total RAM plus swap +because of memory overcommit, which has pretty much been SOP on unix since the DEC VAX version introduced virtual memory. Terry, Why does linux treat this situation as a malloc error? Does UNIX conformance imply acceptance of situations like this that have been addressed by the linux community? Jack _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: what is expected malloc behavior re. limits/insufficient resources?
by IainS 25 Dec '08

25 Dec '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Thanks Terry, for your helpful response, I believe that there are two distinct kinds of overcommit: Kind 2/ Where the system allows a single process to overcommit. I'll do this and copy the radar number here (a) Is malloc() supposed to honor ulimits? Here are my answers, though: have a great Xmas, Iain _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Merry Xmas all round... but just before we head off to over- indulge..... I would welcome your comments on a couple more points. On 25 Dec 2008, at 09:03, Terry Lambert wrote: On Dec 24, 2008, at 3:05 PM, IainS <developer@sandoe- acoustics.co.uk> wrote: On 24 Dec 2008, at 22:47, Terry Lambert wrote: This can exceed total RAM plus swap because of memory overcommit, which has pretty much been SOP on unix since the DEC VAX version introduced virtual memory. Let's define (for the sake of illustration) available = [ram + free swap - wired] ; overcommit as "anything > available". Kind 1/ with multiple competing processes (**all of which which would individually fit within 'available'**). A suitable and successful strategy is to serialize the access to the resources by making processes sleep when a memory demand cannot be met. The assumption is that processes will eventually complete and, although the system might be sluggish owing to swapping, it remains accessible. Some form of prioritization allows at least one process to proceed - such that a deadlock doesn't occur. Everyone's happy. In this case, assuming that the process uses the allocated memory, (and without a grim reaper watchdog process) this will eventually result in a locked system (the amount of time will depend on the offending process' priority). (a) the offending process cannot complete - it will always end up sleeping whilst waiting for some unavailable blocks (which can never be made available, because the disk is full). (b) most likely on the way to this the majority of other processes will be swapped out - and cannot be swapped back in because there is no more backing available for the RAM. (c) in the end all processes that rely on allocation of memory will be asleep - whilst the system has not "technically" crashed - it is actually in a vegetative state and we might as well switch off the life support... (d) anything that needed space on the root partition (including all system and user databases in a default installation) has probably lost data. They mostly did it by having a 32 bit address space, twice as much preconmitted swap as physical RAM. Indeed, I remember this well as a rule-of-thumb. Although, actually, the problem does affect older OSX 32 bit machines with smaller disks - if the same criteria are met (insufficient backing). If that didn't work, then thr approach which was commonly used was to kill programs in order to recover resources. the problem we have here is that it's frequently the case that the system has become so sleep-bound that you cannot obtain an interface to do this - and in any event, there's no automatic way to recover - and a typical non-command-line User is not going to be able to succeed. It's even possible that Activity Monitor would fail to quit the process since it might need memory to pop up the "force-quit, quit, cancel" dialogue. You can also opt a program out of overcommit several ways, but you typically have to replace the malloc, or not use it. Most of these boil down to forcing swap to be preconmitted by dirtying the pages or locking them into physical memory. This is usually inconvenient for both the programmer. Also the other processes sharing the system have to do without those resources, even if they are not being actively used by the greedy precommit process. hm. I wouldn't ask to set aside resources for this - the basis of my thesis here is that there is no point allowing a process to (actually) overcommit (I accept that virtual overcommit is useful) -- somehow one has to draw the line when paging space runs out. I suggest filing a problem report. BTW: in the case that a kernel extension is stupid enough not to wire its memory ... this can also cause a panic (but since the panic is probably down to a 3rd party extension I would not expect it receive much attention). (b) Is malloc() supposed to allow requests in excess of available system resources? Is there any official answer to these? This isn't an official support channel. To get an official answer, you'd need to file a problem report. (a) No. Currently the limits are voluntary. For example, gcc uses getrlimit() to look at the limits and voluntarily limits its working set size by choosing not to exceed the limits. Hard limits can be set by system admin (in my admin days 'normal' users had to request single-user access to machines if they wanted to do stuff like this). Soft voluntary limits would actually solve my problems - so long as malloc honors them. (b) Yes. Allocations that it makes are for virtual, not physical, resources. Virtual resources are effectively near-infinite, and you could be using an algorithm that has sparse utilization of the virtual resources. If so, then there's no reason your algorithm shouldn't be allowed to work, due to an arbitrary administrative limit enforced against a virtual resource. Your argument is good - and I agree it's challenging - but one ought to be able to find a way to "bail out" in the case that a process does not make sparse use of the resource. Perhaps an overcommitted process could be marked as "do bus error, instead of sleeping when backing runs out" (which is similar to one of your suggestions, too). Typically these are one user or single purpose machines. hm .. I run Mail, iTunes, and numerous other background processes which all suffer database corruption if the system blocks this way. OK, it's not a DEC20 timesharing .. but., "single user" is stretching things a bit these days ;-) Back in my admin days, it was also possible to do this on multiuser machines, and our typical reaction was to disable the offending users account. For a single user machine, you could simply reinstall the offending software and contact the vendor for a fix. I can honestly say that this never happened AFAIK on the Vax, Primes and Suns we used (and I've worked in R&D all my life with people persistently trying wacky things :-) ; normal user accounts had restrictive limits. (I do not say it's impossible, just that I don't believe I ever saw it happen ). If it cannot be regarded as a bug, perhaps honoring ulimits or providing RLIMIT_VMEM would be a very useful enhancement? You can file a problem report, but we are unlikely to add an RLIMIT_VMEM for the same reason top can't give you an exact answer on who is responsible for shared pages: cardinality hides this information, even from the OS itself. perhaps then, the desideratum is unachievable (preventing a single processes from over-committing). Limiting your processes utilization of the available (by virtue of addressable bits) virtual address space with RLIMIT_DATA is much more likely, but that would be either voluntary or up to its parent process to set before the program is started. as I said above, admin can set hard limits for RLIMIT_XXXX (I know there's no absolute guarantee - but one can make things considerably more robust). If you choose to raise the limits for a single process or user -- least you then enter the territory with eyes-wide- open (and Mail and iTunes and anything else with a db shut down... ) For a responsible user testing code which is intended to stress the system, voluntary limits are entirely satisfactory -- providing they are honored by malloc. This email sent to site_archiver(a)lists.apple.com

1 0

Re: what is expected malloc behavior re. limits/insufficient resources?
by Terry Lambert 25 Dec '08

25 Dec '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Sorry, I was cut off earlier... I suggest filing a problem report. (a) Is malloc() supposed to honor ulimits? Here are my answers, though: Typically these are one user or single purpose machines. -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Dec 24, 2008, at 3:05 PM, IainS <developer(a)sandoe-acoustics.co.uk> wrote: On 24 Dec 2008, at 22:47, Terry Lambert wrote: This can exceed total RAM plus swap because of memory overcommit, which has pretty much been SOP on unix since the DEC VAX version introduced virtual memory. I'm sure you're right --- but these platforms must have had some way of saving you --- I don't remember having this trouble ever on a VAX or a Sun (or even a Prime)... ;-) They mostly did it by having a 32 bit address space, twice as much preconmitted swap as physical RAM. If that didn't work, then thr approach which was commonly used was to kill programs in order to recover resources. A frequent implementation was to pick the first program that asks you for more of any resource for which the request can't be satisfied as a "volunteer". You can also opt a program out of overcommit several ways, but you typically have to replace the malloc, or not use it. Most of these boil down to forcing swap to be preconmitted by dirtying the pages or locking them into physical memory. This is usually inconvenient for both the programmer. Also the other processes sharing the system have to do without those resources, even if they are not being actively used by the greedy precommit process. Subsequent use of the returned pointer can (and does) bring the system down Define "down". unusable - generally to the extent that a hard reboot is required. once you see a "no free paging space left" message thing are looking grim. Yes the system will spend most of its time attempting to clean pages for you to satisfy your applications demands. AFAICT any process that tries to request memory after the system has run out of paging space is put to sleep. Yes, until some can be reclaimed (see above; the alternative is to kill it when it asks). (b) Is malloc() supposed to allow requests in excess of available system resources? Is there any official answer to these? This isn't an official support channel. To get an official answer, you'd need to file a problem report. (a) No. Currently the limits are voluntary. For example, gcc uses getrlimit() to look at the limits and voluntarily limits its working set size by choosing not to exceed the limits. (b) Yes. Allocations that it makes are for virtual, not physical, resources. Virtual resources are effectively near-infinite, and you could be using an algorithm that has sparse utilization of the virtual resources. If so, then there's no reason your algorithm shouldn't be allowed to work, due to an arbitrary administrative limit enforced against a virtual resource. This behavior is troublesome - because a simple non-privileged program compiled -m64 can easily stop the system non-privileged programs ought not to be able to make the system unusable (IMO). Back in my admin days, it was also possible to do this on multiuser machines, and our typical reaction was to disable the offending users account. For a single user machine, you could simply reinstall the offending software and contact the vendor for a fix. does any of the behavior described constitute a bug -- or perhaps an enhancement ? If it cannot be regarded as a bug, perhaps honoring ulimits or providing RLIMIT_VMEM would be a very useful enhancement? You can file a problem report, but we are unlikely to add an RLIMIT_VMEM for the same reason top can't give you an exact answer on who is responsible for shared pages: cardinality hides this information, even from the OS itself. Limiting your processes utilization of the available (by virtue of addressable bits) virtual address space with RLIMIT_DATA is much more likely, but that would be either voluntary or up to its parent process to set before the program is started. This email sent to site_archiver(a)lists.apple.com

1 0

PureDarwin Xmas
by probono 25 Dec '08

25 Dec '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com The PureDarwin project announces the immediate availability of "PureDarwin Xmas", a developer preview of the upcoming operating system based on Apple's Darwin 9 sources and other Open Source projects such as X11. At the same time, the PureDarwin project would like to invite the community to discuss, participate and contribute. The developer preview is available for download as a pre-configured virtual machine for VMware Fusion 2.0 on Macintosh, and the code used to generate it is available in a Subversion repository. Merry Xmas! -- http://www.puredarwin.org/ _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0