Re: what is expected malloc behavior re. limits/insufficient resources?
Re: what is expected malloc behavior re. limits/insufficient resources?
- Subject: Re: what is expected malloc behavior re. limits/insufficient resources?
- From: Terry Lambert <email@hidden>
- Date: Thu, 25 Dec 2008 01:03:27 -0800
Sorry, I was cut off earlier...
On Dec 24, 2008, at 3:05 PM, IainS <email@hidden>
wrote:
On 24 Dec 2008, at 22:47, Terry Lambert wrote:
This can exceed total RAM plus swap because of memory overcommit,
which has pretty much been SOP on unix since the DEC VAX version
introduced virtual memory.
I'm sure you're right --- but these platforms must have had some way
of saving you --- I don't remember having this trouble ever on a
VAX or a Sun (or even a Prime)... ;-)
They mostly did it by having a 32 bit address space, twice as much
preconmitted swap as physical RAM. If that didn't work, then thr
approach which was commonly used was to kill programs in order to
recover resources. A frequent implementation was to pick the first
program that asks you for more of any resource for which the request
can't be satisfied as a "volunteer".
You can also opt a program out of overcommit several ways, but you
typically have to replace the malloc, or not use it. Most of these
boil down to forcing swap to be preconmitted by dirtying the pages or
locking them into physical memory. This is usually inconvenient for
both the programmer. Also the other processes sharing the system have
to do without those resources, even if they are not being actively
used by the greedy precommit process.
Subsequent use of the returned pointer can (and does) bring the
system down
Define "down".
unusable - generally to the extent that a hard reboot is required.
I suggest filing a problem report.
once you see a "no free paging space left" message thing are
looking grim.
Yes the system will spend most of its time attempting to clean
pages for you to satisfy your applications demands.
AFAICT any process that tries to request memory after the system has
run out of paging space is put to sleep.
Yes, until some can be reclaimed (see above; the alternative is to
kill it when it asks).
(a) Is malloc() supposed to honor ulimits?
(b) Is malloc() supposed to allow requests in excess of available
system resources?
Is there any official answer to these?
This isn't an official support channel. To get an official answer,
you'd need to file a problem report.
Here are my answers, though:
(a) No. Currently the limits are voluntary. For example, gcc uses
getrlimit() to look at the limits and voluntarily limits its working
set size by choosing not to exceed the limits.
(b) Yes. Allocations that it makes are for virtual, not physical,
resources. Virtual resources are effectively near-infinite, and you
could be using an algorithm that has sparse utilization of the virtual
resources. If so, then there's no reason your algorithm shouldn't be
allowed to work, due to an arbitrary administrative limit enforced
against a virtual resource.
This behavior is troublesome - because a simple non-privileged
program compiled -m64 can easily stop the system
non-privileged programs ought not to be able to make the system
unusable (IMO).
Typically these are one user or single purpose machines.
Back in my admin days, it was also possible to do this on multiuser
machines, and our typical reaction was to disable the offending users
account. For a single user machine, you could simply reinstall the
offending software and contact the vendor for a fix.
does any of the behavior described constitute a bug -- or perhaps
an enhancement ?
If it cannot be regarded as a bug, perhaps honoring ulimits or
providing RLIMIT_VMEM would be a very useful enhancement?
You can file a problem report, but we are unlikely to add an
RLIMIT_VMEM for the same reason top can't give you an exact answer on
who is responsible for shared pages: cardinality hides this
information, even from the OS itself.
Limiting your processes utilization of the available (by virtue of
addressable bits) virtual address space with RLIMIT_DATA is much more
likely, but that would be either voluntary or up to its parent process
to set before the program is started.
-- Terry
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Darwin-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden