How limits are enforced (was: various maxproc limits)
How limits are enforced (was: various maxproc limits)
- Subject: How limits are enforced (was: various maxproc limits)
- From: Terry Lambert <email@hidden>
- Date: Thu, 29 Sep 2005 16:22:51 -0700
The stock answers are "Why do you need to do that?" and "Don't do
that unless you know what you are doing or you will be sorry".
Before actually answering your questions, it's likely helpful to set
the stage for the answer. So here is the design information for the
current limit enforcement model on MacOS X.
--
There are six types of limits in MacOS X; in enforcement order, they
are:
1) Physical resources limits
E.g.: you cannot wire more memory down than you have installed
in your machine
These limits are constrained by available physical resources;
you cannot wire
down more memory than you actually have; you cannot start more
processes than
you can create address spaces for, given the pmap
architecture; you cannot
create a non-sparse file that takes up more disk space than
you have installed
in the machine; etc.
2) Compilation time limits
E.g.: The default value for "sysctl kern.maxfiles" is 12288
These limits are implementation limits that are set at
compilation time, either
as a result of compilation manifest constants, or a compile
time computation
based on compile time manifest constants. Some of these
values are stored in
variables and can be modified; some of them are stored in
internal variables,
and cannot be modified. Some of them are inlined in code, and
cannot be modified.
Compilation limits enforce the maximum values of some limits;
sometimes, these
limits default to values less than the maximum, and can be
adjusted up to the
compilation limit; sometimes they default to the maximum, and
can only be set
down (or back up to the limit, if previously set down), and
sometimes, they "just
are": you can't change them up or down.
3) Compilation time default limits
E.g.: kern.maxproc = 532
These are limits which provide defaults which we attempt to
guarantee will not
cause problems for the minimum supported hardware
configuration for a given
version of MacOS X.
The main purpose of compilation time default limits is to keep
you from crashing
your machine. It's possible for some of these to adjust a
limit from a lower
"compilation time default limit" up to a higher "compilation
time limit"; if you
do this on a minimum hardware configuration, however, it's
possible for your
machine to become unstable or crash as a result when you
attempt to exceed a
"physical resources limit" as a result.
One of the primary things we use this for is to default
differences between our
expectation of desktop vs. server usage, based on our
knowledge of the differences
between minimum desktop/laptop machine configurations, and
those os servers (such
as the minimum X Serve hardware configuration).
Both the defaults for kern.maxproc and kern.maxprocperuid,
among others, are set
at this level.
4) Limits controllable by sysctl
E.g.: kern.maxproc, kern.maxprocperuid
These are limits which are changeable from the range of a
minimum compilation
limit up to a maximum compilation limit.
The primary purpose of these limits being adjustable at boot
time in the /etc/rc
files or via a user-supplied /etc/launchd.conf file, is to
permit differential
values for the MacOS X vs. the MacOS X Server products, based
on expected minimum
configurations. For example, the MacOS X server installation
modifies the system
startup files to increase the value of kern.maxproc, with the
knowledge that there
will be sufficient RAM installed in the hardware on which it
is intended to be
loaded to actually support that level of usage.
5) Administrative hard limits
E.g. "max user processes" from "ulimit -Hu" defaults to the
value of kern.maxproc
These limits are limits which a given process is not allowed
to exceed. They
are inherited down based on a parent/child process
relationship from the parent
process. This relationship can be identified using various
system status commands,
such as using "ps -gaxlwww" and looking at the "PPID" coulumn
for a given process.
Inherited hard limits may be set lower by a child process, but
they may not be
raised once lowered, unless a process is privileged (i.e. it
is run by "root" or
it is an SUID "root" program).
6) Administrative soft limits
E.g. "max user processes" from "ulimit -u" defaults to 100 (or
the value of "max
user processes" from "ulimit -Hu", if it is smaller)
These limits are limits which a given process is not allowed
to exceed. Hoever,
unlike "administrative soft limits", an unprivileged process
may adjust these up
and down between the minimum allowable value (if any), and the
"administrative
hard limit", at will. These values are also inherited by
child processes.
On Sep 29, 2005, at 3:05 PM, Yefim Somin wrote:
I have run into process limits on Mac OS X 10.4.2 and looked into
changing various values, but I am unclear about their
relationships, absolute limitations and semantics.
1) kern.maxproc, kern.maxprocperuid
These were initially set to very low values of 532 and 100. I
changed them, apparently successfully in the following way.
administrator$ sudo sysctl -w kern.maxproc=2068
kern.maxproc: 2065 -> 2068
administrator$ sudo sysctl -w kern.maxprocperuid=2068
kern.maxprocperuid: 2000 -> 2068
administrator$ sysctl -a | grep maxproc
kern.maxproc = 2068
kern.maxprocperuid = 2068
Question 1: attempts to set them to values higher than 2068
resulted in rejection. Is this the absolute high limit and can it
be changed?
It is the absolute high limit. On older versions of MacOS X, it's
possible, but inadvisable to change this limit. On newer versions,
it will be completely impossible.
Before I tell you how to hack this on older MacOS X systems, let me
state that whatever you are doing that requires this many processes
would be much better served by using threads in your design instead.
-
One of the main issues we were addressing when we set this limit at
the value we set it to (it is computed off of "MAXUSERS" + 20
reserved slots for required system proceses, which is why it's
defaulted to 512 + 20 and has a hard limit of 2048 + 20) is the
number of 51 bit address spaces it's possible to simultaneously
create on the PPC architecture. Another main issue is that each
process consumes a certain amount of system resources, including
computational resources. The final main issue is that for 64 bit
processes, in order to support addressing more than 51 contiguous
bits worth of space, multiple address spaces must be used. 2068 is
less than the absolute maximum number of address spaces by a factor
of almost 8, but it's the "sweet spot" for average configurations of
RAM, CPU power, and what we expected to be default load.
My *sincere* recommendation is that you rethink the architecture of
your software, and consider using threads instead of processes, where
appropriate, rather than attempt to raise this limit.
You can change this limit by linking a program against libkvm and
modifying the value of the global maxproc. Such modifications of the
running kernel are unsupported, and if your system fails as a result
of such modification, you can expect to be denied support
assistance. The global maxproc is used to enforce the compilation
time limit (#2) against the sysctl controllable limit (#4).
2) Limit maxproc
Limit command revealed that its “maxproc” is still set to 100:
administ% limit
cputime unlimited
filesize unlimited
datasize 6144 kbytes
stacksize 8192 kbytes
coredumpsize 0 kbytes
memoryuse unlimited
descriptors 256
memorylocked unlimited
maxproc 100
Trying to set maxproc to unlimited using “limit” set it also to 2068!
Question 2: how is it related to sysctl limits? Is the one
displayed by “limit” hard or soft one?
I think I answered this one in the introduction.
Question 3: on this list it was implied that if you change limits
by “limit” command they will only be valid for that same shell tree
where it was done, not systemwide. Is that really so?
For administrative limits, hard (#5) or soft (#6), this is true. For
the other limits, this is not true.
A complication which arises is that the initial values of these
limits drive down the inheritance tree for creating processes. This
has impact on your decisions in this regard from three vantage points:
1) The current launchd is limited in that the configuration
changes you might
want to make in this regard are in fact not as persistent as
you would
normally like them to be. For these settings to be inherited
down the
entire process tree, they must be set by launchd. This can be
overcome by
creating an /etc/launchd.conf file to set the hard and soft
resource limits
at the start of the system's life, and thereafter they will be
inherited down.
2) The current launchd is incapable of calling sysctl on your
behalf. As a result,
its administrative limits are capped at the defaults at start
time. There is a
radar report outstanding against launchd in this regard. This
cannot be worked
around in a global context until the radar is resolved by
changing launchd (but
see below).
3) When processes are started from the dock, they are started via
sending a Mach
message to another process in your session, and are not forked
from the dock
itself. The same is true of the shell "open" command line
utility. As a
result, even if you set your administrative limits in a
terminal window and use
"open", the new process will inherit the limits from the
application which forks,
rather than the application which triggered the fork. So the
limits will not
change.
Luckily, you can overcome all of these issues by using an SUID "root"
helper application to adjust the administrative limits of (#5) and
(#6) withing the bounds of the applicable sysctl limits, if any (#4),
and the compilation limits (#2) and (#3), subject to having the
physical resources to support the subsequent operations not causing
your system to become unstable because of your changes.
-
At this point, I will once again suggest you move from "thousands of
processes" to "thousands of threads" instead; you will be happier
with the performance that way anyway.
Question 4: if it’s too hard to answer the above questions without
looking at the source, where could I get that source?
It's not hard to answer, it's just hard to wrap your head all the way
around it. 8-).
The source is, as always, available for download from opendarwin.org.
-- Terry
Thanks in advance,
ys
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Darwin-kernel mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
40apple.com
This email sent to email@hidden
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Darwin-kernel mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden