Hi,
Also, is there an utility to watch the kernel file table state,
i.e. see the current usage in percents and see a list of major
"contributors" to that usage?
And that's the key question. The answer is 'lsof'. Find out which
Hmm. Right now, some 5 hours after the last reboot, we've got around
5552 open handles, with 3140 owned by smbd.
Is this outrageous? Is it standard?
Anyway, given the following:
herr:~ root# sysctl -a | grep -i files
kern.maxfiles = 12288
kern.maxfilesperproc = 10240
I would be very interested to know if he had a series of ssh login
failures anytime in the 6 hours or so before the problem occured.
eg - in secure.log:
Nov 30 07:46:15 arkx com.apple.SecurityServer: authinternal failed to
authenticate user tester.
Nov 30 07:46:15 arkx com.apple.SecurityServer: Failed to authorize right
system.login.tty by process /usr/sbin/sshd for authorization created by
/usr/sbin/sshd.
Though if the problem is as I now believe lifting the max files by an
order of magnitude will likely result in no further problem - as 1000
connections per minute even when under a probing attack is quit excessive.
ArkX:
Nov 30 12:00:59
through to 5pm.
File table is full errors. Perhaps an automatic reboot triggered at 5??
Boot drive mainly filled up with log.smbd (samba log) which as well as
the kernel log was logging the inability to open any more files. Samba
log file > 6GB.
Noted: this morning from 8:27am to 9:05am an ssh probe hammering ...
Steps taken:
1. Moved ssh to port 3822 (same as FlowX). [perhaps the most likely
cause IMO]
2. /etc/sysctl.conf - changed the file limits to the same things I had
set FlowX to.
I'm intrigued about the possibility of sshd being the problem ... the
logs indicate that Apple has buggerised around with the way sshd does
authorisation.
OK - it is definitely ssh causing the server to fall over ... I have
just reported to Apple what I have discovered:
- ssh email@hidden
just press enter at the password prompt...
Hmmm - problem ... it just sits there looking stupid for 2 minutes!
Each such sshd process on the server uses over 50 file handles !!!
With the default max files limit of 12,288 ... that is 245 half open
sshd processes in a 2 minute window before your server falls over.
[and I think an unclean TCP break causes the sshd process to actually
stay open longer than that 2 minute thingy].
Once it hits this hard max files things for the first time it would be
no surprise to see other processes go a little crazy and leave the
server somewhat frozen (and certainly on 100% CPU).
And of course once the machine has managed to have too many file handles
open and services start failing ... the watchdog process should restart
the machine ... but how is it going to do that? Call reboot or
something? Does that call need to open more file handles?
So hope with that change we are all sorted with OS X boxes. [though I'm
giving FlowX a reboot now just for good luck - it hasn't been up for
this long for some time (5.5 days!)]
Well past home time but I think I'd best post this to one of the OS X
server mailing lists. (someone mentioned this problem might be samba a
week or so ago ... I think not!)
