Re: Semi OT: Apache goes into frozen state

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Semi OT: Apache goes into frozen state

Subject: Re: Semi OT: Apache goes into frozen state
From: Johnny Miller <email@hidden>
Date: Wed, 06 Apr 2011 08:38:30 -1000

Hi Johan,

Yeah, I'm not doing anything unusual like creating a DB connection per server.

If I run from the CLI:

sudo lsof | grep httpd | wc -l

The count kind of bounces up and down between 500 - 1,500

If I just look at the contents instead of the count I see a bunch of redundant connections to access_log and the various modules (see below)

To be honest I'm out of my element here, so I'm not sure if this is unusual or not. I've updated the apache's launch daemon plist to increase the number of connections apache is allowed to 2048 and I've gone a couple of days without a failure. But I did do an intentional restart last night. Prior to altering the plist I was seeing apache going into the unresponsive state about every 24 hours.

When I called Apple they told me to run netstat -m and look at this section:

0 requests for memory denied

0 requests for memory delayed

If that goes over 0 you can increase the memory buffer per this article: http://support.apple.com/kb/HT3512 But I've yet to see that number go over zero.

(this prints out several times)

httpd 6511 _www txt REG 14,2 1996352 17733353 /usr/sbin/httpd

httpd 6511 _www txt REG 14,2 393792 17733502 /usr/lib/libaprutil-1.0.2.7.dylib

httpd 6511 _www txt REG 14,2 651248 15133 /usr/lib/libexpat.1.5.0.dylib

httpd 6511 _www txt REG 14,2 4147008 15324 /usr/lib/libiconv.2.dylib

httpd 6511 _www txt REG 14,2 578656 17733501 /usr/lib/libapr-1.0.2.7.dylib

httpd 6511 _www txt REG 14,2 53888 17733291 /usr/libexec/apache2/mod_authn_file.so

httpd 6511 _www txt REG 14,2 66500 17733295 /usr/libexec/apache2/mod_authz_host.so

httpd 6511 _www txt REG 14,2 201360 17733299 /usr/libexec/apache2/mod_cache.so

httpd 6511 _www txt REG 14,2 70660 17733308 /usr/libexec/apache2/mod_dumpio.so

httpd 6511 _www txt REG 14,2 126528 17733321 /usr/libexec/apache2/mod_mem_cache.so

httpd 6511 _www txt REG 14,2 117296 17733307 /usr/libexec/apache2/mod_disk_cache.so

httpd 6511 _www txt REG 14,2 105944 17733311 /usr/libexec/apache2/mod_ext_filter.so

httpd 6511 _www txt REG 14,2 245316 17733316 /usr/libexec/apache2/mod_include.so

httpd 6511 _www txt REG 14,2 91784 17733312 /usr/libexec/apache2/mod_filter.so

httpd 6511 _www txt REG 14,2 121380 17733305 /usr/libexec/apache2/mod_deflate.so

httpd 6511 _www txt REG 14,2 126208 17733318 /usr/libexec/apache2/mod_log_config.so

httpd 6511 _www txt REG 14,2 54200 17733320 /usr/libexec/apache2/mod_logio.so

httpd 6511 _www txt REG 14,2 53956 17733309 /usr/libexec/apache2/mod_env.so

httpd 6511 _www txt REG 14,2 71040 17733310 /usr/libexec/apache2/mod_expires.so

httpd 6511 _www txt REG 14,2 100768 17733313 /usr/libexec/apache2/mod_headers.so

httpd 6511 _www txt REG 14,2 66568 17733314 /usr/libexec/apache2/mod_ident.so

httpd 6511 _www txt REG 14,2 70940 17733334 /usr/libexec/apache2/mod_setenvif.so

httpd 6511 _www txt REG 14,2 378832 17733325 /usr/libexec/apache2/mod_proxy.so

httpd 6511 _www txt REG 14,2 168484 17733330 /usr/libexec/apache2/mod_proxy_http.so

httpd 6511 _www txt REG 14,2 138088 17733327 /usr/libexec/apache2/mod_proxy_balancer.so

httpd 6511 _www txt REG 14,2 786308 17733336 /usr/libexec/apache2/mod_ssl.so

httpd 6511 _www txt REG 14,2 915632 17731671 /usr/lib/libssl.0.9.7.dylib

httpd 6511 _www txt REG 14,2 100588 17733322 /usr/libexec/apache2/mod_mime.so

httpd 6511 _www txt REG 14,2 120612 17733337 /usr/libexec/apache2/mod_status.so

httpd 6511 _www txt REG 14,2 176016 17733298 /usr/libexec/apache2/mod_autoindex.so

httpd 6511 _www txt REG 14,2 54000 17733284 /usr/libexec/apache2/mod_asis.so

httpd 6511 _www txt REG 14,2 101080 17733317 /usr/libexec/apache2/mod_info.so

httpd 6511 _www txt REG 14,2 127384 17733301 /usr/libexec/apache2/mod_cgi.so

httpd 6511 _www txt REG 14,2 70560 17733343 /usr/libexec/apache2/mod_vhost_alias.so

httpd 6511 _www txt REG 14,2 167388 17733324 /usr/libexec/apache2/mod_negotiation.so

httpd 6511 _www txt REG 14,2 66448 17733306 /usr/libexec/apache2/mod_dir.so

httpd 6511 _www txt REG 14,2 95568 17733315 /usr/libexec/apache2/mod_imagemap.so

httpd 6511 _www txt REG 14,2 58204 17733282 /usr/libexec/apache2/mod_actions.so

httpd 6511 _www txt REG 14,2 71144 17733335 /usr/libexec/apache2/mod_speling.so

httpd 6511 _www txt REG 14,2 66460 17733340 /usr/libexec/apache2/mod_userdir.so

httpd 6511 _www txt REG 14,2 79308 17733283 /usr/libexec/apache2/mod_alias.so

httpd 6511 _www txt REG 14,2 448160 698607 /System/Library/WebObjects/Adaptors/Apache2.2/mod_WebObjects.so

httpd 6511 _www txt REG 14,2 305712 17733333 /usr/libexec/apache2/mod_rewrite.so

httpd 6511 _www txt REG 14,2 290736 3429086 /usr/lib/libexslt.0.dylib

httpd 6511 _www txt REG 14,2 1297968 15260 /usr/lib/libiodbc.2.dylib

httpd 6511 _www txt REG 14,2 1046544 17731388 /System/Library/Frameworks/LDAP.framework/Versions/A/LDAP

httpd 6511 _www txt REG 14,2 958864 17733803 /usr/lib/libcurl.4.dylib

httpd 6511 _www txt REG 14,2 401216 17733947 /usr/lib/libsasl2.2.dylib

httpd 6511 _www txt REG 14,2 88356 172780 /usr/libexec/apache2/mod_auth_apple.so

httpd 6511 _www txt REG 14,2 101284 172785 /usr/libexec/apache2/mod_spnego_apple.so

httpd 6511 _www txt REG 14,2 105824 172775 /System/Library/PrivateFrameworks/SpnegoApache.framework/Versions/A/SpnegoApache

httpd 6511 _www txt REG 14,2 684032 17731066 /System/Library/Frameworks/DirectoryService.framework/Versions/A/DirectoryService

httpd 6511 _www txt REG 14,2 135724 172782 /usr/libexec/apache2/mod_digest_apple.so

httpd 6511 _www txt REG 14,2 66280 174412 /usr/libexec/apache2/mod_hfs_apple.so

httpd 6511 _www txt REG 14,2 1380352 18089764 /private/tmp/WOAdaptorState

httpd 6511 _www txt REG 14,2 31641660 17733495 /usr/libexec/apache2/libphp5.so

httpd 6511 _www txt REG 14,2 1380352 18089793 /private/tmp/WOAdaptorState

httpd 6511 _www txt REG 14,2 1059792 8952176 /usr/lib/dyld

httpd 6511 _www txt REG 14,2 98283520 17736835 /private/var/db/dyld/dyld_shared_cache_x86_64

httpd 6511 _www 0r CHR 3,2 0t0 345489284 /dev/null

(same here this repeats many times)

httpd 6511 _www 11w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

httpd 6511 _www 12w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

httpd 6511 _www 13w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

httpd 6511 _www 14w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

httpd 6511 _www 15w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

httpd 6511 _www 16w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

httpd 6511 _www 17w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

httpd 6511 _www 18w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

httpd 6511 _www 19w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

httpd 6511 _www 20w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

httpd 6511 _www 21w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

httpd 6511 _www 22w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

httpd 6511 _www 23w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

httpd 6511 _www 24w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

httpd 6511 _www 25w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

httpd 6511 _www 26w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

httpd 6511 _www 27w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

httpd 6511 _www 28w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

httpd 6511 _www 29w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

httpd 6511 _www 30w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

httpd 6511 _www 31w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

httpd 6511 _www 32w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

httpd 6511 _www 33w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

httpd 6511 _www 34w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

httpd 6511 _www 35w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

httpd 6511 _www 36w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

httpd 6511 _www 37w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log

On Apr 6, 2011, at 7:53 AM, Johan Henselmans wrote:

On 5 apr 2011, at 02:40, Johnny Miller wrote:

Hi,

I was wondering if anyone else had experienced the same thing. I have a X-Serve running OS X 10.5.8 whose apache server is going into a frozen state where it stops serving requests and can only be restarted by rebooting the machine. All other services seem to being running normally.

If I look at the error_log I see many variations of this error:

Too many open files in system:Some/path/on/system

I've seen this thread on the Apple discussion board: http://discussions.apple.com/thread.jspa?threadID=1449787&start=15&tstart=0
And this article in the knowledge base: http://support.apple.com/kb/HT3512

Does anyone have more wisdom to share?

I once had a similar problem when I started my own objectstore coordinator for each session, along the line of:
// this causes all kinds of mayhem: do not do this at home, it creates a new database connection for every session

// EOObjectStoreCoordinator coord = new EOObjectStoreCoordinator();
// myEc = ERXEC.newEditingContext(coord);
// myEc().setDelegate(this);

After that, the database would have so many open connections after a while that the database would have too many open files, and would break down.

I also seem to remember that some apache configuration would freeze with a specific log-rotation setup. (that was apache 1.3 on a FreeBSD machine)

Furthermore I would look for something in apache that opens a file but never closes these. You might use opensnoop for that, see man opensnoop, or lsof might give you some clues.

Thanks in advance,

Johnny Miller
Kahalawai Media Corp
http://www.kahalawai.com

Johan Henselmans
email@hidden

Johnny Miller

Kahalawai Media Corp

http://www.kahalawai.com

 _______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list      (email@hidden)
Help/Unsubscribe/Update your Subscription:

This email sent to email@hidden

Follow-Ups:
- Re: Semi OT: Apache goes into frozen state
  - From: Pascal Robert <email@hidden>

References:
	>Semi OT: Apache goes into frozen state (From: Johnny Miller <email@hidden>)
	>Re: Semi OT: Apache goes into frozen state (From: Johan Henselmans <email@hidden>)

Prev by Date: Re: Migrations not executed at startup problem (SOLVED)
Next by Date: Re: Migrations not executed at startup problem (SOLVED)
Previous by thread: Re: Semi OT: Apache goes into frozen state
Next by thread: Re: Semi OT: Apache goes into frozen state
Index(es):
- Date
- Thread