• Open Menu Close Menu
  • Apple
  • Shopping Bag
  • Apple
  • Mac
  • iPad
  • iPhone
  • Watch
  • TV
  • Music
  • Support
  • Search apple.com
  • Shopping Bag

Lists

Open Menu Close Menu
  • Terms and Conditions
  • Lists hosted on this site
  • Email the Postmaster
  • Tips for posting to public mailing lists
Re: Semi OT: Apache goes into frozen state
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Semi OT: Apache goes into frozen state

  • Subject: Re: Semi OT: Apache goes into frozen state
  • From: Chuck Hill <email@hidden>
  • Date: Wed, 06 Apr 2011 12:06:30 -0700

On Apr 6, 2011, at 11:54 AM, Pascal Robert wrote:


Le 2011-04-06 à 14:38, Johnny Miller a écrit :

Hi Johan,

Yeah, I'm not doing anything unusual like creating a DB connection per server.

If I run from the CLI:

sudo lsof | grep httpd | wc -l

The count kind of bounces up and down between 500 - 1,500

I just got 7652 on a deployed that I would consider small.


If I just look at the contents instead of the count I see a bunch of redundant connections to access_log and the various modules (see below)

To be honest I'm out of my element here, so I'm not sure if this is unusual or not.  I've updated the apache's launch daemon plist to increase the number of connections apache is allowed to 2048

MaxClient and ServerLimit set to 2048??  Are you sure that you need that much clients? I think I never set that value pass 384.

Yeah, that seems waaaay excessive.



How many httpd processes do you have? You can find out like this:

  # lsof -P | grep "/usr/sbin/httpd" | wc -l

That gives me 0 on OS X, this works for me:

ps auxwww | grep httpd | wc -l


Chuck

and I've gone a couple of days without a failure.  But I did do an intentional restart last night.  Prior to altering the plist I was seeing apache going into the unresponsive state about every 24 hours.

When I called Apple they told me to run netstat -m and look at this section:

0 requests for memory denied
0 requests for memory delayed

If that goes over 0 you can increase the memory buffer per this article: http://support.apple.com/kb/HT3512  But I've yet to see that number go over zero.

(this prints out several times)
httpd     6511           _www  txt       REG       14,2    1996352   17733353 /usr/sbin/httpd
httpd     6511           _www  txt       REG       14,2     393792   17733502 /usr/lib/libaprutil-1.0.2.7.dylib
httpd     6511           _www  txt       REG       14,2     651248      15133 /usr/lib/libexpat.1.5.0.dylib
httpd     6511           _www  txt       REG       14,2    4147008      15324 /usr/lib/libiconv.2.dylib
httpd     6511           _www  txt       REG       14,2     578656   17733501 /usr/lib/libapr-1.0.2.7.dylib
httpd     6511           _www  txt       REG       14,2      53888   17733291 /usr/libexec/apache2/mod_authn_file.so
httpd     6511           _www  txt       REG       14,2      66500   17733295 /usr/libexec/apache2/mod_authz_host.so
httpd     6511           _www  txt       REG       14,2     201360   17733299 /usr/libexec/apache2/mod_cache.so
httpd     6511           _www  txt       REG       14,2      70660   17733308 /usr/libexec/apache2/mod_dumpio.so
httpd     6511           _www  txt       REG       14,2     126528   17733321 /usr/libexec/apache2/mod_mem_cache.so
httpd     6511           _www  txt       REG       14,2     117296   17733307 /usr/libexec/apache2/mod_disk_cache.so
httpd     6511           _www  txt       REG       14,2     105944   17733311 /usr/libexec/apache2/mod_ext_filter.so
httpd     6511           _www  txt       REG       14,2     245316   17733316 /usr/libexec/apache2/mod_include.so
httpd     6511           _www  txt       REG       14,2      91784   17733312 /usr/libexec/apache2/mod_filter.so
httpd     6511           _www  txt       REG       14,2     121380   17733305 /usr/libexec/apache2/mod_deflate.so
httpd     6511           _www  txt       REG       14,2     126208   17733318 /usr/libexec/apache2/mod_log_config.so
httpd     6511           _www  txt       REG       14,2      54200   17733320 /usr/libexec/apache2/mod_logio.so
httpd     6511           _www  txt       REG       14,2      53956   17733309 /usr/libexec/apache2/mod_env.so
httpd     6511           _www  txt       REG       14,2      71040   17733310 /usr/libexec/apache2/mod_expires.so
httpd     6511           _www  txt       REG       14,2     100768   17733313 /usr/libexec/apache2/mod_headers.so
httpd     6511           _www  txt       REG       14,2      66568   17733314 /usr/libexec/apache2/mod_ident.so
httpd     6511           _www  txt       REG       14,2      70940   17733334 /usr/libexec/apache2/mod_setenvif.so
httpd     6511           _www  txt       REG       14,2     378832   17733325 /usr/libexec/apache2/mod_proxy.so
httpd     6511           _www  txt       REG       14,2     168484   17733330 /usr/libexec/apache2/mod_proxy_http.so
httpd     6511           _www  txt       REG       14,2     138088   17733327 /usr/libexec/apache2/mod_proxy_balancer.so
httpd     6511           _www  txt       REG       14,2     786308   17733336 /usr/libexec/apache2/mod_ssl.so
httpd     6511           _www  txt       REG       14,2     915632   17731671 /usr/lib/libssl.0.9.7.dylib
httpd     6511           _www  txt       REG       14,2     100588   17733322 /usr/libexec/apache2/mod_mime.so
httpd     6511           _www  txt       REG       14,2     120612   17733337 /usr/libexec/apache2/mod_status.so
httpd     6511           _www  txt       REG       14,2     176016   17733298 /usr/libexec/apache2/mod_autoindex.so
httpd     6511           _www  txt       REG       14,2      54000   17733284 /usr/libexec/apache2/mod_asis.so
httpd     6511           _www  txt       REG       14,2     101080   17733317 /usr/libexec/apache2/mod_info.so
httpd     6511           _www  txt       REG       14,2     127384   17733301 /usr/libexec/apache2/mod_cgi.so
httpd     6511           _www  txt       REG       14,2      70560   17733343 /usr/libexec/apache2/mod_vhost_alias.so
httpd     6511           _www  txt       REG       14,2     167388   17733324 /usr/libexec/apache2/mod_negotiation.so
httpd     6511           _www  txt       REG       14,2      66448   17733306 /usr/libexec/apache2/mod_dir.so
httpd     6511           _www  txt       REG       14,2      95568   17733315 /usr/libexec/apache2/mod_imagemap.so
httpd     6511           _www  txt       REG       14,2      58204   17733282 /usr/libexec/apache2/mod_actions.so
httpd     6511           _www  txt       REG       14,2      71144   17733335 /usr/libexec/apache2/mod_speling.so
httpd     6511           _www  txt       REG       14,2      66460   17733340 /usr/libexec/apache2/mod_userdir.so
httpd     6511           _www  txt       REG       14,2      79308   17733283 /usr/libexec/apache2/mod_alias.so
httpd     6511           _www  txt       REG       14,2     448160     698607 /System/Library/WebObjects/Adaptors/Apache2.2/mod_WebObjects.so
httpd     6511           _www  txt       REG       14,2     305712   17733333 /usr/libexec/apache2/mod_rewrite.so
httpd     6511           _www  txt       REG       14,2     290736    3429086 /usr/lib/libexslt.0.dylib
httpd     6511           _www  txt       REG       14,2    1297968      15260 /usr/lib/libiodbc.2.dylib
httpd     6511           _www  txt       REG       14,2    1046544   17731388 /System/Library/Frameworks/LDAP.framework/Versions/A/LDAP
httpd     6511           _www  txt       REG       14,2     958864   17733803 /usr/lib/libcurl.4.dylib
httpd     6511           _www  txt       REG       14,2     401216   17733947 /usr/lib/libsasl2.2.dylib
httpd     6511           _www  txt       REG       14,2      88356     172780 /usr/libexec/apache2/mod_auth_apple.so
httpd     6511           _www  txt       REG       14,2     101284     172785 /usr/libexec/apache2/mod_spnego_apple.so
httpd     6511           _www  txt       REG       14,2     105824     172775 /System/Library/PrivateFrameworks/SpnegoApache.framework/Versions/A/SpnegoApache
httpd     6511           _www  txt       REG       14,2     684032   17731066 /System/Library/Frameworks/DirectoryService.framework/Versions/A/DirectoryService
httpd     6511           _www  txt       REG       14,2     135724     172782 /usr/libexec/apache2/mod_digest_apple.so
httpd     6511           _www  txt       REG       14,2      66280     174412 /usr/libexec/apache2/mod_hfs_apple.so
httpd     6511           _www  txt       REG       14,2    1380352   18089764 /private/tmp/WOAdaptorState
httpd     6511           _www  txt       REG       14,2   31641660   17733495 /usr/libexec/apache2/libphp5.so
httpd     6511           _www  txt       REG       14,2    1380352   18089793 /private/tmp/WOAdaptorState
httpd     6511           _www  txt       REG       14,2    1059792    8952176 /usr/lib/dyld
httpd     6511           _www  txt       REG       14,2   98283520   17736835 /private/var/db/dyld/dyld_shared_cache_x86_64
httpd     6511           _www    0r      CHR        3,2        0t0  345489284 /dev/null

(same here this repeats many times)
httpd     6511           _www   11w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log
httpd     6511           _www   12w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log
httpd     6511           _www   13w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log
httpd     6511           _www   14w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log
httpd     6511           _www   15w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log
httpd     6511           _www   16w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log
httpd     6511           _www   17w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log
httpd     6511           _www   18w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log
httpd     6511           _www   19w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log
httpd     6511           _www   20w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log
httpd     6511           _www   21w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log
httpd     6511           _www   22w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log
httpd     6511           _www   23w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log
httpd     6511           _www   24w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log
httpd     6511           _www   25w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log
httpd     6511           _www   26w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log
httpd     6511           _www   27w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log
httpd     6511           _www   28w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log
httpd     6511           _www   29w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log
httpd     6511           _www   30w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log
httpd     6511           _www   31w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log
httpd     6511           _www   32w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log
httpd     6511           _www   33w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log
httpd     6511           _www   34w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log
httpd     6511           _www   35w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log
httpd     6511           _www   36w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log
httpd     6511           _www   37w      REG       14,2 5055271697     474173 /private/var/log/apache2/access_log



On Apr 6, 2011, at 7:53 AM, Johan Henselmans wrote:


On 5 apr 2011, at 02:40, Johnny Miller wrote:

Hi,

I was wondering if anyone else had experienced the same thing.  I have a X-Serve running OS X 10.5.8 whose apache server is going into a frozen state where it stops serving requests and can only be restarted by rebooting the machine.  All other services seem to being running normally.

If I look at the error_log I see many variations of this error:

Too many open files in system:Some/path/on/system

I've seen this thread on the Apple discussion board: http://discussions.apple.com/thread.jspa?threadID=1449787&start=15&tstart=0
And this article in the knowledge base: http://support.apple.com/kb/HT3512

Does anyone have more wisdom to share?


I once had a similar problem when I started my own objectstore coordinator for each session, along the line of:
// this causes all kinds of mayhem: do not do this at home, it creates a new database connection for every session

// EOObjectStoreCoordinator coord = new EOObjectStoreCoordinator();
// myEc = ERXEC.newEditingContext(coord);
// myEc().setDelegate(this);


After that, the database would have so many open connections after a while that the database would have too many open files, and would break down. 

I also seem to remember that some apache configuration would freeze with a specific log-rotation setup. (that was apache 1.3 on a FreeBSD machine)

Furthermore I would look for something in apache that opens a file but never closes these. You might use opensnoop for that, see man opensnoop, or lsof might give you some clues. 

Thanks in advance,

Johnny Miller
Kahalawai Media Corp
http://www.kahalawai.com


Johan Henselmans
email@hidden


Johnny Miller
Kahalawai Media Corp
http://www.kahalawai.com


-- 
Chuck Hill             Senior Consultant / VP Development

Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems.    
http://www.global-village.net/products/practical_webobjects







Attachment: smime.p7s
Description: S/MIME cryptographic signature

 _______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list      (email@hidden)
Help/Unsubscribe/Update your Subscription:

This email sent to email@hidden
References: 
 >Semi OT: Apache goes into frozen state (From: Johnny Miller <email@hidden>)
 >Re: Semi OT: Apache goes into frozen state (From: Johan Henselmans <email@hidden>)
 >Re: Semi OT: Apache goes into frozen state (From: Johnny Miller <email@hidden>)
 >Re: Semi OT: Apache goes into frozen state (From: Pascal Robert <email@hidden>)

  • Prev by Date: Re: Migrations not executed at startup problem (SOLVED)
  • Next by Date: Re: Migrations not executed at startup problem (SOLVED)
  • Previous by thread: Re: Semi OT: Apache goes into frozen state
  • Next by thread: Re: Semi OT: Apache goes into frozen state
  • Index(es):
    • Date
    • Thread