Re: Semi OT: Apache goes into frozen state
Re: Semi OT: Apache goes into frozen state
- Subject: Re: Semi OT: Apache goes into frozen state
- From: Pascal Robert <email@hidden>
- Date: Wed, 06 Apr 2011 15:36:02 -0400
Le 2011-04-06 à 15:26, Chuck Hill a écrit :
> You are confusing number of Apache processes with file handles/sockets. The error message
>>>>>> Too many open files in system:Some/path/on/system
> refers to the latter, the number of connections is the former.
And using lsof on Some/path/on/system will tell which processes are holding files on that directory. Example:
# sudo lsof -P +D /private/tmp/
Will find all files opened in /private/tmp
>
> Chuck
>
>
> On Apr 6, 2011, at 12:10 PM, Johnny Miller wrote:
>
>> Mssr. Robert,
>>
>> Thank you for the suggestion. If I run the command you suggested I get a fluctuating count between 3 and 9. While that is way below the threshold you suggested I'm not sure I'm convinced that is the whole story. Otherwise why would I be having a problem?
>>
>> Best,
>>
>> Johnny
>>
>> On Apr 6, 2011, at 8:54 AM, Pascal Robert wrote:
>>
>>>
>>> Le 2011-04-06 à 14:38, Johnny Miller a écrit :
>>>
>>>> Hi Johan,
>>>>
>>>> Yeah, I'm not doing anything unusual like creating a DB connection per server.
>>>>
>>>> If I run from the CLI:
>>>>
>>>> sudo lsof | grep httpd | wc -l
>>>>
>>>> The count kind of bounces up and down between 500 - 1,500
>>>>
>>>> If I just look at the contents instead of the count I see a bunch of redundant connections to access_log and the various modules (see below)
>>>>
>>>> To be honest I'm out of my element here, so I'm not sure if this is unusual or not. I've updated the apache's launch daemon plist to increase the number of connections apache is allowed to 2048
>>>
>>> MaxClient and ServerLimit set to 2048?? Are you sure that you need that much clients? I think I never set that value pass 384.
>>>
>>> How many httpd processes do you have? You can find out like this:
>>>
>>> # lsof -P | grep "/usr/sbin/httpd" | wc -l
>>>
>>>> and I've gone a couple of days without a failure. But I did do an intentional restart last night. Prior to altering the plist I was seeing apache going into the unresponsive state about every 24 hours.
>>>>
>>>> When I called Apple they told me to run netstat -m and look at this section:
>>>>
>>>> 0 requests for memory denied
>>>> 0 requests for memory delayed
>>>>
>>>> If that goes over 0 you can increase the memory buffer per this article: http://support.apple.com/kb/HT3512 But I've yet to see that number go over zero.
>>>>
>>>> (this prints out several times)
>>>> httpd 6511 _www txt REG 14,2 1996352 17733353 /usr/sbin/httpd
>>>> httpd 6511 _www txt REG 14,2 393792 17733502 /usr/lib/libaprutil-1.0.2.7.dylib
>>>> httpd 6511 _www txt REG 14,2 651248 15133 /usr/lib/libexpat.1.5.0.dylib
>>>> httpd 6511 _www txt REG 14,2 4147008 15324 /usr/lib/libiconv.2.dylib
>>>> httpd 6511 _www txt REG 14,2 578656 17733501 /usr/lib/libapr-1.0.2.7.dylib
>>>> httpd 6511 _www txt REG 14,2 53888 17733291 /usr/libexec/apache2/mod_authn_file.so
>>>> httpd 6511 _www txt REG 14,2 66500 17733295 /usr/libexec/apache2/mod_authz_host.so
>>>> httpd 6511 _www txt REG 14,2 201360 17733299 /usr/libexec/apache2/mod_cache.so
>>>> httpd 6511 _www txt REG 14,2 70660 17733308 /usr/libexec/apache2/mod_dumpio.so
>>>> httpd 6511 _www txt REG 14,2 126528 17733321 /usr/libexec/apache2/mod_mem_cache.so
>>>> httpd 6511 _www txt REG 14,2 117296 17733307 /usr/libexec/apache2/mod_disk_cache.so
>>>> httpd 6511 _www txt REG 14,2 105944 17733311 /usr/libexec/apache2/mod_ext_filter.so
>>>> httpd 6511 _www txt REG 14,2 245316 17733316 /usr/libexec/apache2/mod_include.so
>>>> httpd 6511 _www txt REG 14,2 91784 17733312 /usr/libexec/apache2/mod_filter.so
>>>> httpd 6511 _www txt REG 14,2 121380 17733305 /usr/libexec/apache2/mod_deflate.so
>>>> httpd 6511 _www txt REG 14,2 126208 17733318 /usr/libexec/apache2/mod_log_config.so
>>>> httpd 6511 _www txt REG 14,2 54200 17733320 /usr/libexec/apache2/mod_logio.so
>>>> httpd 6511 _www txt REG 14,2 53956 17733309 /usr/libexec/apache2/mod_env.so
>>>> httpd 6511 _www txt REG 14,2 71040 17733310 /usr/libexec/apache2/mod_expires.so
>>>> httpd 6511 _www txt REG 14,2 100768 17733313 /usr/libexec/apache2/mod_headers.so
>>>> httpd 6511 _www txt REG 14,2 66568 17733314 /usr/libexec/apache2/mod_ident.so
>>>> httpd 6511 _www txt REG 14,2 70940 17733334 /usr/libexec/apache2/mod_setenvif.so
>>>> httpd 6511 _www txt REG 14,2 378832 17733325 /usr/libexec/apache2/mod_proxy.so
>>>> httpd 6511 _www txt REG 14,2 168484 17733330 /usr/libexec/apache2/mod_proxy_http.so
>>>> httpd 6511 _www txt REG 14,2 138088 17733327 /usr/libexec/apache2/mod_proxy_balancer.so
>>>> httpd 6511 _www txt REG 14,2 786308 17733336 /usr/libexec/apache2/mod_ssl.so
>>>> httpd 6511 _www txt REG 14,2 915632 17731671 /usr/lib/libssl.0.9.7.dylib
>>>> httpd 6511 _www txt REG 14,2 100588 17733322 /usr/libexec/apache2/mod_mime.so
>>>> httpd 6511 _www txt REG 14,2 120612 17733337 /usr/libexec/apache2/mod_status.so
>>>> httpd 6511 _www txt REG 14,2 176016 17733298 /usr/libexec/apache2/mod_autoindex.so
>>>> httpd 6511 _www txt REG 14,2 54000 17733284 /usr/libexec/apache2/mod_asis.so
>>>> httpd 6511 _www txt REG 14,2 101080 17733317 /usr/libexec/apache2/mod_info.so
>>>> httpd 6511 _www txt REG 14,2 127384 17733301 /usr/libexec/apache2/mod_cgi.so
>>>> httpd 6511 _www txt REG 14,2 70560 17733343 /usr/libexec/apache2/mod_vhost_alias.so
>>>> httpd 6511 _www txt REG 14,2 167388 17733324 /usr/libexec/apache2/mod_negotiation.so
>>>> httpd 6511 _www txt REG 14,2 66448 17733306 /usr/libexec/apache2/mod_dir.so
>>>> httpd 6511 _www txt REG 14,2 95568 17733315 /usr/libexec/apache2/mod_imagemap.so
>>>> httpd 6511 _www txt REG 14,2 58204 17733282 /usr/libexec/apache2/mod_actions.so
>>>> httpd 6511 _www txt REG 14,2 71144 17733335 /usr/libexec/apache2/mod_speling.so
>>>> httpd 6511 _www txt REG 14,2 66460 17733340 /usr/libexec/apache2/mod_userdir.so
>>>> httpd 6511 _www txt REG 14,2 79308 17733283 /usr/libexec/apache2/mod_alias.so
>>>> httpd 6511 _www txt REG 14,2 448160 698607 /System/Library/WebObjects/Adaptors/Apache2.2/mod_WebObjects.so
>>>> httpd 6511 _www txt REG 14,2 305712 17733333 /usr/libexec/apache2/mod_rewrite.so
>>>> httpd 6511 _www txt REG 14,2 290736 3429086 /usr/lib/libexslt.0.dylib
>>>> httpd 6511 _www txt REG 14,2 1297968 15260 /usr/lib/libiodbc.2.dylib
>>>> httpd 6511 _www txt REG 14,2 1046544 17731388 /System/Library/Frameworks/LDAP.framework/Versions/A/LDAP
>>>> httpd 6511 _www txt REG 14,2 958864 17733803 /usr/lib/libcurl.4.dylib
>>>> httpd 6511 _www txt REG 14,2 401216 17733947 /usr/lib/libsasl2.2.dylib
>>>> httpd 6511 _www txt REG 14,2 88356 172780 /usr/libexec/apache2/mod_auth_apple.so
>>>> httpd 6511 _www txt REG 14,2 101284 172785 /usr/libexec/apache2/mod_spnego_apple.so
>>>> httpd 6511 _www txt REG 14,2 105824 172775 /System/Library/PrivateFrameworks/SpnegoApache.framework/Versions/A/SpnegoApache
>>>> httpd 6511 _www txt REG 14,2 684032 17731066 /System/Library/Frameworks/DirectoryService.framework/Versions/A/DirectoryService
>>>> httpd 6511 _www txt REG 14,2 135724 172782 /usr/libexec/apache2/mod_digest_apple.so
>>>> httpd 6511 _www txt REG 14,2 66280 174412 /usr/libexec/apache2/mod_hfs_apple.so
>>>> httpd 6511 _www txt REG 14,2 1380352 18089764 /private/tmp/WOAdaptorState
>>>> httpd 6511 _www txt REG 14,2 31641660 17733495 /usr/libexec/apache2/libphp5.so
>>>> httpd 6511 _www txt REG 14,2 1380352 18089793 /private/tmp/WOAdaptorState
>>>> httpd 6511 _www txt REG 14,2 1059792 8952176 /usr/lib/dyld
>>>> httpd 6511 _www txt REG 14,2 98283520 17736835 /private/var/db/dyld/dyld_shared_cache_x86_64
>>>> httpd 6511 _www 0r CHR 3,2 0t0 345489284 /dev/null
>>>>
>>>> (same here this repeats many times)
>>>> httpd 6511 _www 11w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>> httpd 6511 _www 12w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>> httpd 6511 _www 13w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>> httpd 6511 _www 14w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>> httpd 6511 _www 15w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>> httpd 6511 _www 16w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>> httpd 6511 _www 17w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>> httpd 6511 _www 18w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>> httpd 6511 _www 19w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>> httpd 6511 _www 20w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>> httpd 6511 _www 21w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>> httpd 6511 _www 22w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>> httpd 6511 _www 23w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>> httpd 6511 _www 24w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>> httpd 6511 _www 25w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>> httpd 6511 _www 26w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>> httpd 6511 _www 27w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>> httpd 6511 _www 28w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>> httpd 6511 _www 29w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>> httpd 6511 _www 30w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>> httpd 6511 _www 31w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>> httpd 6511 _www 32w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>> httpd 6511 _www 33w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>> httpd 6511 _www 34w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>> httpd 6511 _www 35w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>> httpd 6511 _www 36w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>> httpd 6511 _www 37w REG 14,2 5055271697 474173 /private/var/log/apache2/access_log
>>>>
>>>>
>>>>
>>>> On Apr 6, 2011, at 7:53 AM, Johan Henselmans wrote:
>>>>
>>>>>
>>>>> On 5 apr 2011, at 02:40, Johnny Miller wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I was wondering if anyone else had experienced the same thing. I have a X-Serve running OS X 10.5.8 whose apache server is going into a frozen state where it stops serving requests and can only be restarted by rebooting the machine. All other services seem to being running normally.
>>>>>>
>>>>>> If I look at the error_log I see many variations of this error:
>>>>>>
>>>>>> Too many open files in system:Some/path/on/system
>>>>>>
>>>>>> I've seen this thread on the Apple discussion board: http://discussions.apple.com/thread.jspa?threadID=1449787&start=15&tstart=0
>>>>>> And this article in the knowledge base: http://support.apple.com/kb/HT3512
>>>>>>
>>>>>> Does anyone have more wisdom to share?
>>>>>>
>>>>>
>>>>> I once had a similar problem when I started my own objectstore coordinator for each session, along the line of:
>>>>> // this causes all kinds of mayhem: do not do this at home, it creates a new database connection for every session
>>>>>
>>>>> // EOObjectStoreCoordinator coord = new EOObjectStoreCoordinator();
>>>>> // myEc = ERXEC.newEditingContext(coord);
>>>>> // myEc().setDelegate(this);
>>>>>
>>>>>
>>>>> After that, the database would have so many open connections after a while that the database would have too many open files, and would break down.
>>>>>
>>>>> I also seem to remember that some apache configuration would freeze with a specific log-rotation setup. (that was apache 1.3 on a FreeBSD machine)
>>>>>
>>>>> Furthermore I would look for something in apache that opens a file but never closes these. You might use opensnoop for that, see man opensnoop, or lsof might give you some clues.
>>>>>
>>>>>> Thanks in advance,
>>>>>>
>>>>>> Johnny Miller
>>>>>> Kahalawai Media Corp
>>>>>> http://www.kahalawai.com
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>> Webobjects-dev mailing list (email@hidden)
>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>>
>>>>>> This email sent to email@hidden
>>>>>
>>>>> Johan Henselmans
>>>>> email@hidden
>>>>>
>>>>>
>>>>>
>>>>
>>>> Johnny Miller
>>>> Kahalawai Media Corp
>>>> http://www.kahalawai.com
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Do not post admin requests to the list. They will be ignored.
>>>> Webobjects-dev mailing list (email@hidden)
>>>> Help/Unsubscribe/Update your Subscription:
>>>>
>>>> This email sent to email@hidden
>>>
>>
>> Johnny Miller
>> Kahalawai Media Corp
>> http://www.kahalawai.com
>>
>>
>>
>> _______________________________________________
>> Do not post admin requests to the list. They will be ignored.
>> Webobjects-dev mailing list (email@hidden)
>> Help/Unsubscribe/Update your Subscription:
>>
>> This email sent to email@hidden
>
> --
> Chuck Hill Senior Consultant / VP Development
>
> Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems.
> http://www.global-village.net/products/practical_webobjects
>
>
>
>
>
>
>
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden