Apple

Hello,

I'm seeing some odd behaviour from JavaMonitor in an EC2-based deployment setting. On a couple of EC2 instances (both running JavaMonitor, wotaskd and the same application, but otherwise not connected, and not identical) JavaMonitor itself dies after some period of time (in the order of days after launch) apparently because it's out of memory:

---

2014-01-03 22:08:56,759 WARN 26.47 MB/30.28 MB [WorkerThread11] logging.ERXNSLogLog4jBridge (ERXNSLogLog4jBridge.java:44) - <er.extensions.appserver.ERXComponentRequestHandler>: Exception occurred

while handling request:

com.webobjects.foundation.NSForwardException [java.lang.reflect.InvocationTargetException] null:java.lang.reflect.InvocationTargetException

2014-01-03 22:08:56,761 WARN 26.47 MB/30.28 MB [WorkerThread11] logging.ERXNSLogLog4jBridge (ERXNSLogLog4jBridge.java:44) - Ran out of memory, killing this instance

2014-01-03 22:08:56,762 FATAL 26.47 MB/30.28 MB [WorkerThread11] appserver.ERXApplication (ERXApplication.java:1947) - Ran out of memory, killing this instance

2014-01-03 22:08:56,763 FATAL 26.47 MB/30.28 MB [WorkerThread11] appserver.ERXApplication (ERXApplication.java:1948) - Ran out of memory, killing this instance

java.lang.OutOfMemoryError: unable to create new native thread

at java.lang.Thread.start0(Native Method)

at java.lang.Thread.start(Thread.java:679)

at com.webobjects.monitor._private.MHost.sendRequestToWotaskdArray(MHost.java:320)

at com.webobjects.monitor.application.WOTaskdHandler.sendRequest(WOTaskdHandler.java:160)

at com.webobjects.monitor.application.WOTaskdHandler.sendQueryToWotaskds(WOTaskdHandler.java:354)

at com.webobjects.monitor.application.WOTaskdHandler.getApplicationStatusForHosts(WOTaskdHandler.java:618)

at com.webobjects.monitor.application.WOTaskdHandler.updateForPage(WOTaskdHandler.java:105)

at com.webobjects.monitor.application.ApplicationsPage.<init>(ApplicationsPage.java:27)

---

(I can post the full stack trace if it's helpful.) By way of background, I'm only seeing this on a couple of specific EC2 instances—I've got other instances that have been running JavaMonitor with uptimes of months if not years.

According to the logging pattern in JavaMonitor's Properties, those log entries for the OutOfMemoryError are claiming 26M used and 30M free, which seems a bit suspect on the face of it. JavaMonitor is being started by /etc/init.d/webobjects using the default heap size, which I assume to be 64M. I've always found that to be sufficient, and can't find any reference to JavaMonitor being a memory hog on the list or elsewhere. The JavaMonitors that are failing in this way are used very lightly—monitoring a few instances at most of a single application on a single host, with occasional stops and starts for deployments.

So, has anyone seen JavaMonitor itself fall over in this way? Is anyone using non-default JVM memory settings for JavaMonitor launch? Does anyone want to have a guess at the root cause here? I can provide the full stack trace if anyone wants to see it.

Paul Hoadley

http://logicsquad.net/

_______________________________________________

Do not post admin requests to the list. They will be ignored.

Webobjects-dev mailing list (email@hidden)

Help/Unsubscribe/Update your Subscription:

This email sent to email@hidden

References:
	>JavaMonitor dying, apparently out of memory (From: Paul Hoadley <email@hidden>)